Replication & Failover

Replication and failover require service support, authorized resources, and an agreed recovery procedure. The SQL below describes the interface for enabled configurations; it is not a customer instruction to deploy a secondary cluster or a recovery SLA.

Gnok supports database replication using a primary-secondary model built on Iceberg snapshot forwarding. Replication enables disaster recovery, read scaling, and cross-region data distribution without copying raw data files.

Overview

Replication in Gnok is asynchronous. The primary database generates Iceberg snapshots as normal during writes, and a replication agent forwards metadata changes (snapshot pointers, schema evolution, manifest lists) to one or more secondary replicas. Because Iceberg stores data in immutable Parquet files on shared object storage, replication transfers only metadata -- not the underlying data files.

Key characteristics:

• Asynchronous: replicas lag behind the primary by the refresh interval (default 10 minutes)
• Metadata-only: data files on S3 are shared; only Iceberg metadata pointers are replicated
• Cross-account: replicas can reside in different Gnok accounts within the same organization
• Read-only replicas: secondary databases accept read queries but reject writes until promoted

Architecture

Primary Account                          Secondary Account
+------------------+                     +-------------------+
|  Primary DB      |                     |  Replica DB       |
|  (read/write)    |                     |  (read-only)      |
|                  |                     |                   |
|  Iceberg Tables  |                     |  Iceberg Tables   |
|  - snapshots     |  --- metadata --->  |  - snapshots      |
|  - manifests     |     forwarding      |  - manifests      |
|  - schema        |                     |  - schema         |
+------------------+                     +-------------------+
         |                                        |
         +-------- Shared Object Storage ---------+
                   (S3 / compatible)

The replication agent runs within the coordinator and:

1. Monitors the primary database for new Iceberg snapshots
2. Serializes metadata changes (new snapshots, schema evolution events, partition spec updates)
3. Forwards metadata to the secondary account's catalog
4. The secondary catalog updates its snapshot pointers to reference the same data files

Because data files are immutable and stored on shared object storage, no file copies are needed. The replica simply updates which snapshots and manifests it reads.

Setup Guide

Step 1: Enable Replication on the Primary

On the primary account, grant replication access to the target account:

ALTER DATABASE analytics ENABLE REPLICATION TO ACCOUNTS org1.account2;

To replicate to multiple accounts:

ALTER DATABASE analytics ENABLE REPLICATION TO ACCOUNTS org1.account2, org1.account3;

Verify replication is enabled:

SHOW REPLICATION DATABASES;

Step 2: Create the Replica on the Secondary

On the secondary account, create a replica database that references the primary:

CREATE DATABASE analytics_replica
  AS REPLICA OF org1.account1.analytics;

The replica is created in a read-only state. All tables, schemas, and views from the primary are visible after the initial sync.

Step 3: Sync the Replica

Trigger an initial synchronization:

ALTER DATABASE analytics_replica REFRESH;

The first refresh performs a full metadata sync. Subsequent refreshes are incremental, transferring only new snapshot deltas.

To set up automatic refresh on a schedule:

ALTER DATABASE analytics_replica SET
  REPLICATION_SCHEDULE = '10 MINUTE';

Step 4: Monitor Replication

Check replication status and lag:

SHOW REPLICATION DATABASES;

This returns:

Column	Description
database_name	Name of the replicated database
account	Source or destination account identifier
replication_state	Current state: ACTIVE, SUSPENDED, FAILING
lag_bytes	Approximate metadata bytes behind the primary
last_refresh_time	Timestamp of the most recent successful sync
next_refresh_time	Scheduled time for the next automatic refresh
error_message	Details if replication_state is FAILING

Failover Procedure

When the primary becomes unavailable, promote a replica to take over as the new primary.

Step 1: Verify the Replica Is Up-to-Date

Before promoting, check that the replica has the latest data:

SHOW REPLICATION DATABASES;
-- Check lag_bytes and last_refresh_time

If the replica is behind, trigger a final refresh (if the primary is still reachable):

ALTER DATABASE analytics_replica REFRESH;

Step 2: Promote the Replica

Promote the replica to a read-write primary:

ALTER DATABASE analytics_replica PRIMARY;

This operation:

• Removes the read-only restriction
• Enables write operations (INSERT, UPDATE, DELETE, DDL)
• Detaches the database from the former primary's replication stream
• Completes in seconds (metadata-only operation)

During promotion, in-flight DML on the former primary (if still running) may produce conflicting snapshots. Any uncommitted transactions on the old primary are not reflected in the promoted replica.

Step 3: Redirect Clients

Update application connection strings to point to the account hosting the promoted database. If using Failover Groups (see below), client redirection can be automated.

Step 4: Re-establish Reverse Replication

Once the former primary is back online, set it up as a replica of the new primary:

-- On the new primary (formerly the replica)
ALTER DATABASE analytics_replica ENABLE REPLICATION TO ACCOUNTS org1.account1;

-- On the former primary
DROP DATABASE analytics;
CREATE DATABASE analytics AS REPLICA OF org1.account2.analytics_replica;
ALTER DATABASE analytics SET REPLICATION_SCHEDULE = '10 MINUTE';

Failover Groups

Failover Groups bundle multiple databases and account-level objects into a single unit that can be failed over together. This avoids promoting databases one at a time.

Create a Failover Group

CREATE FAILOVER GROUP analytics_dr
  OBJECT_TYPES = DATABASES, ROLES, WAREHOUSES
  ALLOWED_DATABASES = analytics, analytics_staging
  ALLOWED_ACCOUNTS = org1.account2
  REPLICATION_SCHEDULE = '10 MINUTE';

Parameters:

Parameter	Description
OBJECT_TYPES	What to replicate: DATABASES, ROLES, WAREHOUSES, RESOURCE_MONITORS
ALLOWED_DATABASES	Databases included in the group
ALLOWED_ACCOUNTS	Target accounts that can host a replica of this group
REPLICATION_SCHEDULE	How often to sync (e.g., 5 MINUTE, 1 HOUR)

Manage Failover Groups

-- View failover groups
SHOW FAILOVER GROUPS;

-- Refresh the secondary group manually
ALTER FAILOVER GROUP analytics_dr REFRESH;

-- Promote the secondary group to primary
ALTER FAILOVER GROUP analytics_dr PRIMARY;

-- Suspend replication
ALTER FAILOVER GROUP analytics_dr SUSPEND;

-- Resume replication
ALTER FAILOVER GROUP analytics_dr RESUME;

-- Drop a failover group
DROP FAILOVER GROUP analytics_dr;

RTO and RPO Characteristics

Recovery time and data-loss exposure depend on the configured service topology, successful refreshes, failure conditions, and account arrangements. A refresh interval alone is not a guaranteed recovery point, and a metadata promotion alone does not guarantee client recovery time. Coordinate failover with Gnok support and confirm the applicable recovery objectives for your account.

Limitations

• Asynchronous only: there is no synchronous replication mode. Replicas always lag by at least one refresh interval.
• No multi-primary: only one database in a replication pair accepts writes at a time. Concurrent writes to both endpoints cause snapshot conflicts.
• DML paused during promotion: the ALTER DATABASE ... PRIMARY command briefly pauses in-flight DML on the replica (typically sub-second) to ensure a consistent transition.
• Shared object storage required: the primary and replica must be able to read the same S3 bucket (or compatible object store). Cross-region replication requires cross-region S3 access or S3 replication.
• Schema changes propagate on next refresh: DDL changes (ALTER TABLE, DROP TABLE) on the primary are visible on the replica only after the next successful refresh.
• Failover Group scope: Failover Groups replicate metadata only for the specified object types. Data sharing policies and network rules are not included.

Further Reading

• Resource Governance -- warehouse and resource monitor management
• Troubleshooting -- diagnosing replication issues
• Runbooks -- operational procedures for failover scenarios