Connecting External Engines

Gnok Catalog is a standard Iceberg REST catalog, so external engines connect using their normal Iceberg REST settings. This page gives working configuration for Spark, Trino, PyIceberg, and DuckDB.

External engine access is set up by request

Connecting an external engine isn't self-service for new organizations yet. Email support@gnok.io to request it. Gnok support provides your Catalog URL and an OAuth client for your engine. https://catalog.example.com below is a placeholder.

What you need

Before configuring a client, gather:

Value	Where it comes from
Catalog URL	The base URL of Gnok Catalog that Gnok support provides (placeholder: https://catalog.example.com). Clients append /v1 automatically.
Catalog (warehouse) name	The Gnok catalog you want to use. Clients pass this as their warehouse setting; the server returns it as the REST prefix.
OAuth2 credential	A client-id:client-secret pair, or a pre-issued bearer token. See Authentication.

The connection model

Two things are specific to how Gnok Catalog works, and every engine config below relies on them:

1. warehouse selects the catalog. On startup a client calls GET /v1/config?warehouse=<name>. The catalog returns defaults.prefix = <name>, and the client then addresses all tables under /v1/<name>/namespaces/.... So set the client's warehouse to your Gnok catalog name.
2. Data access requires vended credentials. Set the request header X-Iceberg-Access-Delegation: vended-credentials so the catalog returns short-lived, location-scoped storage credentials with each table load. Without it, the client receives metadata but no credentials to read or write data files. See credential vending.

Creating an engine credential

Engines authenticate as a service account — a machine principal that holds no password — using an OAuth2 client-credentials secret bound to it. Each engine (or each catalog) can have its own client, and secrets can be rotated or revoked independently.

1. Create the service account the engine will act as, and grant it the access the engine needs (see roles and grants). An organization administrator, or a user with MANAGE SERVICE ACCOUNTS, can run:

   CREATE SERVICE ACCOUNT IF NOT EXISTS trino_svc
       COMMENT = 'Trino reading the analytics catalog';

2. Request an OAuth client for it: email support@gnok.io with the service account's name. Gnok support returns a client-id:client-secret pair (the secret is shown once; store it in your secret manager) and your Catalog URL.

3. Put the credential in the engine config below.

The client-credentials token assumes the service account's identity (its organization and roles), so it can do exactly what that account is granted. To rotate or revoke a client, email support@gnok.io.

Tokens refresh automatically

Configure engines with the credential (client-id:client-secret), not a static token. With a credential, the Iceberg client runs the client-credentials grant itself and refreshes the access token before it expires — no manual rotation. A static token never refreshes and will start failing once it expires.

Engines may present the client id/secret either in the request body or via the HTTP Basic Authorization header (RFC 6749 §2.3.1) — Gnok Catalog accepts both, so Trino, Spark, PyIceberg, and other standard OAuth2 clients all work unchanged.

Apache Spark

Using Spark with the Iceberg runtime, configure a REST catalog named gnok:

spark.sql.catalog.gnok                                   = org.apache.iceberg.spark.SparkCatalog
spark.sql.catalog.gnok.type                              = rest
spark.sql.catalog.gnok.uri                               = https://catalog.example.com
spark.sql.catalog.gnok.warehouse                         = my_catalog
spark.sql.catalog.gnok.credential                        = <client-id>:<client-secret>
spark.sql.catalog.gnok.header.X-Iceberg-Access-Delegation = vended-credentials
spark.sql.catalog.gnok.io-impl                           = org.apache.iceberg.aws.s3.S3FileIO
spark.sql.catalog.gnok.s3.region                         = us-east-1

Then query as usual:

SHOW NAMESPACES IN gnok;
SELECT * FROM gnok.analytics.orders LIMIT 10;

If you already hold a bearer token instead of OAuth client credentials, replace .credential with .token = <access-token>.

Trino

In a catalog properties file (e.g. etc/catalog/gnok.properties):

connector.name                              = iceberg
iceberg.catalog.type                        = rest
iceberg.rest-catalog.uri                    = https://catalog.example.com
iceberg.rest-catalog.warehouse              = my_catalog
iceberg.rest-catalog.security               = OAUTH2
iceberg.rest-catalog.oauth2.credential      = <client-id>:<client-secret>
iceberg.rest-catalog.oauth2.token-refresh-enabled = true
iceberg.rest-catalog.vended-credentials-enabled = true
fs.native-s3.enabled                        = true
s3.region                                   = us-east-1

oauth2.token-refresh-enabled = true makes Trino refresh its access token before it expires; use oauth2.credential (not oauth2.token) so refresh has credentials to use.

SHOW SCHEMAS FROM gnok;
SELECT * FROM gnok.analytics.orders LIMIT 10;

vended-credentials-enabled = true makes Trino send the X-Iceberg-Access-Delegation header for you.

PyIceberg

from pyiceberg.catalog import load_catalog

catalog = load_catalog(
    "gnok",
    type="rest",
    uri="https://catalog.example.com",
    warehouse="my_catalog",
    credential="<client-id>:<client-secret>",
    **{"header.X-Iceberg-Access-Delegation": "vended-credentials"},
)

# List and load
catalog.list_namespaces()
table = catalog.load_table("analytics.orders")
df = table.scan().to_arrow()

DuckDB

DuckDB reads Iceberg REST catalogs through its iceberg extension. Load httpfs too — the extension uses it for object-storage I/O when reading vended-credential data. Tested with DuckDB 1.5.1.

INSTALL iceberg; LOAD iceberg;
INSTALL httpfs; LOAD httpfs;

-- OAuth2 client-credentials: DuckDB fetches and refreshes its own token.
CREATE SECRET gnok_cred (
    TYPE ICEBERG,
    CLIENT_ID '<client-id>',
    CLIENT_SECRET '<client-secret>',
    OAUTH2_SERVER_URI 'https://catalog.example.com/v1/oauth/tokens',
    OAUTH2_SCOPE 'catalog:read namespace:read table:read'
);

ATTACH 'my_catalog' AS gnok (
    TYPE ICEBERG,
    ENDPOINT 'https://catalog.example.com',
    SECRET gnok_cred
);

SELECT * FROM gnok.analytics.orders LIMIT 10;

Here my_catalog (the warehouse) is your Gnok catalog name and gnok (the ATTACH alias) is how you reference it in SQL: gnok.<namespace>.<table>. Two details the extension needs that are easy to miss:

• OAUTH2_SERVER_URI — the token endpoint, <catalog-url>/v1/oauth/tokens. Without it DuckDB has nowhere to exchange the client credentials. OAUTH2_SCOPE must cover catalog reads (e.g. catalog:read namespace:read table:read, the same scopes used in Verifying a connection); you can omit it if your client's default scopes already include them.
• SECRET gnok_cred in the ATTACH — bind the secret explicitly so the right credential is used for the catalog and for vended object-storage access.

Quick start with a short-lived token

For a one-off session or a test, skip the client and paste a bearer token directly (e.g. the one from Verifying a connection):

INSTALL iceberg; LOAD iceberg;
INSTALL httpfs; LOAD httpfs;

CREATE SECRET gnok_tok (TYPE ICEBERG, TOKEN '<bearer-token>');

ATTACH 'my_catalog' AS gnok (
    TYPE ICEBERG, ENDPOINT 'https://catalog.example.com', SECRET gnok_tok
);

SELECT count(*) FROM gnok.analytics.orders;

The token is static, so it won't refresh — fine for short sessions, not for long jobs.

Read through the catalog, not the S3 path

Always go through the REST catalog (ATTACH … TYPE ICEBERG). Don't point DuckDB at the table's storage path with iceberg_scan('s3://…/table') plus unsafe_enable_version_guessing. Gnok writes UUID-named metadata files (no monotonic v1/v2, no version-hint.text), so path-based version guessing can latch onto a superseded metadata file whose snapshot manifest has since been expired — yielding an HTTP 404, or silently reading stale/orphaned data files. The catalog tracks the authoritative current snapshot, so an attached catalog always reads the correct data. DuckDB's Iceberg support is read-oriented and evolving — check your version's Iceberg-extension docs for the credential-vending and write coverage available to you.

StarRocks (and Apache Doris)

StarRocks attaches a Gnok catalog with CREATE EXTERNAL CATALOG (Apache Doris uses CREATE CATALOG — the property names are identical). Both speak the MySQL protocol, so you run these statements from any MySQL client against the FE.

CREATE EXTERNAL CATALOG gnok PROPERTIES (
    "type"                                = "iceberg",
    "iceberg.catalog.type"                = "rest",
    "iceberg.catalog.uri"                 = "https://catalog.example.com",
    "iceberg.catalog.warehouse"           = "my_catalog",
    "iceberg.catalog.security"            = "oauth2",
    "iceberg.catalog.oauth2.credential"   = "<client-id>:<client-secret>",
    "iceberg.catalog.oauth2.server-uri"   = "https://catalog.example.com/v1/oauth/tokens",
    "iceberg.catalog.oauth2.scope"        = "catalog:read namespace:read table:read",
    "header.X-Iceberg-Access-Delegation"  = "vended-credentials",
    "aws.s3.region"                       = "us-east-1"
);

Then query through the catalog name:

SET CATALOG gnok;
SHOW DATABASES;
USE analytics;
SELECT * FROM orders LIMIT 10;          -- or fully-qualified: gnok.analytics.orders

iceberg.catalog.oauth2.credential (the client-id:client-secret you created above) lets StarRocks run the client-credentials grant and refresh its token automatically — use it, not a static token. header.X-Iceberg-Access-Delegation = vended-credentials makes the FE request vended storage credentials so the BE/CN can read data files (see the connection model).

Using a static token? Turn token refresh OFF

StarRocks embeds the Iceberg Java client (iceberg-core), which by default tries to refresh its OAuth2 access token against the catalog's token endpoint. Refreshing needs a credential to refresh with. If you configure a static bearer token (iceberg.catalog.token) and leave refresh enabled, the client exchanges your token and then fails every request with NotAuthorizedException: Token has expired — even immediately after minting a brand-new token. Two safe options:

• Preferred — use iceberg.catalog.oauth2.credential (above). The client refreshes on its own; nothing expires.

• If you must use a static token, disable refresh so the token is sent verbatim until its natural expiry:

  CREATE EXTERNAL CATALOG gnok PROPERTIES (
      "type"="iceberg", "iceberg.catalog.type"="rest",
      "iceberg.catalog.uri"="https://catalog.example.com",
      "iceberg.catalog.warehouse"="my_catalog",
      "iceberg.catalog.token"="<bearer-token>",
      "iceberg.catalog.oauth2.token-refresh-enabled"="false",   -- ← required for static tokens
      "header.X-Iceberg-Access-Delegation"="vended-credentials",
      "aws.s3.region"="us-east-1"
  );

  A static "header.Authorization" = "Bearer <token>" also works — it bypasses the OAuth2 refresh machinery entirely. Static tokens don't refresh, so they're fine for short sessions but not long-running jobs.

Reading data files

StarRocks' BE/CN reads object storage directly, so it needs storage credentials:

• Vended credentials (recommended): the header.X-Iceberg-Access-Delegation = vended-credentials property above makes Gnok Catalog return short-lived, location-scoped credentials with each table load — no static cloud keys in StarRocks.
• Cloud instance role: when StarRocks runs in the same cloud account as the data, you can instead let the BE use the node's role — on AWS, "aws.s3.use_instance_profile" = "true" (drop the delegation header in that case).

To rotate or revoke a catalog after changing its auth, DROP CATALOG gnok; then recreate it; StarRocks caches the connector per catalog name.

Other engines

Any engine that speaks the Iceberg REST catalog protocol with OAuth2 works the same way: point it at the catalog URL, set the warehouse to your Gnok catalog name, give it the client-id:client-secret credential, and enable vended credentials. The property names differ per engine but the values are identical to the examples above.

Apache Flink (Iceberg connector):

CREATE CATALOG gnok WITH (
  'type'='iceberg',
  'catalog-type'='rest',
  'uri'='https://catalog.example.com',
  'warehouse'='my_catalog',
  'credential'='<client-id>:<client-secret>',
  'header.X-Iceberg-Access-Delegation'='vended-credentials',
  'io-impl'='org.apache.iceberg.aws.s3.S3FileIO'
);

StarRocks / Apache Doris — see StarRocks (and Apache Doris) above for a full example, including the static-token refresh caveat.

Dremio / Snowflake / other catalogs — configure an "Iceberg REST" / "external Iceberg" source with the catalog URL, warehouse name, and OAuth2 client credential. Vended credentials require the engine to send X-Iceberg-Access-Delegation: vended-credentials; if an engine cannot, configure it with direct object-storage credentials instead.

In all cases the credential is the client-id:client-secret you created under Creating an engine credential, and the engine refreshes its own token via the client-credentials grant.

Verifying a connection

A quick way to confirm Gnok Catalog is reachable and your credential works, independent of any engine:

# 1. Get a token (client-credentials grant)
TOKEN=$(curl -s -X POST https://catalog.example.com/v1/oauth/tokens \
  -d grant_type=client_credentials \
  -d client_id=<client-id> \
  -d client_secret=<client-secret> \
  -d scope='catalog:read namespace:read table:read' | jq -r .access_token)

# 2. Read catalog config for your warehouse (note the returned defaults.prefix)
curl -s https://catalog.example.com/v1/config?warehouse=my_catalog \
  -H "Authorization: Bearer $TOKEN"

# 3. List namespaces in that catalog
curl -s https://catalog.example.com/v1/my_catalog/namespaces \
  -H "Authorization: Bearer $TOKEN"

See Authentication & credential vending for the full token flow and scope reference, and the REST API reference for the complete endpoint list.