That is fine for the workload Iceberg was built for. It is not fine for an application that edits one order, appends one comment, or increments one counter and then wants to read it back.

Gnok now takes a different path for those writes, and turning it on is one line of DDL:

ALTER TABLE orders SET TBLPROPERTIES ('gnok.write.mode' = 'command');

That write now acknowledges in 5–7 ms.

Where the time goes​

Three paths exist, and the difference between them is when the write is durable enough to acknowledge.

The last one is not a cache and not a queue you can lose. The write is replicated before it is acknowledged; Gnok publishes it into Iceberg in the background shortly after. What changed is the order of those two facts, not whether both happen.

The 712 ms is per statement, not per row — a bulk load of ten million rows pays it once and does not care. This is about the small write, and the small write is what an application does all day.

What the table has to say​

The mode lives on the table, because it is a fact about the data:

The middle value is the one that makes a migration possible. While you are moving writers over, both paths must work on the same table. When the last writer has moved, command_only shuts the door so nothing can modify the table behind the engine's back.

There is no service setting to change and nothing to keep in step elsewhere: every part of Gnok reads the mode from the same place you wrote it.

What a fast write has to prove​

A write that is acknowledged before it reaches the table has to be replayable exactly once, and has to know which row it belongs to. So the engine asks three things of the statement:

BEGIN;
SET LOCAL gnok.idempotency_key = '01a06867-04e0-7f34-a545-d82c7e40e656';
SET LOCAL gnok.command_type    = 'order.edit';      -- optional label
SET LOCAL gnok.expected_version = 4;                -- optional compare-and-swap

MERGE INTO orders x
USING (SELECT CAST(1001 AS BIGINT) AS order_id,
              CAST(99.50 AS DECIMAL(10,2)) AS amount) s
   ON x.order_id = s.order_id
 WHEN MATCHED THEN UPDATE SET amount = s.amount
 WHEN NOT MATCHED THEN INSERT (order_id, amount) VALUES (s.order_id, s.amount);

COMMIT RETURNING;

An explicit transaction with an idempotency key — that key is the opt-in, and a retry with the same key returns the original answer rather than writing twice. A full-primary-key MERGE with every column assigned, so the row's identity and its post-image are both known without reading the table first. And a target the write path owns.

The expected_version carrier is optional and does what a compare-and-swap does: if the row moved since you read it, the write is refused as a stale version and your retry loop re-reads.

Reads see their own writes​

The obvious objection to acknowledging early is that the next query will not see the row. It does. A read against a command table merges in the rows that are ordered but not yet published, so read-your-writes holds without waiting for the catalog. COMMIT RETURNING hands back a commit token if you want to require a specific write to be visible on a particular read.

What gets refused, and why that is the point​

Some statements cannot be replayed exactly once, and the engine says so instead of quietly taking the slow path:

• a MERGE that leaves columns unassigned — the post-image is unknown;
• a MERGE whose ON clause is not the whole primary key — the row's identity is unknown;
• an UPDATE … WHERE over a predicate — it may match anything;
• two writes to the same table in one transaction — the second would have to read the first's uncommitted effect;
• any transaction touching a table outside the write path — refused whole, before anything is admitted, because an acknowledged write that cannot be published would block every write behind it.

Each refusal names what could not be proven and what to change. A write that is admitted is a write the engine can land.

When to leave it alone​

direct is still right for most tables. Bulk loads amortize the catalog commit to nothing. Analytics tables often have no primary key, so there is no identity to key a command on. The low-latency path is for the transactional core of an application — the tables it edits one row at a time and reads back immediately — and for those, the difference between 712 ms and 6 ms is the difference between a background job and an interactive one.

Gnok CDC streams turn the Iceberg snapshots behind a table into a bounded change feed. In this tutorial, we will create an inventory stream, inspect inserts, updates, and deletes, and acknowledge a batch after processing it.

This is outbound table CDC: the source data already lives in Gnok. For inbound data, see Streaming Ingestion. For historical snapshot ranges without a managed watermark, see Snapshots, Time Travel, and Rollback.

The Use Case​

Suppose a retailer stores its current warehouse stock in an Iceberg table. Inventory writes continue normally, while an operations application periodically needs to answer:

• Which products were added?
• Which stock levels changed?
• Which products were removed?
• Has anything changed since the previous acknowledged batch?

The stream stores a watermark, not a second copy of the table. Gnok derives changes between that watermark and a fixed current Iceberg snapshot when the stream is queried.

Create the Inventory Table​

Create a catalog, schema, and table for the example:

CREATE CATALOG retail;
CREATE SCHEMA retail.inventory;

CREATE TABLE retail.inventory.stock_levels (
  sku VARCHAR PRIMARY KEY,
  warehouse VARCHAR,
  on_hand BIGINT,
  updated_at TIMESTAMP
);

The primary key becomes an Iceberg identifier field. That gives CDC a stable logical row identity, which allows Gnok to recognize an update as a related DELETE and INSERT pair.

Seed the current state before creating the stream:

INSERT INTO retail.inventory.stock_levels
  (sku, warehouse, on_hand, updated_at)
VALUES
  ('SKU-100', 'east', 12, TIMESTAMP '2026-08-27 09:00:00'),
  ('SKU-200', 'west',  5, TIMESTAMP '2026-08-27 09:00:00');

Start Tracking Changes​

Create a standard stream in the same catalog and schema as its source table:

CREATE STREAM retail.inventory.stock_changes
ON TABLE retail.inventory.stock_levels;

By default, the current table snapshot becomes the stream's starting watermark. The two seed rows therefore establish the baseline and do not appear as new changes.

Now simulate normal inventory activity:

INSERT INTO retail.inventory.stock_levels
  (sku, warehouse, on_hand, updated_at)
VALUES
  ('SKU-300', 'east', 20, TIMESTAMP '2026-08-27 09:05:00');

UPDATE retail.inventory.stock_levels
SET on_hand = 9,
    updated_at = TIMESTAMP '2026-08-27 09:06:00'
WHERE sku = 'SKU-100';

DELETE FROM retail.inventory.stock_levels
WHERE sku = 'SKU-200';

Each committed write creates a new Iceberg snapshot. The source table continues to represent current inventory; the stream represents the changes since its watermark.

Check and Preview the Pending Batch​

An inexpensive availability check can prevent polling clients from running an empty change query:

SELECT SYSTEM$STREAM_HAS_DATA(
  'retail.inventory.stock_changes'
);

It returns one has_data value, which is now true.

Preview the pending changes with an ordinary SELECT:

SELECT
  sku,
  warehouse,
  on_hand,
  METADATA$ACTION,
  METADATA$ISUPDATE,
  METADATA$ROW_ID
FROM retail.inventory.stock_changes
ORDER BY sku, METADATA$ACTION;

The result has this shape:

Iceberg represents an update as removal of the old row plus insertion of the new row. METADATA$ISUPDATE = true and the shared row ID let a consumer pair those two records. A standalone delete or insert has METADATA$ISUPDATE = false.

Consume the Batch​

After the consumer is ready to acknowledge this bounded batch, use the explicit consuming statement:

CONSUME STREAM retail.inventory.stock_changes;

CONSUME STREAM returns the pending rows and then advances the watermark with a compare-and-swap operation. The watermark advances only after the change query succeeds. A concurrent consumer cannot silently overwrite a different watermark; one of the operations fails and must be retried.

Confirm that the stream is current:

SELECT SYSTEM$STREAM_HAS_DATA(
  'retail.inventory.stock_changes'
);

SELECT *
FROM retail.inventory.stock_changes;

The availability check now returns false, and the preview returns no rows. Future table commits create a new pending batch.

Choose the Right Delivery Pattern​

The stream watermark is transactional inside Gnok, but it cannot participate in the transaction of an unrelated external database, search index, or message broker. That boundary matters when designing a production consumer.

• Use ordinary SELECT for repeatable previews and validation. It never acknowledges data.
• Use CONSUME STREAM when returning the change result is itself the consumption step, or when the surrounding application can reconcile its destination.
• Make downstream operations idempotent where possible. METADATA$ROW_ID, METADATA$ACTION, and the business key are useful deduplication inputs.
• Treat a lost client response as ambiguous: the server may have returned the rows and committed the watermark even if the client did not receive the final response.
• For an external sink that requires replayable, application-managed checkpoints, use explicit READ CHANGES ... BETWEEN SNAPSHOT ... ranges and persist the completed snapshot boundary in that application.

Gnok does not currently make a separate INSERT ... SELECT FROM <stream> or MERGE ... USING <stream> atomic with stream advancement. Do not describe that two-system workflow as exactly-once without an additional transactional handoff design.

Production Checklist​

Before using a stream in a production workflow:

1. Grant least privilege. The principal needs CREATE STREAM on the schema and read access to the source table. Stream reads re-check source-table authorization.
2. Use identifier fields. A primary key or valid Iceberg identifier fields provide stable row IDs and update pairing. Without them, changes remain valid insert/delete records, but updates cannot be distinguished from unrelated deletes and inserts.
3. Keep the stream beside its source. The stream and table must currently share a catalog and schema.
4. Retain required snapshots. A stream derives its delta from Iceberg snapshot history. Coordinate snapshot expiration with the oldest live stream watermark.
5. Monitor the watermark. Use SHOW STREAMS ON TABLE retail.inventory.stock_levels and SYSTEM$STREAM_HAS_DATA to observe lag and pending work.
6. Advance deliberately. ALTER STREAM ... ADVANCE discards the currently pending batch by moving the watermark without returning rows. Reserve it for an intentional skip or recovery procedure.
7. Avoid competing consumers. Compare-and-swap prevents silent watermark corruption, but independent consumers should generally have independent streams.

Clean Up​

When the example is no longer needed:

DROP STREAM retail.inventory.stock_changes;
DROP TABLE retail.inventory.stock_levels;

For syntax, stream variants, operational limits, and task scheduling, see the Streams & Tasks guide.

The shape of the problem​

Three constraints drove the design:

1. The documents are heterogeneous and nested. Datasets like users, orders, events, and sessions share no schema, and individual documents carry sub-objects and arrays. Designing a bespoke relational schema per dataset would have been weeks of work.
2. Some datasets are huge. An event_history-style export alone was 17 GB compressed — far too big to expand onto a laptop and re-process from disk.
3. The data has to get from a laptop to the server. Gnok ingests from object storage, not from the SQL wire. So "loading" really means "get files into S3, then tell Gnok to read them."

The pipeline​

For each dataset we run a single streaming pass:

gunzip -c <dataset>.json.gz \
  | python loader.py <table>     # normalize, chunk, upload, COPY INTO

The loader consumes newline-delimited JSON (NDJSON — one document per line) and never holds a whole dataset on disk: it converts on the fly and keeps exactly one chunk in flight. Here's what each chunk goes through.

1. Stream straight from the compressed file​

Expanding 17 GB of gzipped JSON onto disk just to read it is wasteful — and on a laptop, sometimes impossible. Instead we pipe gunzip -c directly into the loader, so the giant datasets flow through memory a line at a time and never touch local disk in full.

If your export is one big pretty-printed JSON array rather than NDJSON, a streaming parser gets you the same line-at-a-time behavior — jq -c '.[]' (or an ijson pass) emits one compact document per line without materializing the whole array.

This is also a clean path for document databases: MongoDB, for instance, exports straight to NDJSON via mongoexport --type=json, and you can decode a raw .bson dump to the same JSON-lines stream with bsondump — no server to stand up. Either one drops directly into the pipeline below.

2. One schema to rule them all: id + doc VARIANT​

Rather than model each dataset, every table is just two columns:

CREATE TABLE my_db.json.users_raw (id VARCHAR, doc VARCHAR);

id is the document's identifier; doc is the entire document as a JSON string. After load we wrap it in a view that parses the JSON into a queryable VARIANT:

CREATE VIEW my_db.json.users AS
  SELECT id, PARSE_JSON(doc) AS doc FROM my_db.json.users_raw;

Now nested access just works, with zero per-dataset schema work:

SELECT doc:profile:email::varchar AS email,
       doc:address:city::varchar  AS city
FROM my_db.json.users
WHERE doc:status::varchar = 'active';

One normalization detail worth calling out: many document exporters wrap scalars in typed objects rather than emitting plain values — an id might arrive as {"$oid":"..."} and a date as {"$date":{"$numberLong":"..."}}. If your source does this, rewrite those wrappers into plain values ("...", "2026-02-28T23:55:10Z") in the loader before storing, so queries see clean strings and ISO timestamps instead of wrapper objects.

3. The upload: presigned URLs to S3​

This is the part worth understanding. Gnok doesn't accept bulk data over the SQL connection (COPY ... FROM STDIN isn't supported). Data lands in object storage first, then a SQL statement points Gnok at it. The flow per chunk:

POST /v1/stages/presign   { stage, runId, filename }   ->   { url, stageRef }
PUT  <url>  (the file bytes)                            ->   S3 (200 OK)
COPY INTO <table> FROM '<stageRef>' FILE_FORMAT=(TYPE='PARQUET')

A presigned URL is a temporary, signed S3 URL that grants "you may PUT this one object, for the next few minutes." It's signed with Gnok's own cloud credentials, derived from your JWT — so the client uploads directly to S3 and never needs any AWS keys. The returned stageRef (e.g. @stage/uploads/<run>/part_0003.parquet) is the logical handle you hand back to COPY INTO.

4. COPY INTO ingests the staged file​

COPY INTO reads the staged file from S3 across Gnok's workers and writes it into the table's managed Iceberg location, computing column statistics and committing a snapshot. The rows_affected count comes back over the SQL connection — the only thing that actually travels through your database connection in the whole pipeline.

Three things that bit us (and how we fixed them)​

JSON-format COPY INTO timed out — CSV/Parquet didn't​

Our first instinct was FILE_FORMAT=(TYPE='JSON'), since the source was already JSON. Every attempt died with a 60-second catalog timeout, even on a 3-row file. The identical load as CSV or Parquet succeeded in seconds. Lesson: for the whole-document-into-one-column pattern, stage as CSV or Parquet, not JSON — even though the data inside that column is JSON.

We were upload-bound — Parquet + zstd fixed it​

Our first working version staged raw CSV. Each ~400 MB chunk took ~60 seconds, almost entirely upload bandwidth — a home uplink pushing 400 MB to S3. Switching the chunk format to zstd-compressed Parquet shrank a 400 MB chunk to ~20 MB (≈8×) and cut per-chunk time to ~4 seconds. A 20-million-row dataset went from a projected 15 minutes to about 90 seconds. If you're upload-bound, compress before you ship.

Tip: COPY INTO does not transparently decompress gzipped CSV — it reads the gzip bytes as raw text and fails. The "compression" you want lives inside the Parquet file (zstd/snappy), not around the CSV.

Tokens expire; compute warms up​

Two operational realities for a multi-hour load:

• JWTs expire every 15 minutes. The loader refreshes proactively and retries any 401.
• Compute can take a moment to become available. COPY INTO needs distributed workers and can return 503 No compute resources available until they're ready, even while SELECT already works. The loader treats 503 (and transient catalog timeouts) as retryable with backoff, so a long load self-heals instead of aborting.

COPY INTO vs REGISTER FILES: copy or move?​

A natural question: since we already uploaded Parquet, does COPY INTO reuse it or rewrite it? It rewrites — reading the staged file and writing fresh Parquet into the table's managed location, with full column statistics and partition routing. Your staged file is treated as a disposable source.

Gnok also offers REGISTER FILES INTO ... FROM '...', which registers existing Parquet in place — it reads only the footers, leaves the files where they are, and moves no data. The trade-offs:

For an id + doc schema, the missing min/max statistics cost nothing (you can't range-prune a JSON blob), which makes REGISTER FILES an appealing way to skip the rewrite for very large datasets — as long as you're comfortable with the table's data living at the staged path. COPY INTO is the safer default because the staged files remain disposable.

The takeaway recipe​

For landing a nested document dataset into Gnok:

1. Stream gunzip -c straight into the loader — no full decompress, flat disk usage. (Use jq -c '.[]' if your source is a single JSON array.)
2. Normalize any typed wrappers; store each doc as id + doc (JSON string), wrap in a PARSE_JSON view.
3. Chunk into zstd Parquet, presign, PUT to S3, COPY INTO.
4. Make the loader resilient: refresh tokens, retry 503/catalog timeouts, keep one chunk on disk.
5. Reach for REGISTER FILES only when the rewrite cost dominates and the staged-file coupling is acceptable.

The result: a schemaless 50 GB pile of JSON documents, queryable in Gnok SQL — doc:any:nested:path and all — without designing a single table by hand.

This post builds a complete example end to end: read data, explore it, engineer features, train a scikit-learn pipeline, register it, and score new rows from SQL — then let Gnok AI write a cell for us.

The gnok helper​

Every notebook kernel comes with a pre-installed gnok helper. You don't pip install it and you don't pass a host or token — it auto-connects as the signed-in user, so row-level security, masking, and vended credentials apply exactly as they do everywhere else in Gnok.

import gnok
from gnok import ml

# A query → pandas
df = gnok.sql("SELECT 1 AS x, 2 AS y").fetch_df()

gnok wraps the standalone pygnok client and adds a gnok.ml namespace. Full reference: Python Notebooks.

1. Read the data​

We'll predict whether a customer will churn. Start by pulling features into a DataFrame — read_table for a whole table, or gnok.sql(...) for anything shaped:

customers = gnok.sql("""
    SELECT
        tenure_months,
        monthly_charges,
        total_charges,
        support_tickets,
        churned                      -- 1 = churned, 0 = retained
    FROM analytics.customers
    WHERE total_charges IS NOT NULL
""").fetch_df()

print(customers.shape)
customers.head()

2. Explore​

It's a normal pandas DataFrame, so all your usual tools work:

customers.describe()
customers.groupby("churned")[["tenure_months", "monthly_charges"]].mean()

import matplotlib.pyplot as plt
customers.boxplot(column="monthly_charges", by="churned")
plt.title("Monthly charges by churn")
plt.show()

Plots render inline in the cell output.

3. Train a model​

A scikit-learn pipeline — scale the features, then a gradient-boosting model. We use a regressor so the model outputs a continuous churn propensity (0–1), which maps cleanly to a SQL FLOAT:

import numpy as np
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.ensemble import GradientBoostingRegressor
from sklearn.model_selection import train_test_split

FEATURES = ["tenure_months", "monthly_charges", "total_charges", "support_tickets"]
X = customers[FEATURES].to_numpy("float32")
y = customers["churned"].to_numpy("float32")

X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.2, random_state=0)

model = Pipeline([
    ("scale", StandardScaler()),
    ("gbr", GradientBoostingRegressor(n_estimators=200, max_depth=3, random_state=0)),
]).fit(X_tr, y_tr)

print("R2 (holdout):", round(model.score(X_te, y_te), 3))

4. Register it for SQL​

One call exports the fitted pipeline to ONNX, uploads it to an internal stage, and runs CREATE OR REPLACE MODEL — the model is now a SQL function:

ml.register_model(model, "analytics.imported.churn_propensity_v1", X)

A few things worth knowing (the helper handles the rest):

• Use a fully-qualified name (catalog.schema.model). The notebook has no default schema, so an unqualified name would land somewhere ML_PREDICT can't find.
• The input signature (FLOAT × 4) is inferred from X — one float per feature, in column order.
• register_model(..., returns="FLOAT") is the default; a classifier's label output is an integer, so use returns="BIGINT" there.

5. Serve it from SQL​

Now anyone — a dashboard, a scheduled query, a BI tool — can score rows. Note the CAST(... AS REAL): bare decimal literals are typed as DECIMAL, which the ONNX input rejects.

SELECT ML_PREDICT(
  'analytics.imported.churn_propensity_v1',
  CAST(tenure_months AS REAL),
  CAST(monthly_charges AS REAL),
  CAST(total_charges AS REAL),
  CAST(support_tickets AS REAL)
) AS churn_score
FROM analytics.customers
ORDER BY churn_score DESC
LIMIT 20;

Materialize the scores for the rest of the business:

CREATE OR REPLACE TABLE analytics.churn_scores AS
SELECT
  customer_id,
  ML_PREDICT('analytics.imported.churn_propensity_v1',
             CAST(tenure_months AS REAL), CAST(monthly_charges AS REAL),
             CAST(total_charges AS REAL), CAST(support_tickets AS REAL)) AS churn_score
FROM analytics.customers;

…or score back from the notebook and write the result:

scored = gnok.sql("SELECT customer_id, ... FROM analytics.churn_scores").fetch_df()
gnok.write(scored.query("churn_score > 0.7"), "analytics.churn_watchlist")

That's the whole loop: the model trained in Python is now ordinary SQL the rest of the platform can call — no model server, no export/import dance.

6. Let Gnok AI write the cell​

Open the ✨ Gnok AI panel and describe what you want:

"Read 1,000 rows of analytics.customers into a DataFrame and plot churn rate by tenure bucket."

It returns a ready-to-run cell — click ＋ Add as cell and run it. The assistant knows the gnok helper, so it reaches for gnok.sql / read_table rather than generic boilerplate. When a cell errors, ✨ Fix sends the traceback back to the model and proposes a corrected cell.

Recap​

read (gnok.sql / read_table) → explore (pandas) → train (sklearn)
   → register (gnok.ml.register_model) → serve (ML_PREDICT in SQL)

No pip install, no connection string, no data leaving Gnok. Dig into the Python Notebooks reference for the full gnok and gnok.ml API, and ONNX Inference for how registered models execute.

If that line of SQL works, a whole class of "I just need to find the rows where X holds, and X isn't a regex" problems disappears. Gnok's AI_FILTER family lands the natural-language predicate in the place where you'd actually write it — alongside =, LIKE, and BETWEEN.

This post walks through the four new functions, the patterns they unlock, and how to keep cost predictable when LLMs are sitting in your hot path.

The four new functions​

These join the existing AI_* family (AI_COMPLETE, AI_SUMMARIZE, AI_SUMMARIZE_AGG, AI_TRANSLATE, AI_EXTRACT_ANSWER, AI_AGG). The unifying idea: the LLM is just a function. SQL already knows how to push, parallelize, and de-duplicate function calls — let it.

The argument order is prompt first, text second — the prompt is a constant baked into the system prompt, the text is the per-row column.

AI_FILTER as a row predicate​

The classic use case is finding rows that match a fuzzy condition no regex catches:

-- "Show me complaints about shipping"
SELECT review_id, body, rating
FROM   reviews
WHERE  AI_FILTER('is a complaint about shipping or delivery', body)
  AND  rating <= 3
ORDER  BY created_at DESC
LIMIT  20;

Two things to notice:

1. The model only sees rows that pass the cheap filter first. The query planner keeps rating <= 3 predicates ahead of the LLM call, so you only pay for the small slice that matters.
2. De-duplication. Under the hood, AI_FILTER rewrites to LOWER(AI_COMPLETE(<prompt> + body)) = 'yes'. AI_COMPLETE de-dupes identical inputs across a batch — if the same body shows up twice, the model is asked once, so the number of model calls is typically less than the row count.

AI_FILTER_AGG — one call per group​

Some predicates are about the group, not the row. "Show me products where reviews mention battery life":

SELECT product_id
FROM   reviews
GROUP  BY product_id
HAVING AI_FILTER_AGG('reviews mention battery life', body)
  AND  COUNT(*) >= 5;

The aggregate concatenates the rows in each group (delimited by the AISQL separator), runs AI_FILTER once over the combined text, and returns the group-level boolean. One LLM call per product instead of one per review.

AI_CLASSIFY_AGG — group-level labels​

Same pattern, classification flavor:

SELECT
    product_id,
    AI_CLASSIFY_AGG(body, ARRAY['quality_issue', 'shipping_issue', 'usability_issue', 'other']) AS top_complaint
FROM    reviews
WHERE   rating <= 2
GROUP   BY product_id;

You get one label per product. The model sees up to N rows of review text per group (clipped at the engine's per-call token budget; longer groups truncate from the end with a warning).

Patterns that work​

Cheap filters first. Always pin AI_FILTER behind a selective non-AI predicate. This isn't a query-planner hint; it's a cost reality. The cheaper your pre-filter, the less you pay.

Materialize for stability. AI calls are non-deterministic. If you need consistent answers across dashboard refreshes, write the filter to a materialized view:

CREATE MATERIALIZED VIEW shipping_complaints AS
SELECT review_id, body, rating
FROM   reviews
WHERE  rating <= 3
  AND  AI_FILTER('is a complaint about shipping or delivery', body);

REFRESH MATERIALIZED VIEW shipping_complaints;

Stable IDs flow into your downstream joins and the cost is paid once per refresh, not once per dashboard view.

LIMIT before AI_FILTER when you can. If the question is "show me 20 examples of X", an ORDER BY ts DESC LIMIT 1000 ahead of the AI predicate caps your worst-case spend at 1,000 LLM calls. The planner can't always do this for you because it doesn't know AI_FILTER is expensive — be explicit.

Keeping an eye on usage​

AI_* calls draw on your organization's monthly AI token budget. Studio shows the budget's used, reserved, and remaining tokens under Operations → AI Governance. Because identical inputs within a batch are sent to the model once, a query usually makes fewer model calls than it has rows.

Bind-time validation​

The functions are special-cased in the binder so syntax errors fire fast:

• AI_FILTER / AI_FILTER_AGG / AI_AGG require exactly two args: (prompt_literal, text).
• AI_CLASSIFY_AGG requires (text_column, labels[]).
• The prompt / labels arg must be a string literal — it gets baked into the system prompt and isn't allowed to vary per row. Per-row prompts would defeat de-duplication and produce nonsensical audit logs.

If you try AI_FILTER(predicate_col, body), you get a clear "AI_FILTER: first argument must be a literal prompt" error at bind time, not a mysterious runtime miss-classification. Empty literals are caught with a separate "non-empty string literal" error.

Why this matters​

Most "natural language → SQL" stories put the LLM in the parser: take a question, generate a query, execute it. That's a different feature (ASK does it). The AI_FILTER family puts the LLM in the execution path as a cell-by-cell function. You write the SQL; the LLM evaluates predicates the engine can't express.

Both have their place. AI_FILTER is the one you reach for when:

• Your data is already structured (you know the table, the columns, the filter logic).
• The thing the regex / LIKE / classifier can't catch is a semantic predicate.
• You want the result to compose with JOIN, GROUP BY, WINDOW, materialized views — every other SQL feature.

Drop one into a WHERE clause against a table you already know and see what happens.

This post walks through how that path is now wired end-to-end: ALTER TABLE … SET SNAPSHOT actually flips the read pointer, the new is_current / parent_snapshot_id / sequence_number columns surface state honestly, and the $snapshots system view makes Iceberg metadata first-class SQL.

A demo you can re-run​

-- 0. Clean slate
DROP TABLE IF EXISTS analytics.hr.staff;

CREATE TABLE analytics.hr.staff (
  id     INT,
  name   VARCHAR,
  salary INT
) COMMENT = 'snapshot demo';

-- 1. INSERT  → snapshot S1 (seq=1, parent=NULL)
INSERT INTO analytics.hr.staff VALUES
  (1, 'Alice', 100),
  (2, 'Bob',    80),
  (3, 'Carol',  90);

-- 2. UPDATE → snapshot S2 (seq=2, parent=S1)
UPDATE analytics.hr.staff SET salary = 150 WHERE id = 1;

-- 3. DELETE → snapshot S3 (seq=3, parent=S2)
DELETE FROM analytics.hr.staff WHERE id = 2;

SELECT id, name, salary FROM analytics.hr.staff ORDER BY id;
-- Alice=150, Carol=90  (no Bob)

Three commits, three snapshots. SHOW SNAPSHOTS returns the expected timeline:

SHOW SNAPSHOTS FROM analytics.hr.staff;

snapshot_id   | is_current | parent_snapshot_id | seq | operation
1777740163004 | false      | NULL               | 1   | append
1777740187411 | false      | 1777740163004      | 2   | overwrite
1777740201632 | true       | 1777740187411      | 3   | delete

Two columns there are new. is_current says which snapshot read queries hit right now. parent_snapshot_id lets you walk the lineage back without a second SHOW TBLPROPERTIES round-trip.

The rollback​

-- Roll back to S1 — clean DDL form
ALTER TABLE analytics.hr.staff SET SNAPSHOT 1777740163004;

Equivalent forms the parser accepts:

ALTER TABLE analytics.hr.staff SET CURRENT SNAPSHOT 1777740163004;
ALTER TABLE analytics.hr.staff SET VERSION AS OF 1777740163004;
ROLLBACK TABLE analytics.hr.staff TO SNAPSHOT 1777740163004;
ROLLBACK TABLE analytics.hr.staff TO PARENT;        -- one snapshot back
ALTER TABLE analytics.hr.staff SET TIMESTAMP AS OF 1777740164000;

The rollback is declarative — it does not write a new commit. It flips the table's main pointer back to S1. Newer snapshots stay on disk so a roll-forward works:

SELECT id, name, salary FROM analytics.hr.staff ORDER BY id;
-- Alice=100, Bob=80, Carol=90  ← original 3 rows back

-- Roll forward to S3
ROLLBACK TABLE analytics.hr.staff TO SNAPSHOT 1777740201632;

SELECT id, name, salary FROM analytics.hr.staff ORDER BY id;
-- Alice=150, Carol=90  ← back to current state

is_current flips correspondingly. No new snapshots get created during rollback — the history stays clean.

$snapshots — system metadata as queryable SQL​

Inspecting Iceberg metadata used to require shelling out to pyiceberg or running SHOW TBLPROPERTIES and parsing the result. The engine now exposes per-table system views you can SELECT from directly. The full set:

-- Every snapshot of a single table
SELECT snapshot_id, is_current, sequence_number, operation
FROM   analytics.hr.staff.$snapshots
ORDER  BY sequence_number DESC
LIMIT  10;

-- The active head only
SELECT snapshot_id, sequence_number, operation
FROM   analytics.hr.staff.$snapshots
WHERE  is_current = TRUE;

-- Oldest non-current snapshot (next candidate for EXPIRE)
SELECT snapshot_id, sequence_number
FROM   analytics.hr.staff.$snapshots
WHERE  NOT is_current
ORDER  BY sequence_number ASC
LIMIT  1;

The views are scoped per-table (<catalog>.<schema>.<table>.$snapshots), not per-schema. For cross-table inventories use information_schema.tables joined to whichever metadata view applies.

Previously, $snapshots parsed but silently ignored any WHERE / ORDER BY / LIMIT you wrote — the system-table path short-circuited the planner. Now they go through the SQL planner properly: filters push down, orderings hold, projections work.

The metadata views inherit the same RBAC surface as regular tables, so audits work without a separate access path.

Snapshots tab in studio​

The studio Catalog Browser's Snapshots tab consumes the new columns directly — the current badge is sourced from is_current, parent-child lineage is shown alongside each row, and the rollback button calls ALTER TABLE … SET SNAPSHOT under the hood.

If you have a tab open against a rolled-back table, the result panel shows an amber "rolled back" chip below the row count — because reading from a non-newest snapshot is the kind of bug people lose hours to, and silently doing it is worse than the warning.

Why it matters​

Time travel is one of Iceberg's headline features. Half-shipped time travel is a worse experience than no time travel at all — users learn not to trust it. The new path is honest about what it does and exposes the metadata you'd want to verify it.

Try the demo above against your own catalog and watch is_current flip.

The advisors run continuously as part of the service. This post walks through the surface that lets you see what they've learned: three SHOW … RECOMMENDATIONS commands that expose the live state of the join-order, materialized-view, and overall AutoML caches.

SHOW RECOMMENDATIONS — is AutoML running?​

SHOW RECOMMENDATIONS;

enabled | total_queries_recorded | tables_tracked
true    | 12,847                 | 84

That's the AutoML status summary. enabled shows whether AutoML advisors are on; total_queries_recorded counts the plans AutoML has observed; tables_tracked is the number of distinct base tables in the workload profile.

If enabled = false and you expected otherwise, check with an administrator: Studio shows the same status under Operations → AutoML Control. See AutoML Advisors for what is available to your account.

SHOW JOIN ORDER RECOMMENDATIONS​

The join-order advisor watches the planner's choices and remembers which join orders consistently produced the cheapest plans for each table-set:

SHOW JOIN ORDER RECOMMENDATIONS;

tables_hash | recommended_order                      | preferred_root | confidence | obs_count | sql_hint
8a2f1b…     | analytics.sales.orders, …customers, … | orders         | 0.92       | 184       | /*+ JOIN_ORDER(orders, customers, payments) */
b41c…       | analytics.events.clicks, …users        | users          | 0.71       | 36        | /*+ JOIN_ORDER(users, clicks) */

Each row represents one join-shape the advisor has observed enough times to score. confidence is bounded in [0, 1]; the sql_hint is a paste-ready optimizer hint you can drop into a query that the planner is mis-ordering. obs_count tells you how much weight to put on the recommendation — a 0.92-confidence row backed by 184 observations is a different story than a 0.92 backed by 4.

The advisor surfaces these as passive recommendations — the planner already uses the underlying cache when it can; the hint is for the cases where the user wants to pin the choice.

SHOW MV RECOMMENDATIONS​

Repeated query patterns the advisor scores as worth materializing:

SHOW MV RECOMMENDATIONS;

fingerprint | tables                          | filter_columns | repeated_count | savings_score | last_seen           | sql_hint
4a91b3      | analytics.sales.orders          | created_at     | 47             | 22.1          | 2026-05-04T14:22:01 | CREATE MATERIALIZED VIEW mv_4a91b3 AS …
e7203c      | analytics.events.clicks         | event_date,    | 28             | 18.7          | 2026-05-05T10:11:33 | CREATE MATERIALIZED VIEW mv_e7203c AS …
            |                                 | user_id        |                |               |                     |

savings_score is the planner's estimate of "total cost saved across observed queries if this MV existed" — a function of estimated MV scan cost, base-table scan cost, and how often the fingerprint shows up. repeated_count is the raw count; last_seen lets you spot stale recommendations (a workload shift means yesterday's top candidate may not be today's).

The sql_hint column is a paste-ready CREATE MATERIALIZED VIEW statement.

What's actually learned​

The partition and index advisors score candidate strategies for each base table the workload touches. Learned scorers (gradient-boosted models trained on query history) augment the original heuristics:

• The partition advisor scores (column, strategy) pairs across the candidate set. Strategies include identity, hash, bucket, truncate, day, hour-of. Features: column statistics, workload filter/join/group-by frequencies, predicate distribution, estimated build cost.
• The index advisor scores (column, kind) pairs — kinds are btree, bloom, bitmap, hnsw, bm25. Same feature base, different fit checks.

Gnok applies recommendations passively (e.g. the planner consults the join-order cache during its own search, even when no SHOW is called). The SHOW … RECOMMENDATIONS commands surface what has accumulated for human inspection.

What's not in scope (yet)​

• No per-table SQL command. There's no SUGGEST PARTITION FOR <table> form today. The advisor runs over the full workload-profile cache; surfacing per-table is on the roadmap alongside an EXPLAIN ADVISOR introspection command.
• Auto-apply. AutoML doesn't apply schema- changing recommendations on its own. Materialized-view creation and index builds remain explicit actions you take (with the sql_hint as a paste-ready starting point).
• Cross-tenant generalization. The learned scorers start from benchmark-trained baselines; tuning for your workload happens as your queries accumulate.

Try it​

SHOW RECOMMENDATIONS;
SHOW JOIN ORDER RECOMMENDATIONS;
SHOW MV RECOMMENDATIONS;

Run those after your account has been serving real workloads for a few hours. If the second and third return zero rows, either the workload is too uniform to recommend anything useful, or AutoML is disabled (check the first command's enabled column).

The recommendation cache is rebuilt continuously — re-running the SHOWs after a few hundred more queries should show the scores stabilize as the advisor accumulates evidence.

This post walks through the three biggest changes — filtered ANN, offline-backed feature lookups, and the schema-aware ML autocomplete — with the SQL you can run today.

1. Filtered ANN: vector search with WHERE clauses, fast​

Production vector queries are almost never "find me 10 things similar to X". They're "find me 10 things similar to X that are in stock and in my region". Until this release, that WHERE clause forced the planner to brute-force the scan over the entire table — defeating the purpose of having an HNSW index.

Now the planner picks among three strategies based on the predicate's selectivity:

Same query — the planner chooses transparently:

-- IndexOnly: no predicate. Pure ANN.
SELECT id, title
FROM documents
ORDER BY COSINE_DISTANCE(embedding, EMBED('refund policy'))
LIMIT 10;

-- PostFilter: predicate is ~30% selective. The planner emits
-- VectorIndexTopK with k=80 (overfetch=8x), Filter, then
-- LIMIT 10 — bounded at 32x to keep memory predictable.
SELECT id, title
FROM documents
WHERE region IN ('us-east', 'us-west')
ORDER BY COSINE_DISTANCE(embedding, EMBED('refund policy'))
LIMIT 10;

You can inspect the planner's choice with EXPLAIN:

EXPLAIN
SELECT id, title FROM documents
WHERE filed_at > '2026-01-01'
ORDER BY COSINE_DISTANCE(embedding, EMBED('contract dispute'))
LIMIT 50;

A VectorIndexTopK { k = 800, ... } under a Filter → Limit(50) tells you PostFilter fired with 16× overfetch — the query is going to be ~10× faster than the brute-force fallback while still returning the right top-50.

Industries​

• E-commerce. "Headphones similar to this, in stock and under $200." The price + stock predicate is moderately selective → PostFilter wins.
• Legal e-discovery. "Documents semantically similar to this contract, filed in this 90-day window, by these custodians." Custodian + time predicates are very selective → PreFilter gives the best latency.
• Streaming personalization. "Shows like Severance, available in my region." Region is non-selective in most catalogues → PostFilter with modest overfetch.

2. Offline-Backed Feature Store: point-in-time training without leaks​

Online features (the current snapshot, low latency) and offline features (historical, for training) used to be two separate systems with two separate APIs and two opportunities to leak. Gnok's feature store now serves both from a single SQL surface, backed by Iceberg time-travel underneath.

The CREATE side​

Add an OFFLINE TABLE clause to a feature group and every materialization runs an INSERT OVERWRITE into the Iceberg table. The latest snapshot stays in memory for online lookups; older snapshots are reachable through Iceberg's snapshot history.

CREATE FEATURE GROUP user_transaction_features (
    user_id BIGINT PRIMARY KEY
)
MATERIALIZED FROM (
    SELECT user_id,
           COUNT(*)        AS tx_count,
           AVG(amount)     AS avg_amount,
           MAX(created_at) AS last_tx_time
    FROM transactions
    GROUP BY user_id
)
REFRESH EVERY 300
OFFLINE TABLE warehouse.feature_store.user_transaction_features;

The lookup side​

Two UDFs — same SQL surface, different semantics:

-- Online: live snapshot, O(1), microseconds
SELECT t.transaction_id,
       FEATURE_LOOKUP('user_transaction_features',
                      t.user_id,
                      'avg_amount') AS avg_amt
FROM transactions t
WHERE t.created_at > now() - INTERVAL '5 minutes';

-- Point-in-time: rebuilds the snapshot in effect at as_of_ms.
-- First checks the in-memory snapshot ring; if the timestamp is
-- older than the ring retains, falls through to the offline
-- Iceberg table via FOR SYSTEM_TIME AS OF time-travel.
SELECT lbl.user_id, lbl.churned,
       FEATURE_LOOKUP_AS_OF(
           'user_transaction_features',
           lbl.user_id,
           'avg_amount',
           CAST(EXTRACT(EPOCH FROM lbl.observed_at) * 1000 AS BIGINT)
       ) AS avg_at_label_time
FROM training_labels lbl;

For training joins that pull many features per entity, use a SQL JOIN against the offline table directly with explicit time-travel — the planner pushes down better than per-feature UDF calls:

-- Multi-feature join: every column reflects the snapshot in effect on Jan 1
SELECT lbl.user_id, lbl.churned,
       fg.avg_amount, fg.tx_count, fg.last_tx_time
FROM training_labels lbl
LEFT JOIN warehouse.feature_store.user_transaction_features
          FOR SYSTEM_TIME AS OF '2026-01-01T00:00:00Z' AS fg
  ON lbl.user_id = fg.user_id;

Why this matters​

The classic mistake in ML training is temporal leakage: a training row with label "user churned on Feb 1" enriched with features computed today (Apr 28) — the features know things they shouldn't. The new FEATURE_LOOKUP_AS_OF surface makes the correct semantic the easy one: pass the label's observation timestamp, get the features as they existed at that point, no manual snapshot management.

Industries​

• Fintech / underwriting. A claim model joining 20 features for one applicant, all at the application's submission timestamp. No leakage; reproducible.
• Subscription churn modelling. Train on "what did this user look like two weeks before they churned" without time machine hacks.
• Insurance. Auditable point-in-time joins are a regulatory requirement, not a nice-to-have.

3. Schema-Aware ML Autocomplete​

The SQL editor used to suggest ML_PREDICT and FEATURE_LOOKUP as static keywords. Now, when your cursor sits inside the function's string-literal arg, it suggests the actual model and feature-group names registered in your catalog:

ML_PREDICT('fr|
            ↑ cursor here
              suggestions:
                fraud_detector            model · onnx · v3
                fraud_classifier          model · pytorch · v1
                fr_credit_scoring         model · sklearn · v2

Studio looks up the models, vector indexes, and feature groups you can see, matches them by case-insensitive substring, and shows each suggestion's kind, name, and a one-line detail ("model · onnx · v3", "feature group · keys: user_id · 12 features").

The same autocomplete fires inside FEATURE_LOOKUP('|, FEATURE_LOOKUP_AS_OF('|, EXPLAIN MODEL '|, and ALTER MODEL '| — Studio detects which kind of object you're naming and only shows that kind.

Why this matters​

In a tenant with hundreds of models and feature groups, you stop tab-switching between the editor and the catalog browser to remember names. For onboarding new analysts, the editor itself becomes a discoverability surface.

What's next​

Roadmap items that build on this foundation:

• In-traversal attribute filtering so HNSW prunes during the graph walk
• Multi-group point-in-time joins that fan out across feature groups in parallel
• Per-feature regression detection — catch models that go bad on a high-value slice while looking flat in aggregate
• Drag-and-drop feature group composer in Studio for citizen data scientists

If any of the industry scenarios above match yours, let us know — sequencing follows real demand.

Architecture​

Everything runs inside Gnok. No Kafka, no external model servers, no ETL pipelines.

Step 1: Set Up the Schema​

CREATE SCHEMA fraud_demo;
USE SCHEMA fraud_demo;

-- Transaction table
CREATE TABLE transactions (
    tx_id           BIGINT NOT NULL,
    user_id         BIGINT NOT NULL,
    merchant_id     BIGINT,
    amount          DECIMAL(10, 2),
    currency        VARCHAR,
    tx_type         VARCHAR,
    channel         VARCHAR,
    country         VARCHAR,
    is_international BOOLEAN,
    hour_of_day     INT,
    day_of_week     INT,
    tx_timestamp    TIMESTAMP
)
PARTITIONED BY (day(tx_timestamp));

-- Merchant table with embeddings
CREATE TABLE merchants (
    merchant_id     BIGINT NOT NULL,
    name            VARCHAR,
    category        VARCHAR,
    risk_tier       VARCHAR,
    avg_ticket      DECIMAL(10, 2),
    embedding       VECTOR(128)
);

Step 2: Bulk Load Historical Data​

Upload transaction data to a stage, then load it:

-- Load historical transactions from Parquet
COPY INTO transactions
FROM '@stage/fraud_demo/transactions/'
FILE_FORMAT = (TYPE = PARQUET);

-- Load merchant data from CSV
COPY INTO merchants
FROM '@stage/fraud_demo/merchants/'
FILE_FORMAT = (
    TYPE = CSV,
    DELIMITER = ',',
    SKIP_ROWS = 1
);

Gnok's COPY INTO automatically parallelizes across workers and routes rows to the correct partitions. For our transaction table partitioned by day(tx_timestamp), each day's data ends up in optimally-sized files.

Verify the Load​

SELECT COUNT(*) AS total_txns,
       MIN(tx_timestamp) AS earliest,
       MAX(tx_timestamp) AS latest
FROM transactions;

SELECT COUNT(*) AS merchant_count,
       COUNT(DISTINCT category) AS categories
FROM merchants;

Step 3: Build User Behavior Profiles​

Fraud detection works best when you can compare a transaction against the user's normal behavior. Let's compute per-user statistics:

CREATE TABLE user_profiles AS
SELECT
    user_id,
    COUNT(*)                          AS tx_count_total,
    AVG(amount)                       AS avg_amount,
    STDDEV(amount)                    AS stddev_amount,
    PERCENTILE_CONT(0.95) WITHIN GROUP (ORDER BY amount) AS p95_amount,
    COUNT(DISTINCT merchant_id)       AS unique_merchants,
    COUNT(DISTINCT country)           AS unique_countries,
    SUM(CASE WHEN is_international THEN 1 ELSE 0 END) AS intl_tx_count,
    MODE(channel)                     AS preferred_channel,
    MAX(tx_timestamp)                 AS last_tx_time
FROM transactions
GROUP BY user_id;

Check the profile for a specific user:

SELECT * FROM user_profiles WHERE user_id = 12345;

Step 4: Statistical Anomaly Scoring​

Now let's build a scoring query that flags anomalies using multiple signals.

Amount Anomaly (Z-Score)​

Compare each transaction's amount against the user's historical distribution:

SELECT
    t.tx_id,
    t.user_id,
    t.amount,
    u.avg_amount,
    u.stddev_amount,
    -- Z-score: how many standard deviations from the user's mean
    CASE
        WHEN u.stddev_amount > 0
        THEN (t.amount - u.avg_amount) / u.stddev_amount
        ELSE 0
    END AS amount_zscore,
    -- Flag if more than 2.5 standard deviations
    CASE
        WHEN u.stddev_amount > 0
             AND ABS((t.amount - u.avg_amount) / u.stddev_amount) > 2.5
        THEN true
        ELSE false
    END AS amount_anomaly
FROM transactions t
JOIN user_profiles u ON t.user_id = u.user_id
WHERE t.tx_timestamp > CURRENT_TIMESTAMP - INTERVAL '1' HOUR;

Velocity Anomaly​

Flag users with unusual transaction frequency:

WITH recent_velocity AS (
    SELECT
        user_id,
        COUNT(*) AS tx_last_hour,
        SUM(amount) AS total_last_hour
    FROM transactions
    WHERE tx_timestamp > CURRENT_TIMESTAMP - INTERVAL '1' HOUR
    GROUP BY user_id
)
SELECT
    v.user_id,
    v.tx_last_hour,
    v.total_last_hour,
    -- Compare against historical daily average
    u.tx_count_total / 30.0 AS avg_daily_rate,
    -- Velocity ratio: how many times above normal
    v.tx_last_hour / GREATEST(u.tx_count_total / 720.0, 0.1) AS velocity_ratio
FROM recent_velocity v
JOIN user_profiles u ON v.user_id = u.user_id
WHERE v.tx_last_hour / GREATEST(u.tx_count_total / 720.0, 0.1) > 5.0;

Geographic Anomaly​

Flag transactions from countries the user has never used before:

WITH user_countries AS (
    SELECT user_id, country
    FROM transactions
    GROUP BY user_id, country
)
SELECT
    t.tx_id,
    t.user_id,
    t.country AS tx_country,
    true AS new_country_flag
FROM transactions t
LEFT JOIN user_countries uc
    ON t.user_id = uc.user_id AND t.country = uc.country
WHERE t.tx_timestamp > CURRENT_TIMESTAMP - INTERVAL '1' HOUR
  AND uc.country IS NULL;

Step 5: Merchant Similarity Analysis​

Use vector distance to find merchants similar to known high-risk merchants:

-- Find merchants whose embeddings are closest to a known fraudulent merchant
WITH fraud_merchant AS (
    SELECT embedding
    FROM merchants
    WHERE merchant_id = 9999  -- known fraud merchant
)
SELECT
    m.merchant_id,
    m.name,
    m.category,
    COSINE_DISTANCE(m.embedding, fm.embedding) AS similarity_distance
FROM merchants m
CROSS JOIN fraud_merchant fm
WHERE m.merchant_id <> 9999
ORDER BY similarity_distance ASC
LIMIT 20;

Compute average embedding per merchant category for cluster analysis:

SELECT
    category,
    COUNT(*) AS merchant_count,
    VECTOR_AVG(embedding) AS category_centroid
FROM merchants
GROUP BY category;

Step 6: Combined Risk Score​

Pull all the signals together into a single scoring query:

CREATE VIEW fraud_scores AS
WITH amount_scores AS (
    SELECT
        t.tx_id,
        t.user_id,
        t.merchant_id,
        t.amount,
        t.country,
        t.tx_timestamp,
        t.is_international,
        CASE
            WHEN u.stddev_amount > 0
            THEN ABS((t.amount - u.avg_amount) / u.stddev_amount)
            ELSE 0
        END AS amount_zscore,
        t.amount > u.p95_amount AS exceeds_p95
    FROM transactions t
    JOIN user_profiles u ON t.user_id = u.user_id
    WHERE t.tx_timestamp > CURRENT_TIMESTAMP - INTERVAL '1' HOUR
),
velocity_scores AS (
    SELECT
        user_id,
        COUNT(*) AS tx_last_hour
    FROM transactions
    WHERE tx_timestamp > CURRENT_TIMESTAMP - INTERVAL '1' HOUR
    GROUP BY user_id
),
geo_scores AS (
    SELECT DISTINCT t.tx_id, true AS new_country
    FROM transactions t
    LEFT JOIN (
        SELECT user_id, country FROM transactions GROUP BY user_id, country
    ) hist ON t.user_id = hist.user_id AND t.country = hist.country
    WHERE t.tx_timestamp > CURRENT_TIMESTAMP - INTERVAL '1' HOUR
      AND hist.country IS NULL
)
SELECT
    a.tx_id,
    a.user_id,
    a.amount,
    a.country,
    a.tx_timestamp,
    m.category AS merchant_category,
    m.risk_tier AS merchant_risk_tier,

    -- Individual signals
    a.amount_zscore,
    a.exceeds_p95,
    COALESCE(v.tx_last_hour, 0) AS tx_last_hour,
    COALESCE(g.new_country, false) AS new_country,

    -- Combined risk score (weighted sum, 0-100)
    LEAST(100, ROUND(
        CASE WHEN a.amount_zscore > 3.0 THEN 30 WHEN a.amount_zscore > 2.0 THEN 15 ELSE 0 END
        + CASE WHEN a.exceeds_p95 THEN 15 ELSE 0 END
        + CASE WHEN COALESCE(v.tx_last_hour, 0) > 10 THEN 25 WHEN COALESCE(v.tx_last_hour, 0) > 5 THEN 10 ELSE 0 END
        + CASE WHEN COALESCE(g.new_country, false) THEN 20 ELSE 0 END
        + CASE WHEN m.risk_tier = 'high' THEN 15 WHEN m.risk_tier = 'medium' THEN 5 ELSE 0 END
        + CASE WHEN a.is_international THEN 5 ELSE 0 END
    )) AS risk_score,

    -- Risk level classification
    CASE
        WHEN LEAST(100, ROUND(
            CASE WHEN a.amount_zscore > 3.0 THEN 30 WHEN a.amount_zscore > 2.0 THEN 15 ELSE 0 END
            + CASE WHEN a.exceeds_p95 THEN 15 ELSE 0 END
            + CASE WHEN COALESCE(v.tx_last_hour, 0) > 10 THEN 25 WHEN COALESCE(v.tx_last_hour, 0) > 5 THEN 10 ELSE 0 END
            + CASE WHEN COALESCE(g.new_country, false) THEN 20 ELSE 0 END
            + CASE WHEN m.risk_tier = 'high' THEN 15 WHEN m.risk_tier = 'medium' THEN 5 ELSE 0 END
            + CASE WHEN a.is_international THEN 5 ELSE 0 END
        )) >= 60 THEN 'CRITICAL'
        WHEN LEAST(100, ROUND(
            CASE WHEN a.amount_zscore > 3.0 THEN 30 WHEN a.amount_zscore > 2.0 THEN 15 ELSE 0 END
            + CASE WHEN a.exceeds_p95 THEN 15 ELSE 0 END
            + CASE WHEN COALESCE(v.tx_last_hour, 0) > 10 THEN 25 WHEN COALESCE(v.tx_last_hour, 0) > 5 THEN 10 ELSE 0 END
            + CASE WHEN COALESCE(g.new_country, false) THEN 20 ELSE 0 END
            + CASE WHEN m.risk_tier = 'high' THEN 15 WHEN m.risk_tier = 'medium' THEN 5 ELSE 0 END
            + CASE WHEN a.is_international THEN 5 ELSE 0 END
        )) >= 40 THEN 'HIGH'
        WHEN LEAST(100, ROUND(
            CASE WHEN a.amount_zscore > 3.0 THEN 30 WHEN a.amount_zscore > 2.0 THEN 15 ELSE 0 END
            + CASE WHEN a.exceeds_p95 THEN 15 ELSE 0 END
            + CASE WHEN COALESCE(v.tx_last_hour, 0) > 10 THEN 25 WHEN COALESCE(v.tx_last_hour, 0) > 5 THEN 10 ELSE 0 END
            + CASE WHEN COALESCE(g.new_country, false) THEN 20 ELSE 0 END
            + CASE WHEN m.risk_tier = 'high' THEN 15 WHEN m.risk_tier = 'medium' THEN 5 ELSE 0 END
            + CASE WHEN a.is_international THEN 5 ELSE 0 END
        )) >= 20 THEN 'MEDIUM'
        ELSE 'LOW'
    END AS risk_level

FROM amount_scores a
LEFT JOIN velocity_scores v ON a.user_id = v.user_id
LEFT JOIN geo_scores g ON a.tx_id = g.tx_id
LEFT JOIN merchants m ON a.merchant_id = m.merchant_id;

Query the Scores​

-- Top 20 highest-risk transactions in the last hour
SELECT * FROM fraud_scores
ORDER BY risk_score DESC
LIMIT 20;

-- Summary by risk level
SELECT
    risk_level,
    COUNT(*) AS tx_count,
    SUM(amount) AS total_amount,
    AVG(risk_score) AS avg_score
FROM fraud_scores
GROUP BY risk_level
ORDER BY avg_score DESC;

-- Critical alerts with full context
SELECT
    tx_id, user_id, amount, country,
    merchant_category, merchant_risk_tier,
    risk_score,
    amount_zscore,
    tx_last_hour,
    new_country
FROM fraud_scores
WHERE risk_level = 'CRITICAL'
ORDER BY risk_score DESC;

Step 7: Connect a Dashboard​

Connect your BI tool to your account's assigned PostgreSQL-compatible endpoint (see Connect an Application and PostgreSQL wire protocol), or build a Studio dashboard, on the fraud_scores view:

Dashboard panels:

• Risk distribution — pie chart of risk_level counts
• Top flagged transactions — table sorted by risk_score DESC
• Risk by merchant category — bar chart of avg risk_score per category
• Hourly trend — time series of CRITICAL + HIGH transaction counts
• Geographic heat map — risk scores by country

Step 8: Investigate with Natural Language​

When an alert fires, use ASK for quick investigation. ASK uses Gnok's AI provider, so an organization administrator must first enable AI Governance for your organization (see Natural Language Queries):

-- Set context
USE CATALOG my_catalog;
USE SCHEMA fraud_demo;

-- Investigate
ASK 'Show me all transactions for user 12345 in the last 24 hours, sorted by time';

ASK 'What merchant categories have the most critical-risk transactions today?';

ASK 'How does today''s high-risk transaction count compare to the last 7 days?';

ASK 'Find users with more than 10 transactions in the last hour';

Step 9: Time Travel for Forensics​

When investigating a confirmed fraud case, use time travel to reconstruct what happened:

-- What did the user's profile look like before the fraud?
SELECT * FROM user_profiles
AT(TIMESTAMP => '2026-03-15 08:00:00')
WHERE user_id = 12345;

-- Compare with current profile
SELECT * FROM user_profiles
WHERE user_id = 12345;

-- Track how the user's transaction pattern changed
SELECT
    DATE_TRUNC('day', tx_timestamp) AS day,
    COUNT(*) AS tx_count,
    SUM(amount) AS total_amount,
    AVG(amount) AS avg_amount,
    COUNT(DISTINCT country) AS countries
FROM transactions
WHERE user_id = 12345
  AND tx_timestamp > CURRENT_TIMESTAMP - INTERVAL '30' DAY
GROUP BY DATE_TRUNC('day', tx_timestamp)
ORDER BY day;

Step 10: Ongoing Maintenance​

Keep the pipeline running smoothly:

-- Refresh user profiles (run hourly or on a schedule)
CREATE TABLE user_profiles_new AS
SELECT /* same query as Step 3 */ ...;
DROP TABLE user_profiles;
ALTER TABLE user_profiles_new RENAME TO user_profiles;

-- Compact transaction data (run daily)
OPTIMIZE TABLE transactions;

-- Clean up old snapshots (run weekly)
ALTER TABLE transactions EXECUTE EXPIRE_SNAPSHOTS
SET ('older-than' = '2026-03-01T00:00:00Z');
VACUUM TABLE transactions;

Summary​

What we built:

Apart from the ASK investigation step, which calls Gnok's AI provider, all of this runs inside Gnok. The scoring view executes in-engine with partition pruning, SIMD-accelerated aggregation, and distributed MPP execution.

Further Reading​

• AI & ML Tutorials -- model training, ONNX inference, feature groups, streaming scoring
• Vector Operations -- distance functions reference
• Time Travel -- snapshot query syntax
• COPY INTO -- bulk loading reference
• Table Maintenance -- OPTIMIZE, VACUUM schedules
• Connect an Application -- SQL clients and BI tools

This post explains what each feature does, when to use it, and how to get started.

Format Versions at a Glance​

Iceberg defines three format versions, each building on the previous:

Gnok defaults to v2 when creating tables. To use v3 features, specify the format version explicitly:

CREATE TABLE events (
    id BIGINT,
    payload VARIANT,
    location GEOMETRY
) WITH ('format-version' = '3');

You can also upgrade an existing table:

ALTER TABLE events SET TBLPROPERTIES ('format-version' = '3');

Deletion Vectors​

The Problem with Position Deletes​

In v2, every UPDATE or DELETE writes a position delete file -- a Parquet file containing (file_path, pos) pairs that mark which rows are deleted. This works, but delete files accumulate over time. A table with frequent updates can end up with hundreds of delete files, each of which must be read and merged during every scan.

Data file (1M rows)
  ├── Position delete file #1 (50 rows)
  ├── Position delete file #2 (30 rows)
  ├── Position delete file #3 (120 rows)
  └── ... (N more delete files)

Each scan must load all N delete files, build an index, and filter every row against it. At scale this becomes the dominant cost of reads.

How Deletion Vectors Work​

Deletion vectors replace individual delete files with a single, compact Roaring bitmap stored in a Puffin file. Instead of N separate Parquet files with (file_path, pos) pairs, each data file has at most one deletion vector containing the set of deleted row positions.

Data file (1M rows)
  └── Deletion vector (Roaring bitmap, ~2KB) → {pos: 12, 47, 193, ...}

The binary format is compact:

Combined Length (4 bytes)
Magic: 0xD1D33964 (4 bytes)
64-bit Roaring Bitmap (variable)
CRC-32C checksum (4 bytes)

Roaring bitmaps are extremely space-efficient for sparse sets (a few thousand deletes in a million-row file) and support O(1) membership tests. The bitmap is stored in a Puffin blob with LZ4 or Zstd compression.

Current Support​

Gnok fully supports reading deletion vectors. During scan planning, DVs are matched to data files by referenced_data_file path, preloaded, and merged with any existing position deletes. The write path currently produces v2-style position delete files; DV write support is planned.

What This Means for You​

If you produce data with a writer that creates deletion vectors (e.g., Spark 4.0), Gnok can read those tables efficiently today. For Gnok-originated writes, position deletes are used until DV write support ships. Run OPTIMIZE PURGE DELETES periodically to consolidate accumulated delete files.

VARIANT Type​

Semi-Structured Data, Natively​

JSON columns stored as VARCHAR work for simple cases, but they have drawbacks: no type safety, no columnar pruning, and every access requires parsing the entire string. The VARIANT type solves this by storing semi-structured data in Iceberg's native binary variant format with optional shredding -- extracting frequently accessed fields into separate columns for efficient columnar reads.

Creating and Querying VARIANT Data​

-- Create a table with a VARIANT column
CREATE TABLE events (
    event_id BIGINT,
    event_type VARCHAR,
    payload VARIANT
) WITH ('format-version' = '3');

-- Insert JSON data as VARIANT
INSERT INTO events VALUES
    (1, 'click', PARSE_JSON('{"page": "/home", "duration_ms": 1200, "user": "alice"}')),
    (2, 'purchase', PARSE_JSON('{"item": "widget", "price": 29.99, "quantity": 3}')),
    (3, 'click', PARSE_JSON('{"page": "/products", "duration_ms": 800}'));

Accessing Nested Values​

Use VARIANT_GET to extract typed values at a dot-path:

SELECT
    event_id,
    event_type,
    VARIANT_GET(payload, 'page', 'VARCHAR') AS page,
    VARIANT_GET(payload, 'duration_ms', 'INT') AS duration_ms,
    VARIANT_GET(payload, 'price', 'DOUBLE') AS price
FROM events;

Result:

event_id | event_type | page      | duration_ms | price
---------+------------+-----------+-------------+------
       1 | click      | /home     |        1200 | NULL
       2 | purchase   | NULL      |        NULL | 29.99
       3 | click      | /products |         800 | NULL

Function Reference​

When to Use VARIANT vs JSON​

GEOMETRY and GEOGRAPHY Types​

Spatial Analytics in SQL​

v3 adds native spatial types following PostGIS naming conventions. GEOMETRY operates in Cartesian coordinates (planar); GEOGRAPHY uses geodesic calculations (great-circle distances on the WGS84 ellipsoid).

Example: Store Locator​

-- Create a table with spatial columns
CREATE TABLE stores (
    store_id BIGINT,
    name VARCHAR,
    location GEOMETRY
) WITH ('format-version' = '3');

-- Insert store locations
INSERT INTO stores VALUES
    (1, 'Downtown', ST_POINT(-73.9857, 40.7484)),
    (2, 'Midtown', ST_POINT(-73.9787, 40.7614)),
    (3, 'Brooklyn', ST_POINT(-73.9442, 40.6782));

-- Find stores within 2km of a location
SELECT
    name,
    ST_DISTANCE(location, ST_POINT(-73.98, 40.75)) AS distance_m
FROM stores
WHERE ST_DWITHIN(location, ST_POINT(-73.98, 40.75), 2000)
ORDER BY distance_m;

Spatial Pruning​

Gnok applies spatial pruning at the scan level using bounding box statistics stored in file metadata. Queries with ST_WITHIN, ST_CONTAINS, ST_INTERSECTS, or ST_DWITHIN predicates can skip entire data files whose bounding boxes don't overlap the query region.

CRS Support​

Transform between projections with ST_TRANSFORM:

SELECT ST_TRANSFORM(location, 3857) AS web_mercator_point
FROM stores;

Full Function List​

ST_POINT, ST_DISTANCE, ST_WITHIN, ST_CONTAINS, ST_INTERSECTS, ST_AREA, ST_LENGTH, ST_ASTEXT, ST_GEOMFROMTEXT, ST_GEOGFROMTEXT, ST_X, ST_Y, ST_SRID, ST_TRANSFORM, ST_BUFFER, ST_DWITHIN, ST_ENVELOPE