From JSON to Iceberg: Bulk-Loading a Document Dataset into Gnok

We recently needed to land a large pile of JSON documents — roughly 50 GB across ~70 datasets, including a couple of 15+ GB monsters — into Gnok so it could be queried with SQL alongside everything else. The documents were schemaless and deeply nested; Gnok tables are columnar Iceberg. This post walks through the pipeline we landed on, and the handful of gotchas that shaped it.

The shape of the problem

Three constraints drove the design:

1. The documents are heterogeneous and nested. Datasets like users, orders, events, and sessions share no schema, and individual documents carry sub-objects and arrays. Designing a bespoke relational schema per dataset would have been weeks of work.
2. Some datasets are huge. An event_history-style export alone was 17 GB compressed — far too big to expand onto a laptop and re-process from disk.
3. The data has to get from a laptop to the server. Gnok ingests from object storage, not from the SQL wire. So "loading" really means "get files into S3, then tell Gnok to read them."

The pipeline

For each dataset we run a single streaming pass:

gunzip -c <dataset>.json.gz \
  | python loader.py <table>     # normalize, chunk, upload, COPY INTO

The loader consumes newline-delimited JSON (NDJSON — one document per line) and never holds a whole dataset on disk: it converts on the fly and keeps exactly one chunk in flight. Here's what each chunk goes through.

1. Stream straight from the compressed file

Expanding 17 GB of gzipped JSON onto disk just to read it is wasteful — and on a laptop, sometimes impossible. Instead we pipe gunzip -c directly into the loader, so the giant datasets flow through memory a line at a time and never touch local disk in full.

If your export is one big pretty-printed JSON array rather than NDJSON, a streaming parser gets you the same line-at-a-time behavior — jq -c '.[]' (or an ijson pass) emits one compact document per line without materializing the whole array.

This is also a clean path for document databases: MongoDB, for instance, exports straight to NDJSON via mongoexport --type=json, and you can decode a raw .bson dump to the same JSON-lines stream with bsondump — no server to stand up. Either one drops directly into the pipeline below.

2. One schema to rule them all: id + doc VARIANT

Rather than model each dataset, every table is just two columns:

CREATE TABLE my_db.json.users_raw (id VARCHAR, doc VARCHAR);

id is the document's identifier; doc is the entire document as a JSON string. After load we wrap it in a view that parses the JSON into a queryable VARIANT:

CREATE VIEW my_db.json.users AS
  SELECT id, PARSE_JSON(doc) AS doc FROM my_db.json.users_raw;

Now nested access just works, with zero per-dataset schema work:

SELECT doc:profile:email::varchar AS email,
       doc:address:city::varchar  AS city
FROM my_db.json.users
WHERE doc:status::varchar = 'active';

One normalization detail worth calling out: many document exporters wrap scalars in typed objects rather than emitting plain values — an id might arrive as {"$oid":"..."} and a date as {"$date":{"$numberLong":"..."}}. If your source does this, rewrite those wrappers into plain values ("...", "2026-02-28T23:55:10Z") in the loader before storing, so queries see clean strings and ISO timestamps instead of wrapper objects.

3. The upload: presigned URLs to S3

This is the part worth understanding. Gnok doesn't accept bulk data over the SQL connection (COPY ... FROM STDIN isn't supported). Data lands in object storage first, then a SQL statement points Gnok at it. The flow per chunk:

POST /v1/stages/presign   { stage, runId, filename }   ->   { url, stageRef }
PUT  <url>  (the file bytes)                            ->   S3 (200 OK)
COPY INTO <table> FROM '<stageRef>' FILE_FORMAT=(TYPE='PARQUET')

A presigned URL is a temporary, signed S3 URL that grants "you may PUT this one object, for the next few minutes." It's signed with Gnok's own cloud credentials, derived from your JWT — so the client uploads directly to S3 and never needs any AWS keys. The returned stageRef (e.g. @stage/uploads/<run>/part_0003.parquet) is the logical handle you hand back to COPY INTO.

4. COPY INTO ingests the staged file

COPY INTO reads the staged file from S3 across Gnok's workers and writes it into the table's managed Iceberg location, computing column statistics and committing a snapshot. The rows_affected count comes back over the SQL connection — the only thing that actually travels through your database connection in the whole pipeline.

Three things that bit us (and how we fixed them)

JSON-format COPY INTO timed out — CSV/Parquet didn't

Our first instinct was FILE_FORMAT=(TYPE='JSON'), since the source was already JSON. Every attempt died with a 60-second catalog timeout, even on a 3-row file. The identical load as CSV or Parquet succeeded in seconds. Lesson: for the whole-document-into-one-column pattern, stage as CSV or Parquet, not JSON — even though the data inside that column is JSON.

We were upload-bound — Parquet + zstd fixed it

Our first working version staged raw CSV. Each ~400 MB chunk took ~60 seconds, almost entirely upload bandwidth — a home uplink pushing 400 MB to S3. Switching the chunk format to zstd-compressed Parquet shrank a 400 MB chunk to ~20 MB (≈8×) and cut per-chunk time to ~4 seconds. A 20-million-row dataset went from a projected 15 minutes to about 90 seconds. If you're upload-bound, compress before you ship.

Tip: COPY INTO does not transparently decompress gzipped CSV — it reads the gzip bytes as raw text and fails. The "compression" you want lives inside the Parquet file (zstd/snappy), not around the CSV.

Tokens expire; compute warms up

Two operational realities for a multi-hour load:

• JWTs expire every 15 minutes. The loader refreshes proactively and retries any 401.
• Compute can take a moment to become available. COPY INTO needs distributed workers and can return 503 No compute resources available until they're ready, even while SELECT already works. The loader treats 503 (and transient catalog timeouts) as retryable with backoff, so a long load self-heals instead of aborting.

COPY INTO vs REGISTER FILES: copy or move?

A natural question: since we already uploaded Parquet, does COPY INTO reuse it or rewrite it? It rewrites — reading the staged file and writing fresh Parquet into the table's managed location, with full column statistics and partition routing. Your staged file is treated as a disposable source.

Gnok also offers REGISTER FILES INTO ... FROM '...', which registers existing Parquet in place — it reads only the footers, leaves the files where they are, and moves no data. The trade-offs:

	COPY INTO	REGISTER FILES
Data movement	Reads + rewrites into managed location	None (footer read only)
File location	Table's managed path	Stays at the source path
Statistics	Full (min/max, nulls)	Null counts only
Needs prior snapshot	No	Yes

For an id + doc schema, the missing min/max statistics cost nothing (you can't range-prune a JSON blob), which makes REGISTER FILES an appealing way to skip the rewrite for very large datasets — as long as you're comfortable with the table's data living at the staged path. COPY INTO is the safer default because the staged files remain disposable.

The takeaway recipe

For landing a nested document dataset into Gnok:

1. Stream gunzip -c straight into the loader — no full decompress, flat disk usage. (Use jq -c '.[]' if your source is a single JSON array.)
2. Normalize any typed wrappers; store each doc as id + doc (JSON string), wrap in a PARSE_JSON view.
3. Chunk into zstd Parquet, presign, PUT to S3, COPY INTO.
4. Make the loader resilient: refresh tokens, retry 503/catalog timeouts, keep one chunk on disk.
5. Reach for REGISTER FILES only when the rewrite cost dominates and the staged-file coupling is acceptable.

The result: a schemaless 50 GB pile of JSON documents, queryable in Gnok SQL — doc:any:nested:path and all — without designing a single table by hand.