Iceberg v3 in Gnok: Deletion Vectors, VARIANT, Spatial Types, and Row Lineage

March 15, 2026 · 8 min read

Gnok Team

Apache Iceberg format version 3 brings four major capabilities: deletion vectors for more efficient row-level mutations, the VARIANT type for semi-structured data, GEOMETRY/GEOGRAPHY types for spatial analytics, and row lineage for stable row identity across compaction. Gnok now supports all of them.

This post explains what each feature does, when to use it, and how to get started.

Format Versions at a Glance

Iceberg defines three format versions, each building on the previous:

Version	Key Additions
v1	Immutable data files, schema evolution, partition specs, sort orders, column statistics
v2	Position and equality deletes, sequence numbers, partition evolution, snapshot references
v3	Deletion vectors, VARIANT type, GEOMETRY/GEOGRAPHY, row lineage, default column values

Gnok defaults to v2 when creating tables. To use v3 features, specify the format version explicitly:

CREATE TABLE events (
    id BIGINT,
    payload VARIANT,
    location GEOMETRY
) WITH ('format-version' = '3');

You can also upgrade an existing table:

ALTER TABLE events SET TBLPROPERTIES ('format-version' = '3');

caution

Format version upgrades are one-way -- downgrading is not supported. Make sure all consumers of the table support v3 before upgrading.

Deletion Vectors

The Problem with Position Deletes

In v2, every UPDATE or DELETE writes a position delete file -- a Parquet file containing (file_path, pos) pairs that mark which rows are deleted. This works, but delete files accumulate over time. A table with frequent updates can end up with hundreds of delete files, each of which must be read and merged during every scan.

Data file (1M rows)
  ├── Position delete file #1 (50 rows)
  ├── Position delete file #2 (30 rows)
  ├── Position delete file #3 (120 rows)
  └── ... (N more delete files)

Each scan must load all N delete files, build an index, and filter every row against it. At scale this becomes the dominant cost of reads.

How Deletion Vectors Work

Deletion vectors replace individual delete files with a single, compact Roaring bitmap stored in a Puffin file. Instead of N separate Parquet files with (file_path, pos) pairs, each data file has at most one deletion vector containing the set of deleted row positions.

Data file (1M rows)
  └── Deletion vector (Roaring bitmap, ~2KB) → {pos: 12, 47, 193, ...}

The binary format is compact:

Combined Length (4 bytes)
Magic: 0xD1D33964 (4 bytes)
64-bit Roaring Bitmap (variable)
CRC-32C checksum (4 bytes)

Roaring bitmaps are extremely space-efficient for sparse sets (a few thousand deletes in a million-row file) and support O(1) membership tests. The bitmap is stored in a Puffin blob with LZ4 or Zstd compression.

Current Support

Gnok fully supports reading deletion vectors. During scan planning, DVs are matched to data files by referenced_data_file path, preloaded, and merged with any existing position deletes. The write path currently produces v2-style position delete files; DV write support is planned.

Operation	Status
Read DVs during scan	Supported
Merge DVs with position deletes	Supported
Write DVs on UPDATE/DELETE	Planned

What This Means for You

If you produce data with a writer that creates deletion vectors (e.g., Spark 4.0), Gnok can read those tables efficiently today. For Gnok-originated writes, position deletes are used until DV write support ships. Run OPTIMIZE PURGE DELETES periodically to consolidate accumulated delete files.

VARIANT Type

Semi-Structured Data, Natively

JSON columns stored as VARCHAR work for simple cases, but they have drawbacks: no type safety, no columnar pruning, and every access requires parsing the entire string. The VARIANT type solves this by storing semi-structured data in Iceberg's native binary variant format with optional shredding -- extracting frequently accessed fields into separate columns for efficient columnar reads.

Creating and Querying VARIANT Data

-- Create a table with a VARIANT column
CREATE TABLE events (
    event_id BIGINT,
    event_type VARCHAR,
    payload VARIANT
) WITH ('format-version' = '3');

-- Insert JSON data as VARIANT
INSERT INTO events VALUES
    (1, 'click', PARSE_JSON('{"page": "/home", "duration_ms": 1200, "user": "alice"}')),
    (2, 'purchase', PARSE_JSON('{"item": "widget", "price": 29.99, "quantity": 3}')),
    (3, 'click', PARSE_JSON('{"page": "/products", "duration_ms": 800}'));

Accessing Nested Values

Use VARIANT_GET to extract typed values at a dot-path:

SELECT
    event_id,
    event_type,
    VARIANT_GET(payload, 'page', 'VARCHAR') AS page,
    VARIANT_GET(payload, 'duration_ms', 'INT') AS duration_ms,
    VARIANT_GET(payload, 'price', 'DOUBLE') AS price
FROM events;

Result:

event_id | event_type | page      | duration_ms | price
---------+------------+-----------+-------------+------
       1 | click      | /home     |        1200 | NULL
       2 | purchase   | NULL      |        NULL | 29.99
       3 | click      | /products |         800 | NULL

Function Reference

Function	Description
`PARSE_JSON(string)`	Parse JSON into VARIANT
`TO_VARIANT(expr)`	Convert a scalar to VARIANT
`VARIANT_GET(v, path, type)`	Extract a typed value at a dot-path
`VARIANT_EXTRACT_STRING(v, path)`	Extract any value as a string
`VARIANT_TYPE(v)`	Return the runtime type name
`VARIANT_AS_BOOL(v)`	Cast to BOOLEAN
`VARIANT_AS_INT(v)`	Cast to BIGINT
`VARIANT_AS_DOUBLE(v)`	Cast to DOUBLE
`VARIANT_AS_STRING(v)`	Cast to VARCHAR
`IS_VARIANT_NULL(v)`	True if the VARIANT value is null

When to Use VARIANT vs JSON

Scenario	Use
Schema is unknown or evolving rapidly	VARIANT
You need columnar pruning on nested fields	VARIANT (with shredding)
Existing tables with JSON string columns	Keep as-is; use `json_extract()`
Small payloads, infrequent access	Either works

GEOMETRY and GEOGRAPHY Types

Spatial Analytics in SQL

v3 adds native spatial types following PostGIS naming conventions. GEOMETRY operates in Cartesian coordinates (planar); GEOGRAPHY uses geodesic calculations (great-circle distances on the WGS84 ellipsoid).

Example: Store Locator

-- Create a table with spatial columns
CREATE TABLE stores (
    store_id BIGINT,
    name VARCHAR,
    location GEOMETRY
) WITH ('format-version' = '3');

-- Insert store locations
INSERT INTO stores VALUES
    (1, 'Downtown', ST_POINT(-73.9857, 40.7484)),
    (2, 'Midtown', ST_POINT(-73.9787, 40.7614)),
    (3, 'Brooklyn', ST_POINT(-73.9442, 40.6782));

-- Find stores within 2km of a location
SELECT
    name,
    ST_DISTANCE(location, ST_POINT(-73.98, 40.75)) AS distance_m
FROM stores
WHERE ST_DWITHIN(location, ST_POINT(-73.98, 40.75), 2000)
ORDER BY distance_m;

Spatial Pruning

Gnok applies spatial pruning at the scan level using bounding box statistics stored in file metadata. Queries with ST_WITHIN, ST_CONTAINS, ST_INTERSECTS, or ST_DWITHIN predicates can skip entire data files whose bounding boxes don't overlap the query region.

CRS Support

CRS	EPSG Code	Use Case
WGS84	4326	GPS coordinates (lat/lon)
Web Mercator	3857	Web maps (Leaflet, Mapbox)
UTM zones	326xx/327xx	Regional metric projections

Transform between projections with ST_TRANSFORM:

SELECT ST_TRANSFORM(location, 3857) AS web_mercator_point
FROM stores;

Full Function List

ST_POINT, ST_DISTANCE, ST_WITHIN, ST_CONTAINS, ST_INTERSECTS, ST_AREA, ST_LENGTH, ST_ASTEXT, ST_GEOMFROMTEXT, ST_GEOGFROMTEXT, ST_X, ST_Y, ST_SRID, ST_TRANSFORM, ST_BUFFER, ST_DWITHIN, ST_ENVELOPE

Row Lineage

Stable Row Identity

Row lineage provides globally unique, stable row identifiers that survive compaction, schema evolution, and table maintenance. Each row gets a _row_id that doesn't change when the underlying data files are rewritten.

How It Works

Each table tracks a monotonically increasing next_row_id counter in its metadata
When data files are committed, each file is assigned a first_row_id from this counter
A row's _row_id is computed as first_row_id + position_within_file
The counter advances by the total number of rows written

Querying Row Lineage

Two metadata columns are available on v3 tables:

SELECT
    _row_id,
    _last_updated_sequence_number,
    order_id,
    amount
FROM orders
LIMIT 5;

_row_id | _last_updated_sequence_number | order_id | amount
--------+-------------------------------+----------+-------
|                             1 |     1001 | 99.99
|                             1 |     1002 | 49.50
|                             1 |     1003 | 199.00
|                             2 |     1004 | 75.25
|                             2 |     1005 | 320.00

Column	Description
`_row_id`	Globally unique, stable row identifier
`_last_updated_sequence_number`	The commit sequence number that last modified this row

These are metadata columns -- they're excluded from SELECT * but can be requested explicitly.

Use Cases

CDC pipelines: Track which rows changed between commits using _last_updated_sequence_number
Deduplication: Use _row_id as a stable key for exactly-once processing
Audit trails: Record _row_id in downstream systems for lineage tracking
Incremental ETL: Process only rows where _last_updated_sequence_number > last_processed_sequence

Default Column Values

v3 adds support for default values that apply both to new rows (when the column is omitted on INSERT) and to existing rows (when reading data written before the column was added):

-- Add a column with defaults
ALTER TABLE orders ADD COLUMN priority INT DEFAULT 0;
ALTER TABLE orders ADD COLUMN created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP;

Default Type	When Applied
`write_default`	New rows where the column value is omitted
`initial_default`	Existing rows written before the column was added

This makes schema evolution seamless -- old data files don't need to be rewritten when a new column is added.

Migration Guide

Upgrading to v3

Assess compatibility: Ensure all consumers (Spark, Trino, Flink, etc.) support Iceberg v3

Upgrade the table:

ALTER TABLE my_table SET TBLPROPERTIES ('format-version' = '3');

Start using v3 features: Add VARIANT/GEOMETRY columns, query _row_id, etc.

Creating New v3 Tables

CREATE TABLE sensor_data (
    sensor_id BIGINT NOT NULL,
    reading DOUBLE,
    metadata VARIANT,
    location GEOMETRY,
    ts TIMESTAMP
)
PARTITIONED BY (day(ts))
WITH ('format-version' = '3');

What Doesn't Change

All v1 and v2 features continue to work unchanged
Existing queries don't need modification
Position deletes are still written (until DV write support ships)
Schema evolution, partition pruning, and time travel work identically

Format Versions at a Glance​

Deletion Vectors​

The Problem with Position Deletes​

How Deletion Vectors Work​

Current Support​

What This Means for You​

VARIANT Type​

Semi-Structured Data, Natively​

Creating and Querying VARIANT Data​

Accessing Nested Values​

Function Reference​

When to Use VARIANT vs JSON​

GEOMETRY and GEOGRAPHY Types​

Spatial Analytics in SQL​

Example: Store Locator​

Spatial Pruning​

CRS Support​

Full Function List​

Row Lineage​

Stable Row Identity​

How It Works​

Querying Row Lineage​

Use Cases​

Default Column Values​

Migration Guide​

Upgrading to v3​

Creating New v3 Tables​

What Doesn't Change​

Further Reading​