Building Custom WASM UDFs for Gnok: A Step-by-Step Guide

Gnok's WebAssembly UDF system lets you extend SQL with custom functions written in Rust, Go, C, or any language that compiles to WASM. Your functions run inside the query engine with near-native performance, sandboxed execution, and automatic distribution across cluster workers.

This tutorial walks through building a WASM UDF from scratch, registering it, and using it in queries.

What You'll Build

We'll create three UDFs of increasing complexity:

1. mask_pii -- redact personally identifiable information from text
2. haversine_km -- calculate the distance between two GPS coordinates
3. json_extract_field -- extract a field from a JSON string

Prerequisites

• A Gnok account with permission to create functions (you can run the SQL in a Gnok Studio worksheet)
• Rust toolchain with the WASM target:

rustup target add wasm32-wasip2
cargo install cargo-component

How WASM UDFs Work

When you call a WASM UDF in SQL, Gnok:

1. Receives an Arrow RecordBatch (one column per function argument)
2. Serializes it as Arrow IPC bytes
3. Calls your WASM function with the IPC bytes
4. Deserializes the output IPC bytes back into an Arrow column
5. Returns the result as part of the query

Your function signature is always the same: receive Vec<u8> (Arrow IPC input), return Result<Vec<u8>, String> (Arrow IPC output).

Project Setup

Create a new Rust library project:

cargo init --lib gnok-udf-examples
cd gnok-udf-examples

Add the guest library and Arrow dependencies to Cargo.toml:

[package]
name = "gnok-udf-examples"
version = "0.1.0"
edition = "2021"

[lib]
crate-type = ["cdylib"]

[dependencies]
gnok-udf-guest = { version = "0.1" }
arrow = { version = "54", default-features = false, features = ["ipc"] }

UDF 1: PII Masking

Let's start with a simple string transformation that redacts email addresses and phone numbers from text.

Write the Function

Create src/lib.rs:

use gnok_udf_guest::ipc::{decode_input_batch, encode_output_column};
use arrow::array::{ArrayRef, StringArray};

/// Masks email addresses and phone numbers in text.
/// Input: a single TEXT column
/// Output: TEXT column with PII replaced by [REDACTED]
pub fn mask_pii(input_ipc: Vec<u8>) -> Result<Vec<u8>, String> {
    let args = decode_input_batch(&input_ipc)?;
    let texts = args[0]
        .as_any()
        .downcast_ref::<StringArray>()
        .ok_or("expected string array")?;

    let result: StringArray = texts
        .iter()
        .map(|opt_val| {
            opt_val.map(|text| {
                let mut masked = String::with_capacity(text.len());
                let mut i = 0;
                let chars: Vec<char> = text.chars().collect();

                while i < chars.len() {
                    // Simple email detection: look for @ sign
                    if chars[i] == '@' {
                        // Walk back to find start of email
                        let start = masked.rfind(|c: char| c.is_whitespace())
                            .map(|pos| pos + 1)
                            .unwrap_or(0);
                        masked.truncate(start);
                        masked.push_str("[REDACTED]");
                        // Skip to end of email
                        while i < chars.len() && !chars[i].is_whitespace() {
                            i += 1;
                        }
                        continue;
                    }

                    // Simple phone detection: sequences of 10+ digits
                    if chars[i].is_ascii_digit() {
                        let digit_start = i;
                        let mut digit_count = 0;
                        let mut j = i;
                        while j < chars.len()
                            && (chars[j].is_ascii_digit()
                                || chars[j] == '-'
                                || chars[j] == ' '
                                || chars[j] == '('
                                || chars[j] == ')')
                        {
                            if chars[j].is_ascii_digit() {
                                digit_count += 1;
                            }
                            j += 1;
                        }
                        if digit_count >= 10 {
                            masked.push_str("[REDACTED]");
                            i = j;
                            continue;
                        }
                    }

                    masked.push(chars[i]);
                    i += 1;
                }
                masked
            })
        })
        .collect();

    encode_output_column(&result)
}

Build

cargo component build --target wasm32-wasip2 --release

This produces target/wasm32-wasip2/release/gnok_udf_examples.wasm.

Register

Base64-encode the WASM binary and register it:

# Encode the WASM module
WASM_B64=$(base64 -i target/wasm32-wasip2/release/gnok_udf_examples.wasm)

CREATE FUNCTION mask_pii(TEXT) RETURNS TEXT
LANGUAGE wasm
VOLATILITY immutable
AS '<paste-base64-here>';

Use It

-- Mask PII in customer support messages
SELECT
    ticket_id,
    mask_pii(message) AS safe_message
FROM support_tickets
LIMIT 5;

Result:

ticket_id | safe_message
----------+---------------------------------------------
1001      | Please contact me at [REDACTED] about my order
1002      | My phone number is [REDACTED], call me back
1003      | Hi, I'm John from accounting

UDF 2: Haversine Distance

A numeric UDF that calculates the great-circle distance between two GPS coordinates.

Write the Function

use gnok_udf_guest::ipc::{decode_input_batch, encode_output_column};
use arrow::array::{Float64Array};

/// Computes the Haversine distance in kilometers between two
/// lat/lon coordinate pairs.
///
/// Arguments: lat1 FLOAT64, lon1 FLOAT64, lat2 FLOAT64, lon2 FLOAT64
/// Returns:   FLOAT64 (distance in km)
pub fn haversine_km(input_ipc: Vec<u8>) -> Result<Vec<u8>, String> {
    let args = decode_input_batch(&input_ipc)?;

    let lat1 = args[0].as_any().downcast_ref::<Float64Array>()
        .ok_or("expected float64 for lat1")?;
    let lon1 = args[1].as_any().downcast_ref::<Float64Array>()
        .ok_or("expected float64 for lon1")?;
    let lat2 = args[2].as_any().downcast_ref::<Float64Array>()
        .ok_or("expected float64 for lat2")?;
    let lon2 = args[3].as_any().downcast_ref::<Float64Array>()
        .ok_or("expected float64 for lon2")?;

    const EARTH_RADIUS_KM: f64 = 6371.0;

    let result: Float64Array = (0..lat1.len())
        .map(|i| {
            if lat1.is_null(i) || lon1.is_null(i)
                || lat2.is_null(i) || lon2.is_null(i)
            {
                return None;
            }

            let lat1_rad = lat1.value(i).to_radians();
            let lat2_rad = lat2.value(i).to_radians();
            let dlat = (lat2.value(i) - lat1.value(i)).to_radians();
            let dlon = (lon2.value(i) - lon1.value(i)).to_radians();

            let a = (dlat / 2.0).sin().powi(2)
                + lat1_rad.cos() * lat2_rad.cos() * (dlon / 2.0).sin().powi(2);
            let c = 2.0 * a.sqrt().asin();

            Some(EARTH_RADIUS_KM * c)
        })
        .collect();

    encode_output_column(&result)
}

Register and Use

CREATE FUNCTION haversine_km(FLOAT64, FLOAT64, FLOAT64, FLOAT64) RETURNS FLOAT64
LANGUAGE wasm
VOLATILITY immutable
AS '<base64-encoded-wasm>';

-- Find the 10 nearest warehouses to each order's delivery address
SELECT
    o.order_id,
    w.warehouse_name,
    haversine_km(o.delivery_lat, o.delivery_lon, w.lat, w.lon) AS distance_km
FROM orders o
CROSS JOIN warehouses w
ORDER BY o.order_id, distance_km
LIMIT 10;

-- Filter deliveries within a 50km radius
SELECT
    order_id,
    customer_name,
    haversine_km(delivery_lat, delivery_lon, 37.7749, -122.4194) AS distance_to_sf
FROM orders
WHERE haversine_km(delivery_lat, delivery_lon, 37.7749, -122.4194) < 50.0;

Because we declared VOLATILITY immutable, the optimizer knows this function is deterministic. If you reference haversine_km multiple times with the same arguments, Gnok evaluates it once and reuses the result.

UDF 3: JSON Field Extraction

A practical UDF for extracting fields from JSON strings stored in text columns.

Write the Function

Add serde_json to your dependencies:

[dependencies]
serde_json = "1"

use gnok_udf_guest::ipc::{decode_input_batch, encode_output_column};
use arrow::array::StringArray;

/// Extracts a top-level field from a JSON string.
///
/// Arguments: json_text TEXT, field_name TEXT
/// Returns:   TEXT (the field value as a string, or NULL if missing)
pub fn json_extract_field(input_ipc: Vec<u8>) -> Result<Vec<u8>, String> {
    let args = decode_input_batch(&input_ipc)?;

    let json_col = args[0].as_any().downcast_ref::<StringArray>()
        .ok_or("expected string for json_text")?;
    let field_col = args[1].as_any().downcast_ref::<StringArray>()
        .ok_or("expected string for field_name")?;

    let result: StringArray = (0..json_col.len())
        .map(|i| {
            if json_col.is_null(i) || field_col.is_null(i) {
                return None;
            }

            let json_str = json_col.value(i);
            let field = field_col.value(i);

            let parsed: serde_json::Value = match serde_json::from_str(json_str) {
                Ok(v) => v,
                Err(_) => return None,
            };

            parsed.get(field).map(|v| match v {
                serde_json::Value::String(s) => s.clone(),
                other => other.to_string(),
            })
        })
        .collect();

    encode_output_column(&result)
}

Register and Use

CREATE FUNCTION json_extract_field(TEXT, TEXT) RETURNS TEXT
LANGUAGE wasm
VOLATILITY immutable
AS '<base64-encoded-wasm>';

-- Extract user agent from JSON event payloads
SELECT
    event_id,
    json_extract_field(payload, 'user_agent') AS user_agent,
    json_extract_field(payload, 'action') AS action
FROM events
WHERE json_extract_field(payload, 'action') = 'purchase';

Volatility Matters

The VOLATILITY clause tells the optimizer how aggressively it can optimize calls to your function:

Volatility	When to Use	Optimizer Behavior
immutable	Pure functions -- same inputs always produce the same output	Can be constant-folded, pushed through joins
stable	Depends on session state but is consistent within a query	Cached per-query, can be pushed into scans
volatile	Depends on external state or randomness (default)	Re-evaluated every call, never optimized

Use immutable whenever possible -- it gives the optimizer the most freedom. All three UDFs in this tutorial are immutable because they are pure functions.

Sandboxing and Limits

WASM UDFs run in a Wasmtime sandbox with strict resource limits:

Limit	Default	Description
Memory per instance	64 MB	Maximum memory a single UDF invocation can use
Execution timeout	30 seconds	Epoch-based timeout enforcement
Max module size	50 MB	Maximum .wasm binary size

If your UDF exceeds the memory limit, the WASM runtime traps and Gnok returns an error for that row batch. The timeout prevents runaway computation from blocking the query pipeline.

Distributed Execution

When a query runs across multiple workers, Gnok distributes your WASM UDF automatically. Each worker compiles the module on first use and caches it, so later queries reuse the compiled module. You register the function once with SQL; there is nothing to deploy.

Managing UDFs

-- List all registered functions
SHOW FUNCTIONS;

-- Replace an existing function with a new version
CREATE OR REPLACE FUNCTION mask_pii(TEXT) RETURNS TEXT
LANGUAGE wasm
VOLATILITY immutable
AS '<updated-base64>';

-- Remove a function
DROP FUNCTION mask_pii;
DROP FUNCTION IF EXISTS haversine_km;

Tips and Best Practices

1. Keep modules small. Smaller WASM binaries compile faster and distribute quicker across workers. Use wasm-opt to shrink your output:

   wasm-opt -Oz -o optimized.wasm target/wasm32-wasip2/release/gnok_udf_examples.wasm

2. Handle nulls. Arrow arrays can contain null values. Always check is_null(i) before accessing values, and return None for null inputs.

3. Prefer batch operations. Your function receives an entire column batch (up to 8192 rows). Process the whole batch in a loop rather than returning early -- this maximizes throughput.

4. Use immutable volatility for pure functions. This is the single biggest performance lever because it enables constant folding and predicate pushdown.

5. Test locally first. You can unit-test your UDF logic as regular Rust code before compiling to WASM:

   #[cfg(test)]
   mod tests {
       use super::*;
       use arrow::array::StringArray;
   
       #[test]
       fn test_mask_pii_emails() {
           let input = StringArray::from(vec![
               Some("Contact alice@example.com for details"),
               Some("No PII here"),
               None,
           ]);
           // ... test your masking logic
       }
   }

Further Reading

• UDF Reference -- full SQL syntax and type mappings
• ONNX Model Inference -- register ML models as SQL functions
• Service architecture -- how hosted Gnok components fit together