You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by yi...@apache.org on 2026/04/29 17:35:35 UTC
(hudi) branch asf-site updated: docs: add AI-native lakehouse docs with vector search, BLOB, and Lance format (#18591)
This is an automated email from the ASF dual-hosted git repository.
yihua pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/hudi.git
The following commit(s) were added to refs/heads/asf-site by this push:
new b50a503d139b docs: add AI-native lakehouse docs with vector search, BLOB, and Lance format (#18591)
b50a503d139b is described below
commit b50a503d139b6d1e3d3d8409408a6d93c266490f
Author: Y Ethan Guo <et...@gmail.com>
AuthorDate: Wed Apr 29 10:35:28 2026 -0700
docs: add AI-native lakehouse docs with vector search, BLOB, and Lance format (#18591)
Add new "AI & Unstructured Data" docs section and AI Quick Start guide
covering VECTOR type, BLOB unstructured data, hudi_vector_search TVF,
and Lance file format integration.
---
website/docs/ai-quick-start-guide.md | 260 ++++++++++++++++
website/docs/ai_overview.md | 188 +++++++++++
website/docs/blob_unstructured_data.md | 319 +++++++++++++++++++
website/docs/lance_file_format.md | 123 ++++++++
website/docs/overview.mdx | 12 +
website/docs/variant_type.md | 346 +++++++++++++++++++++
website/docs/vector_search.md | 242 ++++++++++++++
website/sidebars.js | 12 +
.../static/assets/images/hudi_read_blob_sample.jpg | Bin 0 -> 20171 bytes
.../assets/images/hudi_vector_search_results.jpg | Bin 0 -> 141176 bytes
10 files changed, 1502 insertions(+)
diff --git a/website/docs/ai-quick-start-guide.md b/website/docs/ai-quick-start-guide.md
new file mode 100644
index 000000000000..9d8fa50ea16f
--- /dev/null
+++ b/website/docs/ai-quick-start-guide.md
@@ -0,0 +1,260 @@
+---
+title: "AI Quick Start"
+keywords: [ hudi, ai, vector search, embeddings, unstructured data, blob, machine learning, image search, similarity]
+summary: "Store embeddings and images in a Hudi table, then find similar images with one SQL query using hudi_vector_search and read_blob"
+toc: true
+last_modified_at: 2026-04-29T00:00:00-00:00
+---
+
+This guide shows how Hudi 1.2.0 brings AI-native data types to the lakehouse. You will store image
+embeddings (`VECTOR`) and raw image bytes (`BLOB`) in a single Hudi table, then run a similarity
+search that finds the top-K nearest neighbors **and** materializes their images — in one SQL query.
+
+:::tip
+Want to try this locally? This guide is also available as an interactive Jupyter notebook.
+[Download the notebook](https://github.com/apache/hudi/blob/master/hudi-examples/hudi-examples-spark/src/test/python/vector_blob_demo/notebooks/00_main_demo.ipynb) and run it end-to-end on your machine.
+:::
+
+
+
+*Example output: a query image of a German Shorthaired pointer (left) and the five most similar images found by `hudi_vector_search`, with cosine similarity scores. Raw image bytes are materialized by `read_blob()` directly in the same query.*
+
+## Prerequisites
+
+| Requirement | Version |
+|:------------|:--------|
+| Java | 11+ |
+| Python | 3.10 – 3.12 |
+| Apache Spark | 3.5.x |
+| Hudi Spark bundle | 1.2.0+ |
+
+```bash
+pip install pyspark==3.5.* pyarrow>=14.0.0 \
+ torch>=2.3.0 torchvision>=0.18.0 timm>=1.0.9 \
+ scikit-learn>=1.4.2 numpy>=1.26.0 pillow>=10.3.0 matplotlib>=3.8.0
+```
+
+## 1. Start Spark with Hudi
+
+```python
+import os
+from pathlib import Path
+from pyspark.sql import SparkSession
+
+HUDI_JAR = os.getenv("HUDI_BUNDLE_JAR", "hudi-spark3.5-bundle_2.12-1.2.0.jar")
+
+spark = (
+ SparkSession.builder
+ .appName("Hudi-AI-QuickStart")
+ .config("spark.jars", HUDI_JAR)
+ .config("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
+ .config("spark.sql.extensions",
+ "org.apache.spark.sql.hudi.HoodieSparkSessionExtension")
+ .config("spark.sql.catalog.spark_catalog",
+ "org.apache.spark.sql.hudi.catalog.HoodieCatalog")
+ .getOrCreate()
+)
+```
+
+## 2. Load images and generate embeddings
+
+Load the [Oxford-IIIT Pet](https://www.robots.ox.ac.uk/~vgg/data/pets/) dataset (37 breeds) and
+generate 1024-dim embeddings with MobileNetV3.
+
+```python
+import io, torch, timm, numpy as np
+from sklearn.preprocessing import normalize
+from PIL import Image
+from torchvision.datasets import OxfordIIITPet
+
+N_SAMPLES = 250
+
+ds = OxfordIIITPet(root="~/.cache/torchvision", split="trainval", download=True)
+indices = np.random.default_rng().choice(len(ds), size=N_SAMPLES, replace=False)
+
+# Collect images as PNG bytes
+data = []
+for idx in indices:
+ img, label = ds[int(idx)]
+ buf = io.BytesIO(); img.convert("RGB").save(buf, format="PNG")
+ data.append({
+ "image_id": f"pets_{int(idx):06d}",
+ "category": ds.classes[label],
+ "label": int(label),
+ "image_bytes_raw": buf.getvalue(),
+ })
+
+# Generate embeddings
+model = timm.create_model("mobilenetv3_small_100", pretrained=True, num_classes=0)
+model.eval()
+cfg = timm.data.resolve_model_data_config(model)
+transform = timm.data.create_transform(**cfg, is_training=False)
+
+batch = torch.stack([
+ transform(Image.open(io.BytesIO(d["image_bytes_raw"])).convert("RGB"))
+ for d in data
+])
+with torch.no_grad():
+ feats = normalize(model(batch).numpy())
+
+for i, d in enumerate(data):
+ d["embedding"] = feats[i].tolist()
+
+DIM = feats.shape[1] # 1024
+```
+
+## 3. Create table and insert data
+
+Declare `VECTOR(1024)` for embeddings and `BLOB` for raw image bytes — first-class Hudi column types.
+
+```python
+from pyspark.sql import Row
+from pyspark.sql.types import *
+
+# Register as a Spark temp view
+schema = StructType([
+ StructField("image_id", StringType(), False),
+ StructField("category", StringType(), False),
+ StructField("label", IntegerType(), False),
+ StructField("image_bytes_raw", BinaryType(), False),
+ StructField("embedding", ArrayType(FloatType(), containsNull=False), False),
+])
+rows = [Row(d["image_id"], d["category"], d["label"],
+ d["image_bytes_raw"], d["embedding"]) for d in data]
+spark.createDataFrame(rows, schema).createOrReplaceTempView("staging")
+```
+
+```sql
+CREATE TABLE pets (
+ image_id STRING,
+ category STRING,
+ label INT,
+ image_bytes BLOB,
+ embedding VECTOR(1024)
+) USING hudi
+LOCATION '/tmp/hudi_pets'
+TBLPROPERTIES (
+ primaryKey = 'image_id',
+ preCombineField = 'image_id',
+ type = 'cow',
+ 'hoodie.table.base.file.format' = 'parquet',
+ 'hoodie.write.record.merge.custom.implementation.classes'
+ = 'org.apache.hudi.DefaultSparkRecordMerger'
+);
+
+INSERT INTO pets
+SELECT image_id, category, label,
+ named_struct(
+ 'type', 'INLINE',
+ 'data', image_bytes_raw,
+ 'reference', cast(null as struct<
+ external_path:string, offset:bigint,
+ length:bigint, managed:boolean>)
+ ) AS image_bytes,
+ embedding
+FROM staging;
+```
+
+Key points:
+- **`VECTOR(1024)`** stores fixed-dimension embeddings for similarity search.
+- **`BLOB`** stores raw image bytes inline. For large objects, use `OUT_OF_LINE` to store a pointer instead — `read_blob()` resolves both modes transparently.
+
+## 4. Materialize a BLOB with `read_blob()`
+
+`read_blob()` is Hudi's BLOB accessor. Pass it a BLOB column, get back raw `binary`. Works the same
+for inline bytes and out-of-line references.
+
+```sql
+SELECT image_id, category,
+ length(read_blob(image_bytes)) AS byte_count
+FROM pets
+LIMIT 5;
+```
+
+```
++-----------+--------------------+----------+
+| image_id| category|byte_count|
++-----------+--------------------+----------+
+|pets_002081| Beagle| 249983|
+|pets_003404| Shiba Inu| 349745|
+|pets_001939| American Bulldog| 267667|
+|pets_002457|English Cocker Sp..| 364492|
+|pets_003538|Staffordshire Bul..| 427728|
++-----------+--------------------+----------+
+```
+
+
+
+*Image bytes retrieved by `read_blob()`, decoded back to a PNG — confirming lossless round-trip through the Hudi BLOB column.*
+
+## 5. Vector search + BLOB retrieval in one query
+
+The headline: `hudi_vector_search` finds the top-K nearest neighbors by cosine similarity, and `read_blob()` materializes image bytes **only for the matching rows**.
+
+```sql
+SELECT image_id,
+ category,
+ read_blob(image_bytes) AS resolved_bytes,
+ _hudi_distance
+FROM hudi_vector_search(
+ '/tmp/hudi_pets', -- table path
+ 'embedding', -- VECTOR column
+ ARRAY(0.12, -0.03, ...), -- query embedding (1024 floats)
+ 5, -- top-K
+ 'cosine' -- distance metric
+)
+ORDER BY _hudi_distance;
+```
+
+```
++-----------+--------------------+-----------+
+| image_id| category| distance |
++-----------+--------------------+-----------+
+|pets_002575| German Shorthaired| 0.378 |
+|pets_000703| German Shorthaired| 0.484 |
+|pets_002562| German Shorthaired| 0.598 |
+|pets_002556| German Shorthaired| 0.607 |
+|pets_003538|Staffordshire Bul..| 0.641 |
++-----------+--------------------+-----------+
+```
+
+
+
+*Query: German Shorthaired pointer (left). Top-5 results ranked by cosine similarity — the search correctly identifies same-breed images as the closest matches.*
+
+One SQL query. No external vector database. Embeddings, raw images, and metadata all live in one transactional Hudi table.
+
+## 6. Visualize results
+
+```python
+import matplotlib.pyplot as plt
+from PIL import Image
+
+fig, axes = plt.subplots(1, len(results) + 1, figsize=(3 * (len(results) + 1), 3.2))
+
+# Query image
+axes[0].imshow(Image.open(io.BytesIO(query_bytes)))
+axes[0].set_title("QUERY", fontweight="bold"); axes[0].axis("off")
+
+# Top-K matches
+for i, row in enumerate(results):
+ img = Image.open(io.BytesIO(bytes(row["resolved_bytes"])))
+ sim = 1.0 - float(row["_hudi_distance"])
+ axes[i+1].imshow(img)
+ axes[i+1].set_title(f"{row['category']}\nSim: {sim:.3f}")
+ axes[i+1].axis("off")
+
+plt.tight_layout()
+plt.savefig("hudi_vector_search_results.png", dpi=150)
+```
+
+## What's next
+
+| Topic | Link |
+|:------|:-----|
+| Full interactive notebook | [00_main_demo.ipynb](https://github.com/apache/hudi/blob/master/hudi-examples/hudi-examples-spark/src/test/python/vector_blob_demo/notebooks/00_main_demo.ipynb) |
+| VECTOR type reference | [Vector Search](vector_search.md) — element types, batch TVF, distance metrics |
+| BLOB type reference | [Unstructured Data](blob_unstructured_data.md) — inline vs. out-of-line, `read_blob()`, configs |
+| VARIANT type | [Semi-Structured Data](variant_type.md) — flexible JSON-like storage for LLM outputs, model metadata |
+| Lance file format | [Lance File Format](lance_file_format.md) — vector-optimized storage |
+| AI overview | [AI-Native Lakehouse](ai_overview.md) — architecture and use cases |
diff --git a/website/docs/ai_overview.md b/website/docs/ai_overview.md
new file mode 100644
index 000000000000..11c695d1bf9c
--- /dev/null
+++ b/website/docs/ai_overview.md
@@ -0,0 +1,188 @@
+---
+title: "AI-Native Lakehouse"
+keywords: [ hudi, ai, machine learning, vector search, embeddings, unstructured data, blob, lance, multimodal, lakehouse]
+summary: "Apache Hudi brings first-class AI and unstructured data support to the data lakehouse — store embeddings, images, audio, and run vector search natively"
+toc: true
+last_modified_at: 2026-04-25T00:00:00-00:00
+---
+
+Modern AI workloads demand more from the data lakehouse than traditional analytics ever did. Teams building
+retrieval-augmented generation (RAG) pipelines, recommendation systems, content moderation, and multimodal
+search need to store and query **embeddings**, **images**, **audio**, **video**, and **documents** alongside
+structured data — all with the transactional guarantees, incremental processing, and table services that
+Hudi is known for.
+
+Apache Hudi's AI-native capabilities bring this vision to life with four foundational building blocks:
+
+## Core Capabilities
+
+### VECTOR Type and Similarity Search
+
+Store high-dimensional embedding vectors as first-class column types and run approximate nearest neighbor (ANN)
+search directly in Spark SQL.
+
+```sql
+-- Declare an embedding column
+CREATE TABLE products (
+ id STRING,
+ name STRING,
+ embedding VECTOR(768)
+) USING hudi;
+
+-- Find the 10 most similar products
+SELECT id, name, _hudi_distance
+FROM hudi_vector_search('products', 'embedding', ARRAY(...), 10, 'cosine');
+```
+
+No external vector database required. Your embeddings live in the same table as your structured data, governed
+by the same transactions, schemas, and access controls.
+
+**[Learn more about Vector Search →](vector_search.md)**
+
+### BLOB Type for Unstructured Data
+
+Store raw binary data (images, PDFs, audio clips, model weights) directly in Hudi tables using the `BLOB` type.
+Choose between two storage modes:
+
+| Mode | How it works | Best for |
+|:-----|:-------------|:---------|
+| **Inline** | Bytes stored directly in the table row | Small objects (thumbnails, short audio clips) |
+| **Out-of-line** | Table stores a pointer; `read_blob()` resolves it on demand | Large objects (high-res images, video, model checkpoints) |
+
+```sql
+CREATE TABLE documents (
+ doc_id STRING,
+ content BLOB,
+ summary STRING
+) USING hudi;
+
+-- Read the raw bytes on demand
+SELECT doc_id, read_blob(content) AS raw_bytes FROM documents WHERE doc_id = 'doc_001';
+```
+
+Out-of-line BLOBs keep your table footprint small (often less than 1% of total data size) while maintaining
+full queryability.
+
+**[Learn more about Unstructured Data →](blob_unstructured_data.md)**
+
+### VARIANT Type for Semi-Structured Data
+
+AI pipelines often deal with data whose shape is not known in advance: LLM outputs, model metadata,
+feature dictionaries, API responses. The `VARIANT` type stores semi-structured, JSON-like data with
+full transactional support — no rigid schema required.
+
+```sql
+CREATE TABLE llm_outputs (
+ request_id STRING,
+ response VARIANT,
+ ts BIGINT
+) USING hudi;
+
+-- Store any JSON structure
+INSERT INTO llm_outputs VALUES (
+ 'req_001',
+ parse_json('{"text": "...", "stop_reason": "end_turn", "tokens": 700}'),
+ 1000
+);
+
+-- Query back as JSON
+SELECT request_id, cast(response as STRING) FROM llm_outputs;
+```
+
+VARIANT supports optional **shredding** to extract hot fields into typed columnar storage for better
+query performance, while keeping the flexibility for everything else.
+
+**[Learn more about Semi-Structured Data →](variant_type.md)**
+
+### Lance File Format
+
+Hudi's pluggable file format architecture supports **Lance**, a modern columnar format purpose-built for
+AI/ML workloads. Lance provides:
+
+- Efficient vector indexing and ANN search
+- Fast random access for training data sampling
+- Optimized storage for high-dimensional arrays and nested structures
+
+```sql
+CREATE TABLE embeddings (...) USING hudi
+TBLPROPERTIES (
+ hoodie.datasource.write.base.file.format = 'lance'
+);
+```
+
+Lance integrates seamlessly with Hudi's table services (compaction, clustering, cleaning) and
+works alongside existing Parquet and ORC tables.
+
+**[Learn more about the Lance File Format →](lance_file_format.md)**
+
+## Why Hudi for AI Workloads?
+
+### Unified Storage for Structured + Unstructured Data
+
+Most AI pipelines today span multiple systems: a data warehouse for metadata, an object store for raw files,
+a vector database for embeddings, and custom glue code to keep them in sync. Hudi collapses this into a single
+table:
+
+```
+┌─────────────────────────────────────────────────┐
+│ Hudi Table │
+│ │
+│ image_id │ breed │ embedding │ image_bytes │
+│ (STRING) │(STRING)│(VECTOR(1024))│ (BLOB) │
+│───────────┼───────┼──────────────┼──────────────│
+│ pet_001 │ Corgi │ [0.12, ...] │ <137 KB PNG> │
+│ pet_002 │ Tabby │ [-.03, ...] │ <89 KB JPEG> │
+└─────────────────────────────────────────────────┘
+```
+
+One table. One set of transactions. One schema. One set of access controls.
+
+### Incremental Processing for Embedding Pipelines
+
+When new data arrives, you do not need to re-embed your entire corpus. Hudi's incremental query capabilities
+let you process only new or changed records:
+
+```sql
+-- Get only new images since the last embedding run
+SELECT * FROM hudi_table_changes('product_images', 'latest_state', '20260101000000');
+```
+
+This can reduce embedding pipeline costs by 10-100x compared to full reprocessing.
+
+### Transactional Guarantees
+
+Embedding updates, metadata changes, and raw data writes happen atomically. No more inconsistent states where
+your vector index points to deleted images or stale embeddings.
+
+### Table Services
+
+Hudi's background table services work on AI tables just like any other:
+
+- **Clustering** — co-locate similar vectors for better search locality
+- **Compaction** — merge incremental updates efficiently
+- **Cleaning** — reclaim storage from old versions
+- **Indexing** — maintain metadata indexes for fast lookups
+
+### Open Ecosystem
+
+Hudi tables are readable by Spark, Flink, Presto, Trino, and the native Python/Rust client (hudi-rs). Your AI
+tables are not locked into a single engine or vendor.
+
+## Use Cases
+
+| Use Case | Hudi Capabilities Used |
+|:---------|:----------------------|
+| **Image/Video Search** | VECTOR embeddings + BLOB storage + cosine similarity search |
+| **RAG (Retrieval-Augmented Generation)** | VECTOR search to retrieve relevant document chunks for LLM context |
+| **LLM Output Management** | VARIANT for flexible response storage, VECTOR for semantic indexing |
+| **Recommendation Systems** | VECTOR similarity for collaborative filtering, incremental re-embedding |
+| **Content Moderation** | BLOB for raw content + VECTOR for content embeddings + incremental processing |
+| **Multimodal Analytics** | Structured metadata + VECTOR embeddings + BLOB raw data in one table |
+| **ML Feature Store** | VECTOR for feature embeddings, VARIANT for sparse feature maps, time-travel for point-in-time retrieval |
+| **Experiment Tracking** | VARIANT for heterogeneous model configs and metrics, incremental queries for latest runs |
+| **Data Labeling Pipelines** | BLOB for raw data, incremental queries for unlabeled data, ACID for label updates |
+
+## Getting Started
+
+The fastest way to try these features is the **[AI Quick Start Guide](ai-quick-start-guide.md)**, which walks
+you through an end-to-end image similarity search pipeline in under 30 minutes.
diff --git a/website/docs/blob_unstructured_data.md b/website/docs/blob_unstructured_data.md
new file mode 100644
index 000000000000..74acc799a230
--- /dev/null
+++ b/website/docs/blob_unstructured_data.md
@@ -0,0 +1,319 @@
+---
+title: "Unstructured Data"
+keywords: [ hudi, blob, unstructured data, images, binary, pdf, audio, video, inline, out-of-line, read_blob]
+summary: "Store and query unstructured data (images, PDFs, audio, video) in Hudi tables using the BLOB type with inline or out-of-line storage"
+toc: true
+last_modified_at: 2026-04-25T00:00:00-00:00
+---
+
+import Tabs from '@theme/Tabs';
+import TabItem from '@theme/TabItem';
+
+The `BLOB` type lets you store raw binary data — images, PDFs, audio clips, video files, model weights —
+directly in Hudi tables. Combined with Hudi's transactional guarantees and table services, BLOB makes
+the data lakehouse a single source of truth for both structured and unstructured data.
+
+## BLOB Type Overview
+
+A BLOB column stores binary data in one of two modes:
+
+| Mode | Storage | Table Footprint | Read Pattern |
+|:-----|:--------|:----------------|:-------------|
+| **Inline** | Raw bytes embedded in the table row | Larger (bytes stored in data files) | Direct read — no external fetch |
+| **Out-of-line** | Pointer to external storage location | Very small (< 1% of data size) | On-demand via `read_blob()` |
+
+Choose **inline** for small objects that are frequently read together (thumbnails, short audio clips,
+small documents). Choose **out-of-line** for large objects where you typically query metadata first and
+fetch raw data selectively (high-res images, video, model checkpoints).
+
+## Creating Tables with BLOB Columns
+
+<Tabs
+groupId="api-style"
+defaultValue="sql"
+values={[
+{ label: 'Spark SQL', value: 'sql', },
+{ label: 'DataFrame API', value: 'dataframe', },
+]}
+>
+<TabItem value="sql">
+
+```sql
+CREATE TABLE media_assets (
+ asset_id STRING,
+ file_name STRING,
+ mime_type STRING,
+ file_size BIGINT,
+ content BLOB
+) USING hudi
+TBLPROPERTIES (
+ primaryKey = 'asset_id',
+ type = 'cow'
+);
+```
+
+The `BLOB` keyword in DDL automatically configures the column with the correct internal structure.
+
+</TabItem>
+<TabItem value="dataframe">
+
+```python
+import pyarrow as pa
+
+schema = pa.schema([
+ pa.field("asset_id", pa.string()),
+ pa.field("file_name", pa.string()),
+ pa.field("mime_type", pa.string()),
+ pa.field("file_size", pa.int64()),
+ pa.field("content", pa.struct([
+ pa.field("type", pa.string()),
+ pa.field("data", pa.binary()),
+ pa.field("reference", pa.struct([
+ pa.field("external_path", pa.string()),
+ pa.field("offset", pa.int64()),
+ pa.field("length", pa.int64()),
+ ])),
+ ]), metadata={b"hudi_type": b"BLOB"}),
+])
+```
+
+The BLOB internal structure is a struct with three fields:
+- `type` — `"INLINE"` or `"OUT_OF_LINE"`
+- `data` — raw bytes (populated for inline, null for out-of-line)
+- `reference` — external storage pointer with subfields:
+ - `external_path` — file path for out-of-line data
+ - `offset` — byte offset in the file (null means read from start)
+ - `length` — byte length to read (null means read to end of file)
+ - `managed` — boolean indicating whether Hudi manages the external file
+
+</TabItem>
+</Tabs>
+
+## Writing Inline BLOBs
+
+Inline BLOBs embed the raw bytes directly in the table row.
+
+<Tabs
+groupId="api-style"
+defaultValue="sql"
+values={[
+{ label: 'Spark SQL', value: 'sql', },
+{ label: 'DataFrame API', value: 'dataframe', },
+]}
+>
+<TabItem value="sql">
+
+```sql
+INSERT INTO media_assets VALUES (
+ 'asset_001',
+ 'logo.png',
+ 'image/png',
+ 45230,
+ named_struct(
+ 'type', 'INLINE',
+ 'data', /* binary literal or column reference */,
+ 'reference', CAST(NULL AS STRUCT<external_path: STRING, offset: BIGINT, length: BIGINT>)
+ )
+);
+```
+
+</TabItem>
+<TabItem value="dataframe">
+
+```python
+from pyspark.sql import Row
+from pyspark.sql.types import *
+
+# Read raw bytes
+with open("logo.png", "rb") as f:
+ raw_bytes = f.read()
+
+row = Row(
+ asset_id="asset_001",
+ file_name="logo.png",
+ mime_type="image/png",
+ file_size=len(raw_bytes),
+ content=Row(type="INLINE", data=raw_bytes, reference=None)
+)
+```
+
+</TabItem>
+</Tabs>
+
+## Writing Out-of-Line BLOBs
+
+Out-of-line BLOBs store a pointer to data in external storage. The actual bytes live elsewhere (e.g.,
+a binary container file on S3, a separate object store path, or a shared filesystem). The table only
+stores the reference metadata.
+
+```sql
+INSERT INTO media_assets VALUES (
+ 'asset_002',
+ 'video.mp4',
+ 'video/mp4',
+ 1073741824, -- 1 GB
+ named_struct(
+ 'type', 'OUT_OF_LINE',
+ 'data', CAST(NULL AS BINARY),
+ 'reference', named_struct(
+ 'external_path', 's3://my-bucket/media/container_001.bin',
+ 'offset', 8388608, -- byte offset in the container
+ 'length', 1073741824 -- number of bytes
+ )
+ )
+);
+```
+
+### Container File Pattern
+
+A common pattern for out-of-line storage is to pack multiple objects into a single binary container file:
+
+```
+container_001.bin
+├── [offset=0, len=45230] → logo.png
+├── [offset=45230, len=89012] → photo.jpg
+├── [offset=134242, len=1073741824] → video.mp4
+└── ...
+```
+
+Each BLOB row stores the `(external_path, offset, length)` triple. This avoids creating millions of
+small files on object storage and enables efficient batch access.
+
+## Reading BLOBs
+
+### Querying Metadata (No Fetch)
+
+Standard queries on a BLOB column return the descriptor — not the raw bytes:
+
+```sql
+SELECT asset_id, file_name, content.type, content.reference.external_path
+FROM media_assets;
+```
+
+```
++----------+-----------+-------------+----------------------------------------+
+| asset_id| file_name| type| external_path|
++----------+-----------+-------------+----------------------------------------+
+| asset_001| logo.png| INLINE| null|
+| asset_002| video.mp4| OUT_OF_LINE| s3://my-bucket/media/container_001.bin |
++----------+-----------+-------------+----------------------------------------+
+```
+
+This is fast and lightweight — no binary data is transferred.
+
+### Resolving Raw Bytes with read_blob()
+
+Use the `read_blob()` SQL function to materialize the actual bytes:
+
+```sql
+-- Returns raw binary data
+SELECT asset_id, read_blob(content) AS raw_bytes
+FROM media_assets
+WHERE asset_id = 'asset_001';
+```
+
+For **inline** BLOBs, `read_blob()` simply extracts the embedded bytes.
+
+For **out-of-line** BLOBs, `read_blob()` reads from the external path at the specified offset and length,
+transparently fetching the data on demand.
+
+:::tip
+Use `read_blob()` selectively — filter first, then resolve. Avoid `SELECT read_blob(content) FROM large_table`
+without a WHERE clause, as this will fetch all raw data.
+:::
+
+## Use Cases
+
+### Image Datasets for Computer Vision
+
+Store training images alongside metadata and embeddings:
+
+```sql
+CREATE TABLE training_images (
+ image_id STRING,
+ label STRING,
+ split STRING, -- 'train', 'val', 'test'
+ embedding VECTOR(1024),
+ raw_image BLOB
+) USING hudi TBLPROPERTIES (...);
+
+-- Get raw images for a specific label
+SELECT image_id, read_blob(raw_image) AS pixels
+FROM training_images
+WHERE label = 'cat' AND split = 'train';
+```
+
+### Document Store for RAG Pipelines
+
+Store PDF documents alongside their chunk embeddings:
+
+```sql
+CREATE TABLE knowledge_base (
+ doc_id STRING,
+ chunk_id STRING,
+ source_url STRING,
+ text STRING,
+ embedding VECTOR(1536),
+ original BLOB -- original PDF bytes
+) USING hudi TBLPROPERTIES (...);
+
+-- Retrieve full document after vector search
+SELECT doc_id, source_url, read_blob(original) AS pdf_bytes
+FROM knowledge_base
+WHERE doc_id IN (SELECT doc_id FROM top_matches);
+```
+
+### Audio/Video Processing Pipelines
+
+```sql
+CREATE TABLE audio_clips (
+ clip_id STRING,
+ transcript STRING,
+ duration DOUBLE,
+ embedding VECTOR(512),
+ audio BLOB
+) USING hudi TBLPROPERTIES (...);
+```
+
+## Storage Efficiency
+
+Out-of-line BLOBs keep the Hudi table footprint extremely small:
+
+| Metric | Inline | Out-of-Line |
+|:-------|:-------|:------------|
+| Table size vs. raw data | ~100% | < 1% |
+| Query metadata without fetch | Requires reading data files | Only reads pointer columns |
+| Random access to raw data | Read full row | Seek to (offset, length) |
+| Best for object size | < 1 MB | > 1 MB |
+
+## Configuration Reference
+
+| Property | Default | Description |
+|:---------|:--------|:------------|
+| `hoodie.read.blob.inline.mode` | `CONTENT` | Controls how INLINE BLOBs are read. `CONTENT` materializes raw bytes in the `data` column. `DESCRIPTOR` surfaces `(position, size)` coordinates rewritten as OUT_OF_LINE references. |
+| `hoodie.blob.batching.max.gap.bytes` | `4096` | Maximum gap (in bytes) between consecutive byte ranges before they are merged into a single read. Larger values reduce I/O calls at the cost of reading some unused bytes. |
+| `hoodie.blob.batching.lookahead.size` | `50` | Number of rows to buffer for batch read detection. Larger values improve batching for sorted data but increase memory usage. |
+
+:::note
+DESCRIPTOR mode is only supported on Lance-backed tables. CONTENT mode is always used for internal
+operations (compaction, merge, log replay) regardless of this setting.
+:::
+
+## Best Practices
+
+1. **Choose the right mode** — Use inline for small, frequently-accessed objects. Use out-of-line for
+ anything over 1 MB.
+
+2. **Filter before resolving** — Always apply WHERE predicates before calling `read_blob()` to avoid
+ unnecessary data transfer.
+
+3. **Batch container files** — When using out-of-line mode, pack multiple objects into container files
+ rather than storing one file per object.
+
+4. **Combine with VECTOR** — Pair BLOB columns with VECTOR columns for powerful "search then retrieve"
+ workflows: vector search narrows candidates, then `read_blob()` fetches just the winners.
+
+5. **Use incremental queries** — Process only new BLOBs by leveraging Hudi's incremental query support:
+ ```sql
+ SELECT * FROM hudi_table_changes('media_assets', 'latest_state', '20260401000000');
+ ```
diff --git a/website/docs/lance_file_format.md b/website/docs/lance_file_format.md
new file mode 100644
index 000000000000..51269107f188
--- /dev/null
+++ b/website/docs/lance_file_format.md
@@ -0,0 +1,123 @@
+---
+title: "Lance File Format"
+keywords: [ hudi, lance, file format, vector, AI, ML, columnar, ANN, indexing]
+summary: "Use the Lance columnar file format with Hudi for vector-optimized storage, ANN indexing, and efficient ML workloads"
+toc: true
+last_modified_at: 2026-04-25T00:00:00-00:00
+---
+
+[Lance](https://lancedb.github.io/lance/) is a modern columnar data format designed for AI and machine learning
+workloads. Hudi's pluggable storage architecture lets you use Lance as the base file format alongside Parquet
+and ORC, unlocking vector indexing, fast random access, and optimized high-dimensional array storage.
+
+## Enabling Lance in Hudi
+
+### Table Creation
+
+Set the base file format to `lance` in table properties:
+
+```sql
+CREATE TABLE my_ai_table (
+ id STRING,
+ embedding VECTOR(768),
+ metadata STRING
+) USING hudi
+TBLPROPERTIES (
+ primaryKey = 'id',
+ type = 'cow',
+ hoodie.record.merger.impls = 'org.apache.hudi.DefaultSparkRecordMerger',
+ hoodie.datasource.write.base.file.format = 'lance'
+);
+```
+
+### DataFrame API
+
+```python
+(df.write
+ .format("hudi")
+ .option("hoodie.table.name", "my_ai_table")
+ .option("hoodie.datasource.write.recordkey.field", "id")
+ .option("hoodie.record.merger.impls",
+ "org.apache.hudi.DefaultSparkRecordMerger")
+ .option("hoodie.datasource.write.base.file.format", "lance")
+ .mode("overwrite")
+ .save("/path/to/my_ai_table"))
+```
+
+### Required Dependencies
+
+Add the Lance Spark bundle to your Spark classpath:
+
+| Component | Maven Coordinates |
+|:----------|:-----------------|
+| Lance Spark Bundle (Spark 3.5) | `org.lance:lance-spark-bundle-3.5_2.12:0.4.0` |
+
+```bash
+export LANCE_BUNDLE_JAR=/path/to/lance-spark-bundle-3.5_2.12-0.4.0.jar
+
+# Include both Hudi and Lance JARs
+spark-shell --jars $HUDI_BUNDLE_JAR,$LANCE_BUNDLE_JAR
+```
+
+## How Hudi + Lance Work Together
+
+Hudi manages the table layer — transactions, schema, timeline, table services — while Lance handles the
+file-level storage:
+
+```
+┌───────────────────────────────────┐
+│ Hudi Table Layer │
+│ Timeline, Metadata, Indexing │
+│ Transactions, Schema Evolution │
+├───────────────────────────────────┤
+│ File Group / File Slice │
+│ (same Hudi concepts as Parquet) │
+├───────────────────────────────────┤
+│ Lance Data Files (.lance) │
+│ IVF-PQ vector index │
+│ Columnar storage │
+│ Fragment-based layout │
+├───────────────────────────────────┤
+│ Storage (S3, GCS, HDFS, FS) │
+└───────────────────────────────────┘
+```
+
+All Hudi table services work with Lance-backed tables:
+
+- **Compaction** — merges log files into Lance base files
+- **Clustering** — reorganizes Lance files for better data locality
+- **Cleaning** — removes old Lance file versions
+- **Metadata indexing** — column stats and bloom filters work across Lance files
+
+## Vector Search with Lance
+
+The `hudi_vector_search` TVF leverages Lance's built-in IVF-PQ index for approximate nearest neighbor search:
+
+```sql
+SELECT id, metadata, _hudi_distance
+FROM hudi_vector_search(
+ 'my_ai_table', 'embedding',
+ ARRAY(0.1, 0.2, ...), -- query vector
+ 10, 'cosine'
+)
+ORDER BY _hudi_distance;
+```
+
+See [Vector Search](vector_search.md) for full documentation on the TVF and distance metrics.
+
+## Configuration Reference
+
+| Property | Description | Default |
+|:---------|:------------|:--------|
+| `hoodie.datasource.write.base.file.format` | Set to `lance` to use Lance as the base file format | `parquet` |
+| `hoodie.record.merger.impls` | Must be `org.apache.hudi.DefaultSparkRecordMerger` for Lance | — |
+
+## Mixed-Format Tables
+
+Hudi does not require all tables in a lakehouse to use the same file format. You can have:
+
+- **Analytical tables** on Parquet — for traditional BI/SQL workloads
+- **AI tables** on Lance — for embeddings, vector search, and ML feature storage
+- **Archival tables** on ORC — for compressed long-term storage
+
+All share the same Hudi catalog, metadata, and tooling.
diff --git a/website/docs/overview.mdx b/website/docs/overview.mdx
index fe320b347e16..46c2d5b4be1c 100644
--- a/website/docs/overview.mdx
+++ b/website/docs/overview.mdx
@@ -51,9 +51,21 @@ Sometimes the fastest way to learn is by doing. Try out these Quick Start resour
- [Spark Quick Start Guide](quick-start-guide.md) – if you primarily use Apache Spark
- [Flink Quick Start Guide](flink-quick-start-guide.md) – if you primarily use Apache Flink
- [Python/Rust Quick Start Guide (Hudi-rs)](python-rust-quick-start-guide.md) - if you primarily use Python or Rust
+- [AI Quick Start Guide](ai-quick-start-guide.md) – if you are building AI/ML pipelines with embeddings, vector search, or unstructured data
If you want to experience Apache Hudi integrated into an end to end demo with Kafka, Spark, Hive, Presto, etc, try out the [Docker Demo](docker_demo.md)
+## AI-Native Capabilities
+
+Hudi brings first-class support for AI and unstructured data workloads to the data lakehouse:
+
+- **[VECTOR type & Similarity Search](vector_search.md)** — Store embeddings and run approximate nearest neighbor search directly in Spark SQL
+- **[BLOB type for Unstructured Data](blob_unstructured_data.md)** — Store images, PDFs, audio, and other binary data with inline or out-of-line storage
+- **[VARIANT type for Semi-Structured Data](variant_type.md)** — Store flexible JSON-like data (LLM outputs, model metadata, feature maps) without rigid schemas
+- **[Lance File Format](lance_file_format.md)** — Vector-optimized columnar format with built-in ANN indexing
+
+See the full [AI-Native Lakehouse Overview](ai_overview.md) for use cases and architecture.
+
## Connect With The Community
Apache Hudi is community-focused and community-led and welcomes new-comers with open arms. Leverage the following
resources to learn more, engage, and get help as you get started.
diff --git a/website/docs/variant_type.md b/website/docs/variant_type.md
new file mode 100644
index 000000000000..bf42390afe2f
--- /dev/null
+++ b/website/docs/variant_type.md
@@ -0,0 +1,346 @@
+---
+title: "Semi-Structured Data (VARIANT)"
+keywords: [ hudi, variant, semi-structured, json, schemaless, shredding, parse_json, flexible schema]
+summary: "Store and query semi-structured JSON-like data in Hudi tables using the VARIANT type, with optional shredding for query performance"
+toc: true
+last_modified_at: 2026-04-25T00:00:00-00:00
+---
+
+import Tabs from '@theme/Tabs';
+import TabItem from '@theme/TabItem';
+
+The `VARIANT` type stores semi-structured, JSON-like data directly in Hudi tables. Unlike rigid schemas
+where every column and type must be declared upfront, VARIANT columns accept arbitrary nested structures —
+objects, arrays, scalars — and store them efficiently in a self-describing binary format.
+
+This is particularly useful for AI and data engineering workloads where data shapes evolve rapidly:
+model metadata, LLM outputs, feature dictionaries, API responses, and event payloads.
+
+## Overview
+
+VARIANT represents semi-structured data as two binary fields:
+
+| Field | Description |
+|:------|:------------|
+| `metadata` | Encodes field names, types, and structure for efficient access |
+| `value` | The actual data payload |
+
+This encoding supports any JSON-compatible value: objects, arrays, strings, numbers, booleans, and null.
+Unlike storing raw JSON strings, VARIANT provides type-aware binary storage with optional **shredding**
+for columnar query performance.
+
+## Creating Tables with VARIANT Columns
+
+<Tabs
+groupId="api-style"
+defaultValue="sql4"
+values={[
+{ label: 'Spark SQL (4.0+)', value: 'sql4', },
+{ label: 'Spark SQL (3.x)', value: 'sql3', },
+{ label: 'DataFrame API', value: 'dataframe', },
+]}
+>
+<TabItem value="sql4">
+
+Spark 4.0+ has native `VARIANT` type support:
+
+```sql
+CREATE TABLE events (
+ event_id STRING,
+ payload VARIANT,
+ ts BIGINT
+) USING hudi
+TBLPROPERTIES (
+ primaryKey = 'event_id',
+ preCombineField = 'ts'
+);
+```
+
+</TabItem>
+<TabItem value="sql3">
+
+On Spark 3.x, use the struct representation with metadata tagging:
+
+```sql
+CREATE TABLE events (
+ event_id STRING,
+ payload STRUCT<value: BINARY, metadata: BINARY>,
+ ts BIGINT
+) USING hudi
+TBLPROPERTIES (
+ primaryKey = 'event_id',
+ preCombineField = 'ts'
+);
+```
+
+Hudi recognizes this struct pattern and treats it as a logical VARIANT.
+
+</TabItem>
+<TabItem value="dataframe">
+
+```python
+from pyspark.sql.types import *
+
+# Option 1: Spark 4.0+ native VariantType
+# schema = StructType([
+# StructField("event_id", StringType()),
+# StructField("payload", VariantType()),
+# StructField("ts", LongType()),
+# ])
+
+# Option 2: Struct with metadata tag (works on Spark 3.x and 4.0+)
+from pyspark.sql.types import MetadataBuilder
+variant_metadata = MetadataBuilder() \
+ .putString("hudi_type", "VARIANT") \
+ .build()
+variant_struct = StructType([
+ StructField("metadata", BinaryType()),
+ StructField("value", BinaryType()),
+])
+schema = StructType([
+ StructField("event_id", StringType()),
+ StructField("payload", variant_struct, metadata=variant_metadata),
+ StructField("ts", LongType()),
+])
+```
+
+</TabItem>
+</Tabs>
+
+## Writing VARIANT Data
+
+<Tabs
+groupId="api-style"
+defaultValue="sql4"
+values={[
+{ label: 'Spark SQL (4.0+)', value: 'sql4', },
+{ label: 'DataFrame API', value: 'dataframe', },
+]}
+>
+<TabItem value="sql4">
+
+Use `parse_json()` to convert JSON strings into VARIANT values:
+
+```sql
+INSERT INTO events VALUES
+ ('evt_001', parse_json('{"action": "click", "x": 120, "y": 450}'), 1000),
+ ('evt_002', parse_json('{"action": "purchase", "items": ["sku_a", "sku_b"], "total": 59.99}'), 1001),
+ ('evt_003', parse_json('"simple string value"'), 1002);
+```
+
+VARIANT accepts any valid JSON: objects, arrays, strings, numbers, booleans, and null.
+
+</TabItem>
+<TabItem value="dataframe">
+
+```python
+df = spark.sql("""
+ SELECT
+ 'evt_001' as event_id,
+ parse_json('{"action": "click", "x": 120, "y": 450}') as payload,
+ 1000 as ts
+""")
+
+df.write.format("hudi") \
+ .option("hoodie.table.name", "events") \
+ .option("hoodie.datasource.write.recordkey.field", "event_id") \
+ .option("hoodie.datasource.write.precombine.field", "ts") \
+ .mode("append") \
+ .save("/path/to/events")
+```
+
+</TabItem>
+</Tabs>
+
+## Querying VARIANT Data
+
+### Reading VARIANT as JSON
+
+Cast VARIANT to string for JSON output:
+
+```sql
+SELECT event_id, cast(payload as STRING) as payload_json
+FROM events;
+```
+
+```
++----------+--------------------------------------------------------------+
+| event_id| payload_json|
++----------+--------------------------------------------------------------+
+| evt_001| {"action":"click","x":120,"y":450} |
+| evt_002| {"action":"purchase","items":["sku_a","sku_b"],"total":59.99}|
+| evt_003| "simple string value" |
++----------+--------------------------------------------------------------+
+```
+
+### Full DML Support
+
+VARIANT columns support all standard DML operations:
+
+```sql
+-- UPDATE
+UPDATE events SET payload = parse_json('{"action": "click", "x": 200}')
+WHERE event_id = 'evt_001';
+
+-- DELETE
+DELETE FROM events WHERE event_id = 'evt_003';
+
+-- MERGE
+MERGE INTO events target
+USING new_events source ON target.event_id = source.event_id
+WHEN MATCHED THEN UPDATE SET payload = source.payload, ts = source.ts
+WHEN NOT MATCHED THEN INSERT *;
+```
+
+### Table Type Support
+
+VARIANT works with both COW (Copy-on-Write) and MOR (Merge-on-Read) table types. MOR tables
+generate log files for VARIANT changes, which are merged during compaction.
+
+## VARIANT Shredding
+
+Shredding extracts frequently-accessed fields from VARIANT into dedicated typed Parquet columns. This
+gives columnar query performance for known fields while retaining the flexibility of VARIANT for
+the rest.
+
+### Unshredded (Default)
+
+```
+group variant {
+ required binary metadata;
+ required binary value;
+}
+```
+
+All data is in the opaque binary fields. Every query must decode the full VARIANT.
+
+### Shredded (Scalar)
+
+```
+group variant {
+ required binary metadata;
+ optional binary value;
+ optional int64 typed_value; -- extracted scalar field
+}
+```
+
+The `typed_value` column stores a typed extraction that Spark can read directly without decoding
+the binary payload.
+
+### Shredded (Object)
+
+```
+group variant {
+ required binary metadata;
+ optional binary value;
+ optional group typed_value {
+ optional group action {
+ optional binary value;
+ optional string typed_value; -- "click", "purchase", etc.
+ }
+ optional group total {
+ optional binary value;
+ optional double typed_value; -- 59.99
+ }
+ }
+}
+```
+
+Each known field gets its own sub-column. Fields not present in a given row fall back to the
+binary `value` field.
+
+## Cross-Engine Compatibility
+
+| Engine | VARIANT Support |
+|:-------|:---------------|
+| **Spark 4.0+** | Native `VariantType` — full read/write/query |
+| **Spark 3.x** | Reads as `STRUCT<value: BINARY, metadata: BINARY>` — backward compatible |
+| **Flink** | Reads as `ROW<metadata BYTES, value BYTES>` — cross-engine compatible |
+
+A VARIANT table written by Spark 4.0 can be read by Spark 3.x or Flink, and vice versa. The
+binary encoding is engine-independent.
+
+## Use Cases for AI Workloads
+
+### LLM Output Storage
+
+Store raw LLM responses with varying structures:
+
+```sql
+CREATE TABLE llm_outputs (
+ request_id STRING,
+ model STRING,
+ response VARIANT,
+ tokens_used INT,
+ ts BIGINT
+) USING hudi TBLPROPERTIES (primaryKey = 'request_id', preCombineField = 'ts');
+
+INSERT INTO llm_outputs VALUES (
+ 'req_001', 'claude-sonnet',
+ parse_json('{"text": "...", "stop_reason": "end_turn", "usage": {"input": 500, "output": 200}}'),
+ 700, 1000
+);
+```
+
+### Model Metadata & Experiment Tracking
+
+Store heterogeneous model configurations and metrics:
+
+```sql
+CREATE TABLE experiments (
+ run_id STRING,
+ model_config VARIANT, -- hyperparameters, architecture, etc.
+ metrics VARIANT, -- loss, accuracy, custom metrics
+ ts BIGINT
+) USING hudi TBLPROPERTIES (primaryKey = 'run_id', preCombineField = 'ts');
+```
+
+### Feature Dictionaries
+
+Store sparse or variable-length feature maps for ML:
+
+```sql
+CREATE TABLE user_features (
+ user_id STRING,
+ features VARIANT, -- {"age": 25, "interests": [...], "embedding": [...]}
+ updated_at BIGINT
+) USING hudi TBLPROPERTIES (primaryKey = 'user_id', preCombineField = 'updated_at');
+```
+
+### API Response Archival
+
+Ingest and store API responses with evolving schemas:
+
+```sql
+CREATE TABLE api_responses (
+ request_id STRING,
+ endpoint STRING,
+ status_code INT,
+ body VARIANT,
+ ts BIGINT
+) USING hudi TBLPROPERTIES (primaryKey = 'request_id', preCombineField = 'ts');
+```
+
+## Best Practices
+
+1. **Use VARIANT for evolving schemas** — When the data shape changes frequently or varies across
+ records, VARIANT avoids constant schema migrations.
+
+2. **Prefer typed columns for frequently-queried fields** — If you always filter on a specific field,
+ make it a top-level typed column rather than burying it in VARIANT. Combine: use typed columns for
+ stable fields and VARIANT for the rest.
+
+3. **Consider shredding for hot fields** — If certain VARIANT fields are queried heavily, shredding
+ extracts them into columnar storage for better performance.
+
+4. **Use `parse_json()` for ingestion** — On Spark 4.0+, `parse_json()` is the standard way to create
+ VARIANT values from JSON strings.
+
+5. **Use `cast(v as STRING)` for export** — Convert VARIANT back to JSON for downstream consumers
+ that expect text.
+
+## Limitations
+
+- Native `VARIANT` keyword in DDL requires Spark 4.0+. On Spark 3.x, use the struct representation.
+- VARIANT shredding configuration is determined at write time based on the schema definition.
+- Complex path expressions within VARIANT may require casting to STRING and then using JSON functions.
diff --git a/website/docs/vector_search.md b/website/docs/vector_search.md
new file mode 100644
index 000000000000..9c53443161a7
--- /dev/null
+++ b/website/docs/vector_search.md
@@ -0,0 +1,242 @@
+---
+title: "Vector Search"
+keywords: [ hudi, vector, search, embeddings, similarity, cosine, ANN, nearest neighbor, VECTOR type]
+summary: "Store embedding vectors in Hudi tables and run approximate nearest neighbor search using the VECTOR type and hudi_vector_search TVF"
+toc: true
+last_modified_at: 2026-04-25T00:00:00-00:00
+---
+
+import Tabs from '@theme/Tabs';
+import TabItem from '@theme/TabItem';
+
+Hudi's `VECTOR` type and `hudi_vector_search` table-valued function (TVF) bring native similarity search
+to the data lakehouse. Store embeddings alongside your structured data and query them with familiar Spark SQL —
+no external vector database required.
+
+## VECTOR Type
+
+The `VECTOR(dim[, elementType])` type declares a column that stores fixed-dimensional embedding vectors.
+Dimension metadata enables the query engine to validate inputs and optimize search.
+
+### Element Types
+
+| Element Type | Description | Storage | Use Case |
+|:-------------|:------------|:--------|:---------|
+| **FLOAT** (default) | 32-bit float | `ArrayType(FloatType)` | Standard embeddings (OpenAI, Cohere, etc.) |
+| **DOUBLE** | 64-bit double | `ArrayType(DoubleType)` | High-precision scientific embeddings |
+| **INT8** / **BYTE** | 8-bit signed integer | `ArrayType(ByteType)` | Quantized embeddings for storage efficiency |
+
+```sql
+-- Default (FLOAT)
+embedding VECTOR(768)
+
+-- Explicit element types
+embedding VECTOR(768, FLOAT)
+embedding VECTOR(768, DOUBLE)
+embedding VECTOR(256, INT8)
+```
+
+### Declaring VECTOR Columns
+
+<Tabs
+groupId="api-style"
+defaultValue="sql"
+values={[
+{ label: 'Spark SQL', value: 'sql', },
+{ label: 'DataFrame API', value: 'dataframe', },
+]}
+>
+<TabItem value="sql">
+
+```sql
+CREATE TABLE products (
+ product_id STRING,
+ name STRING,
+ description STRING,
+ embedding VECTOR(768)
+) USING hudi
+TBLPROPERTIES (
+ primaryKey = 'product_id',
+ type = 'cow',
+ hoodie.record.merger.impls = 'org.apache.hudi.DefaultSparkRecordMerger',
+ hoodie.datasource.write.base.file.format = 'parquet'
+);
+```
+
+When using SQL DDL, Hudi's parser automatically stamps the `VECTOR(dim)` metadata on the column.
+
+</TabItem>
+<TabItem value="dataframe">
+
+```python
+import pyarrow as pa
+
+schema = pa.schema([
+ pa.field("product_id", pa.string()),
+ pa.field("name", pa.string()),
+ pa.field("embedding", pa.list_(pa.float32()),
+ metadata={b"hudi_type": b"VECTOR(768)"}),
+])
+```
+
+When using the DataFrame API, you must manually stamp `hudi_type` metadata on the column via PyArrow.
+This metadata is what distinguishes a `VECTOR` column from a regular array column.
+
+</TabItem>
+</Tabs>
+
+### Writing Vectors
+
+Vectors are written as arrays of floats. Both the DataFrame API and SQL accept standard array syntax:
+
+```sql
+INSERT INTO products VALUES (
+ 'prod_001', 'Running Shoes', 'Lightweight trail runner',
+ ARRAY(0.123, -0.456, 0.789, ...) -- 768 floats
+);
+```
+
+## hudi_vector_search TVF
+
+The `hudi_vector_search` table-valued function performs approximate nearest neighbor (ANN) search
+over a VECTOR column.
+
+### Syntax
+
+```sql
+SELECT *
+FROM hudi_vector_search(
+ table_name, -- STRING: name of the Hudi table
+ vector_column, -- STRING: name of the VECTOR column
+ query_vector, -- ARRAY<FLOAT>: the query embedding
+ top_k, -- INT: number of nearest neighbors to return
+ [distance_metric], -- STRING: 'cosine' (default), 'l2', or 'dot_product'
+ [algorithm] -- STRING: 'brute_force' (default)
+)
+```
+
+### Parameters
+
+| Parameter | Type | Default | Description |
+|:----------|:-----|:--------|:------------|
+| `table_name` | STRING | (required) | The Hudi table to search. Can be a registered table name or a path. |
+| `vector_column` | STRING | (required) | The name of the VECTOR column to search against. |
+| `query_vector` | ARRAY<FLOAT> | (required) | The query embedding. Must match the declared dimension and element type of the VECTOR column. |
+| `top_k` | INT | (required) | The number of nearest neighbors to return. Must be a positive integer. |
+| `distance_metric` | STRING | `'cosine'` | Distance metric: `'cosine'`, `'l2'`, or `'dot_product'`. |
+| `algorithm` | STRING | `'brute_force'` | Search algorithm. Currently only `'brute_force'` is supported. |
+
+### Return Schema
+
+The TVF returns all columns from the source table (excluding the embedding column) plus:
+
+| Column | Type | Description |
+|:-------|:-----|:------------|
+| `_hudi_distance` | DOUBLE | The computed distance between the query vector and each result. Lower values indicate greater similarity. |
+
+Results are ordered by `_hudi_distance` ascending — closest matches first.
+
+## hudi_vector_search_batch TVF
+
+For searching with multiple query vectors at once, use the batch variant:
+
+### Syntax
+
+```sql
+SELECT *
+FROM hudi_vector_search_batch(
+ corpus_table, -- STRING: table to search
+ corpus_embedding_col, -- STRING: VECTOR column in corpus
+ query_table, -- STRING: table containing query vectors
+ query_embedding_col, -- STRING: VECTOR column in query table
+ top_k, -- INT: neighbors per query
+ [distance_metric], -- STRING: 'cosine' (default)
+ [algorithm] -- STRING: 'brute_force' (default)
+)
+```
+
+### Return Schema (Batch)
+
+Returns corpus columns + query columns + distance info:
+
+| Column | Type | Description |
+|:-------|:-----|:------------|
+| `_hudi_distance` | DOUBLE | Distance between query and corpus vector |
+| `_hudi_query_index` | LONG | Index identifying which query vector produced this result |
+
+If corpus and query tables share column names, query columns are prefixed with `_hudi_query_`.
+
+### Distance Metrics
+
+| Metric | Formula | Range | Best for |
+|:-------|:--------|:------|:---------|
+| **cosine** | 1 - cos(a, b), clamped to [0, 2] | [0, 2] | Normalized embeddings (most common). Returns 1.0 for zero vectors. |
+| **l2** | sqrt(sum((a[i] - b[i])^2)) | [0, +inf) | Raw (unnormalized) embeddings |
+| **dot_product** | -(a · b) | (-inf, +inf) | Maximum inner product search. Negated so ascending sort = most similar. |
+
+:::tip
+For best results with cosine distance, **L2-normalize your embeddings** before writing them to the table.
+Most embedding models (OpenAI, Cohere, sentence-transformers) output normalized vectors by default.
+If yours does not, normalize during ingestion:
+
+```python
+embedding = embedding / np.linalg.norm(embedding)
+```
+:::
+
+### Examples
+
+**Find similar products:**
+
+```sql
+SELECT product_id, name, _hudi_distance AS distance
+FROM hudi_vector_search(
+ 'products', 'embedding',
+ ARRAY(0.12, -0.03, 0.87, ...), -- query embedding
+ 10, -- top 10
+ 'cosine'
+)
+ORDER BY distance;
+```
+
+**RAG context retrieval:**
+
+```sql
+-- Retrieve the 5 most relevant document chunks for an LLM prompt
+SELECT chunk_id, text_content, _hudi_distance
+FROM hudi_vector_search(
+ 'document_chunks', 'embedding',
+ ARRAY(...), -- embedding of the user's question
+ 5, 'cosine'
+)
+WHERE _hudi_distance < 0.3; -- optional distance threshold
+```
+
+**Cross-modal search (text-to-image):**
+
+```sql
+-- Using CLIP embeddings, find images matching a text query
+SELECT image_id, caption, _hudi_distance
+FROM hudi_vector_search(
+ 'image_catalog', 'clip_embedding',
+ ARRAY(...), -- text embedding from CLIP
+ 20, 'cosine'
+);
+```
+
+## Best Practices
+
+1. **Normalize embeddings** — Pre-normalize embeddings (L2 norm = 1) for cosine distance. This yields
+ more consistent results and slightly faster search.
+
+2. **Right-size your dimensions** — Higher dimensions capture more information but increase storage and
+ search cost. Many use cases work well with 384–1024 dimensions.
+
+3. **Use incremental processing** — When new data arrives, only embed and write the new records.
+ Hudi's incremental query capabilities make this straightforward.
+
+## Constraints
+
+- VECTOR columns must be **top-level fields** — nesting inside STRUCT, ARRAY, or MAP is not supported.
+- The query vector's element type must **exactly match** the corpus embedding's element type (no implicit casting).
+- VECTOR dimension and element type **cannot be changed** after table creation via schema evolution.
diff --git a/website/sidebars.js b/website/sidebars.js
index 4f0dfac146e8..b4e8379b571b 100644
--- a/website/sidebars.js
+++ b/website/sidebars.js
@@ -16,11 +16,23 @@ module.exports = {
'quick-start-guide',
'flink-quick-start-guide',
'python-rust-quick-start-guide',
+ 'ai-quick-start-guide',
'docker_demo',
"notebooks",
'use_cases',
],
},
+ {
+ type: 'category',
+ label: 'AI & Unstructured Data',
+ items: [
+ 'ai_overview',
+ 'vector_search',
+ 'blob_unstructured_data',
+ 'variant_type',
+ 'lance_file_format',
+ ],
+ },
{
type: 'category',
label: 'Design & Concepts',
diff --git a/website/static/assets/images/hudi_read_blob_sample.jpg b/website/static/assets/images/hudi_read_blob_sample.jpg
new file mode 100644
index 000000000000..bde0fa8f65b5
Binary files /dev/null and b/website/static/assets/images/hudi_read_blob_sample.jpg differ
diff --git a/website/static/assets/images/hudi_vector_search_results.jpg b/website/static/assets/images/hudi_vector_search_results.jpg
new file mode 100644
index 000000000000..1edcf592f080
Binary files /dev/null and b/website/static/assets/images/hudi_vector_search_results.jpg differ