You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "yihua (via GitHub)" <gi...@apache.org> on 2026/04/26 00:30:48 UTC
[PR] docs: add AI-native lakehouse documentation [hudi]
yihua opened a new pull request, #18591:
URL: https://github.com/apache/hudi/pull/18591
### Describe the issue this Pull Request addresses
Add documentation for Hudi's AI-native capabilities to attract AI/ML users to the project.
### Summary and Changelog
Add a new "AI & Unstructured Data" docs section and an AI Quick Start guide:
- **AI Quick Start** (under Getting Started): hands-on tutorial for image similarity search using VECTOR, BLOB, and Lance
- **AI-Native Lakehouse Overview**: landing page positioning Hudi for AI workloads (RAG, image search, feature stores, etc.)
- **Vector Search**: VECTOR type reference, `hudi_vector_search` TVF syntax, distance metrics, embedding model compatibility
- **Unstructured Data**: BLOB type reference covering inline vs out-of-line storage and `read_blob()` function
- **Lance File Format**: Lance integration guide with setup, architecture, and comparison to Parquet for AI workloads
- Updated `overview.mdx` and `sidebars.js` to link the new pages
### Impact
Documentation-only change. No code changes.
### Risk Level
none
### Documentation Update
This PR is the documentation update itself. Pages are added to the current (next) version only, not backported to any released version.
### Contributor's checklist
- [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [x] Enough context is provided in the sections above
- [x] Adequate tests were added if applicable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] docs: add AI-native lakehouse documentation [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #18591:
URL: https://github.com/apache/hudi/pull/18591#discussion_r3162909598
##########
website/docs/ai-quick-start-guide.md:
##########
@@ -0,0 +1,343 @@
+---
+title: "AI Quick Start"
+keywords: [ hudi, ai, vector search, embeddings, unstructured data, blob, lance, machine learning]
+summary: "Get started with Hudi's AI-native capabilities: vector search, unstructured data storage, and embedding-powered similarity queries"
+toc: true
+last_modified_at: 2026-04-25T00:00:00-00:00
+---
+
+import Tabs from '@theme/Tabs';
+import TabItem from '@theme/TabItem';
+
+This guide walks you through Hudi's AI-native capabilities using a hands-on image similarity search example.
+You will store image embeddings and raw image bytes in a Hudi table, then run vector search to find visually
+similar images — all using familiar Spark SQL or DataFrame APIs.
+
+## What You Will Build
+
+By the end of this guide you will have:
+
+1. Created a Hudi table with **VECTOR** columns (for embeddings) and **BLOB** columns (for raw image bytes)
+2. Generated image embeddings using a pre-trained neural network (MobileNetV3)
+3. Written embeddings and images to a Hudi table backed by the **Lance** columnar format
+4. Run **vector similarity search** to find the top-K most similar images to a query
+
+## Prerequisites
+
+- Java 11+
+- Python 3.10 – 3.12 (Python 3.13+ is not yet supported)
+- Apache Spark 3.5.x
+- Hudi 1.2.0+ (with Spark bundle)
+
+### Python Dependencies
+
+```bash
+pip install pyspark==3.5.* pyarrow>=14.0.0 \
+ torch>=2.3.0 torchvision>=0.18.0 timm>=1.0.9 \
+ scikit-learn>=1.4.2 numpy>=1.26.0 pillow>=10.3.0 matplotlib>=3.8.0
+```
+
+### Required JARs
+
+You need two JAR files on the Spark classpath:
+
+| JAR | How to get it |
+|:----|:--------------|
+| Hudi Spark bundle | Build from source: `mvn clean package -pl packaging/hudi-spark-bundle -am -DskipTests -Dspark3.5` or download from [releases](/releases/download) |
+| Lance Spark bundle | Maven Central: `org.lance:lance-spark-bundle-3.5_2.12:0.4.0` |
+
+```bash
+export HUDI_BUNDLE_JAR=/path/to/hudi-spark3.5-bundle_2.12-1.2.0-SNAPSHOT.jar
+export LANCE_BUNDLE_JAR=/path/to/lance-spark-bundle-3.5_2.12-0.4.0.jar
+```
+
+## Step 1: Launch Spark with Hudi + Lance
+
+```python
+from pyspark.sql import SparkSession
+
+spark = (SparkSession.builder
+ .appName("HudiAIQuickStart")
+ .master("local[*]")
+ .config("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
+ .config("spark.sql.catalog.spark_catalog",
+ "org.apache.spark.sql.hudi.catalog.HoodieCatalog")
+ .config("spark.sql.extensions",
+ "org.apache.spark.sql.hudi.HoodieSparkSessionExtension")
+ .config("spark.jars", f"{HUDI_BUNDLE_JAR},{LANCE_BUNDLE_JAR}")
+ .config("spark.driver.memory", "8g")
+ .getOrCreate())
+```
+
+## Step 2: Prepare Image Embeddings
+
+We use the [Oxford-IIIT Pet dataset](https://www.robots.ox.ac.uk/~vgg/data/pets/) (37 dog and cat breeds) and
+generate embeddings with MobileNetV3, a lightweight convolutional neural network.
+
+```python
+import torch, timm, numpy as np
+from torchvision import datasets, transforms
+from torch.utils.data import DataLoader
+
+# Load the pre-trained model
+model = timm.create_model("mobilenetv3_large_100", pretrained=True, num_classes=0)
+model.eval()
+
+# Image pre-processing pipeline
+transform = transforms.Compose([
+ transforms.Resize(256),
+ transforms.CenterCrop(224),
+ transforms.ToTensor(),
+ transforms.Normalize(mean=[0.485, 0.456, 0.406],
+ std=[0.229, 0.224, 0.225]),
+])
+
+# Download and load the dataset
+dataset = datasets.OxfordIIITPet(root="/tmp/pets", split="trainval",
+ download=True, transform=transform)
+loader = DataLoader(dataset, batch_size=64, shuffle=False)
+
+# Generate embeddings
+embeddings, labels = [], []
+with torch.no_grad():
+ for images, targets in loader:
+ emb = model(images)
+ # L2-normalize for cosine similarity
+ emb = emb / emb.norm(dim=1, keepdim=True)
+ embeddings.append(emb.numpy())
+ labels.extend(targets.tolist())
+
+embeddings = np.vstack(embeddings)
+print(f"Generated {len(embeddings)} embeddings of dimension {embeddings.shape[1]}")
+```
+
+## Step 3: Create a Hudi Table with VECTOR and BLOB Columns
+
+<Tabs
+groupId="api-style"
+defaultValue="sql"
+values={[
+{ label: 'Spark SQL', value: 'sql', },
+{ label: 'DataFrame API', value: 'dataframe', },
+]}
+>
+<TabItem value="sql">
+
+```sql
+CREATE TABLE pet_vectors (
+ image_id STRING,
+ breed STRING,
+ embedding VECTOR(1024),
+ image_bytes BLOB
+)
+USING hudi
+TBLPROPERTIES (
+ primaryKey = 'image_id',
+ type = 'cow',
+ hoodie.record.merger.impls = 'org.apache.hudi.DefaultSparkRecordMerger',
+ hoodie.datasource.write.base.file.format = 'lance'
+);
+```
+
+Key things to note:
+- **`VECTOR(1024)`** declares a column that stores 1024-dimensional embedding vectors, enabling ANN (approximate nearest neighbor) search.
+- **`BLOB`** declares a column that can store raw binary data (images, PDFs, audio, etc.) either inline or as out-of-line references.
+- **`lance`** as the base file format enables efficient vector indexing and retrieval.
+
+</TabItem>
+<TabItem value="dataframe">
+
+```python
+import pyarrow as pa
+
+# Build a PyArrow schema with VECTOR metadata
+arrow_schema = pa.schema([
+ pa.field("image_id", pa.string()),
+ pa.field("breed", pa.string()),
+ pa.field("embedding", pa.list_(pa.float32()),
+ metadata={b"hudi_type": b"VECTOR(1024)"}),
+ pa.field("image_bytes", pa.struct([
+ pa.field("type", pa.string()),
+ pa.field("data", pa.binary()),
+ pa.field("reference", pa.struct([
+ pa.field("external_path", pa.string()),
+ pa.field("offset", pa.int64()),
+ pa.field("length", pa.int64()),
+ ])),
+ ]), metadata={b"hudi_type": b"BLOB"}),
+])
+```
+
+When using the DataFrame API, you stamp `hudi_type` metadata on columns to declare
+VECTOR and BLOB semantics. The SQL DDL handles this automatically.
+
+</TabItem>
+</Tabs>
+
+## Step 4: Write Data to the Hudi Table
+
+<Tabs
+groupId="api-style"
+defaultValue="sql"
+values={[
+{ label: 'Spark SQL', value: 'sql', },
+{ label: 'DataFrame API', value: 'dataframe', },
+]}
+>
+<TabItem value="sql">
+
+```sql
+-- Stage embeddings and image bytes into a temp view, then INSERT
+INSERT INTO pet_vectors
+SELECT
+ image_id,
+ breed,
+ embedding,
+ named_struct(
+ 'type', 'INLINE',
+ 'data', image_bytes,
+ 'reference', CAST(NULL AS STRUCT<external_path: STRING, offset: BIGINT, length: BIGINT>)
+ ) AS image_bytes
+FROM staging_view;
+```
+
+</TabItem>
+<TabItem value="dataframe">
+
+```python
+# Prepare rows with inline BLOB structs
+rows = []
+for i in range(len(embeddings)):
+ img_path = dataset._images[i]
+ with open(img_path, "rb") as f:
+ raw_bytes = f.read()
+
+ rows.append({
+ "image_id": f"pet_{i:05d}",
+ "breed": dataset.classes[labels[i]],
+ "embedding": embeddings[i].tolist(),
+ "image_bytes": {"type": "INLINE", "data": raw_bytes, "reference": None},
+ })
+
+# Write via PyArrow staging (avoids PySpark serialization issues)
+table = pa.Table.from_pylist(rows, schema=arrow_schema)
+staging_path = "/tmp/hudi_ai_staging"
+import pyarrow.parquet as pq
+pq.write_table(table, f"{staging_path}/data.parquet")
+
+df = spark.read.parquet(staging_path)
+
+(df.write
+ .format("hudi")
+ .option("hoodie.table.name", "pet_vectors")
+ .option("hoodie.datasource.write.recordkey.field", "image_id")
+ .option("hoodie.record.merger.impls",
+ "org.apache.hudi.DefaultSparkRecordMerger")
+ .option("hoodie.datasource.write.base.file.format", "lance")
+ .mode("overwrite")
+ .save("/tmp/hudi_pet_vectors"))
+```
+
+</TabItem>
+</Tabs>
+
+## Step 5: Run Vector Similarity Search
+
+Now for the exciting part — find images similar to a query image using cosine similarity:
+
+<Tabs
+groupId="api-style"
+defaultValue="sql"
+values={[
+{ label: 'Spark SQL', value: 'sql', },
+{ label: 'DataFrame API', value: 'dataframe', },
+]}
+>
+<TabItem value="sql">
+
+```sql
+-- Pick a query image and find the 5 most similar pets
+SELECT image_id, breed, _hudi_distance AS distance
+FROM hudi_vector_search(
+ 'pet_vectors', -- table name
+ 'embedding', -- vector column
+ ARRAY(0.12, -0.03, ...), -- query embedding (1024 floats)
+ 5, -- top-K results
+ 'cosine' -- distance metric
+)
+ORDER BY distance;
+```
+
+</TabItem>
+<TabItem value="dataframe">
+
+```python
+# Generate embedding for a query image
+query_img = transform(dataset[0][0] if isinstance(dataset[0][0],
+ torch.Tensor) else transform(dataset[0][0]))
+with torch.no_grad():
+ query_emb = model(query_img.unsqueeze(0))
+ query_emb = query_emb / query_emb.norm(dim=1, keepdim=True)
+
+query_list = query_emb.squeeze().tolist()
+
+# Run vector search using the TVF (table-valued function)
+array_literal = ", ".join(str(round(v, 8)) for v in query_list)
+results = spark.sql(f"""
+ SELECT image_id, breed, _hudi_distance AS distance
+ FROM hudi_vector_search(
+ 'pet_vectors', 'embedding',
+ ARRAY({array_literal}), 5, 'cosine'
+ )
+ ORDER BY distance
+""")
+results.show()
+```
+
+</TabItem>
+</Tabs>
+
+Example output:
+
+```
++----------+-----------------+----------+
+| image_id| breed| distance|
++----------+-----------------+----------+
+| pet_00000| Abyssinian| 0.000|
+| pet_00412| Abyssinian| 0.127|
+| pet_00831| Abyssinian| 0.134|
+| pet_00205| Siamese| 0.198|
+| pet_00617| Bengal | 0.213|
++----------+-----------------+----------+
+```
+
+The query returns the most visually similar images ranked by cosine distance. A distance of 0 is an
+exact match (the query image itself).
+
+## Step 6: Retrieve Stored Images
+
+Use the `read_blob()` function to materialize BLOB data from the table:
+
+```sql
+SELECT image_id, breed, read_blob(image_bytes) AS raw_image
+FROM pet_vectors
+WHERE image_id = 'pet_00412';
+```
+
+For **out-of-line** BLOBs (references to external storage), `read_blob()` transparently resolves the
+pointer and fetches the bytes on demand, keeping your table footprint small.
+
+## What's Next
+
+You have just built a complete AI-powered image search pipeline on Apache Hudi. Here is what to explore next:
+
+- **[Vector Search](vector_search.md)** — Deep dive into the VECTOR type, distance metrics, and search tuning
+- **[Unstructured Data](blob_unstructured_data.md)** — Learn about BLOB inline vs. out-of-line storage for images, PDFs, audio, and more
+- **[Semi-Structured Data](variant_type.md)** — Store flexible JSON-like data with the VARIANT type
+- **[Lance File Format](lance_file_format.md)** — Understand why Lance is optimized for vector and ML workloads
+- **[AI-Native Lakehouse Overview](ai_overview.md)** — See the full picture of Hudi's AI capabilities
+
+:::tip
+The complete demo scripts (DataFrame API, SQL, and BLOB-only variants) are available in the
+[hudi-examples](https://github.com/apache/hudi/tree/master/hudi-examples/hudi-examples-spark/src/test/python/vector_blob_demo) repository.
Review Comment:
Fixed now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] docs: add AI-native lakehouse documentation [hudi]
Posted by "voonhous (via GitHub)" <gi...@apache.org>.
voonhous commented on code in PR #18591:
URL: https://github.com/apache/hudi/pull/18591#discussion_r3148980751
##########
website/docs/blob_unstructured_data.md:
##########
@@ -0,0 +1,319 @@
+---
+title: "Unstructured Data"
+keywords: [ hudi, blob, unstructured data, images, binary, pdf, audio, video, inline, out-of-line, read_blob]
+summary: "Store and query unstructured data (images, PDFs, audio, video) in Hudi tables using the BLOB type with inline or out-of-line storage"
+toc: true
+last_modified_at: 2026-04-25T00:00:00-00:00
+---
+
+import Tabs from '@theme/Tabs';
+import TabItem from '@theme/TabItem';
+
+The `BLOB` type lets you store raw binary data — images, PDFs, audio clips, video files, model weights —
+directly in Hudi tables. Combined with Hudi's transactional guarantees and table services, BLOB makes
+the data lakehouse a single source of truth for both structured and unstructured data.
+
+## BLOB Type Overview
+
+A BLOB column stores binary data in one of two modes:
+
+| Mode | Storage | Table Footprint | Read Pattern |
+|:-----|:--------|:----------------|:-------------|
+| **Inline** | Raw bytes embedded in the table row | Larger (bytes stored in data files) | Direct read — no external fetch |
+| **Out-of-line** | Pointer to external storage location | Very small (< 1% of data size) | On-demand via `read_blob()` |
+
+Choose **inline** for small objects that are frequently read together (thumbnails, short audio clips,
+small documents). Choose **out-of-line** for large objects where you typically query metadata first and
+fetch raw data selectively (high-res images, video, model checkpoints).
+
+## Creating Tables with BLOB Columns
+
+<Tabs
+groupId="api-style"
+defaultValue="sql"
+values={[
+{ label: 'Spark SQL', value: 'sql', },
+{ label: 'DataFrame API', value: 'dataframe', },
+]}
+>
+<TabItem value="sql">
+
+```sql
+CREATE TABLE media_assets (
+ asset_id STRING,
+ file_name STRING,
+ mime_type STRING,
+ file_size BIGINT,
+ content BLOB
+) USING hudi
+TBLPROPERTIES (
+ primaryKey = 'asset_id',
+ type = 'cow'
+);
+```
+
+The `BLOB` keyword in DDL automatically configures the column with the correct internal structure.
+
+</TabItem>
+<TabItem value="dataframe">
+
+```python
+import pyarrow as pa
+
+schema = pa.schema([
+ pa.field("asset_id", pa.string()),
+ pa.field("file_name", pa.string()),
+ pa.field("mime_type", pa.string()),
+ pa.field("file_size", pa.int64()),
+ pa.field("content", pa.struct([
+ pa.field("type", pa.string()),
+ pa.field("data", pa.binary()),
+ pa.field("reference", pa.struct([
+ pa.field("external_path", pa.string()),
+ pa.field("offset", pa.int64()),
+ pa.field("length", pa.int64()),
+ ])),
+ ]), metadata={b"hudi_type": b"BLOB"}),
+])
+```
+
+The BLOB internal structure is a struct with three fields:
+- `type` — `"INLINE"` or `"OUT_OF_LINE"`
+- `data` — raw bytes (populated for inline, null for out-of-line)
+- `reference` — external storage pointer with subfields:
+ - `external_path` — file path for out-of-line data
+ - `offset` — byte offset in the file (null means read from start)
+ - `length` — byte length to read (null means read to end of file)
+ - `managed` — boolean indicating whether Hudi manages the external file
+
+</TabItem>
+</Tabs>
+
+## Writing Inline BLOBs
+
+Inline BLOBs embed the raw bytes directly in the table row.
+
+<Tabs
+groupId="api-style"
+defaultValue="sql"
+values={[
+{ label: 'Spark SQL', value: 'sql', },
+{ label: 'DataFrame API', value: 'dataframe', },
+]}
+>
+<TabItem value="sql">
+
+```sql
+INSERT INTO media_assets VALUES (
+ 'asset_001',
+ 'logo.png',
+ 'image/png',
+ 45230,
+ named_struct(
+ 'type', 'INLINE',
+ 'data', /* binary literal or column reference */,
+ 'reference', CAST(NULL AS STRUCT<external_path: STRING, offset: BIGINT, length: BIGINT>)
+ )
+);
+```
+
+</TabItem>
+<TabItem value="dataframe">
+
+```python
+from pyspark.sql import Row
+from pyspark.sql.types import *
+
+# Read raw bytes
+with open("logo.png", "rb") as f:
+ raw_bytes = f.read()
+
+row = Row(
+ asset_id="asset_001",
+ file_name="logo.png",
+ mime_type="image/png",
+ file_size=len(raw_bytes),
+ content=Row(type="INLINE", data=raw_bytes, reference=None)
+)
+```
+
+</TabItem>
+</Tabs>
+
+## Writing Out-of-Line BLOBs
+
+Out-of-line BLOBs store a pointer to data in external storage. The actual bytes live elsewhere (e.g.,
+a binary container file on S3, a separate object store path, or a shared filesystem). The table only
+stores the reference metadata.
+
+```sql
+INSERT INTO media_assets VALUES (
+ 'asset_002',
+ 'video.mp4',
+ 'video/mp4',
+ 1073741824, -- 1 GB
+ named_struct(
+ 'type', 'OUT_OF_LINE',
+ 'data', CAST(NULL AS BINARY),
+ 'reference', named_struct(
+ 'external_path', 's3://my-bucket/media/container_001.bin',
+ 'offset', 8388608, -- byte offset in the container
+ 'length', 1073741824 -- number of bytes
+ )
+ )
+);
+```
+
+### Container File Pattern
+
+A common pattern for out-of-line storage is to pack multiple objects into a single binary container file:
+
+```
+container_001.bin
+├── [offset=0, len=45230] → logo.png
+├── [offset=45230, len=89012] → photo.jpg
+├── [offset=134242, len=1073741824] → video.mp4
+└── ...
+```
+
+Each BLOB row stores the `(external_path, offset, length)` triple. This avoids creating millions of
+small files on object storage and enables efficient batch access.
+
+## Reading BLOBs
+
+### Querying Metadata (No Fetch)
+
+Standard queries on a BLOB column return the descriptor — not the raw bytes:
+
+```sql
+SELECT asset_id, file_name, content.type, content.reference.external_path
+FROM media_assets;
+```
+
+```
++----------+-----------+-------------+----------------------------------------+
+| asset_id| file_name| type| external_path|
++----------+-----------+-------------+----------------------------------------+
+| asset_001| logo.png| INLINE| null|
+| asset_002| video.mp4| OUT_OF_LINE| s3://my-bucket/media/container_001.bin |
++----------+-----------+-------------+----------------------------------------+
+```
+
+This is fast and lightweight — no binary data is transferred.
+
+### Resolving Raw Bytes with read_blob()
+
+Use the `read_blob()` SQL function to materialize the actual bytes:
+
+```sql
+-- Returns raw binary data
+SELECT asset_id, read_blob(content) AS raw_bytes
+FROM media_assets
+WHERE asset_id = 'asset_001';
+```
+
+For **inline** BLOBs, `read_blob()` simply extracts the embedded bytes.
+
+For **out-of-line** BLOBs, `read_blob()` reads from the external path at the specified offset and length,
+transparently fetching the data on demand.
+
+:::tip
+Use `read_blob()` selectively — filter first, then resolve. Avoid `SELECT read_blob(content) FROM large_table`
+without a WHERE clause, as this will fetch all raw data.
+:::
+
+## Use Cases
+
+### Image Datasets for Computer Vision
+
+Store training images alongside metadata and embeddings:
+
+```sql
+CREATE TABLE training_images (
+ image_id STRING,
+ label STRING,
+ split STRING, -- 'train', 'val', 'test'
+ embedding VECTOR(1024),
+ raw_image BLOB
+) USING hudi TBLPROPERTIES (...);
+
+-- Get raw images for a specific label
+SELECT image_id, read_blob(raw_image) AS pixels
+FROM training_images
+WHERE label = 'cat' AND split = 'train';
+```
+
+### Document Store for RAG Pipelines
+
+Store PDF documents alongside their chunk embeddings:
+
+```sql
+CREATE TABLE knowledge_base (
+ doc_id STRING,
+ chunk_id STRING,
+ source_url STRING,
+ text STRING,
+ embedding VECTOR(1536),
+ original BLOB -- original PDF bytes
+) USING hudi TBLPROPERTIES (...);
+
+-- Retrieve full document after vector search
+SELECT doc_id, source_url, read_blob(original) AS pdf_bytes
+FROM knowledge_base
+WHERE doc_id IN (SELECT doc_id FROM top_matches);
+```
+
+### Audio/Video Processing Pipelines
+
+```sql
+CREATE TABLE audio_clips (
+ clip_id STRING,
+ transcript STRING,
+ duration DOUBLE,
+ embedding VECTOR(512),
+ audio BLOB
+) USING hudi TBLPROPERTIES (...);
+```
+
+## Storage Efficiency
+
+Out-of-line BLOBs keep the Hudi table footprint extremely small:
+
+| Metric | Inline | Out-of-Line |
+|:-------|:-------|:------------|
+| Table size vs. raw data | ~100% | < 1% |
+| Query metadata without fetch | Requires reading data files | Only reads pointer columns |
+| Random access to raw data | Read full row | Seek to (offset, length) |
+| Best for object size | < 1 MB | > 1 MB |
+
+## Configuration Reference
+
+| Property | Default | Description |
+|:---------|:--------|:------------|
+| `hoodie.read.blob.inline.mode` | `CONTENT` | Controls how INLINE BLOBs are read. `CONTENT` materializes raw bytes in the `data` column. `DESCRIPTOR` surfaces `(position, size)` coordinates rewritten as OUT_OF_LINE references. |
+| `hoodie.blob.batching.max.gap.bytes` | `4096` | Maximum gap (in bytes) between consecutive byte ranges before they are merged into a single read. Larger values reduce I/O calls at the cost of reading some unused bytes. |
+| `hoodie.blob.batching.lookahead.size` | `50` | Number of rows to buffer for batch read detection. Larger values improve batching for sorted data but increase memory usage. |
Review Comment:
There are some characteristics/nuance with DESCRIPTOR that i think @yihua summarised very nicely into a table:
```
**Storage (immutable):**
| Storage type | `data` | `reference` |
|---|---|---|
| `INLINE` | bytes | `null` |
| `OUT_OF_LINE` | `null` | `{path, offset, length, managed}` |
**`SELECT blob_col` (no `read_blob()`) — controlled by `hoodie.read.blob.inline.mode`:**
| Mode | Inline blob behavior | Out-of-line (unaffected) |
|---|---|---|
| **Eager** | Read struct as-is, `data` has bytes | Reference as-is |
| **Lazy** (default) | Prune `data` from read (no I/O), populate `reference` with pos+size | Reference as-is |
**`SELECT read_blob(blob_col)` — always materializes bytes, mode-unaware:**
| Storage type | Behavior |
|---|---|
| `INLINE` | Reader reads full struct (no pruning), `read_blob()` returns `data` bytes directly |
| `OUT_OF_LINE` | `read_blob()` does batched range read via `reference` |
**Merge/compaction**: eager mode (need actual bytes for rewriting).
**Format support for lazy mode:**
- Lance: works now (DESCRIPTOR mode provides pos+size natively)
- Parquet: separate PR to extract pos+size within the Parquet file for the reference transformation
```
Is it possible to make this more visual? i.e. create a mermaid chart or something so it's easier for users to follow. Docusaurus supports mermaid charts as of now.
##########
website/docs/ai-quick-start-guide.md:
##########
@@ -0,0 +1,343 @@
+---
+title: "AI Quick Start"
+keywords: [ hudi, ai, vector search, embeddings, unstructured data, blob, lance, machine learning]
+summary: "Get started with Hudi's AI-native capabilities: vector search, unstructured data storage, and embedding-powered similarity queries"
+toc: true
+last_modified_at: 2026-04-25T00:00:00-00:00
+---
+
+import Tabs from '@theme/Tabs';
+import TabItem from '@theme/TabItem';
+
+This guide walks you through Hudi's AI-native capabilities using a hands-on image similarity search example.
+You will store image embeddings and raw image bytes in a Hudi table, then run vector search to find visually
+similar images — all using familiar Spark SQL or DataFrame APIs.
+
+## What You Will Build
+
+By the end of this guide you will have:
+
+1. Created a Hudi table with **VECTOR** columns (for embeddings) and **BLOB** columns (for raw image bytes)
+2. Generated image embeddings using a pre-trained neural network (MobileNetV3)
+3. Written embeddings and images to a Hudi table backed by the **Lance** columnar format
+4. Run **vector similarity search** to find the top-K most similar images to a query
+
+## Prerequisites
+
+- Java 11+
+- Python 3.10 – 3.12 (Python 3.13+ is not yet supported)
+- Apache Spark 3.5.x
+- Hudi 1.2.0+ (with Spark bundle)
+
+### Python Dependencies
+
+```bash
+pip install pyspark==3.5.* pyarrow>=14.0.0 \
+ torch>=2.3.0 torchvision>=0.18.0 timm>=1.0.9 \
+ scikit-learn>=1.4.2 numpy>=1.26.0 pillow>=10.3.0 matplotlib>=3.8.0
+```
+
+### Required JARs
+
+You need two JAR files on the Spark classpath:
+
+| JAR | How to get it |
+|:----|:--------------|
+| Hudi Spark bundle | Build from source: `mvn clean package -pl packaging/hudi-spark-bundle -am -DskipTests -Dspark3.5` or download from [releases](/releases/download) |
+| Lance Spark bundle | Maven Central: `org.lance:lance-spark-bundle-3.5_2.12:0.4.0` |
+
+```bash
+export HUDI_BUNDLE_JAR=/path/to/hudi-spark3.5-bundle_2.12-1.2.0-SNAPSHOT.jar
+export LANCE_BUNDLE_JAR=/path/to/lance-spark-bundle-3.5_2.12-0.4.0.jar
+```
+
+## Step 1: Launch Spark with Hudi + Lance
+
+```python
+from pyspark.sql import SparkSession
+
+spark = (SparkSession.builder
+ .appName("HudiAIQuickStart")
+ .master("local[*]")
+ .config("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
+ .config("spark.sql.catalog.spark_catalog",
+ "org.apache.spark.sql.hudi.catalog.HoodieCatalog")
+ .config("spark.sql.extensions",
+ "org.apache.spark.sql.hudi.HoodieSparkSessionExtension")
+ .config("spark.jars", f"{HUDI_BUNDLE_JAR},{LANCE_BUNDLE_JAR}")
+ .config("spark.driver.memory", "8g")
+ .getOrCreate())
+```
+
+## Step 2: Prepare Image Embeddings
+
+We use the [Oxford-IIIT Pet dataset](https://www.robots.ox.ac.uk/~vgg/data/pets/) (37 dog and cat breeds) and
+generate embeddings with MobileNetV3, a lightweight convolutional neural network.
+
+```python
+import torch, timm, numpy as np
+from torchvision import datasets, transforms
+from torch.utils.data import DataLoader
+
+# Load the pre-trained model
+model = timm.create_model("mobilenetv3_large_100", pretrained=True, num_classes=0)
+model.eval()
+
+# Image pre-processing pipeline
+transform = transforms.Compose([
+ transforms.Resize(256),
+ transforms.CenterCrop(224),
+ transforms.ToTensor(),
+ transforms.Normalize(mean=[0.485, 0.456, 0.406],
+ std=[0.229, 0.224, 0.225]),
+])
+
+# Download and load the dataset
+dataset = datasets.OxfordIIITPet(root="/tmp/pets", split="trainval",
+ download=True, transform=transform)
+loader = DataLoader(dataset, batch_size=64, shuffle=False)
+
+# Generate embeddings
+embeddings, labels = [], []
+with torch.no_grad():
+ for images, targets in loader:
+ emb = model(images)
+ # L2-normalize for cosine similarity
+ emb = emb / emb.norm(dim=1, keepdim=True)
+ embeddings.append(emb.numpy())
+ labels.extend(targets.tolist())
+
+embeddings = np.vstack(embeddings)
+print(f"Generated {len(embeddings)} embeddings of dimension {embeddings.shape[1]}")
+```
+
+## Step 3: Create a Hudi Table with VECTOR and BLOB Columns
+
+<Tabs
+groupId="api-style"
+defaultValue="sql"
+values={[
+{ label: 'Spark SQL', value: 'sql', },
+{ label: 'DataFrame API', value: 'dataframe', },
+]}
+>
+<TabItem value="sql">
+
+```sql
+CREATE TABLE pet_vectors (
+ image_id STRING,
+ breed STRING,
+ embedding VECTOR(1024),
+ image_bytes BLOB
+)
+USING hudi
+TBLPROPERTIES (
+ primaryKey = 'image_id',
+ type = 'cow',
+ hoodie.record.merger.impls = 'org.apache.hudi.DefaultSparkRecordMerger',
+ hoodie.datasource.write.base.file.format = 'lance'
+);
+```
+
+Key things to note:
+- **`VECTOR(1024)`** declares a column that stores 1024-dimensional embedding vectors, enabling ANN (approximate nearest neighbor) search.
+- **`BLOB`** declares a column that can store raw binary data (images, PDFs, audio, etc.) either inline or as out-of-line references.
+- **`lance`** as the base file format enables efficient vector indexing and retrieval.
+
+</TabItem>
+<TabItem value="dataframe">
+
+```python
+import pyarrow as pa
+
+# Build a PyArrow schema with VECTOR metadata
+arrow_schema = pa.schema([
+ pa.field("image_id", pa.string()),
+ pa.field("breed", pa.string()),
+ pa.field("embedding", pa.list_(pa.float32()),
+ metadata={b"hudi_type": b"VECTOR(1024)"}),
+ pa.field("image_bytes", pa.struct([
+ pa.field("type", pa.string()),
+ pa.field("data", pa.binary()),
+ pa.field("reference", pa.struct([
+ pa.field("external_path", pa.string()),
+ pa.field("offset", pa.int64()),
+ pa.field("length", pa.int64()),
+ ])),
+ ]), metadata={b"hudi_type": b"BLOB"}),
+])
+```
+
+When using the DataFrame API, you stamp `hudi_type` metadata on columns to declare
+VECTOR and BLOB semantics. The SQL DDL handles this automatically.
+
+</TabItem>
+</Tabs>
+
+## Step 4: Write Data to the Hudi Table
+
+<Tabs
+groupId="api-style"
+defaultValue="sql"
+values={[
+{ label: 'Spark SQL', value: 'sql', },
+{ label: 'DataFrame API', value: 'dataframe', },
+]}
+>
+<TabItem value="sql">
+
+```sql
+-- Stage embeddings and image bytes into a temp view, then INSERT
+INSERT INTO pet_vectors
+SELECT
+ image_id,
+ breed,
+ embedding,
+ named_struct(
+ 'type', 'INLINE',
+ 'data', image_bytes,
+ 'reference', CAST(NULL AS STRUCT<external_path: STRING, offset: BIGINT, length: BIGINT>)
+ ) AS image_bytes
+FROM staging_view;
+```
+
+</TabItem>
+<TabItem value="dataframe">
+
+```python
+# Prepare rows with inline BLOB structs
+rows = []
+for i in range(len(embeddings)):
+ img_path = dataset._images[i]
+ with open(img_path, "rb") as f:
+ raw_bytes = f.read()
+
+ rows.append({
+ "image_id": f"pet_{i:05d}",
+ "breed": dataset.classes[labels[i]],
+ "embedding": embeddings[i].tolist(),
+ "image_bytes": {"type": "INLINE", "data": raw_bytes, "reference": None},
+ })
+
+# Write via PyArrow staging (avoids PySpark serialization issues)
+table = pa.Table.from_pylist(rows, schema=arrow_schema)
+staging_path = "/tmp/hudi_ai_staging"
+import pyarrow.parquet as pq
+pq.write_table(table, f"{staging_path}/data.parquet")
+
+df = spark.read.parquet(staging_path)
+
+(df.write
+ .format("hudi")
+ .option("hoodie.table.name", "pet_vectors")
+ .option("hoodie.datasource.write.recordkey.field", "image_id")
+ .option("hoodie.record.merger.impls",
+ "org.apache.hudi.DefaultSparkRecordMerger")
+ .option("hoodie.datasource.write.base.file.format", "lance")
+ .mode("overwrite")
+ .save("/tmp/hudi_pet_vectors"))
+```
+
+</TabItem>
+</Tabs>
+
+## Step 5: Run Vector Similarity Search
+
+Now for the exciting part — find images similar to a query image using cosine similarity:
+
+<Tabs
+groupId="api-style"
+defaultValue="sql"
+values={[
+{ label: 'Spark SQL', value: 'sql', },
+{ label: 'DataFrame API', value: 'dataframe', },
+]}
+>
+<TabItem value="sql">
+
+```sql
+-- Pick a query image and find the 5 most similar pets
+SELECT image_id, breed, _hudi_distance AS distance
+FROM hudi_vector_search(
+ 'pet_vectors', -- table name
+ 'embedding', -- vector column
+ ARRAY(0.12, -0.03, ...), -- query embedding (1024 floats)
+ 5, -- top-K results
+ 'cosine' -- distance metric
+)
+ORDER BY distance;
+```
+
+</TabItem>
+<TabItem value="dataframe">
+
+```python
+# Generate embedding for a query image
+query_img = transform(dataset[0][0] if isinstance(dataset[0][0],
+ torch.Tensor) else transform(dataset[0][0]))
+with torch.no_grad():
+ query_emb = model(query_img.unsqueeze(0))
+ query_emb = query_emb / query_emb.norm(dim=1, keepdim=True)
+
+query_list = query_emb.squeeze().tolist()
+
+# Run vector search using the TVF (table-valued function)
+array_literal = ", ".join(str(round(v, 8)) for v in query_list)
+results = spark.sql(f"""
+ SELECT image_id, breed, _hudi_distance AS distance
+ FROM hudi_vector_search(
+ 'pet_vectors', 'embedding',
+ ARRAY({array_literal}), 5, 'cosine'
+ )
+ ORDER BY distance
+""")
+results.show()
+```
+
+</TabItem>
+</Tabs>
+
+Example output:
+
+```
++----------+-----------------+----------+
+| image_id| breed| distance|
++----------+-----------------+----------+
+| pet_00000| Abyssinian| 0.000|
+| pet_00412| Abyssinian| 0.127|
+| pet_00831| Abyssinian| 0.134|
+| pet_00205| Siamese| 0.198|
+| pet_00617| Bengal | 0.213|
++----------+-----------------+----------+
+```
+
+The query returns the most visually similar images ranked by cosine distance. A distance of 0 is an
+exact match (the query image itself).
+
+## Step 6: Retrieve Stored Images
+
+Use the `read_blob()` function to materialize BLOB data from the table:
+
+```sql
+SELECT image_id, breed, read_blob(image_bytes) AS raw_image
+FROM pet_vectors
+WHERE image_id = 'pet_00412';
+```
+
+For **out-of-line** BLOBs (references to external storage), `read_blob()` transparently resolves the
+pointer and fetches the bytes on demand, keeping your table footprint small.
+
+## What's Next
+
+You have just built a complete AI-powered image search pipeline on Apache Hudi. Here is what to explore next:
+
+- **[Vector Search](vector_search.md)** — Deep dive into the VECTOR type, distance metrics, and search tuning
+- **[Unstructured Data](blob_unstructured_data.md)** — Learn about BLOB inline vs. out-of-line storage for images, PDFs, audio, and more
+- **[Semi-Structured Data](variant_type.md)** — Store flexible JSON-like data with the VARIANT type
+- **[Lance File Format](lance_file_format.md)** — Understand why Lance is optimized for vector and ML workloads
+- **[AI-Native Lakehouse Overview](ai_overview.md)** — See the full picture of Hudi's AI capabilities
+
+:::tip
+The complete demo scripts (DataFrame API, SQL, and BLOB-only variants) are available in the
+[hudi-examples](https://github.com/apache/hudi/tree/master/hudi-examples/hudi-examples-spark/src/test/python/vector_blob_demo) repository.
Review Comment:
These are not available as of now, let's remind ourselves to prepare the demo as a prerequisite before publishing this to asf-site.
##########
website/docs/lance_file_format.md:
##########
@@ -0,0 +1,156 @@
+---
+title: "Lance File Format"
+keywords: [ hudi, lance, file format, vector, AI, ML, columnar, ANN, indexing]
+summary: "Use the Lance columnar file format with Hudi for vector-optimized storage, ANN indexing, and efficient ML workloads"
+toc: true
+last_modified_at: 2026-04-25T00:00:00-00:00
+---
+
+[Lance](https://lancedb.github.io/lance/) is a modern columnar data format designed for AI and machine learning
+workloads. Hudi's pluggable storage architecture lets you use Lance as the base file format alongside Parquet
+and ORC, unlocking vector indexing, fast random access, and optimized high-dimensional array storage.
+
+## Why Lance?
+
+Traditional columnar formats like Parquet are optimized for analytical queries — filtering, aggregation, and
+scan-heavy workloads on scalar data. AI workloads have different access patterns:
+
+| Requirement | Parquet | Lance |
+|:------------|:--------|:------|
+| Vector ANN indexing | Not supported | Built-in IVF-PQ indexes |
+| Random row access | Slow (scan-based) | O(1) row lookup |
+| High-dimensional arrays | Generic list encoding | Optimized fixed-size vector encoding |
+| Append + update | Full file rewrite | Fragment-based append and update |
+| Nested struct columns | Supported | Supported with better random access |
+
+Lance is not a replacement for Parquet — it is a complementary format for workloads where vector search,
+random access, and high-dimensional data are priorities.
+
+## Enabling Lance in Hudi
+
+### Table Creation
+
+Set the base file format to `lance` in table properties:
+
+```sql
+CREATE TABLE my_ai_table (
+ id STRING,
+ embedding VECTOR(768),
+ metadata STRING
+) USING hudi
+TBLPROPERTIES (
+ primaryKey = 'id',
+ type = 'cow',
+ hoodie.record.merger.impls = 'org.apache.hudi.DefaultSparkRecordMerger',
+ hoodie.datasource.write.base.file.format = 'lance'
+);
+```
+
+### DataFrame API
+
+```python
+(df.write
+ .format("hudi")
+ .option("hoodie.table.name", "my_ai_table")
+ .option("hoodie.datasource.write.recordkey.field", "id")
+ .option("hoodie.record.merger.impls",
+ "org.apache.hudi.DefaultSparkRecordMerger")
+ .option("hoodie.datasource.write.base.file.format", "lance")
+ .mode("overwrite")
+ .save("/path/to/my_ai_table"))
+```
+
+### Required Dependencies
+
+Add the Lance Spark bundle to your Spark classpath:
+
+| Component | Maven Coordinates |
+|:----------|:-----------------|
+| Lance Spark Bundle (Spark 3.5) | `org.lance:lance-spark-bundle-3.5_2.12:0.4.0` |
+
+```bash
+export LANCE_BUNDLE_JAR=/path/to/lance-spark-bundle-3.5_2.12-0.4.0.jar
+
+# Include both Hudi and Lance JARs
+spark-shell --jars $HUDI_BUNDLE_JAR,$LANCE_BUNDLE_JAR
+```
+
+## How Hudi + Lance Work Together
+
+Hudi manages the table layer — transactions, schema, timeline, table services — while Lance handles the
+file-level storage:
+
+```
+┌───────────────────────────────────┐
+│ Hudi Table Layer │
+│ Timeline, Metadata, Indexing │
+│ Transactions, Schema Evolution │
+├───────────────────────────────────┤
+│ File Group / File Slice │
+│ (same Hudi concepts as Parquet) │
+├───────────────────────────────────┤
+│ Lance Data Files (.lance) │
+│ IVF-PQ vector index │
+│ Columnar storage │
+│ Fragment-based layout │
+├───────────────────────────────────┤
+│ Storage (S3, GCS, HDFS, FS) │
+└───────────────────────────────────┘
+```
+
+All Hudi table services work with Lance-backed tables:
+
+- **Compaction** — merges log files into Lance base files
+- **Clustering** — reorganizes Lance files for better data locality
+- **Cleaning** — removes old Lance file versions
+- **Metadata indexing** — column stats and bloom filters work across Lance files
+
+## Vector Search with Lance
+
+The `hudi_vector_search` TVF leverages Lance's built-in IVF-PQ index for approximate nearest neighbor search:
+
+```sql
+SELECT id, metadata, _hudi_distance
+FROM hudi_vector_search(
+ 'my_ai_table', 'embedding',
+ ARRAY(0.1, 0.2, ...), -- query vector
+ 10, 'cosine'
+)
+ORDER BY _hudi_distance;
+```
+
+See [Vector Search](vector_search.md) for full documentation on the TVF and distance metrics.
+
+## Configuration Reference
+
+| Property | Description | Default |
+|:---------|:------------|:--------|
+| `hoodie.datasource.write.base.file.format` | Set to `lance` to use Lance as the base file format | `parquet` |
+| `hoodie.record.merger.impls` | Must be `org.apache.hudi.DefaultSparkRecordMerger` for Lance | — |
+
+## Mixed-Format Tables
+
+Hudi does not require all tables in a lakehouse to use the same file format. You can have:
+
+- **Analytical tables** on Parquet — for traditional BI/SQL workloads
+- **AI tables** on Lance — for embeddings, vector search, and ML feature storage
+- **Archival tables** on ORC — for compressed long-term storage
+
+All share the same Hudi catalog, metadata, and tooling.
+
+## Known Limitations
Review Comment:
We can use the danger/warning admonition for this part.
https://docusaurus.io/docs/markdown-features/admonitions
##########
website/docs/ai-quick-start-guide.md:
##########
@@ -0,0 +1,343 @@
+---
+title: "AI Quick Start"
+keywords: [ hudi, ai, vector search, embeddings, unstructured data, blob, lance, machine learning]
+summary: "Get started with Hudi's AI-native capabilities: vector search, unstructured data storage, and embedding-powered similarity queries"
+toc: true
+last_modified_at: 2026-04-25T00:00:00-00:00
+---
+
+import Tabs from '@theme/Tabs';
+import TabItem from '@theme/TabItem';
+
+This guide walks you through Hudi's AI-native capabilities using a hands-on image similarity search example.
+You will store image embeddings and raw image bytes in a Hudi table, then run vector search to find visually
+similar images — all using familiar Spark SQL or DataFrame APIs.
+
+## What You Will Build
+
+By the end of this guide you will have:
+
+1. Created a Hudi table with **VECTOR** columns (for embeddings) and **BLOB** columns (for raw image bytes)
+2. Generated image embeddings using a pre-trained neural network (MobileNetV3)
+3. Written embeddings and images to a Hudi table backed by the **Lance** columnar format
+4. Run **vector similarity search** to find the top-K most similar images to a query
+
+## Prerequisites
+
+- Java 11+
+- Python 3.10 – 3.12 (Python 3.13+ is not yet supported)
+- Apache Spark 3.5.x
+- Hudi 1.2.0+ (with Spark bundle)
+
+### Python Dependencies
+
+```bash
+pip install pyspark==3.5.* pyarrow>=14.0.0 \
+ torch>=2.3.0 torchvision>=0.18.0 timm>=1.0.9 \
+ scikit-learn>=1.4.2 numpy>=1.26.0 pillow>=10.3.0 matplotlib>=3.8.0
Review Comment:
Another idea is that we could translate everything in here into a TestContainer test too for validation / E2E.
One problem i usually encounter is that test-scripts and code-base are usually decoupled. i.e. It is written at a point in time that corresponds to a snapshot of the state of our code.
We don't do a sanity check after changing our code, so, many things break and quickstarts no longer work after awhile.
##########
website/docs/blob_unstructured_data.md:
##########
@@ -0,0 +1,319 @@
+---
+title: "Unstructured Data"
+keywords: [ hudi, blob, unstructured data, images, binary, pdf, audio, video, inline, out-of-line, read_blob]
+summary: "Store and query unstructured data (images, PDFs, audio, video) in Hudi tables using the BLOB type with inline or out-of-line storage"
+toc: true
+last_modified_at: 2026-04-25T00:00:00-00:00
+---
+
+import Tabs from '@theme/Tabs';
+import TabItem from '@theme/TabItem';
+
+The `BLOB` type lets you store raw binary data — images, PDFs, audio clips, video files, model weights —
+directly in Hudi tables. Combined with Hudi's transactional guarantees and table services, BLOB makes
+the data lakehouse a single source of truth for both structured and unstructured data.
+
+## BLOB Type Overview
+
+A BLOB column stores binary data in one of two modes:
+
Review Comment:
It would be good if we can add hud's blob spec here. So users can easily correlate and see where the `INLINE`/`OUT-OF-LINE` type will be used later.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] docs: add AI-native lakehouse documentation [hudi]
Posted by "rahil-c (via GitHub)" <gi...@apache.org>.
rahil-c commented on code in PR #18591:
URL: https://github.com/apache/hudi/pull/18591#discussion_r3159166996
##########
website/docs/ai-quick-start-guide.md:
##########
@@ -0,0 +1,343 @@
+---
+title: "AI Quick Start"
+keywords: [ hudi, ai, vector search, embeddings, unstructured data, blob, lance, machine learning]
+summary: "Get started with Hudi's AI-native capabilities: vector search, unstructured data storage, and embedding-powered similarity queries"
+toc: true
+last_modified_at: 2026-04-25T00:00:00-00:00
+---
+
+import Tabs from '@theme/Tabs';
+import TabItem from '@theme/TabItem';
+
+This guide walks you through Hudi's AI-native capabilities using a hands-on image similarity search example.
+You will store image embeddings and raw image bytes in a Hudi table, then run vector search to find visually
+similar images — all using familiar Spark SQL or DataFrame APIs.
+
+## What You Will Build
+
+By the end of this guide you will have:
+
+1. Created a Hudi table with **VECTOR** columns (for embeddings) and **BLOB** columns (for raw image bytes)
+2. Generated image embeddings using a pre-trained neural network (MobileNetV3)
+3. Written embeddings and images to a Hudi table backed by the **Lance** columnar format
Review Comment:
we should change this to say either parquet or lance
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] docs: add AI-native lakehouse documentation [hudi]
Posted by "voonhous (via GitHub)" <gi...@apache.org>.
voonhous commented on code in PR #18591:
URL: https://github.com/apache/hudi/pull/18591#discussion_r3148947689
##########
website/docs/ai-quick-start-guide.md:
##########
@@ -0,0 +1,343 @@
+---
+title: "AI Quick Start"
+keywords: [ hudi, ai, vector search, embeddings, unstructured data, blob, lance, machine learning]
+summary: "Get started with Hudi's AI-native capabilities: vector search, unstructured data storage, and embedding-powered similarity queries"
+toc: true
+last_modified_at: 2026-04-25T00:00:00-00:00
+---
+
+import Tabs from '@theme/Tabs';
+import TabItem from '@theme/TabItem';
+
+This guide walks you through Hudi's AI-native capabilities using a hands-on image similarity search example.
+You will store image embeddings and raw image bytes in a Hudi table, then run vector search to find visually
+similar images — all using familiar Spark SQL or DataFrame APIs.
+
+## What You Will Build
+
+By the end of this guide you will have:
+
+1. Created a Hudi table with **VECTOR** columns (for embeddings) and **BLOB** columns (for raw image bytes)
+2. Generated image embeddings using a pre-trained neural network (MobileNetV3)
+3. Written embeddings and images to a Hudi table backed by the **Lance** columnar format
+4. Run **vector similarity search** to find the top-K most similar images to a query
+
+## Prerequisites
+
+- Java 11+
+- Python 3.10 – 3.12 (Python 3.13+ is not yet supported)
+- Apache Spark 3.5.x
+- Hudi 1.2.0+ (with Spark bundle)
+
+### Python Dependencies
+
+```bash
+pip install pyspark==3.5.* pyarrow>=14.0.0 \
+ torch>=2.3.0 torchvision>=0.18.0 timm>=1.0.9 \
+ scikit-learn>=1.4.2 numpy>=1.26.0 pillow>=10.3.0 matplotlib>=3.8.0
Review Comment:
**NOTE**: This is a non-blocker
Another idea is that we could translate everything in here into a TestContainer test too for validation / E2E.
One problem i usually encounter is that test-scripts and code-base are usually decoupled. i.e. It is written at a point in time that corresponds to a snapshot of the state of our code.
We don't do a sanity check after changing our code, so, many things break and quickstarts no longer work after awhile.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] docs: add AI-native lakehouse documentation [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua merged PR #18591:
URL: https://github.com/apache/hudi/pull/18591
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] docs: add AI-native lakehouse documentation [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #18591:
URL: https://github.com/apache/hudi/pull/18591#discussion_r3162919936
##########
website/docs/ai-quick-start-guide.md:
##########
@@ -0,0 +1,343 @@
+---
+title: "AI Quick Start"
+keywords: [ hudi, ai, vector search, embeddings, unstructured data, blob, lance, machine learning]
+summary: "Get started with Hudi's AI-native capabilities: vector search, unstructured data storage, and embedding-powered similarity queries"
+toc: true
+last_modified_at: 2026-04-25T00:00:00-00:00
+---
+
+import Tabs from '@theme/Tabs';
+import TabItem from '@theme/TabItem';
+
+This guide walks you through Hudi's AI-native capabilities using a hands-on image similarity search example.
+You will store image embeddings and raw image bytes in a Hudi table, then run vector search to find visually
+similar images — all using familiar Spark SQL or DataFrame APIs.
+
+## What You Will Build
+
+By the end of this guide you will have:
+
+1. Created a Hudi table with **VECTOR** columns (for embeddings) and **BLOB** columns (for raw image bytes)
+2. Generated image embeddings using a pre-trained neural network (MobileNetV3)
+3. Written embeddings and images to a Hudi table backed by the **Lance** columnar format
Review Comment:
I remove the mention of file formats in the quick start and instead focus on the use cases for clarity.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] docs: add AI-native lakehouse documentation [hudi]
Posted by "rahil-c (via GitHub)" <gi...@apache.org>.
rahil-c commented on code in PR #18591:
URL: https://github.com/apache/hudi/pull/18591#discussion_r3150321126
##########
website/docs/blob_unstructured_data.md:
##########
@@ -0,0 +1,319 @@
+---
+title: "Unstructured Data"
+keywords: [ hudi, blob, unstructured data, images, binary, pdf, audio, video, inline, out-of-line, read_blob]
+summary: "Store and query unstructured data (images, PDFs, audio, video) in Hudi tables using the BLOB type with inline or out-of-line storage"
+toc: true
+last_modified_at: 2026-04-25T00:00:00-00:00
+---
+
+import Tabs from '@theme/Tabs';
+import TabItem from '@theme/TabItem';
+
+The `BLOB` type lets you store raw binary data — images, PDFs, audio clips, video files, model weights —
+directly in Hudi tables. Combined with Hudi's transactional guarantees and table services, BLOB makes
+the data lakehouse a single source of truth for both structured and unstructured data.
+
+## BLOB Type Overview
+
+A BLOB column stores binary data in one of two modes:
+
Review Comment:
Yea nice point
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org