You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by vi...@apache.org on 2026/07/14 20:33:21 UTC
(hudi) branch asf-site updated: docs(blog): explainer wave 1: open table format, upsert, incremental ETL, ACID (#19269)
This is an automated email from the ASF dual-hosted git repository.
vinothchandar pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/hudi.git
The following commit(s) were added to refs/heads/asf-site by this push:
new 4ba23d1275fc docs(blog): explainer wave 1: open table format, upsert, incremental ETL, ACID (#19269)
4ba23d1275fc is described below
commit 4ba23d1275fcdb0a7434a448b75be2e57e9e73b1
Author: vinoth chandar <vi...@users.noreply.github.com>
AuthorDate: Tue Jul 14 13:33:14 2026 -0700
docs(blog): explainer wave 1: open table format, upsert, incremental ETL, ACID (#19269)
---
...18-hudi-incremental-processing-on-data-lakes.md | 4 +
.../2026-07-14-what-is-an-open-table-format.md | 113 +++++++++++++++
.../2026-07-15-what-is-upsert-on-a-data-lake.md | 116 +++++++++++++++
...07-16-what-is-incremental-etl-on-a-data-lake.md | 156 +++++++++++++++++++++
.../blog/2026-07-17-what-is-acid-on-a-data-lake.md | 128 +++++++++++++++++
website/static/llms.txt | 4 +
6 files changed, 521 insertions(+)
diff --git a/website/blog/2020-08-18-hudi-incremental-processing-on-data-lakes.md b/website/blog/2020-08-18-hudi-incremental-processing-on-data-lakes.md
index a39e3f612db1..5461c357c128 100644
--- a/website/blog/2020-08-18-hudi-incremental-processing-on-data-lakes.md
+++ b/website/blog/2020-08-18-hudi-incremental-processing-on-data-lakes.md
@@ -17,6 +17,10 @@ Hudi joined the Apache incubator for incubation in January 2019, and was promote
of Hudi to the data lake from the perspective of "incremental processing". More information about Apache Hudi's framework functions, features, usage scenarios, and
latest developments can be found at [QCon Global Software Development Conference (Shanghai Station) 2020](https://qconplus.infoq.cn/2020/shanghai/presentation/2646).
+:::info
+For a current, ground-up explainer of this idea, see [What is Incremental ETL on a Data Lake?](/blog/2026/07/16/what-is-incremental-etl-on-a-data-lake).
+:::
+
### NOTE: This article is a translation of the infoq.cn article, found [here](https://www.infoq.cn/article/CAgIDpfJBVcJHKJLSbhe), with minor edits
<!--truncate-->
diff --git a/website/blog/2026-07-14-what-is-an-open-table-format.md b/website/blog/2026-07-14-what-is-an-open-table-format.md
new file mode 100644
index 000000000000..601c6efdeb22
--- /dev/null
+++ b/website/blog/2026-07-14-what-is-an-open-table-format.md
@@ -0,0 +1,113 @@
+---
+title: "What is an Open Table Format?"
+excerpt: "What open table formats like Apache Iceberg, Apache Hudi and Delta Lake actually are — table metadata plus a transaction protocol that turns files on object storage into transactional tables."
+description: "An open table format is table metadata plus a transaction protocol that turns files on cloud storage into database tables with ACID commits, schema evolution and time travel — Apache Iceberg, Hudi, Delta Lake and Paimon compared."
+authors: [vinoth-chandar]
+category: how-to
+image: /assets/images/blog/dlh_1200.png
+tags:
+- table format
+- data lakehouse
+- apache iceberg
+- delta lake
+- open architecture
+- apache xtable
+---
+
+An open table format is an openly governed specification with two inseparable parts: **table metadata** — which files make up the table, what schema they follow, what statistics they carry, at every version — and a **transaction protocol** — the rules by which writers commit changes atomically and readers see consistent snapshots. Together they turn a collection of files on cloud object storage into a database-like table with ACID transactions, schema evolution, and time travel. Apache H [...]
+Because the specification is open, any query engine — Apache Spark, Apache Flink, Trino, and many others — can read and write the same table without going through a proprietary system.
+
+Open table formats emerged to fix a specific gap. Cloud object stores such as Amazon S3, Azure Blob Storage, and Google Cloud Storage made it cheap to store virtually unlimited data, and columnar file formats made it efficient to scan. But a pile of Parquet files is not a table: no atomic commits, no record-level updates or deletes, no history to query, and no protection for a reader scanning a directory mid-rewrite.
+
+Table formats close that gap by layering a transaction log and rich table metadata over the files, and in doing so became the foundational component of the [data lakehouse](/blog/2024/07/11/what-is-a-data-lakehouse) architecture — warehouse-grade reliability and management over open data on the lake.
+
+This post covers the problem, the anatomy of a table format, the major open projects, and how the pieces fit together.
+
+## The Problem Before Table Formats
+
+Until roughly a decade ago, the de-facto "table" abstraction on data lakes was the Apache Hive model: a table was a directory, partitions were subdirectories, and the Hive Metastore recorded the schema and the list of partitions. This worked well enough on HDFS for append-only batch workloads, but it had structural weaknesses that cloud object storage made acute.
+
+- **No atomicity.** A failed job left partial files behind, and readers could not distinguish committed data from garbage; jobs resorted to fragile rename tricks that object stores do not support atomically.
+- **No record-level mutations.** Updating or deleting a record meant rewriting entire partitions, making change data capture (CDC) from operational databases and GDPR deletions painful and expensive.
+- **Coarse, expensive planning.** Query planning relied on listing directories — slow and costly on object stores — and on partition values as the only pruning mechanism; statistics about the actual data in each file lived nowhere.
+- **No isolation or history.** A query could read an inconsistent mix of old and new files while a pipeline rewrote the table, and with no table versions there was no time travel, rollback, or incremental "what changed since yesterday" queries.
+
+The fix was to stop treating a table as a directory listing and treat it instead as versioned metadata plus a commit protocol: a transactionally maintained record of the files that make up the table. Apache Hudi pioneered this idea in production, built at Uber in 2016 for near-real-time ingestion with updates, deletes, and incremental processing on one of the world's largest data lakes, and described publicly in [Uber's engineering blog](https://www.uber.com/blog/hoodie/) in early 2017. [...]
+
+## What a Table Format Actually Consists Of
+
+Strip away the branding and every open table format is a specification for a handful of cooperating pieces of metadata stored alongside the data files.
+
+
+<p align = "center">Figure: The table format as a metadata layer above open data file formats</p>
+
+**A log of changes to the table.** The heart of a table format is an ordered record of every commit — data writes, deletes, schema changes, and maintenance operations. In Hudi this is the [timeline](/docs/timeline); Delta Lake calls it the transaction log; Iceberg maintains a chain of snapshots referenced from table metadata files. Whatever the name, this log is the source of truth: a change is part of the table if and only if it is committed to the log, and commit is an atomic operation [...]
+
+**Snapshots and versioning.** Every commit produces a new consistent view of the table, so readers can query it "as of" any retained version — enabling time travel for debugging and audits, rollback of bad writes, and, where the format tracks record-level changes, incremental queries that return only what changed between two points in time.
+
+**File-level statistics and indexes.** Table formats track per-file and per-column statistics — min/max values, null counts, record counts — that engines use to skip files that cannot contain matching rows, often eliminating most I/O for selective queries. Hudi generalizes this into a [multi-modal indexing subsystem](/docs/indexes) stored in an internal [metadata table](/docs/metadata), including column statistics, bloom filters, and record-level indexes that also accelerate the write pa [...]
+
+**Schema tracking and evolution.** The format stores the table's schema and defines rules for [evolving it](/docs/schema_evolution) — adding, renaming, or promoting column types — without rewriting existing data files, plus schema enforcement on write to catch pipeline breakages before bad data lands in the table.
+
+**Concurrency control.** Finally, the specification defines how multiple writers and background jobs coexist. All major formats give readers snapshot isolation — a query sees a consistent version of the table regardless of in-flight writes — while for concurrent writers they define mechanisms from optimistic concurrency control to non-blocking approaches for jobs touching disjoint or reconcilable data; Hudi's [concurrency control](/docs/concurrency_control) documentation covers these tra [...]
+
+How data is physically organized under this metadata varies too. Hudi, for example, defines two [table types](/docs/table_types): Copy-on-Write, which rewrites base files on update for maximum read performance, and Merge-on-Read, which writes changes to compact log files that are periodically merged — a trade-off between write latency and read cost that other formats have since adopted variants of.
+
+Note, though, that everything above is *specification* — passive metadata and protocol rules. A table format defines what a valid table looks like; it does not keep the table healthy. In database terms, that job belongs to a **storage engine**: the machinery that compacts small files, clusters data, cleans up old versions, and builds and maintains indexes. Every table on a lake needs this running somewhere — built into the project, supplied by the query engine, or bought from a vendor — [...]
+
+## The Major Open Table Formats
+
+Four open source projects dominate the space today. All are openly governed (three at the Apache Software Foundation, one at the Linux Foundation), and all deliver the core capabilities above; they differ in design center and scope.
+
+**[Apache Hudi](https://hudi.apache.org)** originated at Uber in 2016 for fast, mutable ingestion — upserts, deletes, and incremental processing — on the data lake. Hudi is accurately described as a table format plus a storage engine: beyond the storage specification (timeline, file layout, metadata table), it ships a writer runtime with pluggable indexing for fast updates, and built-in [table services](/docs/hudi_stack) — compaction, clustering, cleaning, and indexing — that run inline [...]
+
+**[Apache Iceberg](https://iceberg.apache.org)** originated at Netflix and was donated to the ASF in 2018. Iceberg's design center is an engine-agnostic specification for table metadata, with immutable snapshots, hidden partitioning (partition values derived from column transforms), and well-defined metadata evolution. It deliberately scopes itself to the format and libraries, leaving table maintenance to engines and vendors, and has attracted the broadest catalog and vendor ecosystem vi [...]
+
+**[Delta Lake](https://delta.io)** was created at Databricks and open-sourced in 2019; it is governed under the Linux Foundation. Delta centers on a JSON-based transaction log with periodic Parquet checkpoints, offering ACID transactions, schema enforcement, and time travel. Like Hudi, it has always been more than a format on its home platform, pairing the log with maintenance operations such as optimize and vacuum. It is the native format of Databricks and has first-class Spark integrat [...]
+
+**[Apache Paimon](https://paimon.apache.org)** is the newest of the four, grown out of the Flink Table Store subproject and a top-level Apache project since 2024. Paimon is narrower in scope than the other three: its design center is LSM-tree (log-structured merge tree) storage on the lake, optimizing write speed for high-frequency updates and changelog-producing tables consumed by Apache Flink, while remaining readable by batch engines like Spark and Trino. The trade-off is a classic on [...]
+
+An honest summary: the formats have converged substantially on core capabilities — ACID commits, schema evolution, time travel, statistics-based pruning — and continue to borrow ideas from one another. But write performance and read performance remain distinct problems with different trade-offs, even today: a design that makes writes cheap taxes readers unless compaction and read-side indexing keep pace, while one that optimizes purely for scans makes every update a rewrite. Because read [...]
+
+
+## How Query Engines Integrate
+
+A table format is only useful insofar as engines speak it. Integration happens at two levels.
+
+At the **data plane**, each format ships libraries or specifications that engines embed. Apache Spark and Apache Flink integrate with all four formats for both reading and writing; distributed SQL engines such as Trino and Presto ship native connectors. Cloud warehouses — Amazon Athena and Redshift, Google BigQuery, Snowflake — can query open table formats directly, with capabilities varying by format and vendor, and Python-native engines (DuckDB, Daft, Ray) extend the same reach to ligh [...]
+
+At the **control plane**, a catalog tracks which tables exist and points engines at each table's current metadata. Options range from the venerable Hive Metastore and AWS Glue Data Catalog to newer catalogs like Apache Polaris, Unity Catalog, and Apache Gravitino. The catalog is what turns `SELECT * FROM orders` into "resolve `orders` to a storage location and metadata pointer, then plan the scan." Hudi syncs its table metadata to these catalogs through catalog sync tools, keeping tables [...]
+
+## Interoperability: You Don't Have to Pick One Forever
+
+A common source of anxiety is that choosing a table format is a one-way door. Increasingly, it is not. [Apache XTable](https://xtable.apache.org) (incubating) is an open source project that translates table metadata between Hudi, Iceberg, and Delta Lake — in any direction — without copying or rewriting the underlying data files. Because all three formats ultimately describe Parquet files plus metadata, XTable can read one format's metadata and write out the equivalent for another, as a o [...]
+
+This is how major vendors ship interoperability today. XTable was co-launched by Microsoft, Google, and Onehouse before its donation to the ASF; [Microsoft OneLake](https://xtable.apache.org/docs/fabric/) in Fabric uses it under the hood to surface tables across Delta and Iceberg formats, and Onehouse uses it to keep customer tables continuously queryable as Hudi, Iceberg, and Delta at once. Databricks takes the same one-copy approach within Delta Lake through UniForm, which generates Ic [...]
+
+## Table Format vs File Format vs Lakehouse
+
+These three terms get conflated constantly, so a short disambiguation is worth the space.
+
+- A **file format** (Apache Parquet, ORC, Avro) defines how records are encoded *inside a single file* — columnar layout, compression, per-file statistics. It knows nothing about other files.
+- A **table format** (Hudi, Iceberg, Delta Lake, Paimon) defines how *many files plus metadata* form a versioned, transactional table. It sits directly above the file format.
+- A **data lakehouse** is the overall *architecture* that combines cloud object storage, open file formats, a table format, catalogs, and query engines into a warehouse-like platform on the lake. The table format is its load-bearing component, alongside storage engines/table services, catalogs, and compute.
+
+
+<p align = "center">Figure: Where the table format sits in a lakehouse architecture</p>
+
+So Parquet is not a table format, and a table format alone is not a lakehouse — it is the layer that makes one possible. For the architectural view, see the companion post on [what a data lakehouse is](/blog/2024/07/11/what-is-a-data-lakehouse).
+
+## Conclusion
+
+An open table format — table metadata plus a transaction protocol — turns cheap, scalable object storage into a real analytical database substrate: files become tables, jobs become transactions, and a directory of Parquet becomes something you can update, evolve, audit, and query as of any point in time, from any engine that implements the open specification. Apache Hudi proved the idea in production first while building the transactional data lake; Apache Iceberg, Delta Lake, and Apache [...]
+
+## FAQ
+
+<PostFAQ heading={null} items={[
+ {question: 'What is an open table format?', answer: 'An open table format is an openly governed specification combining table metadata (which files form the table, their schema and statistics, at every version) with a transaction protocol (how writers commit atomically and readers see consistent snapshots). Together they turn files on cloud object storage into a database-like table with ACID transactions, schema evolution, and time travel. Apache Hudi, Apache Iceberg, Delta Lake, and A [...]
+ {question: 'Is Parquet an open table format?', answer: 'No. Apache Parquet is an open file format: it defines how records are encoded inside a single file. A table format like Apache Hudi or Apache Iceberg sits above Parquet and defines how many files plus transactional metadata form a versioned table.'},
+ {question: 'What is the difference between a table format and a file format?', answer: 'A file format defines the byte layout of one file, including encoding, compression, and per-file statistics. A table format defines how a collection of such files behaves as a single transactional table, tracking committed files, schema, versions, and statistics across the whole table. They are complementary layers, not alternatives.'},
+ {question: 'Which open table format should I use?', answer: 'It depends on your workloads, and note that write and read performance involve different trade-offs. Hudi is strongest for mutable, incremental ingestion at high volume, with built-in indexing for fast writes and automated compaction and clustering to keep reads fast; Iceberg emphasizes an engine-neutral specification with broad catalog support; Delta Lake offers deep Spark and Databricks integration; Paimon targets LSM-based [...]
+ {question: 'Do open table formats support ACID transactions?', answer: 'Yes. All major open table formats provide atomic commits and snapshot isolation for readers, so queries always see a consistent version of the table. They differ in how they handle concurrent writers, using techniques such as optimistic concurrency control and non-blocking concurrency mechanisms.'},
+ {question: 'Is Apache Hudi just a table format?', answer: 'No. Hudi includes an open table format, but pairs it with a built-in storage engine: a writer runtime with pluggable indexing plus table services such as compaction, clustering, and cleaning that keep tables optimized automatically. Hudi was also the first open table format in production, built at Uber in 2016 for its transactional data lake.'},
+]} />
diff --git a/website/blog/2026-07-15-what-is-upsert-on-a-data-lake.md b/website/blog/2026-07-15-what-is-upsert-on-a-data-lake.md
new file mode 100644
index 000000000000..0c3851ddaf19
--- /dev/null
+++ b/website/blog/2026-07-15-what-is-upsert-on-a-data-lake.md
@@ -0,0 +1,116 @@
+---
+title: "What is Upsert on a Data Lake?"
+excerpt: "An upsert updates a record if it exists and inserts it if it does not. This post explains why that was historically hard on data lakes, and how open table formats like Apache Hudi make it fast."
+description: "What an upsert is, why record updates were historically hard on data lakes, and how table formats like Apache Hudi implement fast, indexed upserts."
+authors: [vinoth-chandar]
+category: how-to
+image: /assets/images/blog/2023-05-10-top-3-things-you-can-do-to-get-fast-upsert-performance-in-apache-hudi.png
+tags:
+- upsert
+- dml
+- indexing
+- data lakehouse
+- cow
+- mor
+---
+
+An upsert is a write operation that updates a record if it already exists and inserts it if it does not — a single atomic operation that combines *update* and *insert*, keyed on a unique record identifier. Databases have offered this for decades (`MERGE` in SQL, `ON CONFLICT DO UPDATE` in PostgreSQL, `replace into` in MySQL). On a data lake, however, upserts were effectively impossible for most of the technology's history, and making them fast is still what separates data lake storage sy [...]
+
+## Why Data Lakes Couldn't Update Records
+
+Data lakes are built on files — typically columnar formats like Apache Parquet or ORC — sitting on distributed file systems or cloud object storage such as Amazon S3. Two properties of this foundation make record-level updates hard.
+
+First, the files themselves are immutable. A Parquet file is written once, with column chunks, compression, and statistics computed over the whole file; there is no way to reach into it and change one row in place. Second, the storage layer underneath offers no transactional semantics across files. Early cloud object stores were even only eventually consistent, so a reader listing a directory mid-write could see a partial, corrupt view of the table. There were no primary keys, no notion [...]
+
+The Hive-era workaround was coarse-grained rewriting: partition the table (say, by day), and when any record in a partition changed, recompute and overwrite the *entire partition*. Pipelines re-read hours or days of data to apply what might be a handful of changed rows, then swapped directories and hoped no query was reading at that moment. This pattern wasted enormous compute, delayed data freshness from minutes to hours or days, and provided no isolation guarantees. It is precisely the [...]
+
+## How Open Table Formats Make Upserts Possible
+
+Open table formats — [Apache Hudi](https://hudi.apache.org), Apache Iceberg, and Delta Lake — introduced a transactional metadata layer on top of immutable files, turning a directory of Parquet files into a proper table. That metadata layer is what makes upserts possible, through a few key mechanisms (covered more broadly in [What is a Data Lakehouse?](/blog/2024/07/11/what-is-a-data-lakehouse)):
+
+- **Atomic commits.** Every write produces new file versions, and a commit protocol atomically publishes them. In Hudi, each write is an action on a [timeline](/docs/timeline); queries only ever see fully committed data, so a failed or in-flight write is invisible to readers.
+- **Snapshot isolation and versioning.** Because old file versions are retained until cleaned, readers see a consistent snapshot while writers produce the next one. Updating a record no longer risks corrupting a running query.
+- **Record keys.** Hudi brings the database notion of a primary key to the lake: every record has a key (record key, plus optionally a partition path) that uniquely identifies it. This is the precondition for "update if exists" to even be well defined.
+- **Fine-grained file management.** Instead of rewriting a whole partition, the system can identify exactly which files contain the affected records and rewrite — or log changes against — only those.
+
+With these pieces in place, an upsert on a data lake becomes what it is in a database: hand the system a batch of records, and it transactionally reconciles them against what is already stored. In Hudi, `upsert` is the [default write operation](/docs/write_operations#upsert): incoming records are tagged as inserts or updates via an index lookup, sized into files, written, and committed atomically. The target table never shows duplicates.
+
+## The Anatomy of an Upsert
+
+Understanding the cost of an upsert requires walking through what actually happens. Hudi's [write path](/docs/write_operations#write-path) breaks it into distinct steps.
+
+**1. Key definition and deduplication.** Each incoming record carries a record key. Since an input batch may itself contain multiple versions of the same key (common with change streams), records are first combined by key, keeping the winner according to configured ordering fields.
+
+**2. Index lookup.** The system must answer: *does this key already exist, and if so, in which file?* Hudi maintains a mapping from each record key to a *file group* — the unit of storage that holds all versions of a set of records. The [index lookup](/docs/indexes) tags each incoming record as an update (routed to its existing file group) or an insert (assigned to a new or under-sized file group). This step is the heart of the upsert, and — as the next section shows — the main determina [...]
+
+**3. Merge and write.** Tagged records are then written, and here the [table type](/docs/table_types) determines the strategy:
+
+- **Copy-on-Write (CoW)** rewrites the affected base files: the writer reads the current base file of each touched file group, merges in the updates, and writes a new versioned base file. Queries stay simple and fast — they read pure columnar data with zero merge overhead — but write amplification is high: updating even a few records in a file means rewriting the whole file. CoW suits read-heavy tables with moderate update rates.
+- **Merge-on-Read (MoR)** instead appends the changed records to compact delta log files attached to each file group, deferring the expensive base-file rewrite. Writes become lightweight and fast, enabling commits every few minutes; queries merge the logs with base files at read time, and a background [compaction](/docs/compaction) process periodically folds logs into new base files to restore pure-columnar read performance. MoR suits high-frequency updates and streaming ingestion, tradi [...]
+
+The trade-off is summarized well by the [comparison table](/docs/table_types#comparison) in the Hudi docs: CoW pays `O(file groups written)` in write amplification for zero read amplification; MoR pays `O(records changed)` on both sides.
+
+**4. Commit.** Finally, the write — new base files, new log blocks, and index updates — is committed atomically to the timeline, at which point queries can see it.
+
+## Why Indexing Is the Difference Between Minutes and Hours
+
+Step 2 above deserves its own section, because it is where naive upsert implementations fall down. Without an index, the only way to find which files contain the incoming keys is to *join the input batch against the entire table* — reading key columns from every file, every time you write. For a large table receiving a small trickle of updates, this means the cost of each write is proportional to the size of the table, not the size of the change.
+
+
+<p align = "center">Figure: Merge cost for updates (dark blue) against base files (light blue), with and without an index</p>
+
+Hudi treats [indexing](/docs/indexes) as an integral part of its storage engine and offers several index types tuned to different workloads:
+
+- **Bloom index**: stores bloom filters and key ranges (in file footers, or centrally in the metadata table) so most files can be pruned without reading data. Works very well when keys have some ordering — e.g., event tables with timestamp-prefixed keys — where range pruning eliminates most candidate files.
+- **Simple index**: a lean join against keys extracted from existing files; a better fit for random-update workloads like dimension tables, where bloom filters would return true for nearly every file anyway.
+- **Bucket index**: hashes keys directly to file groups, eliminating the lookup entirely at the cost of a fixed (or consistent-hashed) bucket layout.
+- **[Record Level Index](/blog/2023/11/01/record-level-index)**: a scalable, exact key-to-file-group mapping stored in Hudi's metadata table, sharded by hash. Instead of pruning candidate files probabilistically, it answers the location question with a point lookup, delivering order-of-magnitude speedups on large deployments where index lookup dominates write latency.
+
+The practical impact is dramatic. In a table with tens of thousands of files, an untagged upsert must consider all of them; a well-chosen index narrows the work to only the file groups that actually contain updated records. Merge-on-Read benefits doubly: the index bounds how many change records any base file must be merged against at query time, not just at write time. For concrete tuning guidance, see [Top 3 Things You Can Do to Get Fast Upsert Performance in Apache Hudi](/blog/2023/05/ [...]
+
+## What Upserts Unlock
+
+Fast record-level mutation is not a niche feature; it changes what a data lake can be used for.
+
+- **Change data capture (CDC) replication.** Streaming a database's change log (via Debezium, AWS DMS, or similar) into the lake requires applying inserts, updates, and deletes in order, continuously. Upserts make it possible to maintain a query-ready mirror of an operational database with minute-level freshness, instead of nightly full dumps.
+- **Compliance deletes (GDPR/CCPA).** "Delete this user's data" is a targeted mutation across potentially every partition of a table. With keyed [deletes](/docs/write_operations#delete) — implemented on the same indexed path as upserts — this becomes a routine operation rather than a table-rewriting project.
+- **Deduplication.** Event pipelines deliver duplicates; upserting by event key means the table enforces uniqueness on write, so consumers never need to deduplicate at query time.
+- **Mutable fact and dimension tables.** Late-arriving corrections, order status changes, transaction settlements, slowly changing dimensions — all become incremental updates against the affected records, rather than periodic partition rewrites.
+- **Incremental pipelines.** Because upserts record exactly what changed and when, downstream jobs can consume just the changes via [incremental queries](/docs/table_types#query-types), replacing full-table rescans with streaming-style processing.
+
+## How Hudi, Iceberg, and Delta Lake Handle Mutation
+
+All three major open table formats support upserts today, but with different designs and different cost profiles. Delta Lake and Iceberg expose mutation primarily through `MERGE INTO`: the engine joins the source batch against the target table to find affected files, then either rewrites them (copy-on-write) or, in newer versions, writes delete files/deletion vectors that mark rows as removed (merge-on-read style reads). Neither maintains a persistent record-level index, so locating affe [...]
+
+## Conclusion
+
+An upsert — update if the record exists, insert if it does not — is the operation that turns a data lake from an append-only archive into a live, continuously updated foundation for analytics. It was historically out of reach because lakes stored immutable files with no transactions, keys, or record-level metadata, forcing wasteful partition rewrites. Open table formats supplied the transactional layer; what determines real-world performance is everything layered on top: how records are [...]
+
+## FAQ
+
+<PostFAQ heading={null} items={[
+ {
+ question: 'What is the difference between upsert and merge?',
+ answer: 'They achieve similar outcomes through different interfaces. MERGE is a SQL statement where you write an explicit join condition and specify what happens on match or no match, giving full control including conditional logic. Upsert is a keyed write operation: the system uses the table\'s record key to decide automatically whether each incoming record is an update or an insert. In Apache Hudi, upsert is the default write operation and uses an index to locate existing records, [...]
+ },
+ {
+ question: 'Can you update a record in a Parquet file?',
+ answer: 'Not in place. Parquet files are immutable once written, so there is no way to modify a single row inside one. Table formats work around this by rewriting the affected file with the change applied, or by logging the change separately and merging it with the base file when the data is read. Either way, the update is expressed as new files plus a metadata commit, never as an in-place edit.',
+ },
+ {
+ question: 'Is upsert slow on a data lake?',
+ answer: 'It depends almost entirely on how the system finds existing records and how it applies changes. Without an index, every upsert must scan or join against the whole table to locate matching records, so write cost grows with table size. With an index such as Hudi\'s bloom index or record level index, the write only touches file groups that actually contain updated records. Choosing Merge-on-Read further reduces write latency by logging changes instead of rewriting columnar file [...]
+ },
+ {
+ question: 'What is the difference between upsert and insert in Apache Hudi?',
+ answer: 'Upsert performs an index lookup to tag each incoming record as an update or an insert, guaranteeing the table never shows duplicate keys. Insert skips the index lookup entirely, which makes it faster but allows duplicates unless the input is pre-deduplicated. Upsert is recommended for change data capture and other update-heavy workloads, while insert suits append-mostly data that can tolerate or separately handle duplicates.',
+ },
+ {
+ question: 'How do GDPR deletes work on a data lake?',
+ answer: 'A GDPR or CCPA delete request means removing specific records, identified by key, from potentially every partition of a table. Table formats implement this on the same path as upserts: an index lookup finds the files containing the keys, and the delete is applied either by rewriting those files or by logging delete markers that compaction later reconciles. Hudi supports both soft deletes, which null out non-key fields, and hard deletes, which remove the records entirely.',
+ },
+ {
+ question: 'Do Apache Iceberg and Delta Lake support upserts?',
+ answer: 'Yes. Both support row-level mutation, primarily through the SQL MERGE INTO statement, with copy-on-write file rewrites or merge-on-read style delete files and deletion vectors. The main design difference from Apache Hudi is that they do not maintain a persistent record-level index, so finding affected files relies on joining the source data against the table with statistics-based pruning. Hudi instead defines record keys as a table-level concept and maintains indexes that ma [...]
+ },
+]} />
diff --git a/website/blog/2026-07-16-what-is-incremental-etl-on-a-data-lake.md b/website/blog/2026-07-16-what-is-incremental-etl-on-a-data-lake.md
new file mode 100644
index 000000000000..c3811c6e6923
--- /dev/null
+++ b/website/blog/2026-07-16-what-is-incremental-etl-on-a-data-lake.md
@@ -0,0 +1,156 @@
+---
+title: "What is Incremental ETL on a Data Lake?"
+excerpt: "Incremental ETL is a pipeline pattern that processes only the data that changed since the last run. This post explains how it works, what the storage layer must provide, and how it compares to batch and stream processing."
+description: "Incremental ETL processes only changed data instead of recomputing entire tables. Learn how it works on a data lake and how Apache Hudi enables it."
+authors: [vinoth-chandar]
+category: how-to
+image: /assets/images/blog/incr-processing/image7.png
+tags:
+- incremental processing
+- etl
+- streaming
+- data lakehouse
+- cdc
+---
+
+Incremental ETL is a pipeline pattern that processes only the data that changed since the last run, instead of recomputing entire tables or partitions on every execution. Each run of an incremental pipeline consumes a delta — the records inserted, updated, or deleted after a known checkpoint — applies its transformation logic to just that delta, and merges the results into the target table. The pipeline's work is proportional to how much the data changed, not how large the tables have grown.
+
+That distinction matters because most data pipelines today still run as full-recompute batch jobs: every night (or every hour), they re-scan source tables end to end and rewrite the derived tables downstream, even when only a tiny fraction of the rows actually changed. As data volumes grow, that recomputation gets slower and more expensive at exactly the moment businesses are asking for fresher data.
+
+Incremental ETL occupies the middle ground between classic batch processing and full stream processing. It keeps the familiar economics and tooling of batch — scheduled jobs, SQL or DataFrame transformations, cheap [data lakehouse](/blog/2024/07/11/what-is-a-data-lakehouse) storage — while borrowing the core idea of stream processing: treat data as a continuous flow of changes, and process each change roughly once.
+
+## The Problem with Full Recomputation
+
+Consider a typical medallion-style warehouse or lakehouse: raw data lands in bronze tables, gets cleaned into silver tables, and is aggregated into gold tables for reporting. In a full-recompute design, each layer is rebuilt by scanning its inputs wholesale. Three problems compound as the tables grow:
+
+- **Cost.** A daily job that scans a multi-terabyte fact table to pick up one day's worth of changes does orders of magnitude more I/O and compute than the size of the change justifies. Joins are the worst offenders: rewriting a large joined table means re-shuffling both sides in full, every run. Organizations routinely spend a large share of their data platform budget on pipelines that mostly reprocess data they already processed yesterday.
+
+- **Latency.** If rebuilding a table takes four hours, the data in it can never be fresher than four hours plus the scheduling interval. Teams respond by running the job less often (making data staler) or throwing more compute at it (making it more expensive). Neither addresses the root cause: the work grows with table size rather than change size.
+
+- **Missed SLAs and cascading delays.** Derived pipelines form DAGs. When an upstream full-recompute job runs long, every downstream job either starts late or reads stale inputs. Late-arriving data makes this worse: when records for an old partition show up, the whole partition — and every downstream partition derived from it — must be recomputed, silently blowing through freshness SLAs.
+
+Stream processing solves the latency problem but replaces batch's cost profile with a different one: always-on clusters, state stores to manage, and a second storage system (typically a log store like Kafka) holding another copy of the data. For the large class of pipelines where "fresh within a few minutes" is sufficient, that operational overhead buys latency headroom no one uses.
+
+## What Makes ETL "Incremental"
+
+An ETL pipeline is incremental when both its input and its output are expressed as changes:
+
+1. **Change streams in.** Instead of scanning the full source table, the pipeline asks the storage layer a question a plain file listing cannot answer: *"give me all records that changed since commit X."* The result is a delta — new inserts, the latest values of updated records, and deletes — along with a new checkpoint to resume from next run.
+
+2. **Incremental writes out.** Instead of rewriting the target table, the pipeline applies its transformed delta using record-level operations — upserts and deletes — so only the affected file groups are rewritten. Crucially, this makes the *output* table itself an incremental source: the next pipeline in the DAG can consume its changes the same way.
+
+Because deltas flow in and deltas flow out, the pattern composes. A chain of incremental pipelines behaves like a topology of stream processors — each stage consuming a change stream and emitting one — except the "streams" are ordinary tables on cheap cloud storage, queryable by any engine at any time. This is the [duality between change logs and tables](/blog/2020/08/18/hudi-incremental-processing-on-data-lakes) that stream processing literature has long described, realized directly on [...]
+
+The efficiency win is easy to see with numbers. If a 10 TB fact table receives 50 GB of changes per hour, an hourly incremental pipeline processes 50 GB per run instead of 10 TB — a 200x reduction in data scanned, while *improving* freshness from daily to hourly.
+
+## What the Storage Layer Must Provide
+
+Incremental ETL is not primarily a compute-engine feature; it is a storage contract. Plain files in cloud storage — even columnar formats like Parquet — cannot tell a consumer what changed, and cannot accept record-level updates without rewriting whole files or partitions. Open table formats close this gap to varying degrees. Concretely, the storage layer must provide three primitives:
+
+- **Record-level change retrieval.** The table must be able to answer "what changed between time T1 and T2" efficiently, without scanning everything. This comes in two flavors: *incremental queries*, which return the latest value of each changed record (ideal for driving downstream upserts), and *CDC queries*, which return before/after images of every change (needed when the transformation must see each individual mutation). Apache Hudi supports [both query types](/docs/sql_queries#incre [...]
+
+- **Upserts and deletes to apply changes.** Producing a delta is only useful if the next table can absorb it cheaply. That requires primary keys and indexes to locate which files contain the affected records, so a small change touches a small amount of storage. Hudi's [write operations](/docs/write_operations) are built around indexed upserts and deletes, with merge modes to handle out-of-order and late-arriving data correctly.
+
+- **Ordering and commit points.** Consumers need a reliable notion of "where I left off." Hudi's [timeline](/docs/timeline) records every action on the table as an atomically committed instant; incremental queries use commit completion times to define exact, repeatable change windows, and checkpoints are managed automatically by tools like Hudi Streamer's incremental sources. Hudi additionally tracks record-level metadata (commit time and sequence number on every record), so change histo [...]
+
+These primitives — upsert and incremental consumption — are exactly the pair that defines the pattern: with them, you can maintain one table from a change stream, then derive another table from *its* change stream, end to end. The [use cases documentation](/docs/use_cases) covers how this plays out across ingestion, warehouse offload, and derived pipelines.
+
+### Frequent Table Maintenance Becomes the Norm
+
+There is a fourth requirement that follows directly from the first three: the format must be designed for frequent table maintenance, because incremental processing fundamentally increases how often the table is written. A table that once absorbed one nightly batch now commits every few minutes — producing small files, accumulating deltas in log files, and continuously churning the storage layout. At that write frequency, maintenance work — [compaction](/docs/compaction), clustering, cle [...]
+
+And it must run *without blocking the writers it serves*. If compacting or clustering a table means pausing ingestion, or if maintenance jobs contend with writers as just another optimistic transaction — conflicting and forcing retries on exactly the hot file groups that change most — then every maintenance pass reintroduces the latency and cost the pipeline went incremental to eliminate. Hudi builds table services into the storage engine for this reason: compaction, clustering, cleaning [...]
+
+## Incremental ETL vs. Stream Processing
+
+Incremental ETL and stream processing share semantics but differ sharply in infrastructure and economics:
+
+| | Stream processing (Flink, Kafka Streams) | Incremental ETL (Hudi on a data lake) |
+|---|---|---|
+| Latency | Sub-second to seconds | Minutes |
+| Compute model | Always-on jobs with managed state | Scheduled or continuous mini-batch jobs; state lives in tables |
+| Storage | Event log (Kafka) + state stores + eventual sink | One copy: lakehouse tables on cloud object storage |
+| Reprocessing/backfill | Replay the log, rebuild state | Re-run the job over a commit range or full table |
+| Query access to intermediate results | Requires sinking to another system | Every intermediate table is directly queryable |
+
+A useful way to summarize the trade: **incremental ETL delivers streaming semantics at minute-level batch economics.** If a use case genuinely needs sub-second reactions — fraud blocking, operational alerting — a stream processor is the right tool. But a large majority of analytics pipelines have freshness requirements in the minutes-to-hour range, where paying for always-on stream infrastructure and a second copy of the data buys nothing. The two also compose rather than compete: Flink [...]
+
+## A Concrete Pipeline Walkthrough
+
+Here is the pattern end to end, using a familiar CDC scenario: an operational database (say, PostgreSQL) whose changes must feed an analytics table.
+
+**Step 1 — CDC into a bronze table.** A capture tool such as Debezium streams the database's change log into Kafka, and an ingestion job — for example [Hudi Streamer](/docs/hoodie_streaming_ingestion) — continuously upserts those changes into a bronze Hudi table. The bronze table is now a queryable, transactional mirror of the source, and every commit on its timeline is a consumable change point.
+
+**Step 2 — incrementally read the bronze table.** The silver pipeline wakes up on its schedule (or runs continuously) and pulls only what changed since its last checkpoint:
+
+```scala
+// Read only the records that changed since the last run
+val ordersDelta = spark.read.format("hudi").
+ option("hoodie.datasource.query.type", "incremental").
+ option("hoodie.datasource.read.begin.instanttime", lastCheckpoint).
+ load(bronzeOrdersPath)
+```
+
+The same read is available in SQL through the `hudi_table_changes` table-valued function:
+
+```sql
+SELECT * FROM hudi_table_changes('bronze_orders', 'latest_state', '20260716083000000');
+```
+
+**Step 3 — transform and upsert into a silver table.** The transformation runs over just the delta, and the result is applied as an upsert keyed on the record key:
+
+```scala
+val silverDelta = ordersDelta.
+ filter(col("order_status") =!= "CANCELLED").
+ withColumn("order_total", col("quantity") * col("unit_price"))
+
+silverDelta.write.format("hudi").
+ option("hoodie.datasource.write.operation", "upsert").
+ option("hoodie.table.name", "silver_orders").
+ mode("append").
+ save(silverOrdersPath)
+```
+
+The job then persists the new checkpoint (the latest commit completion time it consumed) for the next run. Because the silver table was written with upserts, it exposes its own change stream — a gold-layer aggregation job can consume `silver_orders` incrementally in exactly the same way, re-aggregating only the keys whose inputs changed.
+
+
+<p align = "center">Figure: Changes are upserted into an upstream table and incrementally consumed to maintain a derived table</p>
+
+One honest caveat: not every transformation is trivially incremental. Simple projections, filters, and enrichments map one-to-one onto deltas. Aggregations require merging new partial results into existing ones, and some full-table operations (percentiles, complex window functions) may still warrant periodic full recomputation. A pragmatic design runs the same logic incrementally for freshness and occasionally in full for correction — on the same tables and infrastructure.
+
+## A Brief History: Pioneered at Uber
+
+Incremental processing on the data lake is not a recent idea bolted onto table formats — it is the problem Hudi was created to solve. In 2016, Uber's data team faced exactly the trade-off described above: petabyte-scale Hadoop tables that could only be rewritten wholesale, and business use cases (like driver-partner incentives and trip pricing) that needed data in minutes, not hours. The case for a third processing style — incremental, on top of batch storage — was laid out in the O'Reil [...]
+
+## Conclusion
+
+Incremental ETL replaces "recompute everything, always" with "process what changed, continuously." It requires a storage layer that can serve record-level change streams, absorb upserts and deletes efficiently, and expose reliable commit points to checkpoint against — the primitives Apache Hudi has provided on data lakes since its inception. The payoff is pipelines whose cost tracks the rate of change in the business rather than the accumulated size of history, with data freshness measur [...]
+
+## FAQ
+
+<PostFAQ heading={null} items={[
+ {
+ question: 'What is incremental ETL?',
+ answer: 'Incremental ETL is a data pipeline pattern that processes only the records that were inserted, updated, or deleted since the pipeline\'s last run, instead of re-scanning and rewriting entire tables. Each run consumes a delta from a checkpoint, transforms it, and merges the result into the target table with upserts and deletes. This makes pipeline cost proportional to the volume of change rather than total table size.'
+ },
+ {
+ question: 'What is the difference between incremental ETL and streaming?',
+ answer: 'Both process data as a flow of changes, but stream processors like Apache Flink run always-on jobs with managed state and typically require a separate event log such as Kafka, delivering sub-second latency. Incremental ETL runs as scheduled or continuous mini-batch jobs directly against lakehouse tables on cloud storage, delivering minute-level freshness at much lower infrastructure cost. A common summary is that incremental ETL provides streaming semantics at batch economics.'
+ },
+ {
+ question: 'How does incremental ETL reduce cost?',
+ answer: 'It eliminates repeated scans and rewrites of data that did not change. If a large table receives a small volume of changes between runs, an incremental pipeline reads and processes only that delta instead of the whole table, often reducing data scanned per run by orders of magnitude. Compute clusters also only run for the duration of each small job rather than staying on permanently.'
+ },
+ {
+ question: 'What does the storage layer need to support incremental ETL?',
+ answer: 'Four things: the ability to efficiently retrieve record-level changes between two points in time, support for upserts and deletes so downstream tables can absorb deltas cheaply, an ordered log of commits that consumers can checkpoint against, and table maintenance that runs without blocking writers — because incremental processing multiplies write frequency, compaction, clustering, and cleaning become continuous background work rather than occasional jobs. Plain Parquet file [...]
+ },
+ {
+ question: 'Can every transformation be made incremental?',
+ answer: 'No. Projections, filters, enrichments, and key-based joins map naturally onto change streams, and aggregations can be maintained by merging partial results for the affected keys. Some computations, like exact percentiles or complex full-table window functions, are hard to update incrementally, so teams often run those periodically in full while keeping the rest of the pipeline incremental.'
+ },
+ {
+ question: 'Which Apache Hudi features enable incremental ETL?',
+ answer: 'Hudi provides incremental queries that return the latest values of changed records after a given commit, CDC queries that return before and after images of each change, and indexed upsert and delete write operations to apply deltas downstream. Its timeline records every commit with completion times that define exact change windows, and record-level metadata preserves change history across compaction and clustering. Ingestion tools like Hudi Streamer manage incremental checkp [...]
+ }
+]} />
diff --git a/website/blog/2026-07-17-what-is-acid-on-a-data-lake.md b/website/blog/2026-07-17-what-is-acid-on-a-data-lake.md
new file mode 100644
index 000000000000..237c64225182
--- /dev/null
+++ b/website/blog/2026-07-17-what-is-acid-on-a-data-lake.md
@@ -0,0 +1,128 @@
+---
+title: "What is ACID on a Data Lake?"
+excerpt: "What atomicity, consistency, isolation and durability actually mean for tables on object storage, and how lakehouse table formats like Apache Hudi deliver them."
+description: "What ACID means for tables on S3, GCS or ADLS: how table formats like Apache Hudi deliver atomic commits, snapshot isolation and safe concurrent writes."
+authors: [sivabalan]
+category: how-to
+image: /assets/images/blog/acid.png
+tags:
+- acid
+- concurrency control
+- hudi timeline
+- data lakehouse
+---
+
+ACID on a data lake means that writes to tables stored on object storage are Atomic, Consistent, Isolated and Durable — the same transactional guarantees a database provides, applied to files in S3, GCS or ADLS. That sentence is easy to say and surprisingly hard to deliver. A database owns its storage engine end to end; a data lake is a pile of Parquet files on an object store that offers durable single-object PUTs and nothing more. There is no multi-file atomic operation, no rename that [...]
+
+This post walks through what each of the four letters means concretely at the file level, how lakes worked before table formats existed, how formats like [Apache Hudi](https://hudi.apache.org) deliver these guarantees, and — just as important — what ACID on a lake does *not* promise.
+
+## What A, C, I and D Mean on Object Storage
+
+The textbook definitions of ACID were written for databases processing many small transactions. On a data lake, the transactions are large batch or streaming writes that produce files, so it is worth restating each property in file terms.
+
+**Atomicity** means a write either publishes completely or not at all. Consider a Spark job writing 500 Parquet files into a table and dying at file 320 — an executor is lost, a spot instance is reclaimed, someone kills the job. Atomicity requires that those 320 orphaned files are never visible to any query, and that the failed write can be cleanly rolled back. Without it, every downstream consumer inherits a partially written dataset and no reliable way to detect it.
+
+**Consistency** means every write moves the table from one valid state to another valid state. On a lake this covers schema enforcement (a write with an incompatible schema is rejected rather than quietly producing unreadable files), key constraints like upsert semantics (a record with the same key updates in place rather than duplicating), and the invariant that the table's metadata always describes exactly the set of files that constitute the table.
+
+**Isolation** means concurrent readers and writers do not observe each other's intermediate states. Two failure modes matter here. First, two writers targeting the same partition at the same time — say, a backfill job and a streaming ingest job — must not interleave their files such that the table ends up with a mix of both writes' partial output. Second, a reader that starts listing files mid-write must not see half of a commit; it should see the table exactly as it was at some committe [...]
+
+**Durability** means a committed write survives failures. Object stores already provide excellent durability for individual objects — S3, GCS and ADLS replicate objects across zones. The lake-specific part of durability is that the *commit record itself* is persisted to the same durable storage, so that "was this write committed?" has a crash-proof answer, and a committed transaction is never silently rolled back by a subsequent cleanup process.
+
+## Life Before ACID on Data Lakes
+
+The Hadoop and early cloud-lake era answered these problems with conventions, not guarantees. It is worth remembering how fragile those conventions were, because they explain why table formats exist.
+
+The classic Hive pattern was the **partition swap**: write data into a staging directory, then "atomically" move it into place and update the Hive metastore partition location. On HDFS a directory rename was genuinely atomic, so this mostly worked. On object stores there are no directories and no atomic rename — a "rename" is a copy-then-delete of every object — so the same pattern became a slow, non-atomic window during which readers could see anything.
+
+The **`_SUCCESS` file** convention had MapReduce and Spark drop an empty marker file when a job finished, and downstream jobs would poll for it. This signaled completion but protected nothing: readers that listed the directory directly saw partial data, a rerun of a failed job could mix old and new files, and there was no rollback story at all — cleanup after a failed job meant a human with a delete command deciding which files were orphans.
+
+Layered on top of this, early S3 offered only **eventual consistency** for listings, so even a fully written directory might not be fully *visible* yet. Whole projects (S3Guard, committers with DynamoDB bookkeeping) existed just to paper over that. S3 became strongly consistent in 2020, which removed one class of bugs — but strong single-object consistency still does not give you multi-file atomicity, isolation between writers, or rollback. Those require a transaction log.
+
+## How Table Formats Deliver ACID
+
+The shared insight behind Apache Hudi, Apache Iceberg and Delta Lake is that the table's state should be defined by a **metadata log with an atomic publish step**, not by whatever files happen to be in a directory. Data files are written first, invisibly; then a single, small metadata operation makes them visible all at once. Because that final operation is one atomic object-store action (a single PUT, or a conditional/compare-and-swap write), the entire multi-file write inherits its atomicity.
+
+In Hudi, that log is the [timeline](/docs/timeline) — an ordered event log of every action performed on the table, stored under the table's `.hoodie/timeline` directory. Each write is an action that moves through explicit states: `REQUESTED` (the write is planned), `INFLIGHT` (data files are being produced), and `COMPLETED` (the write is published). The transition to `COMPLETED` is a single atomic object-store operation, and Hudi timestamps every action with monotonically increasing, [Tr [...]
+
+
+<p align = "center">Figure: Writes and table services recorded as actions on the Hudi timeline</p>
+
+This structure delivers each letter directly:
+
+- **Atomicity**: our 500-file Spark job that dies at file 320 leaves an `INFLIGHT` action on the timeline and 320 uncommitted files that no query will ever read, because queries resolve visible files through completed timeline actions — never by listing directories. A rollback action later removes the orphaned files. Nothing partial is ever served.
+- **Consistency**: the timeline is the single source of truth for table state, and schema and key constraints are enforced as part of the write before the commit is published.
+- **Isolation**: readers get **snapshot isolation**. A query binds to the set of completed commits as of its start and sees exactly that state for its entire duration, no matter how many writes complete concurrently. Multi-version concurrency control (MVCC) keeps prior file versions around so long-running queries are never yanked out from under.
+- **Durability**: the commit metadata lives on the same replicated object storage as the data, so a completed instant is as durable as the data it describes.
+
+For a deeper walkthrough of how the timeline underpins each transaction state transition, see the earlier post [Hoodie Timeline: Foundational pillar for ACID transactions](/blog/2023/07/09/Hoodie-Timeline-Foundational-pillar-for-ACID-transactions).
+
+To be fair to the broader ecosystem: Apache Iceberg and Delta Lake achieve ACID through the same fundamental idea with different mechanics. Iceberg builds immutable snapshot trees of manifest files and commits by atomically swapping a pointer to the new table metadata (typically via a catalog's compare-and-swap). Delta Lake maintains a JSON transaction log (`_delta_log`) where writing log entry *N+1* is the atomic commit point, using conditional writes on the object store. All three give [...]
+
+## Concurrency Control Models
+
+Atomic commits solve the single-writer case. The harder question — and where table formats differ most — is what happens when multiple processes write to the same table. Hudi's [concurrency control](/docs/concurrency_control) documentation lays out a spectrum of models:
+
+**Single writer with table services.** Most pipelines have one writer, plus background table services (compaction, clustering, cleaning) that rewrite data for performance. Hudi distinguishes these by design: table services run lock-free alongside the writer, coordinated through the timeline via MVCC, either inline or fully async in the same process. For this very common case, no external lock infrastructure is needed at all.
+
+**Optimistic concurrency control (OCC).** When genuinely separate jobs write to one table — a streaming ingest plus a batch backfill, say — Hudi supports OCC at file-group granularity. Each writer proceeds without blocking, and at commit time takes a short-lived distributed lock to check whether a concurrently completed commit touched the same files. Non-overlapping writes both succeed; overlapping writes cause one to abort and retry. The lock is held only around the commit-time critical [...]
+
+
+<p align = "center">Figure: Optimistic concurrency control — conflicts detected at commit time</p>
+
+OCC works well when conflicts are rare, but it has an honest weakness for streaming: two high-frequency writers that routinely touch the same file groups will repeatedly abort each other, wasting compute and starving progress.
+
+**Non-blocking concurrency control (NBCC).** For multi-writer streaming, Hudi supports a mode where multiple writers can write into the *same* file groups simultaneously without aborting each other. Instead of preventing the conflict, resolution is deferred: commits are ordered by completion time on the timeline, and the query reader and the compactor merge overlapping writes deterministically. A lock is needed only for the instant-metadata write to the timeline itself. NBCC currently ta [...]
+
+## What ACID on a Data Lake Does Not Mean
+
+Honesty matters here, because "ACID" invites comparisons to OLTP databases that lakes are not designed to win.
+
+- **It is not OLTP.** Commit latency on a lakehouse is measured in seconds — an object-store round trip plus metadata work — not the milliseconds of a transactional database. Lakehouse transactions are designed for batches and micro-batches of records, not single-row point writes at high QPS.
+- **Transactions span one table.** The unit of atomicity in Hudi (and its peers, by and large) is a single table's commit. Multi-table transactions, cross-table foreign keys, and interactive multi-statement transactions with `BEGIN`/`COMMIT` semantics are not part of the model.
+- **Isolation is snapshot isolation, not serializability of arbitrary read-write transactions.** Writers are serialized against each other per table, and readers see consistent snapshots — which is exactly what analytical workloads need — but you should not port an OLTP application's locking assumptions onto a lake.
+- **ACID does not fix bad pipelines.** Atomic commits guarantee that what was written is either fully visible or invisible; they do not validate that the data itself was correct.
+
+If your workload is high-QPS point reads and writes, use an operational database and land the changes onto the lake via CDC — which, conveniently, is a workload where lake ACID shines.
+
+## Why It Matters
+
+These guarantees are not academic; they are what make several load-bearing patterns possible at all.
+
+**CDC correctness.** Replicating a database table into the lake via change data capture means a continuous stream of upserts and deletes. Without atomic commits and key-based upserts, a mid-batch failure leaves the lake copy in a state the source database never had — and there is no way to reconcile. With ACID, every commit is an all-or-nothing application of a batch of changes, and incremental consumers downstream see change batches in a well-defined order, never partially.
+
+**Concurrent ingestion and table services.** Real tables need continuous compaction, clustering and cleaning to stay queryable. Running those alongside ingestion without transactions means either stopping the world or risking corruption. MVCC on the timeline lets Hudi rewrite data in the background while writers write and readers read, with each process pinned to a consistent snapshot.
+
+**Regulatory deletes.** GDPR and CCPA erasure requests require deleting specific records from immutable files — physically, a read-modify-rewrite of affected files. ACID makes each deletion pass atomic and auditable: the timeline records exactly when the delete committed, queries never see a half-deleted state, and the commit history serves as evidence of compliance.
+
+## Conclusion
+
+ACID on a data lake is a commit protocol over object storage: data files are written invisibly, then published through one atomic metadata operation, with snapshot isolation for readers and explicit concurrency control between writers. Hudi implements this with its [timeline](/docs/timeline) — an ordered, durable event log of every action on the table — plus a spectrum of [concurrency control](/docs/concurrency_control) modes ranging from lock-free single-writer deployments to OCC and no [...]
+
+## FAQ
+
+<PostFAQ heading={null} items={[
+ {
+ question: 'Does S3 support ACID transactions?',
+ answer: 'Not by itself. S3 provides durable, strongly consistent operations on individual objects, but no atomic operation spanning multiple objects and no isolation between concurrent writers. ACID transactions on S3 come from a table format layered on top, such as Apache Hudi, Apache Iceberg or Delta Lake, which publishes multi-file writes through a single atomic metadata commit.'
+ },
+ {
+ question: 'Is a data lake ACID compliant?',
+ answer: 'A raw data lake of Parquet or ORC files is not ACID compliant: readers can observe partial writes, concurrent jobs can corrupt each other, and failed jobs leave orphaned files. A data lake becomes ACID compliant when tables are managed by a transactional table format, at which point it is usually called a lakehouse.'
+ },
+ {
+ question: 'How does Hudi guarantee atomicity?',
+ answer: 'Hudi records every write as an action on its timeline, moving through requested, inflight and completed states. Data files are written first but remain invisible; the transition to completed is a single atomic object-store operation. Queries only read files referenced by completed actions, so a failed write is never visible and is later rolled back cleanly.'
+ },
+ {
+ question: 'What is snapshot isolation on a data lake?',
+ answer: 'Snapshot isolation means every query reads the table as of a specific committed state and sees exactly that state for its entire run, even while new writes commit concurrently. Table formats implement it with multi-version concurrency control, keeping prior file versions available until no reader needs them.'
+ },
+ {
+ question: 'Can multiple jobs write to the same lakehouse table at once?',
+ answer: 'Yes, with concurrency control. Hudi supports optimistic concurrency control, where writers proceed in parallel and conflicts on overlapping files are detected at commit time using a short-lived lock, and non-blocking concurrency control, where multiple writers can even target the same file group and conflicts are resolved at read and compaction time.'
+ },
+ {
+ question: 'Is ACID on a data lake the same as ACID in a database like PostgreSQL?',
+ answer: 'The guarantees are the same in kind but different in scope and latency. Lakehouse transactions cover batches of records within one table and commit in seconds, whereas an OLTP database offers millisecond, multi-statement, often multi-table transactions. Lakes are built for analytical and streaming workloads, not high-QPS point reads and writes.'
+ }
+]} />
diff --git a/website/static/llms.txt b/website/static/llms.txt
index 138ed0fcc51f..72ac2077ebd7 100644
--- a/website/static/llms.txt
+++ b/website/static/llms.txt
@@ -39,6 +39,10 @@ It can also synchronize your data to half dozen data catalogs to keep table cons
- [Apache Hudi: 21 Unique Differentiators](https://hudi.apache.org/blog/2025/03/05/hudi-21-unique-differentiators): The technical crux of what sets Hudi apart — indexing, non-blocking concurrency control, async compaction, read-side indexes and built-in table services.
- [Apache Hudi 1.2: The Open Lakehouse for AI and Multimodal Data](https://hudi.apache.org/blog/2026/06/07/apache-hudi-release-1-2-announcement): First-class support for vectors, blobs/unstructured data, and semi-structured data on the lakehouse — for Spark pipelines doing unstructured data processing or generating embeddings.
- [What is a Data Lakehouse & How does it Work?](https://hudi.apache.org/blog/2024/07/11/what-is-a-data-lakehouse): The lakehouse architecture explained — evolution, components, use cases and key technologies.
+- [What is an Open Table Format?](https://hudi.apache.org/blog/2026/07/14/what-is-an-open-table-format): How open table formats like Hudi, Iceberg and Delta Lake bring database semantics to files on object storage.
+- [What is Upsert on a Data Lake?](https://hudi.apache.org/blog/2026/07/15/what-is-upsert-on-a-data-lake): Record-level updates and deletes on immutable cloud storage, and how Hudi makes them fast.
+- [What is Incremental ETL on a Data Lake?](https://hudi.apache.org/blog/2026/07/16/what-is-incremental-etl-on-a-data-lake): Processing only the data that changed instead of recomputing whole tables.
+- [What is ACID on a Data Lake?](https://hudi.apache.org/blog/2026/07/17/what-is-acid-on-a-data-lake): What atomicity, consistency, isolation and durability mean when your database is a bucket of files.
## Blog — Migration guides