You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by mo...@apache.org on 2026/05/12 06:45:00 UTC
(doris-website) branch master updated: [refactor](next) opt key features (#3634)
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris-website.git
The following commit(s) were added to refs/heads/master by this push:
new 9c3de314568 [refactor](next) opt key features (#3634)
9c3de314568 is described below
commit 9c3de31456824eb9790170529c02df69e6f4c642
Author: Mingyu Chen (Rayner) <mo...@163.com>
AuthorDate: Mon May 11 23:44:54 2026 -0700
[refactor](next) opt key features (#3634)
---
docs-next/key-features/analytic-functions.mdx | 33 ++---
docs-next/key-features/batch-load.mdx | 18 +--
docs-next/key-features/binlog-table-stream.mdx | 22 ++--
docs-next/key-features/bm25.mdx | 38 +++---
docs-next/key-features/catalog-integrations.mdx | 31 +++--
docs-next/key-features/columnar-storage.mdx | 20 ++--
docs-next/key-features/compute-group.mdx | 19 +--
docs-next/key-features/condition-cache.mdx | 23 ++--
docs-next/key-features/data-cache-page-cache.mdx | 21 ++--
docs-next/key-features/data-compaction.mdx | 21 ++--
docs-next/key-features/data-lineage.mdx | 19 +--
docs-next/key-features/data-model.mdx | 23 ++--
docs-next/key-features/data-pruning.mdx | 21 ++--
docs-next/key-features/data-update-delete.mdx | 31 +++--
docs-next/key-features/embedding.mdx | 32 ++---
docs-next/key-features/full-text-search.mdx | 22 ++--
docs-next/key-features/group-commit.mdx | 19 +--
.../key-features/high-concurrency-point-query.mdx | 32 +++--
docs-next/key-features/hybrid-search.mdx | 26 ++--
docs-next/key-features/iceberg.mdx | 129 ++++++++++++++++++++
.../key-features/incremental-materialized-view.mdx | 31 +++--
docs-next/key-features/inverted-index.mdx | 22 ++--
docs-next/key-features/kafka-cdc-integration.mdx | 20 ++--
docs-next/key-features/llm-sql-functions.mdx | 18 +--
docs-next/key-features/load-transaction.mdx | 21 ++--
docs-next/key-features/managing-lake-table.mdx | 39 +++---
docs-next/key-features/mcp-server.mdx | 28 +++--
docs-next/key-features/metadata-cache.mdx | 23 ++--
docs-next/key-features/mpp.mdx | 120 +++++++++++++++++++
docs-next/key-features/multi-catalog.mdx | 39 +++---
.../key-features/parquet-reader-optimization.mdx | 21 ++--
.../key-features/partitioning-and-bucketing.mdx | 20 ++--
.../key-features/pipeline-execution-engine.mdx | 21 ++--
docs-next/key-features/pluggable-auth.mdx | 20 ++--
.../key-features/preaggregation-and-rollup.mdx | 24 ++--
docs-next/key-features/prepared-statement.mdx | 23 ++--
docs-next/key-features/query-cache.mdx | 19 +--
docs-next/key-features/reciprocal-rank-fusion.mdx | 20 ++--
docs-next/key-features/resource-group.mdx | 20 ++--
docs-next/key-features/spill-to-disk.mdx | 21 ++--
docs-next/key-features/stream-load.mdx | 27 +++--
docs-next/key-features/unique-key.mdx | 133 +++++++++++++++++++++
docs-next/key-features/variant-data-type.mdx | 20 ++--
docs-next/key-features/vector-index.mdx | 28 +++--
docs-next/key-features/vectorized-execution.mdx | 19 +--
docs-next/key-features/vertical-compaction.mdx | 19 +--
docs-next/key-features/workload-group.mdx | 19 +--
docusaurus.config.js | 8 +-
.../use-cases-next/CustomerFacingAnalyticsNext.tsx | 2 +
.../use-cases-next/DataWarehousingNext.tsx | 2 +
src/theme/DocItem/Layout/pathTransfer.ts | 6 +
static/images/next/key-features/iceberg.jpg | Bin 0 -> 217298 bytes
static/images/next/key-features/mpp.jpg | Bin 0 -> 216977 bytes
static/images/next/key-features/unique-key.jpg | Bin 0 -> 184624 bytes
54 files changed, 1017 insertions(+), 436 deletions(-)
diff --git a/docs-next/key-features/analytic-functions.mdx b/docs-next/key-features/analytic-functions.mdx
index 7c766392a5c..e5aaf684220 100644
--- a/docs-next/key-features/analytic-functions.mdx
+++ b/docs-next/key-features/analytic-functions.mdx
@@ -20,22 +20,22 @@ featureCard:
- performance
---
-> **TL;DR** Analytic functions (window functions) compute a value for every row of a result set without collapsing rows the way `GROUP BY` does. Doris supports the standard set, `ROW_NUMBER`, `RANK`, `DENSE_RANK`, `NTILE`, `LAG`, `LEAD`, `FIRST_VALUE`, `LAST_VALUE`, `NTH_VALUE`, `PERCENT_RANK`, `CUME_DIST`, plus every aggregate (`SUM`, `AVG`, `COUNT`, `MIN`, `MAX`) used with `OVER`. The Nereids optimizer also rewrites `WHERE row_num <= K` into a partition-aware Top-N below the window, so [...]
+> **TL;DR** Apache Doris analytic functions (window functions) compute a value for every row of a result set without collapsing rows the way `GROUP BY` does. Apache Doris supports the standard set, `ROW_NUMBER`, `RANK`, `DENSE_RANK`, `NTILE`, `LAG`, `LEAD`, `FIRST_VALUE`, `LAST_VALUE`, `NTH_VALUE`, `PERCENT_RANK`, `CUME_DIST`, plus every aggregate (`SUM`, `AVG`, `COUNT`, `MIN`, `MAX`) used with `OVER`. The Nereids optimizer also rewrites `WHERE row_num <= K` into a partition-aware Top-N [...]
-## Why it matters {#why}
+## Why use analytic functions in Apache Doris? {#why}
-Three questions show up in almost every analytics workload, and `GROUP BY` answers none of them on its own.
+Apache Doris analytic functions answer three questions that show up in almost every analytics workload and that `GROUP BY` cannot answer on its own.
- "Number every order per customer in time order, then keep the first 10." A self-join on `(customer_id, order_date)` works but reads the table twice.
- "What was the running total of sales by day, and the 7-day moving average?" `GROUP BY` collapses the rows you still want to see.
- "How does each row compare to the previous row?" Year-over-year, period-over-period, gap-between-events, this is the lag/lead pattern, and writing it without window functions means a self-join with an offset predicate that the planner cannot optimize.
-Analytic functions answer all three in one pass over the data, with no self-join and the original rows preserved.
+Apache Doris analytic functions answer all three in one pass over the data, with no self-join and the original rows preserved.
-## What it is {#what}
+## What are Apache Doris analytic functions? {#what}
-An analytic function is a SQL function that, for each input row, computes a value over a *window* of related rows defined by an `OVER` clause. The window can be the whole partition, a fixed range around the current row, or anything in between. The output has the same number of rows as the input.
+An Apache Doris analytic function is a SQL function that, for each input row, computes a value over a *window* of related rows defined by an `OVER` clause. The window can be the whole partition, a fixed range around the current row, or anything in between. The output has the same number of rows as the input.
```
function(args) OVER ( [PARTITION BY ...] [ORDER BY ...] [<frame>] )
@@ -43,19 +43,21 @@ function(args) OVER ( [PARTITION BY ...] [ORDER BY ...] [<frame>] )
**Key terms**
-- **`OVER` clause**: tells Doris this call is a window function rather than a regular aggregate. Required.
+- **`OVER` clause**: tells Apache Doris this call is a window function rather than a regular aggregate. Required.
- **`PARTITION BY`**: splits the input into independent groups. Each partition is computed on its own. Different from table partitions, this is a runtime concept.
- **`ORDER BY` (inside `OVER`)**: orders rows within each partition. `LAG`, `LEAD`, `ROW_NUMBER`, `RANK`, and any frame with `PRECEDING`/`FOLLOWING` need it.
-- **Window frame**: the slice of the partition the function reads for the current row. Doris supports `ROWS BETWEEN ... PRECEDING/FOLLOWING/CURRENT ROW/UNBOUNDED ...` and a restricted form of `RANGE`.
+- **Window frame**: the slice of the partition the function reads for the current row. Apache Doris supports `ROWS BETWEEN ... PRECEDING/FOLLOWING/CURRENT ROW/UNBOUNDED ...` and a restricted form of `RANGE`.
- **PartitionTopN**: an internal operator the planner inserts when it can prove a `WHERE rank <= K` filter only needs the top K rows per partition. Source: `CreatePartitionTopNFromWindow.java`.
-## How it works {#how}
+## How do Apache Doris analytic functions work? {#how}
+
+Apache Doris analytic functions run through a five-step pipeline: plan, shuffle, sort, evaluate, and push down.
1. **Plan.** The optimizer parses the `OVER` clause into a `WindowExpression`, normalizes the frame (`CheckAndStandardizeWindowFunctionAndFrame`), and groups window calls that share the same `PARTITION BY` plus `ORDER BY` into a single physical window operator. Calls that share a partition and order do not pay for an extra sort.
2. **Shuffle by partition.** If `PARTITION BY` is present, the engine shuffles rows so all rows for the same partition land on the same backend. With no `PARTITION BY`, the whole window runs in a single pipeline (the parallelism upper bound).
3. **Sort within partition.** Each backend sorts its partitions by the `ORDER BY` columns. Ties produce a non-deterministic row order unless the `ORDER BY` is unique, which is why the docs warn that `SUM() OVER (ORDER BY date_col)` can return different results on tied dates.
-4. **Evaluate per row.** Doris walks each partition once, maintaining the window frame as it goes. Ranking functions emit one integer per row; aggregate functions over `ROWS UNBOUNDED PRECEDING` keep a running total; sliding-window aggregates add the new row and drop the row that fell off the back.
-5. **Push down filters and Top-N.** `CreatePartitionTopNFromWindow` turns `WHERE row_number() OVER (PARTITION BY a ORDER BY b) <= K` into a `PartitionTopN(K)` operator below the window, so each partition only carries the top K rows into the window operator. `PushDownFilterThroughWindow` lifts filters on `PARTITION BY` columns past the window, so Doris filters before sorting instead of after.
+4. **Evaluate per row.** Apache Doris walks each partition once, maintaining the window frame as it goes. Ranking functions emit one integer per row; aggregate functions over `ROWS UNBOUNDED PRECEDING` keep a running total; sliding-window aggregates add the new row and drop the row that fell off the back.
+5. **Push down filters and Top-N.** `CreatePartitionTopNFromWindow` turns `WHERE row_number() OVER (PARTITION BY a ORDER BY b) <= K` into a `PartitionTopN(K)` operator below the window, so each partition only carries the top K rows into the window operator. `PushDownFilterThroughWindow` lifts filters on `PARTITION BY` columns past the window, so Apache Doris filters before sorting instead of after.
## Quick start {#quick-start}
@@ -92,11 +94,13 @@ FROM orders;
One pass, three windowed columns, no self-joins. The original five rows are preserved. `LAG(..., 1, 0)` returns `0` instead of `NULL` for the first row of each partition.
-## When to use it {#when}
+## When should you use Apache Doris analytic functions? {#when}
+
+Apache Doris analytic functions fit any row-preserving computation that depends on neighboring rows, including Top-N per group, running totals, lag/lead comparisons, percentile bucketing, and share-of-total ratios.
**Good fit**
-- Top-N per group: "top 10 orders per customer," "best-selling product per region." Add a `WHERE rn <= 10` filter and Doris pushes a partition Top-N below the window.
+- Top-N per group: "top 10 orders per customer," "best-selling product per region." Add a `WHERE rn <= 10` filter and Apache Doris pushes a partition Top-N below the window.
- Running totals, moving averages, and centered moving averages with `ROWS BETWEEN n PRECEDING AND m FOLLOWING`.
- Year-over-year, day-over-day, and gap-between-events analysis with `LAG` and `LEAD`. One pass over the table replaces a self-join.
- Bucketing for percentile or quartile reports with `NTILE`.
@@ -105,7 +109,7 @@ One pass, three windowed columns, no self-joins. The original five rows are pres
**Not a good fit**
- A pure aggregation that collapses rows. `SELECT category, SUM(amount) FROM t GROUP BY category` does the same work without sorting and without keeping every row in memory. Reach for `GROUP BY` first, and only switch to `OVER` when you also need the unaggregated columns.
-- `RANGE` frames with numeric offsets, like `RANGE BETWEEN 5 PRECEDING AND CURRENT ROW`. Doris's `RANGE` frame is restricted to `UNBOUNDED` boundaries or `CURRENT ROW`; arbitrary `RANGE n PRECEDING/FOLLOWING` is not supported. Use `ROWS` if you need a numeric offset.
+- `RANGE` frames with numeric offsets, like `RANGE BETWEEN 5 PRECEDING AND CURRENT ROW`. The Apache Doris `RANGE` frame is restricted to `UNBOUNDED` boundaries or `CURRENT ROW`; arbitrary `RANGE n PRECEDING/FOLLOWING` is not supported. Use `ROWS` if you need a numeric offset.
- A window with no `PARTITION BY`. The whole result set lands on one pipeline, and that pipeline becomes the bottleneck on large inputs. Add a partition key whenever the workload allows it.
- `ORDER BY` on a non-unique column when you care about deterministic output. `SUM(x) OVER (ORDER BY day)` can return different cumulative totals across runs when several rows share the same day. Add a tie-breaker; see [Window Functions Overview](../sql-manual/sql-functions/window-functions/overview).
- Recomputing the same window result on every refresh. If the same `ROW_NUMBER()` query runs every minute against a slow-moving table, an [async materialized view](../query-acceleration/materialized-view/async-materialized-view/overview) with a partial refresh is cheaper than re-windowing each time.
@@ -118,3 +122,4 @@ One pass, three windowed columns, no self-joins. The original five rows are pres
- [Pipeline Execution Engine: how partitioned windows get parallelized across BEs](./pipeline-execution-engine)
- [Vectorized Execution](./vectorized-execution): the engine that runs each window's batch through SIMD-accelerated operators.
- [Async Materialized View: precompute window results that re-run every minute](../query-acceleration/materialized-view/async-materialized-view/overview)
+- [MPP Architecture](./mpp): how window/analytic operators are shuffled and partitioned across BEs.
diff --git a/docs-next/key-features/batch-load.mdx b/docs-next/key-features/batch-load.mdx
index d936a58c843..a7a92dd1881 100644
--- a/docs-next/key-features/batch-load.mdx
+++ b/docs-next/key-features/batch-load.mdx
@@ -20,12 +20,12 @@ featureCard:
- bulk-ingestion
---
-> **TL;DR** Batch Load is how you move large files from S3, HDFS, or another warehouse into Doris in one shot. You submit a `LOAD LABEL` statement, the FE plans the work and fans it out to BEs, and you check progress with `SHOW LOAD`. The job is asynchronous, so the client can disconnect; the label is what lets you find the result and what dedupes a retry. For [external catalogs](./multi-catalog) and TVF reads, the synchronous `INSERT INTO ... SELECT` form covers the same ground with sim [...]
+> **TL;DR** Apache Doris Batch Load moves large files from S3, HDFS, or another warehouse into Doris in one shot. You submit a `LOAD LABEL` statement, the FE plans the work and fans it out to BEs, and you check progress with `SHOW LOAD`. The job is asynchronous, so the client can disconnect; the label is what lets you find the result and what dedupes a retry. For [external catalogs](./multi-catalog) and TVF reads, the synchronous `INSERT INTO ... SELECT` form covers the same ground with [...]
-## Why it matters {#why}
+## Why use batch load in Apache Doris? {#why}
-Real-time loaders are the right tool for events trickling in, but they fall apart when you need to move a TB of historical data, or rebuild a fact table from a Hive snapshot, or backfill last quarter from Parquet on S3. A [Stream Load](./stream-load) over HTTP times out. A per-row `INSERT` would take days. The job has to run server-side, retry on its own, and survive the client losing its connection.
+Apache Doris Batch Load is the right tool when a single ingestion job runs for hours and the warehouse, not the client, has to own its lifecycle. Real-time loaders are the right tool for events trickling in, but they fall apart when you need to move a TB of historical data, or rebuild a fact table from a Hive snapshot, or backfill last quarter from Parquet on S3. A [Stream Load](./stream-load) over HTTP times out. A per-row `INSERT` would take days. The job has to run server-side, retry [...]
- A nightly ETL has to land a few hundred GB of Parquet from S3 before the morning dashboards refresh.
- A migration from Snowflake or Hive needs to copy whole partitions, with column-level filters and type casts.
@@ -33,9 +33,9 @@ Real-time loaders are the right tool for events trickling in, but they fall apar
Batch Load handles all three by treating large bulk ingestion as a long-running, label-tracked job that the warehouse owns end to end.
-## What it is {#what}
+## What is Apache Doris batch load? {#what}
-Batch Load is the family of asynchronous bulk-load methods in Doris. The main entry point is the `LOAD LABEL ... WITH S3|HDFS|BROKER` statement, historically called Broker Load, which today covers S3, HDFS, and any storage system reachable through a Broker process. The synchronous `INSERT INTO ... SELECT` variant covers the same ingestion needs when the source is an external catalog (Hive, Iceberg, JDBC) or a file behind a TVF such as `S3()` or `HDFS()`. Both share the same transaction m [...]
+Apache Doris Batch Load is the family of asynchronous bulk-load methods in Doris. The main entry point is the `LOAD LABEL ... WITH S3|HDFS|BROKER` statement, historically called Broker Load, which today covers S3, HDFS, and any storage system reachable through a Broker process. The synchronous `INSERT INTO ... SELECT` variant covers the same ingestion needs when the source is an external catalog (Hive, Iceberg, JDBC) or a file behind a TVF such as `S3()` or `HDFS()`. Both share the same [...]
**Key terms**
@@ -45,7 +45,9 @@ Batch Load is the family of asynchronous bulk-load methods in Doris. The main en
- **TVF (Table-Valued Function)**: `S3(...)`, `HDFS(...)`, `LOCAL(...)`, `iceberg_meta(...)`, and the like. They expose files or external metadata as a table you can `SELECT` from, then pipe into `INSERT INTO ... SELECT` for synchronous ingestion.
- **`max_filter_ratio`**: the per-job tolerance for malformed rows. Default is zero, so a single bad row cancels the load.
-## How it works {#how}
+## How does Apache Doris batch load work? {#how}
+
+Apache Doris Batch Load registers a labeled job on the FE, parallelizes file scans across BEs, and atomically commits the result once every replica publishes the new version.
1. **Submit and label.** A `LOAD LABEL my_db.daily_orders ...` request lands on the FE. The FE registers the label, opens a transaction, and the load enters `PENDING`. Resubmitting the same label inside the retention window short-circuits to the original job.
2. **Plan and fan out.** The FE estimates file sizes (one BE handles between `min_bytes_per_broker_scanner` and `max_bytes_per_broker_scanner`, default 64 MB to 500 GB), splits the work, and dispatches scanner tasks to BEs. Job state moves through `LOADING`.
@@ -87,7 +89,9 @@ PROPERTIES ("timeout" = "3600", "max_filter_ratio" = "0.01");
`SHOW LOAD ORDER BY CreateTime DESC LIMIT 1` returns when the job is `FINISHED`. Up to 1% of malformed rows are tolerated; anything more cancels the job and the URL field points at a sample of the offending rows.
-## When to use it {#when}
+## When should you use Apache Doris batch load? {#when}
+
+Use Apache Doris Batch Load when a single ingestion run handles tens of GB or more from object storage, an external warehouse, or a Hive-partitioned directory.
**Good fit**
diff --git a/docs-next/key-features/binlog-table-stream.mdx b/docs-next/key-features/binlog-table-stream.mdx
index 3aa1d9b277c..59bb7825673 100644
--- a/docs-next/key-features/binlog-table-stream.mdx
+++ b/docs-next/key-features/binlog-table-stream.mdx
@@ -22,16 +22,16 @@ featureCard:
- coming-soon
---
-> **TL;DR** Table Stream is the upcoming public API for reading row-level changes (insert, update, delete) out of a Doris table, the way you read a MySQL binlog. The plumbing is already in the codebase: a row-format binlog, a `STREAM` object that pins offsets per partition, and consumption metadata. It is not GA yet. Today the same binlog powers [Cross-Cluster Replication](../admin-manual/data-admin/ccr/overview); for general CDC out of Doris, use the [Flink Doris Connector](../connectio [...]
+> **TL;DR** Apache Doris Table Stream is the upcoming public API for reading row-level changes (insert, update, delete) out of a Doris table, the way you read a MySQL binlog. The plumbing is already in the codebase: a row-format binlog, a `STREAM` object that pins offsets per partition, and consumption metadata. The feature is not GA yet, and the default `binlog.ttl_seconds` is 86,400 seconds (one day). Today the same binlog powers [Cross-Cluster Replication](../admin-manual/data-admin/c [...]
:::caution Status: Coming Soon
The `STREAM` object, `binlog.format = ROW`, and the consumption metadata views described here are present in the Doris source tree but not yet released as a stable, user-facing API. Syntax and semantics may change before GA. Production CDC pipelines today should use [Kafka and CDC Integration](./kafka-cdc-integration) or [CCR](../admin-manual/data-admin/ccr/overview).
:::
-## Why it matters {#why}
+## Why use binlog and table stream in Apache Doris? {#why}
-Doris is good at [ingesting change data](./kafka-cdc-integration). Getting change data back out is harder. If a downstream system (a Flink job, an audit trail, another Doris cluster, a search index) needs to react to every insert, [update, and delete](./data-update-delete), the options today are awkward:
+Apache Doris binlog and table stream give downstream consumers a row-level change feed, so a Flink job, audit trail, or search index can react to every insert, update, and delete without re-reading the table. Apache Doris is good at [ingesting change data](./kafka-cdc-integration). Getting change data back out is harder. If a downstream system (a Flink job, an audit trail, another Doris cluster, a search index) needs to react to every insert, [update, and delete](./data-update-delete), t [...]
- Re-read the table on a timer and diff. Cheap to write, expensive to run, never real-time.
- Have producers double-write to Kafka and Doris. Two sinks, two failure modes.
@@ -39,11 +39,11 @@ Doris is good at [ingesting change data](./kafka-cdc-integration). Getting chang
Binlog and Table Stream close that gap. The binlog records every committed change at the row level. The Table Stream is a named cursor over that log: a consumer attaches to a stream, reads the changes since its last offset, and acknowledges. Same idea as MySQL binlog, Snowflake Streams, or a Postgres logical replication slot, expressed as a first-class Doris object.
-## What it is {#what}
+## What is the Apache Doris binlog and table stream? {#what}
-Binlog is the per-table change log Doris already maintains for replication. It comes in two formats. `STATEMENT_AND_SNAPSHOT` is what CCR has shipped on for years: it records DDL plus snapshots, and the CCR Syncer applies them downstream. The new `ROW` format records every committed row delta, which is what a generic CDC consumer needs.
+Apache Doris binlog is the per-table change log Doris already maintains for replication, and Apache Doris table stream is the consumer-facing object that exposes it as row-level CDC. The binlog comes in two formats. `STATEMENT_AND_SNAPSHOT` is what CCR has shipped on for years: it records DDL plus snapshots, and the CCR Syncer applies them downstream. The new `ROW` format records every committed row delta, which is what a generic CDC consumer needs.
-Table Stream is the consumer-facing object on top of the row-format binlog. A stream is created with `CREATE STREAM <name> ON <base_table>`, owns its own offsets per partition, and exposes a SQL surface: `SHOW CREATE STREAM` and an `information_schema` view of consumption progress.
+A table stream is the consumer-facing object on top of the row-format binlog. A stream is created with `CREATE STREAM <name> ON <base_table>`, owns its own offsets per partition, and exposes a SQL surface: `SHOW CREATE STREAM` and an `information_schema` view of consumption progress.
**Key terms**
@@ -54,7 +54,9 @@ Table Stream is the consumer-facing object on top of the row-format binlog. A st
- **Stream consume type**: `default`, `append_only`, or `min_delta`. Controls whether the stream emits full row deltas, inserts only, or the minimal change set.
- **Stream lag**: visible version of the partition minus the stream's recorded offset. Surfaced as the `LAG` column.
-## How it works {#how}
+## How does the Apache Doris binlog and table stream work? {#how}
+
+The Apache Doris binlog records row-level deltas at the BE, and a table stream wraps that log with per-partition offsets so a downstream consumer can read changes incrementally and acknowledge them.
1. **Turn the binlog on.** `ALTER TABLE t SET ("binlog.enable" = "true", "binlog.format" = "ROW")`. The BE starts retaining row-level deltas alongside tablet data, governed by `binlog.ttl_seconds` and the size and history caps. CCR users may already have `binlog.enable = true` with the older `STATEMENT_AND_SNAPSHOT` format; Table Stream needs `ROW`.
2. **Create a stream over the table.** `CREATE STREAM s ON t PROPERTIES ("type" = "default")`. The FE materializes a `STREAM` object, snaps the current visible version of every partition as the starting offset, and registers it under the database. `show_initial_rows = true` flips that so the first read returns the historical state.
@@ -94,7 +96,9 @@ WHERE STREAM_NAME = 'orders_cdc';
`LAG = 0` means the consumer is caught up to the partition's visible version. As writes commit upstream, `LAG` grows; as the consumer reads, it shrinks. A non-`N/A` `STALE_REASON` means retention rolled past the offset and a re-snapshot is needed.
-## When to use it {#when}
+## When should you use the Apache Doris binlog and table stream? {#when}
+
+Once GA, use the Apache Doris binlog and table stream when a downstream system needs row-level change events from a Doris table without re-reading the whole table.
**Good fit (once GA)**
@@ -118,3 +122,5 @@ WHERE STREAM_NAME = 'orders_cdc';
- [CCR feature matrix](../admin-manual/data-admin/ccr/feature): which DDL and DML operations the binlog already captures and replays, a preview of what a row-format stream will surface.
- [CCR quickstart](../admin-manual/data-admin/ccr/quickstart): how `binlog.enable` and `enable_feature_binlog` get configured today.
- [Data Update and Delete](./data-update-delete): how Unique Key Merge-on-Write tables represent the deletes and out-of-order updates a row stream will carry.
+- [Iceberg](./iceberg): an external destination for replicated tables — land CDC into Iceberg via SQL without leaving Doris.
+- [Unique Key](./unique-key): the upstream table model that captures one-row-per-key state cleanly for downstream binlog consumers.
diff --git a/docs-next/key-features/bm25.mdx b/docs-next/key-features/bm25.mdx
index faad32c8578..fad45302998 100644
--- a/docs-next/key-features/bm25.mdx
+++ b/docs-next/key-features/bm25.mdx
@@ -20,39 +20,41 @@ featureCard:
- ranking
---
-> **TL;DR** Doris ranks [full-text search](./full-text-search) hits with BM25, the same scoring algorithm Lucene and Elasticsearch use. Add `score()` to a SELECT, keep a `MATCH_*` predicate in the WHERE clause, and `ORDER BY` the score with a `LIMIT`. The planner pushes the top-K computation into the [inverted index](./inverted-index) and returns the most relevant rows first.
+> **TL;DR** Apache Doris ranks [full-text search](./full-text-search) hits with BM25, the same scoring algorithm Lucene and Elasticsearch use. Add `score()` to a SELECT, keep a `MATCH_*` predicate in the WHERE clause, and `ORDER BY` the score with a `LIMIT`. The Apache Doris planner pushes the top-K computation into the [inverted index](./inverted-index) and returns the most relevant rows first.
-## Why it matters {#why}
+## Why use BM25 scoring in Apache Doris? {#why}
-Equality filters tell you which rows match. They don't tell you which rows are *good* matches. A query like `WHERE content MATCH_ANY 'apache doris real-time analytics'` over a million-row log table can easily return 50,000 hits, and the user only wants the best ten. Without a relevance score, the database has no good way to pick those ten:
+Apache Doris BM25 scoring turns full-text matches into a ranked top-K result inside SQL, so a million-row corpus can return the best ten hits without a separate search engine. Equality filters tell you which rows match. They don't tell you which rows are *good* matches. A query like `WHERE content MATCH_ANY 'apache doris real-time analytics'` over a million-row log table can easily return 50,000 hits, and the user only wants the best ten. Without a relevance score, the database has no go [...]
- Sort by date and you bury the perfect match from last year.
- Sort by an arbitrary column and you ship random results.
- Compute relevance in the application and you've now built half a search engine on top of your warehouse.
-BM25 closes that gap inside SQL. Doris gives each matching row a score based on how often the query terms appear in it, how rare those terms are across the table, and how long the row's text is. Sort by that score and you get search-engine-style ranking from a SELECT.
+BM25 closes that gap inside SQL. Apache Doris gives each matching row a score based on how often the query terms appear in it, how rare those terms are across the table, and how long the row's text is. Sort by that score and you get search-engine-style ranking from a SELECT.
-## What it is {#what}
+## What is Apache Doris BM25 scoring? {#what}
-BM25 (Best Matching 25) is a relevance ranking function in the TF-IDF family. It takes a query term and a document and returns a positive number: higher means more relevant. Three things drive the score. Rows that contain the term more often score higher. Rare terms count for more than common ones. And longer rows are normalized down so a 200-word row doesn't automatically beat a 20-word row that says the same thing.
+Apache Doris BM25 scoring is the SQL-exposed implementation of the BM25 (Best Matching 25) relevance ranking function from the TF-IDF family. The function takes a query term and a document and returns a positive number: higher means more relevant. Three things drive the score. Rows that contain the term more often score higher. Rare terms count for more than common ones. And longer rows are normalized down so a 200-word row doesn't automatically beat a 20-word row that says the same thing.
-Doris exposes BM25 through the `score()` SQL function. The inverted index already stores the term frequencies, document frequencies, and field lengths the formula needs, so once the index exists you don't pay an extra scan to score.
+Apache Doris exposes BM25 through the `score()` SQL function. The inverted index already stores the term frequencies, document frequencies, and field lengths the formula needs, so once the index exists you don't pay an extra scan to score.
**Key terms**
-- **`score()`**: a Doris scalar function that returns the BM25 relevance score of the current row against the query's `MATCH_*` predicates. Returns `FLOAT`.
+- **`score()`**: an Apache Doris scalar function that returns the BM25 relevance score of the current row against the query's `MATCH_*` predicates. Returns `FLOAT`.
- **TF (term frequency)**: how many times the query term appears in the row.
- **IDF (inverse document frequency)**: a measure of how rare the term is across the table. Common words score low, rare words score high.
-- **`k1` and `b`**: BM25's tuning constants. Doris ships with `k1 = 1.2` and `b = 0.75`, the Lucene defaults. They are not currently exposed as session variables.
+- **`k1` and `b`**: BM25's tuning constants. Apache Doris ships with `k1 = 1.2` and `b = 0.75`, the Lucene defaults. They are not currently exposed as session variables.
-## How it works {#how}
+## How does Apache Doris BM25 scoring work? {#how}
-1. **The inverted index records the statistics.** When data is loaded into a column with a tokenized inverted index (parsers like `english`, `standard`, or `chinese`), Doris stores the term dictionary, per-row term frequencies, and field lengths. The same statistics that power `MATCH_*` filtering also feed BM25.
-2. **The planner recognizes the scoring shape.** When a query has `score()` in the SELECT, a `MATCH_*` predicate in the WHERE, and an `ORDER BY score()` with a `LIMIT`, Nereids rewrites the plan to push top-K scoring into the storage layer. If you use `score()` outside that exact shape, Doris fails the query with `score() function requires WHERE clause with MATCH function, ORDER BY and LIMIT for optimization`. That constraint exists to keep you from accidentally scoring every row in a bi [...]
+Apache Doris evaluates BM25 inside the inverted-index reader: the planner recognizes the `score()` + `MATCH_*` + `ORDER BY` + `LIMIT` shape, the BE scores rows while it scans postings, and a bounded min-heap of size `LIMIT` keeps only the top-K.
+
+1. **The inverted index records the statistics.** When data is loaded into a column with a tokenized inverted index (parsers like `english`, `standard`, or `chinese`), Apache Doris stores the term dictionary, per-row term frequencies, and field lengths. The same statistics that power `MATCH_*` filtering also feed BM25.
+2. **The planner recognizes the scoring shape.** When a query has `score()` in the SELECT, a `MATCH_*` predicate in the WHERE, and an `ORDER BY score()` with a `LIMIT`, Nereids rewrites the plan to push top-K scoring into the storage layer. If you use `score()` outside that exact shape, Apache Doris fails the query with `score() function requires WHERE clause with MATCH function, ORDER BY and LIMIT for optimization`. That constraint exists to keep you from accidentally scoring every row [...]
3. **The BE scores while it scans.** The inverted-index reader walks the postings for each query term. For every candidate row, it computes `IDF * (tf * (k1 + 1)) / (tf + k1 * (1 - b + b * |d| / avgdl))`, where `|d|` is the row's tokenized length and `avgdl` is the table's average. For multi-term queries, the per-term scores are added.
4. **A heap keeps the top-K.** The BE keeps a bounded min-heap of size `LIMIT`, so it never has to hold the full match set in memory. That's the difference between scoring 50,000 rows and then sorting versus stream-scoring and keeping only the best ten.
-5. **Doris returns the ranked rows.** The FE merges the already-sorted top-K from each tablet and returns the result.
+5. **Apache Doris returns the ranked rows.** The FE merges the already-sorted top-K from each tablet and returns the result.
## Quick start {#quick-start}
@@ -89,7 +91,9 @@ LIMIT 3;
Row 1 wins because it matches all three query terms. Row 2 matches one rare term (`analytics`). Row 3 only matches `apache`, the most common term in this corpus, so it scores lowest. The absolute numbers don't mean anything on their own. What matters is the ordering inside this one result set.
-## When to use it {#when}
+## When should you use Apache Doris BM25 scoring? {#when}
+
+Apache Doris BM25 scoring fits top-K full-text ranking and the text leg of hybrid search; it is not the right tool for exact-equality lookups, aggregations on relevance, or comparing scores across different queries.
**Good fit**
@@ -101,14 +105,14 @@ Row 1 wins because it matches all three query terms. Row 2 matches one rare term
**Not a good fit**
- **Exact-equality lookups.** A non-tokenized inverted index does not compute scores. Use `=` or `IN` and skip `score()` entirely.
-- **Aggregations over scored results.** Doris rejects `score()` inside `GROUP BY`, inside aggregate functions, and in any plan without `ORDER BY score() LIMIT`. If you need analytics on relevance, materialize the top-K first and aggregate over that result set.
+- **Aggregations over scored results.** Apache Doris rejects `score()` inside `GROUP BY`, inside aggregate functions, and in any plan without `ORDER BY score() LIMIT`. If you need analytics on relevance, materialize the top-K first and aggregate over that result set.
- **Comparing scores across queries.** BM25 is corpus- and query-dependent. A score of 3.0 from one query and 3.0 from another mean different things. Use a rank-based fusion such as RRF instead. See [Reciprocal Rank Fusion](./reciprocal-rank-fusion).
-- **Tuning `k1` and `b` per query.** Doris uses Lucene's defaults and does not currently expose these as runtime parameters. If your workload really needs custom tuning, file an issue rather than working around it with hand-rolled scoring.
+- **Tuning `k1` and `b` per query.** Apache Doris uses Lucene's defaults and does not currently expose these as runtime parameters. If your workload really needs custom tuning, file an issue rather than working around it with hand-rolled scoring.
- **Pure vector ranking.** If the query is "find the 10 nearest embeddings, no keyword constraint at all", use an ANN distance function and skip BM25.
## Further reading {#further-reading}
-- [Inverted index: how Doris stores the data BM25 reads from](./inverted-index)
+- [Inverted index: how Apache Doris stores the data BM25 reads from](./inverted-index)
- [Full-text search: tokenizers, MATCH operators, and the SEARCH function](./full-text-search)
- [Hybrid search: combining BM25 with vector similarity in one query](./hybrid-search)
- [Reciprocal Rank Fusion: a query-independent way to fuse BM25 and vector rankings](./reciprocal-rank-fusion)
diff --git a/docs-next/key-features/catalog-integrations.mdx b/docs-next/key-features/catalog-integrations.mdx
index 39c166056c9..628563a51e9 100644
--- a/docs-next/key-features/catalog-integrations.mdx
+++ b/docs-next/key-features/catalog-integrations.mdx
@@ -22,22 +22,22 @@ featureCard:
- catalog
---
-> **TL;DR** Doris talks to most of the catalog services your lake already uses. Hive Metastore, AWS Glue, the Iceberg REST spec (Tabular, Polaris, Lakekeeper, Snowflake Open Catalog, Gravitino, Unity), AWS S3 Tables, Aliyun DLF, file-based Hadoop, and a JDBC backend all sit behind one `CREATE CATALOG` statement. Pick the right `iceberg.catalog.type` (or `hive.metastore.type`), supply credentials and a warehouse, and the same SQL works against every one of them.
+> **TL;DR** Apache Doris catalog integrations cover most of the catalog services a lakehouse already uses: Hive Metastore, AWS Glue, the Iceberg REST spec (Tabular, Polaris, Lakekeeper, Snowflake Open Catalog, Gravitino, Unity), AWS S3 Tables, Aliyun DLF, file-based Hadoop, and a JDBC backend. Every backend sits behind one `CREATE CATALOG` statement, selected by `iceberg.catalog.type` or `hive.metastore.type`. The same SQL works against every one of them once credentials and a warehouse [...]
-## Why it matters {#why}
+## Why use Apache Doris catalog integrations? {#why}
-Catalog choice is rarely up to the analytics engine. The metastore comes from somewhere else: a Hadoop cluster that picked Hive Metastore years ago, an AWS account that standardized on Glue, a platform team that just rolled out Polaris, a Databricks workspace that lives in Unity. If your query engine only speaks one of those, you either build a translation layer or limit the lake to whatever the engine supports.
+Apache Doris catalog integrations let one engine speak every common lakehouse catalog dialect, so the metastore choice does not dictate the query layer. Catalog choice is rarely up to the analytics engine. The metastore comes from somewhere else: a Hadoop cluster that picked Hive Metastore years ago, an AWS account that standardized on Glue, a platform team that just rolled out Polaris, a Databricks workspace that lives in Unity. If your query engine only speaks one of those, you either [...]
-Doris takes the opposite stance: stay plural about the catalog, keep the SQL flat. You type `CREATE CATALOG`, pick a type, and get `SHOW DATABASES`, `SELECT`, joins, writes, and time travel against the same Iceberg or Hive table no matter where the metadata lives. Pair it with [multi-catalog](./multi-catalog) federation and a unified [metadata cache](./metadata-cache), and you can also [manage lake tables](./managing-lake-table) end to end.
+Apache Doris takes the opposite stance: stay plural about the catalog, keep the SQL flat. You type `CREATE CATALOG`, pick a type, and get `SHOW DATABASES`, `SELECT`, joins, writes, and time travel against the same Iceberg or Hive table no matter where the metadata lives. Pair it with [multi-catalog](./multi-catalog) federation and a unified [metadata cache](./metadata-cache), and you can also [manage lake tables](./managing-lake-table) end to end.
- One engine for many metastores. No per-catalog query layer to operate.
- New catalog services arrive as another backend type. Iceberg REST, S3 Tables, Polaris, and Unity all reuse the same connection scaffolding.
- Migrating off a legacy metastore is a `CREATE CATALOG` away, not a re-platform.
-## What it is {#what}
+## What is an Apache Doris catalog integration? {#what}
-A catalog integration is a backend driver that maps Doris's external-catalog interface to a specific metadata service. For Iceberg tables the type is selected with `iceberg.catalog.type`; for Hive-style tables the choice flows through `type = 'hms'` plus `hive.metastore.type`. Storage credentials (S3, OSS, COS, HDFS) are layered on top of whichever catalog you pick.
+An Apache Doris catalog integration is a backend driver that maps the external-catalog interface to a specific metadata service. For Iceberg tables the type is selected with `iceberg.catalog.type`; for Hive-style tables the choice flows through `type = 'hms'` plus `hive.metastore.type`. Storage credentials (S3, OSS, COS, HDFS) are layered on top of whichever catalog you pick.
The shipped backends today:
@@ -55,16 +55,18 @@ The shipped backends today:
- **`type`**: the catalog family. `iceberg`, `hms`, `paimon`, `hudi`, `jdbc`, and similar.
- **`iceberg.catalog.type`**: the metastore driver inside the Iceberg family. One of `hms`, `rest`, `glue`, `dlf`, `jdbc`, `hadoop`, `s3tables`.
- **`hive.metastore.type`**: the equivalent switch for Hive-style catalogs. `hms` (default) or `glue` or `dlf`.
-- **Vended credentials**: a Doris feature that asks the REST catalog for short-lived storage credentials per table, so you do not hand-roll AKs in the catalog properties.
+- **Vended credentials**: an Apache Doris feature that asks the REST catalog for short-lived storage credentials per table, so you do not hand-roll AKs in the catalog properties.
- **Warehouse**: the storage root the catalog writes new tables under. Required for `hadoop`, optional for most others.
-## How it works {#how}
+## How does an Apache Doris catalog integration work? {#how}
-1. **Pick a backend.** Doris's `IcebergExternalCatalogFactory` switches on `iceberg.catalog.type` and instantiates one of `IcebergHMSExternalCatalog`, `IcebergGlueExternalCatalog`, `IcebergRestExternalCatalog`, `IcebergDLFExternalCatalog`, `IcebergJdbcExternalCatalog`, `IcebergHadoopExternalCatalog`, or `IcebergS3TablesExternalCatalog`. Hive-family catalogs route the same way through the `hms` driver and `hive.metastore.type`.
+An Apache Doris catalog integration dispatches metadata calls through a backend driver chosen at `CREATE CATALOG` time, then funnels reads and writes through the same unified storage layer the BEs already use.
+
+1. **Pick a backend.** The Apache Doris `IcebergExternalCatalogFactory` switches on `iceberg.catalog.type` and instantiates one of `IcebergHMSExternalCatalog`, `IcebergGlueExternalCatalog`, `IcebergRestExternalCatalog`, `IcebergDLFExternalCatalog`, `IcebergJdbcExternalCatalog`, `IcebergHadoopExternalCatalog`, or `IcebergS3TablesExternalCatalog`. Hive-family catalogs route the same way through the `hms` driver and `hive.metastore.type`.
2. **Resolve metadata.** The chosen backend handles `SHOW DATABASES`, schema fetches, snapshot lookups, and partition listing through its native protocol (Thrift for HMS, AWS SDK for Glue, HTTP for REST).
3. **Resolve storage.** Whatever credentials you pass (`s3.access_key`, `oss.endpoint`, vended creds from a REST catalog) flow into a unified storage layer the BEs use to read Parquet, ORC, or Avro files.
4. **Cache for speed.** Schemas, table objects, manifests, and views are cached per catalog under `meta.cache.iceberg.*` keys. See [Metadata Cache](./metadata-cache) for the eviction model.
-5. **Plan and execute.** From there it is the same MPP pipeline as a Doris-internal query, reusing the same data cache, materialized view rewrite, and vectorized execution.
+5. **Plan and execute.** From there it is the same MPP pipeline as an Apache Doris internal query, reusing the same data cache, materialized view rewrite, and vectorized execution.
## Quick start {#quick-start}
@@ -100,11 +102,13 @@ SELECT region, SUM(amount) FROM sales.orders WHERE dt = '2026-05-01' GROUP BY re
`SHOW CATALOGS` lists `iceberg_rest` next to `internal`. The same query shape works after you swap the PROPERTIES block for a Glue, HMS, or S3 Tables backend; only the connection block changes.
-## When to use it {#when}
+## When should you use Apache Doris catalog integrations? {#when}
+
+Use Apache Doris catalog integrations whenever the lakehouse metadata already lives in an external service and you want one engine to read and write it without moving data.
**Good fit**
-- An existing lake that already runs on Hive Metastore, Glue, DLF, or an Iceberg REST service, and you want Doris to read and write it without moving data.
+- An existing lake that already runs on Hive Metastore, Glue, DLF, or an Iceberg REST service, and you want Apache Doris to read and write it without moving data.
- Cross-region or cross-vendor work, for example reading a Glue catalog in one account and joining against a Polaris catalog in another. See [Multi Catalog](./multi-catalog) for the federation story.
- Migrations between catalog services. Bind both old and new as separate catalogs, run them side by side, and switch when ready.
- Cloud-native deployments that need vended credentials so the engine never sees long-lived storage keys.
@@ -113,7 +117,7 @@ SELECT region, SUM(amount) FROM sales.orders WHERE dt = '2026-05-01' GROUP BY re
**Not a good fit**
- Operational metadata stores like Confluent Schema Registry. They describe wire formats, not tables.
-- Treating Doris as the system of record for a lake's metadata. Writes go back to the bound catalog, but Doris is not the catalog itself; for pure Doris-managed tables, use the `internal` catalog.
+- Treating Apache Doris as the system of record for a lake's metadata. Writes go back to the bound catalog, but Apache Doris is not the catalog itself; for pure Apache Doris-managed tables, use the `internal` catalog.
- Reading raw files when there is no catalog at all. Use [Table Value Functions](../lakehouse/file-analysis) (`s3()`, `hdfs()`, `local()`) instead of standing up an empty catalog.
## Further reading {#further-reading}
@@ -124,3 +128,4 @@ SELECT region, SUM(amount) FROM sales.orders WHERE dt = '2026-05-01' GROUP BY re
- [AWS Glue](../lakehouse/metastores/aws-glue), [Iceberg REST Catalog](../lakehouse/metastores/iceberg-rest), [Hive Metastore](../lakehouse/metastores/hive-metastore), [Aliyun DLF](../lakehouse/metastores/aliyun-dlf): full per-backend connection guides with examples.
- [Catalog Overview](../lakehouse/catalog-overview): common properties (`include_database_list`, case sensitivity, refresh) that apply across every backend.
- [Metadata Cache](./metadata-cache): the cache model that keeps repeat queries fast without losing freshness.
+- [Iceberg](./iceberg): the engine-level capabilities Doris exposes on top of any of the seven Iceberg catalog backends.
diff --git a/docs-next/key-features/columnar-storage.mdx b/docs-next/key-features/columnar-storage.mdx
index ad8008f8752..5055969e917 100644
--- a/docs-next/key-features/columnar-storage.mdx
+++ b/docs-next/key-features/columnar-storage.mdx
@@ -20,12 +20,12 @@ featureCard:
- performance
---
-> **TL;DR** Every Doris OLAP table writes its data in column-oriented Segment files. Each column gets its own encoding (PLAIN, dictionary, RLE, bit-shuffle, frame-of-reference), each page is compressed independently (LZ4 by default, ZSTD optional), and per-page zone maps and bloom filters let queries [skip pages](./data-pruning) without decoding them. The V3 footer added in 4.1 cuts segment-open latency on wide tables by 16x. There's nothing to enable; this is just how Doris [stores data [...]
+> **TL;DR** Every Apache Doris OLAP table writes its data in column-oriented Segment files. Each column gets its own encoding (PLAIN, dictionary, RLE, bit-shuffle, frame-of-reference), and each page is compressed independently (LZ4 by default, ZSTD optional). Per-page zone maps and bloom filters let queries [skip pages](./data-pruning) without decoding them, and the V3 footer added in 4.1 cuts segment-open latency on wide tables by 16x.
-## Why it matters {#why}
+## Why use columnar storage in Apache Doris? {#why}
-Three problems show up the moment you put analytics on a row-oriented store.
+Apache Doris stores every OLAP table in column-oriented Segment files because the columnar layout cuts I/O to the columns a query touches, lets each column pick its strongest codec, and gives skip indexes a natural home. Three problems show up the moment you put analytics on a row-oriented store.
- A `GROUP BY country` over a 50-column table still reads bytes for the other 49 columns. I/O cost scales with row width, not query width.
- Compression is weak. Adjacent rows mix unrelated types, so dictionary, run-length, and delta codecs have nothing to chew on.
@@ -33,9 +33,9 @@ Three problems show up the moment you put analytics on a row-oriented store.
A column-oriented layout fixes all three. Values from the same column sit together on disk, so the codec can pick a transform that fits the type. And the page indexes give the reader something to test predicates against before it [decompresses anything](./vectorized-execution).
-## What it is {#what}
+## What is Apache Doris columnar storage? {#what}
-Columnar storage in Doris is the on-disk segment file format that every OLAP table (Duplicate, Unique, Aggregate) shares. Each load produces one or more Segment files. Inside a segment, the writer handles each column on its own: it streams values into ~64 KB pages, encodes and compresses each page, and writes the skip indexes alongside. A footer at the end of the file points to everything.
+Apache Doris columnar storage is the on-disk segment file format that every OLAP table (Duplicate, Unique, Aggregate) shares. Each load produces one or more Segment files. Inside a segment, the writer handles each column on its own: it streams values into ~64 KB pages, encodes and compresses each page, and writes the skip indexes alongside. A footer at the end of the file points to everything.
**Key terms**
@@ -45,9 +45,9 @@ Columnar storage in Doris is the on-disk segment file format that every OLAP tab
- **`Page indexes`**: ordinal index, short key index, zone map (per-page min/max plus has-null), and bloom filter. All written into the segment alongside the data.
- **`Footer`**: protobuf record at the end of the file describing every column, every encoding, every page pointer, and every index location. V3 (4.1+) externalizes column metadata so opening a wide segment doesn't have to read all of it.
-## How it works {#how}
+## How does Apache Doris columnar storage work? {#how}
-The lifecycle of one column on disk:
+Apache Doris writes each column independently inside a segment: encode, page into ~64 KB chunks, compress, index, and footer. The lifecycle of one column on disk:
1. **Encode.** The writer picks an encoding from the column's type and value distribution. An `INT` column with low cardinality gets bit-shuffle; a string column with few distinct values gets a dictionary built into a separate dict page.
2. **Page and compress.** Encoded values stream into pages of about 64 KB each. Each page goes through LZ4F by default, or ZSTD / Snappy / Zlib if the table sets `compression`. Compression is skipped if the saving falls below 10%.
@@ -95,11 +95,13 @@ ORDER BY hits DESC;
The query reads only the `ts` and `country` columns. The reader skips pages whose zone map says `max(ts) < 2026-05-01` before any decompression runs. ZSTD typically shrinks the table 5x to 10x, and the V3 footer keeps segment-open cost flat as the column count grows.
-## When to use it {#when}
+## When should you use Apache Doris columnar storage? {#when}
+
+Apache Doris columnar storage is the default and only OLAP format, so every analytic table uses it; the trade-offs come up only at the edges (high-QPS point lookups, codec tuning).
**Good fit**
-- Every analytic table you run through Doris. Columnar storage isn't optional; it's the default and only OLAP format.
+- Every analytic table you run through Apache Doris. Columnar storage isn't optional; it's the default and only OLAP format.
- Wide tables and dashboard queries. The wider the table and the narrower the column projection, the bigger the savings over a row store.
- Tables you load into frequently. Per-column compression and per-page zone maps both improve as more rows pile up under the same schema.
- Wide tables with thousands of columns. Set `"storage_format" = "V3"` (4.1+) to keep segment-open latency and memory bounded.
diff --git a/docs-next/key-features/compute-group.mdx b/docs-next/key-features/compute-group.mdx
index c3347cdbc7e..fa7ef26b321 100644
--- a/docs-next/key-features/compute-group.mdx
+++ b/docs-next/key-features/compute-group.mdx
@@ -19,20 +19,20 @@ featureCard:
- storage-compute-decoupled
---
-> **TL;DR** Compute Group is the storage-compute decoupled equivalent of [Resource Group](./resource-group). You split BE nodes into named groups, grant users access, and bind each session to a group; only those BEs serve the query. Because storage is shared (S3, OSS, COS, HDFS), adding a group costs you compute, not a replica. A BE failure inside a group still does not affect the others, and decommissioning a whole group leaves the data untouched.
+> **TL;DR** Apache Doris Compute Group is the storage-compute decoupled equivalent of [Resource Group](./resource-group). You split BE nodes into named groups, grant users access, and bind each session to a group; only those BEs serve the query. Because storage is shared (S3, OSS, COS, HDFS), adding a group costs you compute, not a replica. A BE failure inside one group does not affect the others, and decommissioning a whole group leaves the data untouched.
-## Why it matters {#why}
+## Why use Apache Doris compute groups? {#why}
-[Resource Group](./resource-group) buys you BE-level isolation in compute-storage integrated mode, but the bill arrives as storage. Five groups that all need to read the same table means five replicas of that table. Two new tenants means rebuilding replica allocation. Spinning up a temporary pool for tomorrow's batch run means moving data around first.
+Apache Doris compute groups give you BE-level workload isolation without multiplying storage, because every group reads from the same shared object store. [Resource Group](./resource-group) buys you BE-level isolation in compute-storage integrated mode, but the bill arrives as storage. Five groups that all need to read the same table means five replicas of that table. Two new tenants means rebuilding replica allocation. Spinning up a temporary pool for tomorrow's batch run means moving d [...]
In a storage-compute decoupled deployment, the BE is stateless. It caches hot tablets locally for speed, but the source of truth lives in shared object storage. So adding another set of BEs costs you compute and a cold cache, nothing else. There is one logical copy of the data, regardless of how many compute groups read from it.
Compute Group is the feature that turns that into workload isolation. You divide BEs into groups the way you would with Resource Group, but the storage budget stops scaling with the number of groups. Spin up an ETL pool for a load window, run BI on a different pool that never feels the load, decommission a group at midnight. The data does not move.
-## What it is {#what}
+## What is an Apache Doris compute group? {#what}
-Compute Group is a node-level isolation feature in Doris's storage-compute decoupled (cloud) mode. Each BE belongs to exactly one compute group, identified by a name. The same object-storage manifest backs every group, so adding a group means adding compute and a cache that needs warming, not adding a replica. Workload Group then sits inside a compute group, partitioning CPU and memory across the queries running there.
+An Apache Doris compute group is a node-level isolation primitive in storage-compute decoupled (cloud) mode. Each BE belongs to exactly one compute group, identified by a name. The same object-storage manifest backs every group, so adding a group means adding compute and a cache that needs warming, not adding a replica. Workload Group then sits inside a compute group, partitioning CPU and memory across the queries running there.
**Key terms**
@@ -42,7 +42,9 @@ Compute Group is a node-level isolation feature in Doris's storage-compute decou
- **`USAGE_PRIV ON COMPUTE GROUP`**: the access privilege. Granted with `GRANT USAGE_PRIV ON COMPUTE GROUP <name> TO <user>`. Without it, the user cannot select the group.
- **File cache**: the local on-BE cache of hot tablet blocks. A new group starts cold and pays an object-storage round-trip on every miss until it warms up.
-## How it works {#how}
+## How does an Apache Doris compute group work? {#how}
+
+An Apache Doris compute group tags each BE at registration, grants `USAGE_PRIV` to specific users, binds sessions to a group either by default property or by `USE @<group>`, and restricts query fragment placement to that group's BEs while shared object storage backs every group.
1. **Tag BEs at registration.** `ALTER SYSTEM ADD BACKEND 'host:9050' PROPERTIES ("tag.compute_group_name" = "etl")` adds a backend into the `etl` group. Omitting the property lands the BE in `default_compute_group`. The MetaService persists the assignment, and `SHOW COMPUTE GROUPS` reports the membership.
2. **Grant access.** A regular user has no compute groups by default. `GRANT USAGE_PRIV ON COMPUTE GROUP etl TO 'etl_user'` is what lets them target the group at all. Admin users see every group.
@@ -85,7 +87,9 @@ SELECT region, SUM(amount) FROM orders WHERE dt = CURRENT_DATE() GROUP BY region
`SHOW COMPUTE GROUPS` confirms the layout. The aggregation runs only on `be2` and `be3`. ETL and BI share the same `orders` data on object storage, but they share no BEs and no local cache. Decommission `be1` and the etl group is empty; the data is untouched.
-## When to use it {#when}
+## When should you use an Apache Doris compute group? {#when}
+
+Apache Doris compute groups fit storage-compute decoupled clusters that need workload isolation, read/write separation, or elastic peak/off-peak compute, but not integrated deployments or cross-group queries.
**Good fit**
@@ -111,3 +115,4 @@ SELECT region, SUM(amount) FROM orders WHERE dt = CURRENT_DATE() GROUP BY region
- [Resource Group](./resource-group): the integrated-mode counterpart, with its replica-multiplication tradeoff.
- [Workload Group](./workload-group): the in-process layer you usually combine with Compute Group.
- [File cache best practices for read-write separation](../compute-storage-decoupled/rw/file-cache-rw-compute-group-best-practice): warm-up strategies that keep query groups fast even when writes happen elsewhere.
+- [MPP Architecture](./mpp): the shared-nothing execution model — every BE you add brings another fragment-instance slot.
diff --git a/docs-next/key-features/condition-cache.mdx b/docs-next/key-features/condition-cache.mdx
index c1b648217e4..1c585a24038 100644
--- a/docs-next/key-features/condition-cache.mdx
+++ b/docs-next/key-features/condition-cache.mdx
@@ -19,22 +19,22 @@ featureCard:
- caching
---
-> **TL;DR** Condition Cache stores the result of a `WHERE` clause on each segment as a compressed bit vector keyed by `(predicate digest, key range)`. The next query that uses the same filter on the same segment skips the granules the bit vector already says cannot match, even when the SELECT list, GROUP BY, or aggregation is different. Five dashboard panels that all share `WHERE region = 'ASIA' AND order_date >= '2023-01-01'` evaluate that filter once per segment and reuse it across eve [...]
+> **TL;DR** Apache Doris Condition Cache stores the result of a `WHERE` clause on each segment as a compressed bit vector keyed by `(predicate digest, key range)`. The next query that uses the same filter on the same segment skips the granules the bit vector already says cannot match, even when the SELECT list, GROUP BY, or aggregation is different. Five dashboard panels that all share `WHERE region = 'ASIA' AND order_date >= '2023-01-01'` evaluate that filter once per segment and reuse [...]
-## Why it matters {#why}
+## Why use the Apache Doris Condition Cache? {#why}
-Most analytical workloads filter a lot more than they aggregate. A dashboard built on a single `orders` table will have ten panels that all share `WHERE region = ? AND order_date >= ?` and differ only in the SELECT list. An application backend reissues the same predicate every few seconds with a new aggregation. A T+1 report runs different aggregations against [partitions](./partitioning-and-bucketing) that have not changed since last night.
+The Apache Doris Condition Cache reuses predicate evaluation across queries that share a `WHERE` clause but differ in everything else, plugging the gap between SQL Cache and Query Cache. Most analytical workloads filter a lot more than they aggregate. A dashboard built on a single `orders` table will have ten panels that all share `WHERE region = ? AND order_date >= ?` and differ only in the SELECT list. An application backend reissues the same predicate every few seconds with a new aggr [...]
The cluster pays the predicate evaluation cost on every one of those queries: read the column, decode it, run the comparison, build the row mask, then carry on with the rest of the plan.
SQL Cache helps when the SQL text is byte-identical and gives up the moment the SELECT list shifts. Query Cache helps for aggregation queries that share whole tablets. Neither helps the case where the WHERE is shared but the SELECT list is not.
-Condition Cache fits between them. It caches the predicate evaluation step at the segment level, separate from whatever the surrounding query does with the rows.
+The Apache Doris Condition Cache fits between them, caching the predicate evaluation step at the segment level, separate from whatever the surrounding query does with the rows.
-## What it is {#what}
+## What is the Apache Doris Condition Cache? {#what}
-Condition Cache is a BE-side cache of predicate evaluation results, stored per segment as a granule-level bit vector. When a scan starts on a segment, the BE computes a 64-bit digest from the predicate expression and the segment's key range, then probes the cache. On a hit, it skips every granule whose bit is `0` and only scans the granules whose bit is `1`. On a miss, it runs the predicate as usual and writes the resulting bit vector back to the cache.
+The Apache Doris Condition Cache is a BE-side cache of predicate evaluation results, stored per segment as a granule-level bit vector. When a scan starts on a segment, the BE computes a 64-bit digest from the predicate expression and the segment's key range, then probes the cache. On a hit, it skips every granule whose bit is `0` and only scans the granules whose bit is `1`. On a miss, it runs the predicate as usual and writes the resulting bit vector back to the cache.
The cache is independent of projections, aggregations, joins, and sort. Two queries with identical WHERE clauses share entries even when one runs `SELECT *` and the other runs `SELECT COUNT(DISTINCT customer_id)`.
@@ -46,13 +46,15 @@ The cache is independent of projections, aggregations, joins, and sort. Two quer
- **Predicate digest**: a 64-bit hash of the conjunctive predicate plus the segment's key range. Semantically equivalent predicates produce the same digest.
- **`condition_cache_limit`**: per-BE memory cap for the cache, in MB. Defaults to `1024`. Lives in `be.conf`.
-## How it works {#how}
+## How does the Apache Doris Condition Cache work? {#how}
+
+The Apache Doris Condition Cache hashes each conjunctive predicate plus the segment's key range into a 64-bit digest, looks the digest up in a per-BE LRU, and skips every granule whose stored bit is `0`.
1. **Compile the digest (BE).** When a scan operator starts on a segment, it normalizes its conjunctive predicates and the segment's key range into a 64-bit digest. Different SELECT lists, GROUP BYs, and aggregations all produce the same digest because none of those affect the predicate.
2. **Probe per segment.** The BE looks up `(digest, segment_id)` in a sharded LRU cache. If the segment carries a delete marker, or if a TopN runtime filter built the predicate at execution time, the BE skips the lookup and falls back to a normal scan.
3. **On hit.** The scan reads the bit vector, skips every granule whose bit is `0`, and only decodes the columns for the granules whose bit is `1`. A row-level filter pass on those granules confirms the matches.
4. **On miss.** The scan evaluates the predicate normally, builds the bit vector as it goes, and writes the result back keyed on `(digest, segment_id)`. Subsequent queries with the same digest land on a hit.
-5. **Invalidation by immutability.** Segments do not mutate. A new INSERT or compaction produces a new segment with a new ID, and the old cache entry ages out of the LRU on its own. Doris does not need an explicit invalidation step.
+5. **Invalidation by immutability.** Segments do not mutate. A new INSERT or compaction produces a new segment with a new ID, and the old cache entry ages out of the LRU on its own. Apache Doris does not need an explicit invalidation step.
## Quick start {#quick-start}
@@ -80,7 +82,9 @@ ConditionCacheFilteredRows: 18,400,219
The first run scans every segment, evaluates the predicate, and writes one bit vector per segment. The second run shares the WHERE so the digest matches; every segment hits the cache, and the BE skips the granules already known to be empty for `region = 'ASIA' AND order_date >= '2023-01-01'`.
-## When to use it {#when}
+## When should you use the Apache Doris Condition Cache? {#when}
+
+Turn the Apache Doris Condition Cache on for read-heavy workloads where many queries reuse the same `WHERE` clause and the table is not under continuous write pressure.
**Good fit**
@@ -106,3 +110,4 @@ The first run scans every segment, evaluates the predicate, and writes one bit v
- [Caching in Doris: choosing the right strategy](../query-acceleration/caching-intro)
- [Data Cache & Page Cache](./data-cache-page-cache): the storage-tier caches that sit below Condition Cache for the bytes and decoded pages themselves.
- [Metadata Cache](./metadata-cache): the external-catalog metadata cache that complements the data-side caches.
+- [MPP Architecture](./mpp): the execution model whose plan the condition cache hooks into to skip already-evaluated predicates.
diff --git a/docs-next/key-features/data-cache-page-cache.mdx b/docs-next/key-features/data-cache-page-cache.mdx
index b2e88ebea21..762702f23d1 100644
--- a/docs-next/key-features/data-cache-page-cache.mdx
+++ b/docs-next/key-features/data-cache-page-cache.mdx
@@ -20,12 +20,12 @@ featureCard:
- performance
---
-> **TL;DR** Doris caches storage at two layers. The Data Cache keeps bytes from S3 or HDFS on a BE's local disk so repeat scans skip the network. The Page Cache keeps decompressed column pages in memory so repeat scans also skip decompression and decoding. The first run after a deploy is slow; the next run is local-disk fast; the run after that is RAM-fast. `WARM UP` pre-populates both.
+> **TL;DR** Apache Doris caches lakehouse storage at two layers. The Data Cache keeps bytes from S3 or HDFS on a BE's local disk (1 MB blocks) so repeat scans skip the network. The Page Cache keeps decompressed column pages in memory so repeat scans also skip decompression and decoding. `WARM UP` pre-populates both, turning a cold deploy into a local-disk-fast second run and a RAM-fast third run.
-## Why it matters {#why}
+## Why use the data cache and page cache in Apache Doris? {#why}
-A query against a [Hive or Iceberg table](./multi-catalog) on S3 pays for every byte three times: once to fetch it (HTTP round trip per ranged GET), once to decompress it (snappy, zstd, gzip), and once to decode it (RLE, dictionary, plain). The fetch is usually the worst offender. S3 GET tail latency runs 30 ms to 500 ms; a [Parquet scan](./parquet-reader-optimization) that issues thousands of ranged reads can spend most of its wall clock waiting for HTTP, before any CPU work starts.
+The Apache Doris Data Cache and Page Cache cut lakehouse query latency by removing repeated network fetches and repeated decompression on hot data. A query against a [Hive or Iceberg table](./multi-catalog) on S3 pays for every byte three times: once to fetch it (HTTP round trip per ranged GET), once to decompress it (snappy, zstd, gzip), and once to decode it (RLE, dictionary, plain). The fetch is usually the worst offender. S3 GET tail latency runs 30 ms to 500 ms; a [Parquet scan](./p [...]
A few common patterns make this hurt:
@@ -35,9 +35,11 @@ A few common patterns make this hurt:
The two caches sit at different layers. A Data Cache hit removes the network. A Page Cache hit removes the network and the decompression, but only after some earlier read populated it.
-## What it is {#what}
+## What are the Apache Doris Data Cache and Page Cache? {#what}
-**Data Cache** is a per-BE local-disk cache for remote file bytes. When a Doris BE reads a Parquet or ORC file from a Hive, Iceberg, Hudi, or Paimon table, the bytes are pulled in 1 MB blocks and written to a configured local directory. Subsequent reads of the same offset hit the disk instead of S3.
+The Apache Doris Data Cache and Page Cache are two complementary storage caches: a per-BE local-disk cache for remote file bytes and an in-memory cache for column pages.
+
+**Data Cache** is a per-BE local-disk cache for remote file bytes. When an Apache Doris BE reads a Parquet or ORC file from a Hive, Iceberg, Hudi, or Paimon table, the bytes are pulled in 1 MB blocks and written to a configured local directory. Subsequent reads of the same offset hit the disk instead of S3.
**Page Cache** is an in-memory cache for column pages. There are two flavors. The **Storage Page Cache** caches decompressed pages from internal Doris segments. The **Parquet Page Cache** (4.1+) does the same for external Parquet files, and is smart about whether to keep the page compressed or decompressed depending on its compression ratio.
@@ -49,9 +51,9 @@ The two caches sit at different layers. A Data Cache hit removes the network. A
- **`WARM UP`**: SQL syntax that pre-populates either cache by running a synthetic scan in the background.
- **`Admission control`**: a policy that decides whether a given query is even allowed to write into the Data Cache, used to keep one-shot scans from polluting it.
-## How it works {#how}
+## How do the Apache Doris Data Cache and Page Cache work? {#how}
-A scan flows through the caches in order, top to bottom.
+Apache Doris funnels every lakehouse scan through the Page Cache first, then the Data Cache, and only falls back to S3/HDFS on a miss. A scan flows through the caches in order, top to bottom.
1. **Page Cache lookup (memory).** Before issuing any IO, the reader checks whether the column page it needs is already in the Page Cache, keyed by file path, file size, and offset. A hit returns ready-to-decode (or already-decoded) bytes; the reader skips both IO and decompression.
2. **Data Cache lookup (local disk).** On a Page Cache miss, the file reader checks the Data Cache for the 1 MB block covering the requested byte range. A hit reads from local NVMe at GB/s; the reader still has to decompress and decode, but the network is bypassed.
@@ -96,7 +98,9 @@ VFileScanNode
The first run populated the Data Cache from S3. The second run answered the query without leaving the BE's local disk, and decompressed pages were served from RAM.
-## When to use it {#when}
+## When should you use the Apache Doris Data Cache and Page Cache? {#when}
+
+Use the Apache Doris Data Cache and Page Cache for repeat queries on hot lakehouse partitions, high-concurrency dashboards, and compute-group elasticity. Skip them when working sets exceed the cache, on one-shot ETL, or on mechanical disks where local IO is no faster than the remote source.
**Good fit**
@@ -122,3 +126,4 @@ The first run populated the Data Cache from S3. The second run answered the quer
- [Parquet Reader Optimization](./parquet-reader-optimization): the reader that decides which bytes to ask for in the first place. Caches and pruning compose.
- [Query Cache](./query-cache) and [Condition Cache](./condition-cache): result-level and predicate-level caches that sit above the storage caches.
- [WARM UP statement](../sql-manual/sql-statements/cluster-management/storage-management/WARM-UP): the SQL surface for pre-populating caches across compute groups.
+- [Iceberg](./iceberg): a primary consumer of the page cache — Iceberg scans read remote Parquet over S3-style object storage.
diff --git a/docs-next/key-features/data-compaction.mdx b/docs-next/key-features/data-compaction.mdx
index 3bb9e393e18..3e5bd297028 100644
--- a/docs-next/key-features/data-compaction.mdx
+++ b/docs-next/key-features/data-compaction.mdx
@@ -19,22 +19,22 @@ featureCard:
- performance
---
-> **TL;DR** Every load into Doris produces a new immutable rowset. Compaction merges those rowsets in the background so queries don't have to walk through hundreds of versions. Two policies (`size_based` and `time_series`), plus [vertical](./vertical-compaction) and segment compaction, let you tune the write-amplification vs. read-amplification trade-off per table.
+> **TL;DR** Every load into Apache Doris produces a new immutable rowset. Apache Doris compaction merges those rowsets in the background so queries don't have to walk through hundreds of versions. Two policies (`size_based` and `time_series`), plus [vertical](./vertical-compaction) and segment compaction, let you tune the write-amplification vs. read-amplification trade-off per table.
-## Why it matters {#why}
+## Why use Apache Doris compaction? {#why}
-Doris stores data the way an LSM tree does: a write never updates a file in place. It lands as a brand-new rowset, and the rowset count on a tablet keeps climbing until something cleans up. High-frequency ingest paths like [Stream Load](./stream-load) and [Group Commit](./group-commit) make that climb especially fast. Without compaction:
+Apache Doris compaction keeps queries fast under high-frequency loads by merging the immutable rowsets each write produces, before the version pile-up turns into the dreaded `-235` ingest failure. Apache Doris stores data the way an LSM tree does: a write never updates a file in place. It lands as a brand-new rowset, and the rowset count on a tablet keeps climbing until something cleans up. High-frequency ingest paths like [Stream Load](./stream-load) and [Group Commit](./group-commit) m [...]
- Once a tablet's version count exceeds `max_tablet_version_num` (default 500), loads fail with error -235. This is one of the most common production incidents on the Doris user list.
- Every query has to merge all of a tablet's rowsets on the fly. Going from 5 rowsets to 50 turns a one-box lookup into a fifty-box one.
- Deleted rows linger as tombstones, and small files cost more in metadata than the data they hold.
-Compaction is the background process that handles all three. Doris's policies exist because doing it well also means not burning so much CPU that ingestion slows down.
+Apache Doris compaction is the background process that handles all three. The shipped policies exist because doing it well also means not burning so much CPU that ingestion slows down.
-## What it is {#what}
+## What is Apache Doris compaction? {#what}
-Compaction is a tablet-local procedure that picks a window of rowsets, merges their rows, applies deletes, and writes a single replacement rowset. The Backend (BE) runs it on a thread pool, scoring tablets by how messy they are and grabbing the worst offenders first.
+Apache Doris compaction is a tablet-local procedure that picks a window of rowsets, merges their rows, applies deletes, and writes a single replacement rowset. The Backend (BE) runs it on a thread pool, scoring tablets by how messy they are and grabbing the worst offenders first.
**Key terms**
@@ -45,7 +45,9 @@ Compaction is a tablet-local procedure that picks a window of rowsets, merges th
- **`Compaction policy`**: per-table choice of `size_based` (default) or `time_series`. Picks which rowsets to merge and when.
- **`Vertical compaction`**: column-group merging for wide tables. Cuts memory use by ~10x compared with row-wise merging.
-## How it works {#how}
+## How does Apache Doris compaction work? {#how}
+
+Apache Doris compaction scores every tablet by version pressure, picks rowsets per the active policy, and merges them on a BE thread pool while the original files stay readable for in-flight queries.
1. **Score every tablet.** The BE keeps a running compaction score for each tablet based on how many rowsets sit above the cumulative point. Tablets with higher scores get scheduled first.
2. **Pick the rowsets.** The active policy decides the window. `size_based` groups by power-of-two sizes; `time_series` waits until rowsets in a partition cross a goal size, file count, or age threshold, then takes them in one bite.
@@ -84,7 +86,9 @@ Query OK, 0 rows affected
You've created a partitioned log table that uses the time-series policy: small rowsets get merged once they cross 1 GB total, hit 2000 files, or sit for an hour. The `ADMIN COMPACT` line forces a manual cumulative pass on the active partition, which is useful for benchmarks and post-load cleanup. Watch progress with `SELECT * FROM information_schema.doris_be_compaction_tasks WHERE TABLE_ID = ...`.
-## When to use it {#when}
+## When should you use Apache Doris compaction? {#when}
+
+Apache Doris compaction runs on every table by default; the real choice is which policy to set and when to override the defaults.
**Good fit**
@@ -112,3 +116,4 @@ You've created a partitioned log table that uses the time-series policy: small r
- [Compaction tuning: vertical, segment, single-replica, time-series](../admin-manual/trouble-shooting/compaction)
- [`ADMIN COMPACT TABLE` syntax reference](../sql-manual/sql-statements/table-and-view/data-and-status-management/COMPACT-TABLE)
- [Understanding Data Compaction in 3 Minutes (blog)](https://doris.apache.org/blog/Understanding-Data-Compaction-in-3-Minutes/)
+- [Unique Key](./unique-key): the table model where compaction reclaims rows marked in the per-rowset delete bitmap.
diff --git a/docs-next/key-features/data-lineage.mdx b/docs-next/key-features/data-lineage.mdx
index c119d8559d8..c4cde5dc35d 100644
--- a/docs-next/key-features/data-lineage.mdx
+++ b/docs-next/key-features/data-lineage.mdx
@@ -20,12 +20,12 @@ featureCard:
- metadata
---
-> **TL;DR** Doris extracts column-level lineage from the analyzed plan of every `INSERT`, `INSERT OVERWRITE`, and `CREATE TABLE AS SELECT`, including JOIN, FILTER, GROUP BY, WINDOW, and CASE WHEN dependencies. Events flow through a SPI to whichever lineage backend you ship: a custom emitter, an OpenLineage receiver, Apache Atlas, DataHub. For table-level lineage from ad-hoc queries, the `audit_log` system table records the source tables and chosen materialized views per query.
+> **TL;DR** Apache Doris extracts column-level lineage from the analyzed Nereids plan of every `INSERT`, `INSERT OVERWRITE`, and `CREATE TABLE AS SELECT`, including JOIN, FILTER, GROUP BY, WINDOW, and CASE WHEN dependencies. Events flow through a SPI to whichever lineage backend you ship: a custom emitter, an OpenLineage receiver, Apache Atlas, or DataHub. For table-level lineage from ad-hoc queries, the `audit_log` system table records the source tables and chosen materialized views per query.
-## Why it matters {#why}
+## Why use Apache Doris data lineage? {#why}
-You change a column type on a base table, and four hours later a dashboard starts returning nulls. Or someone asks where the value in `tgt_region_revenue.revenue` actually comes from, and the honest answer is "somewhere in three years of `INSERT` statements." The dependencies are real and the SQL is in the system, but nobody has stitched it into a graph you can query. Writes that traverse [multi-catalog](./multi-catalog) federation or land into [incremental materialized views](./incremen [...]
+Apache Doris data lineage hands you the column-level dependencies the planner already computed, so impact analysis and audit questions can be answered without writing a SQL parser inside your governance tool. You change a column type on a base table, and four hours later a dashboard starts returning nulls. Or someone asks where the value in `tgt_region_revenue.revenue` actually comes from, and the honest answer is "somewhere in three years of `INSERT` statements." The dependencies are re [...]
Writing your own SQL parser inside the governance tool is the wrong fix. The database already parsed the SQL. What you want is for it to hand you the dependencies it found, at column granularity, with the transformation that produced each output column attached. Then the lineage matches what ran, not a static reading of the DDL.
@@ -35,9 +35,9 @@ Typical questions this answers:
- Where did this number on the dashboard come from?
- Which regulated tables did this user touch last week?
-## What it is {#what}
+## What is Apache Doris data lineage? {#what}
-Data Lineage in Doris is column-level lineage extraction built into the Nereids planner, plus a SPI that ships the extracted events to whichever governance backend you run. There is no built-in viewer. Doris is the producer; your existing tool (Atlas, DataHub, OpenMetadata, an internal service) is the consumer. For table-level lineage derived from query history, the `audit_log` table already records source tables and any matched materialized views per query, so you can build a coarse gra [...]
+Apache Doris data lineage is column-level lineage extraction built into the Nereids planner, plus a SPI that ships the extracted events to whichever governance backend you run. There is no built-in viewer. Apache Doris is the producer; your existing tool (Atlas, DataHub, OpenMetadata, an internal service) is the consumer. For table-level lineage derived from query history, the `audit_log` table already records source tables and any matched materialized views per query, so you can build a [...]
**Key terms**
@@ -47,7 +47,9 @@ Data Lineage in Doris is column-level lineage extraction built into the Nereids
- **`audit_log.queried_tables_and_views`**: an `ARRAY<STRING>` column in `__internal_schema.audit_log` that lists the catalog-qualified tables touched by each query.
- **`audit_log.chosen_m_views`**: companion column listing the materialized views the optimizer rewrote into.
-## How it works {#how}
+## How does Apache Doris data lineage work? {#how}
+
+Apache Doris hooks the Nereids planner before optimization, walks the analyzed plan to resolve each output column back to its sources, records the JOIN/FILTER/GROUP BY/WINDOW/CASE WHEN dependencies, and ships the event off the query thread to every registered `LineagePlugin`.
1. **Hook the planner.** Each write command (`INSERT INTO`, `INSERT OVERWRITE`, `CTAS`) registers an analyze-plan hook. The hook grabs the analyzed `LogicalPlan` before optimization, so column references still carry their source slots and lineage matches the SQL the user wrote.
2. **Extract direct lineage.** `LineageInfoExtractor` walks the plan and resolves every output column back to its source expressions. A pure column reference becomes `IDENTITY`. A scalar expression becomes `TRANSFORMATION`. An aggregate function becomes `AGGREGATION`. CTE consumer slots get pre-resolved to producer slots so the chain does not stop at a `WITH` clause boundary.
@@ -88,7 +90,9 @@ ORDER BY time DESC LIMIT 5;
The row holds catalog-qualified table names and any MV the optimizer rewrote into. That is enough to draw a table-level graph for ad-hoc queries. For column-level lineage on writes, implement `LineagePlugin`, drop the jar in `plugin_dir/lineage/`, and add the plugin name to `activate_lineage_plugin`. From then on, every `INSERT` and `CTAS` produces a `LineageInfo` event.
-## When to use it {#when}
+## When should you use Apache Doris data lineage? {#when}
+
+Apache Doris data lineage fits impact analysis, compliance trails, MV governance, and feeding external catalogs (Atlas, DataHub, OpenMetadata) with column-level events; it is not a built-in lineage browser and does not cover pure `SELECT` traffic at column granularity.
**Good fit**
@@ -115,3 +119,4 @@ The row holds catalog-qualified table names and any MV the optimizer rewrote int
- [OpenLineage specification](https://openlineage.io/docs/spec/overview/): the open standard for column-level lineage events; a useful target schema for a `LineagePlugin` implementation.
- [Managing Lake Tables](./managing-lake-table): writes against Iceberg, Paimon, and Hive produce lineage events too; this card covers the write paths involved.
- [Apache Atlas overview](https://atlas.apache.org/): a common downstream sink that accepts OpenLineage today and renders the column-level graph.
+- [Iceberg](./iceberg): an external table format whose snapshot history feeds into the lineage graph alongside Doris internal tables.
diff --git a/docs-next/key-features/data-model.mdx b/docs-next/key-features/data-model.mdx
index 47f59a4000c..0c3f322723f 100644
--- a/docs-next/key-features/data-model.mdx
+++ b/docs-next/key-features/data-model.mdx
@@ -19,12 +19,12 @@ featureCard:
- table-design
---
-> **TL;DR** Every Doris OLAP table picks one of three models when you create it: Duplicate Key keeps every row, Unique Key keeps the latest row per primary key, and Aggregate Key folds rows together by [per-column functions like `SUM`](./preaggregation-and-rollup). The `KEY(...)` clause names the columns Doris sorts on, and in Unique and Aggregate models those same columns also enforce uniqueness. The model is fixed at table creation, so the choice up front decides what writes, updates, [...]
+> **TL;DR** Every Apache Doris OLAP table picks one of three models when you create it: Duplicate Key keeps every row, Unique Key keeps the latest row per primary key, and Aggregate Key folds rows together by [per-column functions like `SUM`](./preaggregation-and-rollup). The `KEY(...)` clause names the columns Apache Doris sorts on, and in Unique and Aggregate models those same columns also enforce uniqueness. The model is fixed at `CREATE TABLE` time, so the choice up front decides wha [...]
-## Why it matters {#why}
+## Why use the Apache Doris data model? {#why}
-The first decision in any Doris schema is which table model to use, and it is the one decision you cannot reverse. `ALTER TABLE` can add columns, drop columns, [repartition](./partitioning-and-bucketing), and swap encodings. It cannot turn a Duplicate table into a Unique table. If you get the model wrong, the fix is `CREATE TABLE` and a full reload.
+The Apache Doris data model is the first decision in any schema, and it is the one decision you cannot reverse. `ALTER TABLE` can add columns, drop columns, [repartition](./partitioning-and-bucketing), and swap encodings. It cannot turn a Duplicate table into a Unique table. If you get the model wrong, the fix is `CREATE TABLE` and a full reload.
- A logs table created as Aggregate loses the raw rows and cannot answer detail queries.
- An orders table created as Duplicate rejects `UPDATE`, so a CDC pipeline has nowhere to land row changes.
@@ -32,9 +32,9 @@ The first decision in any Doris schema is which table model to use, and it is th
The model decides whether duplicate keys are allowed, what `UPDATE` and `DELETE` mean, and what the storage engine does with rows that share a key. Pick by what the data is (raw events, current state, or pre-aggregated facts), not by the shape of the dashboard you are building this week.
-## What it is {#what}
+## What is the Apache Doris data model? {#what}
-A Doris table model is the contract the storage engine enforces on rows that share the same key columns. Doris ships three: `DUPLICATE KEY`, `UNIQUE KEY`, and `AGGREGATE KEY`. The FE catalog calls them `DUP_KEYS`, `UNIQUE_KEYS`, and `AGG_KEYS` (see `KeysType.java`). The clause sits right after the column list in `CREATE TABLE` and names the key columns:
+The Apache Doris data model is the contract the storage engine enforces on rows that share the same key columns. Apache Doris ships three: `DUPLICATE KEY`, `UNIQUE KEY`, and `AGGREGATE KEY`. The FE catalog calls them `DUP_KEYS`, `UNIQUE_KEYS`, and `AGG_KEYS` (see `KeysType.java`). The clause sits right after the column list in `CREATE TABLE` and names the key columns:
```sql
... DUPLICATE KEY(log_time, log_type)
@@ -51,12 +51,12 @@ A Doris table model is the contract the storage engine enforces on rows that sha
- **Unique Key model**: one row per key. New writes overwrite older rows with the same key. Supports SQL `UPDATE`, `DELETE`, and partial-column upserts via Merge-on-Write.
- **Aggregate Key model**: rows that share a key are merged using each value column's declared function. The merged row is what you read.
-## How it works {#how}
+## How does the Apache Doris data model work? {#how}
-Same DDL shape, three different per-row contracts.
+The Apache Doris data model enforces one of three per-row contracts at load time and again at compaction time, using the same DDL shape.
-1. **Pick a model.** The `DUPLICATE KEY`, `UNIQUE KEY`, or `AGGREGATE KEY` clause sets the `KeysType` for the table. Doris validates the rest of the DDL against it: Aggregate value columns must declare a function, Unique tables pick up the merge-on-write property, and Duplicate tables reject aggregation functions outright.
-2. **Order columns.** Key columns must come before any value column in the column list. Doris sorts data inside each tablet by the key columns and builds a sparse [prefix index](../table-design/index/prefix-index) over the first 36 bytes of the sort key. That index is what makes range and equality predicates on the leading key columns fast.
+1. **Pick a model.** The `DUPLICATE KEY`, `UNIQUE KEY`, or `AGGREGATE KEY` clause sets the `KeysType` for the table. Apache Doris validates the rest of the DDL against it: Aggregate value columns must declare a function, Unique tables pick up the merge-on-write property, and Duplicate tables reject aggregation functions outright.
+2. **Order columns.** Key columns must come before any value column in the column list. Apache Doris sorts data inside each tablet by the key columns and builds a sparse [prefix index](../table-design/index/prefix-index) over the first 36 bytes of the sort key. That index is what makes range and equality predicates on the leading key columns fast.
3. **Apply the contract on load.** Duplicate appends. Unique looks up the existing row by key, flips it on in a per-rowset delete bitmap, and writes the new row. Aggregate merges incoming rows against existing ones using each value column's function (`SUM`, `MAX`, `REPLACE`, and so on).
4. **Apply it again at compaction.** Background compaction merges rowsets using the same rules. Duplicate concatenates. Unique drops rows the delete bitmap covers. Aggregate folds duplicates again with the value-column functions.
5. **Read.** Queries see the model's view of the data: every appended row in Duplicate, the latest row per key in Unique, one merged row per key in Aggregate.
@@ -95,7 +95,9 @@ SHOW CREATE TABLE ad_stats\G -- prints "AGGREGATE KEY(`dt`, `ad_id`)"
Identical column lists, but the storage engine treats writes very differently. Insert the same row into `logs` twice and you get two rows. Do the same on `orders` and you get one. Insert two rows into `ad_stats` with matching `(dt, ad_id)` and the `clicks` column sums.
-## When to use it {#when}
+## When should you use the Apache Doris data model? {#when}
+
+Pick the Apache Doris data model by what the data is (raw events, current state, or pre-aggregated facts), not by the shape of the dashboard you are building this week.
**Good fit**
@@ -124,3 +126,4 @@ Identical column lists, but the storage engine treats writes very differently. I
- [Data Update and Delete](./data-update-delete): SQL UPDATE, DELETE, partial-column upsert, and the Merge-on-Write internals that the Unique Key model unlocks.
- [Preaggregation and Rollup](./preaggregation-and-rollup): the Aggregate Key model in depth, plus ROLLUP and sync materialized views.
- [Partitioning and Bucketing](./partitioning-and-bucketing): how the same key columns drive partition pruning and tablet placement.
+- [Unique Key](./unique-key): the primary key (one-row-per-key) variant alongside Duplicate and Aggregate — pick this when sources do upserts.
diff --git a/docs-next/key-features/data-pruning.mdx b/docs-next/key-features/data-pruning.mdx
index 12709f30486..eceb7d6c915 100644
--- a/docs-next/key-features/data-pruning.mdx
+++ b/docs-next/key-features/data-pruning.mdx
@@ -19,12 +19,12 @@ featureCard:
- performance
---
-> **TL;DR** A query in Doris doesn't really scan a table; it scans whatever's left after pruning. [Partition keys, distribution keys](./partitioning-and-bucketing), ZoneMap min/max statistics, BloomFilter and NGram BloomFilter indexes each peel off another layer of data the engine doesn't have to read. Most of the win is automatic; you control it through DDL choices and watch it land in EXPLAIN output.
+> **TL;DR** Apache Doris data pruning means a query never really scans a table — it scans whatever survives a stack of pruning layers. [Partition keys, distribution keys](./partitioning-and-bucketing), ZoneMap min/max statistics, BloomFilter and NGram BloomFilter indexes each peel off another layer of data the engine doesn't have to read. Most of the win is automatic; the operator controls it through DDL choices and verifies it in EXPLAIN output.
-## Why it matters {#why}
+## Why use data pruning in Apache Doris? {#why}
-The fastest way to make an OLAP query faster is to read less. A wide events table at 50 GB per day stretches into petabytes over a year, and most queries care about a sliver: one day, one user, one product. Without pruning, the planner fans the query out across every partition and every tablet, and the executor walks every page from disk to evaluate `WHERE`.
+Apache Doris data pruning lets a query skip most of the table before reading it, turning multi-terabyte scans into reads of a few partitions or tablets. The fastest way to make an OLAP query faster is to read less. A wide events table at 50 GB per day stretches into petabytes over a year, and most queries care about a sliver: one day, one user, one product. Without pruning, the planner fans the query out across every partition and every tablet, and the executor walks every page from disk [...]
This shows up as three concrete pains:
@@ -32,11 +32,11 @@ This shows up as three concrete pains:
- Hash-distributed point lookups (`WHERE user_id = 12345`) become full-tablet scans when the optimizer can't tie the predicate to the bucket key.
- High-cardinality filters on non-key columns (`WHERE phone = '...'`) read every segment because there's no statistic to rule pages out.
-Doris addresses each of these with a different pruning layer, and the layers compose: the planner trims partitions and tablets before the scan starts, then the BE skips segments and pages inside whatever survived.
+Apache Doris addresses each of these with a different pruning layer, and the layers compose: the planner trims partitions and tablets before the scan starts, then the BE skips segments and pages inside whatever survived.
-## What it is {#what}
+## What is Apache Doris data pruning? {#what}
-Data pruning is the cumulative result of every step in the read path that lets Doris skip data without reading it. The Frontend planner does the coarse work using table metadata: partitions and tablets. The Backend does the fine work during scan: ZoneMap min/max, BloomFilter, NGram BloomFilter, and the inverted index decide whether a segment, a column page, or even a row range can be skipped.
+Apache Doris data pruning is the cumulative result of every step in the read path that lets the engine skip data without reading it, spanning partitions, tablets, segments, and column pages. The Frontend planner does the coarse work using table metadata: partitions and tablets. The Backend does the fine work during scan: ZoneMap min/max, BloomFilter, NGram BloomFilter, and the inverted index decide whether a segment, a column page, or even a row range can be skipped.
**Key terms**
@@ -47,9 +47,9 @@ Data pruning is the cumulative result of every step in the read path that lets D
- **`NGram BloomFilter`**: opt-in index that turns substring `LIKE '%abc%'` predicates into bloom probes on n-grams.
- **`Runtime filter`**: a join-time filter generated from one side of a join and pushed down to prune the other side.
-## How it works {#how}
+## How does Apache Doris data pruning work? {#how}
-The five layers run roughly top-down, with each one shrinking the input the next layer has to consider.
+Apache Doris data pruning works in five top-down layers — partition, tablet, segment/page, index probe, and runtime filter — with each layer shrinking the input the next layer has to consider.
1. **Partition pruning (FE).** The Nereids planner runs `PruneOlapScanPartition`, matches `WHERE` predicates against the table's range or list partition tree, and produces the surviving set. EXPLAIN shows it as `partitions=2/4 (p2,p3)`.
2. **Tablet pruning (FE).** `PruneOlapScanTablet` extracts equality predicates on the bucketing column and asks `HashDistributionPruner` which buckets they hash to. EXPLAIN shows `tablets=1/32`.
@@ -90,7 +90,9 @@ WHERE ts >= '2026-05-08' AND user_id = 42 AND action = 'click';
The planner kept one partition out of two (date range) and one tablet out of sixteen (hash on `user_id`). At read time, the BE will use ZoneMap on `ts` plus the BloomFilter on `action` to skip pages inside that surviving tablet. Four pruning layers handle three predicates without any manual tuning.
-## When to use it {#when}
+## When should you use Apache Doris data pruning? {#when}
+
+Apache Doris data pruning is always on — the question is which layers your DDL and predicates actually activate. Time-bounded analytical queries, high-concurrency point lookups, substring search on long text columns, and star-schema joins benefit the most; misapplied BloomFilters and catch-all partitions are the common anti-patterns.
**Good fit**
@@ -116,3 +118,4 @@ The planner kept one partition out of two (date range) and one tablet out of six
- [Partitioning and Bucketing](./partitioning-and-bucketing): the DDL choices that decide which partitions and tablets the planner can prune in the first place.
- [Inverted Index](./inverted-index): the index-probe layer that turns text and high-cardinality equality predicates into posting-list lookups.
- [Deep dive: Data Pruning in Apache Doris (blog)](https://doris.apache.org/blog/data-pruning-250908/)
+- [MPP Architecture](./mpp): where pruning runs — partition/bucket prune happens in the Nereids planner before fragments ship.
diff --git a/docs-next/key-features/data-update-delete.mdx b/docs-next/key-features/data-update-delete.mdx
index cf7a495ce90..3bee6679bc7 100644
--- a/docs-next/key-features/data-update-delete.mdx
+++ b/docs-next/key-features/data-update-delete.mdx
@@ -19,43 +19,45 @@ featureCard:
- delete
---
-> **TL;DR** Doris speaks SQL `UPDATE` and `DELETE` like a transactional database, plus partial-column upserts from [streaming loads](./stream-load), predicate-based bulk deletes, and atomic partition swaps. On Unique Key tables, the [Merge-on-Write engine](./data-model) keeps query latency low by marking deleted rows in a per-rowset bitmap at write time instead of merging versions at read time.
+> **TL;DR** Apache Doris supports SQL `UPDATE` and `DELETE` like a transactional database, with partial-column upserts from [streaming loads](./stream-load), predicate-based bulk deletes, and atomic partition swaps. On Unique Key tables, the [Merge-on-Write engine](./data-model) keeps query latency low by marking deleted rows in a per-rowset bitmap at write time instead of merging versions at read time. Merge-on-Write became the Unique Key default in version 2.1.
-## Why it matters {#why}
+## Why use data update and delete in Apache Doris? {#why}
-Most analytical engines were built around the idea that warehouses are append-only. Then CDC happened, and so did GDPR, and so did the boring fact that production data has bugs. If your warehouse can't fix a single bad row without rewriting a partition, every late-arriving correction turns into an ops ticket.
+Apache Doris data update and delete brings transactional row-level mutations to an analytical engine, so CDC, GDPR erasure, and one-off corrections do not turn into partition rewrites. Most analytical engines were built around the idea that warehouses are append-only. Then CDC happened, and so did GDPR, and so did the boring fact that production data has bugs. If your warehouse can't fix a single bad row without rewriting a partition, every late-arriving correction turns into an ops ticket.
- A [Flink CDC job](./kafka-cdc-integration) watches MySQL and needs every row update to land within seconds.
- A user requests deletion under GDPR. You need it gone from the warehouse, not "gone after the next compaction."
- One report shows the wrong number because a single row in a 10-billion-row fact table has a typo.
-Doris handles all three with first-class SQL: `UPDATE`, `DELETE FROM ... WHERE`, partial-column upserts from Stream Load, and `TRUNCATE` or atomic partition replace for bulk cleanup. None of them require the table to be small, and none rely on you to schedule a rewrite job afterward.
+Apache Doris handles all three with first-class SQL: `UPDATE`, `DELETE FROM ... WHERE`, partial-column upserts from Stream Load, and `TRUNCATE` or atomic partition replace for bulk cleanup. None of them require the table to be small, and none rely on you to schedule a rewrite job afterward.
-## What it is {#what}
+## What is Apache Doris data update and delete? {#what}
-Doris exposes four user-facing ways to change data after it lands: row-level `UPDATE` SQL on Unique Key tables, predicate-based `DELETE FROM ... WHERE` on any table, partial-column upserts driven by ingestion (the `partial_columns: true` Stream Load header or the matching session variable), and partition-level operations like `TRUNCATE PARTITION` and atomic temp-partition replacement. Underneath, the Unique Key model uses Merge-on-Write (MoW) so deletes and updates take effect at write t [...]
+Apache Doris data update and delete is a four-part surface — `UPDATE` SQL, predicate `DELETE`, partial-column upserts, and partition swaps — backed by the Merge-on-Write engine on Unique Key tables. Row-level `UPDATE` SQL works on Unique Key tables, predicate-based `DELETE FROM ... WHERE` works on any table, partial-column upserts are driven by ingestion (the `partial_columns: true` Stream Load header or the matching session variable), and partition-level operations include `TRUNCATE PAR [...]
**Key terms**
- **`Unique Key model`**: a table model where one row per primary key is kept. Required for `UPDATE` and for batch upserts.
-- **`Merge-on-Write (MoW)`**: at write time, Doris looks up each new key in the existing data, marks the old row in a delete bitmap, and writes the new row to a fresh rowset.
+- **`Merge-on-Write (MoW)`**: at write time, Apache Doris looks up each new key in the existing data, marks the old row in a delete bitmap, and writes the new row to a fresh rowset.
- **`Delete bitmap`**: a per-rowset bitmap of row IDs that queries should skip, replacing per-key version merging at read time.
-- **`Partial column update`**: a write that only touches some columns. Doris reads the missing values from existing rows and writes a complete row back.
+- **`Partial column update`**: a write that only touches some columns. Apache Doris reads the missing values from existing rows and writes a complete row back.
- **`Sequence column`** (`function_column.sequence_col`): a user-chosen column that tells MoW which version of a key wins when out-of-order writes arrive.
- **`__DORIS_DELETE_SIGN__`**: a hidden column you set to `1` to soft-delete a row through any load path.
-## How it works {#how}
+## How does Apache Doris data update and delete work? {#how}
+
+Apache Doris data update and delete runs in five stages on a Unique Key MoW table: plan and read, look up keys, flip delete-bitmap bits on old rows, write new rows to a fresh rowset, and publish the new version.
Take an `UPDATE` against a Unique Key MoW table:
1. **Plan and read.** The FE parses the statement, resolves the WHERE predicate, and routes a read across BE replicas. For partial-column writes, BEs additionally fetch the missing column values for each affected key.
2. **Look up keys.** Each BE consults the in-memory primary key index per rowset to find which rowset and row IDs the affected keys currently live in.
3. **Mark old rows.** Every rowset that holds an affected key has the corresponding row IDs flipped on in its delete bitmap. The old rows stay on disk until compaction sweeps them up.
-4. **Write new rows.** New values, plus any unchanged columns Doris had to read for partial updates, land in a fresh rowset with the next version number.
+4. **Write new rows.** New values, plus any unchanged columns Apache Doris had to read for partial updates, land in a fresh rowset with the next version number.
5. **Publish.** The transaction commits, the new version becomes visible, and queries from that point on filter through the merged delete bitmap. Old data is reclaimed at compaction.
-`DELETE FROM ... WHERE` follows the same flow without the write step. On Aggregate or Duplicate tables, Doris records the predicate in tablet metadata instead, and queries apply it at read time.
+`DELETE FROM ... WHERE` follows the same flow without the write step. On Aggregate or Duplicate tables, Apache Doris records the predicate in tablet metadata instead, and queries apply it at read time.
## Quick start {#quick-start}
@@ -90,11 +92,13 @@ DELETE FROM orders WHERE order_id = 2;
Order 1 was rewritten in place; order 2 is gone. The `function_column.sequence_col` property protects the table from out-of-order writes: a CDC event with an older `updated` timestamp would lose to the row that's already there, even if it lands after the newer one.
-## When to use it {#when}
+## When should you use Apache Doris data update and delete? {#when}
+
+Apache Doris data update and delete fits CDC upserts into Unique Key tables, hot-column updates on wide tables, GDPR-style row-level deletes, atomic backfill swaps, and multi-stream wide-table assembly. It is not a fit for `UPDATE` on Aggregate or Duplicate tables, single-row updates fired in a tight loop, primary-key renames, or massive predicate `DELETE`s that a partition replace could handle in one shot.
**Good fit**
-- Flink and Spark CDC pipelines that upsert into a Unique Key table, with the source primary key as the Doris key.
+- Flink and Spark CDC pipelines that upsert into a Unique Key table, with the source primary key as the Apache Doris key.
- Hot-column updates on wide tables (order status, last-seen timestamp, current balance) via partial-column Stream Load.
- GDPR-style row-level deletes by user ID, including on append-only Duplicate tables.
- Backfills that swap a corrected partition in atomically with `ALTER TABLE ... REPLACE PARTITION`.
@@ -118,3 +122,4 @@ Order 1 was rewritten in place; order 2 is gone. The `function_column.sequence_c
- [High-Concurrency Point Query](./high-concurrency-point-query): the short-circuit path that Unique Key Merge-on-Write tables unlock for KV-style reads.
- [Binlog Table Stream](./binlog-table-stream): how committed row-level changes are exposed to downstream consumers.
- [Apache Doris 2.1.0: Merge-on-Write becomes the Unique Key default](https://doris.apache.org/blog/release-note-2.1.0/)
+- [Unique Key](./unique-key): the table model that powers SQL UPDATE/DELETE — Merge-on-Write resolves each upsert at write time.
diff --git a/docs-next/key-features/embedding.mdx b/docs-next/key-features/embedding.mdx
index e4cd0fa07e9..6c28c7404ad 100644
--- a/docs-next/key-features/embedding.mdx
+++ b/docs-next/key-features/embedding.mdx
@@ -20,12 +20,12 @@ featureCard:
- search
---
-> **TL;DR** Doris ships a built-in `EMBED()` SQL function that sends text (or an image, video, audio file) to an external embedding model and returns an `ARRAY<FLOAT>` you can store in a column or feed straight into a [distance function](./vector-index). You configure the model once as an AI `RESOURCE`, then write `EMBED('your text')` anywhere a value is allowed. No UDFs, no client-side glue, no second service to operate.
+> **TL;DR** Apache Doris ships a built-in `EMBED()` SQL function that sends text (or an image, video, audio file) to an external embedding model and returns an `ARRAY<FLOAT>` ready for a column or a [distance function](./vector-index). The model is configured once as an AI `RESOURCE`, then `EMBED('your text')` works anywhere a value is allowed. No UDFs, no client-side glue, no second service to operate.
-## Why it matters {#why}
+## Why use the EMBED function in Apache Doris? {#why}
-Vector search needs vectors, and vectors come from an embedding model. The usual setup looks like this: an application server pulls rows out of the database, sends them to OpenAI or a local model, gets vectors back, and writes them to a vector table. At query time the same round trip happens for the search string. You end up maintaining a small ETL service whose only job is to call an HTTP API and stash the result somewhere.
+The Apache Doris `EMBED()` function collapses the usual application-side embedding pipeline into one SQL call, so vectors are generated, stored, and queried inside the database without a separate ETL service. Vector search needs vectors, and vectors come from an embedding model. The usual setup looks like this: an application server pulls rows out of the database, sends them to OpenAI or a local model, gets vectors back, and writes them to a vector table. At query time the same round tri [...]
A few things tend to go wrong:
@@ -33,25 +33,27 @@ A few things tend to go wrong:
- Backfills are jobs in their own right, often slower and more expensive than the original ingest.
- Two teams own one logical step. The data team owns the rows, the application team owns the vectors, and nobody owns the gap.
-`EMBED()` pulls the whole loop back into SQL. The model is a Doris resource, the call is a function, and the result is a column you can persist with `INSERT` or compute on the fly.
+`EMBED()` pulls the whole loop back into SQL. The model is an Apache Doris resource, the call is a function, and the result is a column you can persist with `INSERT` or compute on the fly.
-## What it is {#what}
+## What is the Apache Doris EMBED function? {#what}
-`EMBED()` is an [AI function](../sql-manual/sql-functions/ai-functions/overview) that takes a piece of text (or a multimodal file reference) and asks an external embedding provider for a vector. The return type is `ARRAY<FLOAT>`. Doris reaches the provider through an AI resource you set up once with `CREATE RESOURCE`. The same function call works at load time, so you can persist embeddings into a column, and at query time, so you can embed the user's search string on the fly without leav [...]
+The Apache Doris `EMBED()` function is a built-in [AI function](../sql-manual/sql-functions/ai-functions/overview) that takes text or a multimodal file reference, asks an external embedding provider for a vector, and returns an `ARRAY<FLOAT>`. Apache Doris reaches the provider through an AI resource you set up once with `CREATE RESOURCE`. The same function call works at load time, so you can persist embeddings into a column, and at query time, so you can embed the user's search string on [...]
**Key terms**
- **AI resource**: a named connection to an embedding provider. `EMBED()` ships dedicated adapters for `openai`, `gemini`, `voyageai`, `jina`, `qwen`, `minimax`, plus a `local` option for in-house model servers. Stores the endpoint, model name, API key, and optional dimensions.
-- **`EMBED([resource_name], input)`**: the SQL function. With one argument, Doris uses the session default; with two, the first names the resource explicitly.
+- **`EMBED([resource_name], input)`**: the SQL function. With one argument, Apache Doris uses the session default; with two, the first names the resource explicitly.
- **`default_ai_resource`**: a session variable that picks a resource for the rest of the session.
-- **Multimodal input**: a JSON value describing a file (URI, content type, optional S3 credentials). Doris presigns S3 URLs and forwards them to providers that support image, video, or audio embeddings.
+- **Multimodal input**: a JSON value describing a file (URI, content type, optional S3 credentials). Apache Doris presigns S3 URLs and forwards them to providers that support image, video, or audio embeddings.
-## How it works {#how}
+## How does the Apache Doris EMBED function work? {#how}
-1. **Register the model.** `CREATE RESOURCE` stores the provider type, endpoint, model name, and credentials. Doris validates the resource with a probe call to the provider, except when `provider_type = "local"`, where the check is skipped.
+The Apache Doris `EMBED()` function works in five stages: register the AI resource, plan the call on the FE, batch rows on the BE, retry on failures, and return one `ARRAY<FLOAT>` per row.
+
+1. **Register the model.** `CREATE RESOURCE` stores the provider type, endpoint, model name, and credentials. Apache Doris validates the resource with a probe call to the provider, except when `provider_type = "local"`, where the check is skipped.
2. **Plan the call.** When the planner sees `EMBED()`, it sends the work to the BE. The function processes the input column row by row, but it does not call the provider row by row.
3. **Batch on the BE.** The BE accumulates rows into a batch, capped by `embed_max_batch_size` (default 5 inputs) and by `ai_context_window_size` (default 128 KB of accumulated text). Each batch becomes one HTTP request, which keeps round trips and per-minute rate limits in check.
-4. **Retry and fail loudly.** The provider call honors `ai.max_retries` and `ai.retry_delay_second` from the resource. If the API returns the wrong number of vectors, Doris fails the query rather than risk misaligning rows with vectors.
+4. **Retry and fail loudly.** The provider call honors `ai.max_retries` and `ai.retry_delay_second` from the resource. If the API returns the wrong number of vectors, Apache Doris fails the query rather than risk misaligning rows with vectors.
5. **Return a column.** Every row gets its own `ARRAY<FLOAT>`. You can store it, feed it to `cosine_distance`, or load it into an ANN index for vector search.
## Quick start {#quick-start}
@@ -89,11 +91,13 @@ FROM notes ORDER BY d ASC LIMIT 1;
The `INSERT` precomputes one vector per row. The `SELECT` embeds the query string once at runtime, then ranks rows by cosine distance. Two API calls in total, both hidden inside SQL.
-## When to use it {#when}
+## When should you use the Apache Doris EMBED function? {#when}
+
+The Apache Doris `EMBED()` function fits RAG and semantic search over corpora that already live in Apache Doris, backfills that keep the work inside SQL, hybrid pipelines that pair vectors with keyword filters, and multimodal embeddings whose source files live in S3. It is not a fit for per-row embedding over millions of rows on the hot path, models outside the supported provider list, or air-gapped clusters with no path to the provider.
**Good fit**
-- RAG and semantic search where the corpus already lives in Doris and you want one system to handle ingestion, embedding, and retrieval.
+- RAG and semantic search where the corpus already lives in Apache Doris and you want one system to handle ingestion, embedding, and retrieval.
- Backfills over existing tables: `UPDATE t SET vec = EMBED(body) WHERE vec IS NULL` keeps the work inside the database.
- Hybrid pipelines that pair `EMBED()` with an ANN index for storage and a `MATCH_*` predicate for keyword filtering. See [Hybrid Search](./hybrid-search).
- Multimodal embeddings (image, video, audio) where the file lives in S3 and you would rather not write a download-then-upload script.
@@ -101,7 +105,7 @@ The `INSERT` precomputes one vector per row. The `SELECT` embeds the query strin
**Not a good fit**
- Real-time queries that embed at the per-row level over millions of rows. Each row is one provider call, so the bill and the latency scale with row count. Precompute once, reuse forever.
-- Workloads that need a specific embedding model not on the supported provider list. The providers with an embedding adapter are OpenAI, Gemini, VoyageAI, Jina, Qwen, and MiniMax, plus `local` for an in-house model server. Doris's broader AI resource list also accepts DeepSeek, MoonShot, Anthropic, Zhipu, and Baichuan, but those route through the [LLM SQL Functions](./llm-sql-functions) — chat / completion only, not `EMBED()`. Anything outside the embedding-adapter set needs to expose an [...]
+- Workloads that need a specific embedding model not on the supported provider list. The providers with an embedding adapter are OpenAI, Gemini, VoyageAI, Jina, Qwen, and MiniMax, plus `local` for an in-house model server. Apache Doris's broader AI resource list also accepts DeepSeek, MoonShot, Anthropic, Zhipu, and Baichuan, but those route through the [LLM SQL Functions](./llm-sql-functions) — chat / completion only, not `EMBED()`. Anything outside the embedding-adapter set needs to ex [...]
- Air-gapped clusters with no path to the provider. Use `provider_type = "local"` against an in-house model server, or precompute vectors in your own pipeline.
- Columns where the embedding model's dimension does not match the ANN index. The ARRAY length is fixed by the model (or by `ai.dimensions` for models that allow truncation); the index is fixed at table creation. Mismatch fails at write time.
diff --git a/docs-next/key-features/full-text-search.mdx b/docs-next/key-features/full-text-search.mdx
index eb79ba8afea..f43ed6fa907 100644
--- a/docs-next/key-features/full-text-search.mdx
+++ b/docs-next/key-features/full-text-search.mdx
@@ -20,24 +20,24 @@ featureCard:
- sql
---
-> **TL;DR** Doris turns text columns into searchable terms with an [inverted index](./inverted-index), then exposes a family of `MATCH_*` operators in SQL: `MATCH_ANY`, `MATCH_ALL`, `MATCH_PHRASE`, `MATCH_PHRASE_PREFIX`, `MATCH_PHRASE_EDGE`, and `MATCH_REGEXP`. You pick a tokenizer when you create the index, write a normal SQL query, and the planner pushes the predicate down to the index.
+> **TL;DR** Apache Doris full-text search turns text columns into searchable terms with an [inverted index](./inverted-index) and exposes them through six `MATCH_*` operators in SQL: `MATCH_ANY`, `MATCH_ALL`, `MATCH_PHRASE`, `MATCH_PHRASE_PREFIX`, `MATCH_PHRASE_EDGE`, and `MATCH_REGEXP`. You pick a tokenizer when you create the index and write a normal SQL query. The Apache Doris planner pushes the predicate down to the index, so no separate search cluster is required.
-## Why it matters {#why}
+## Why use full-text search in Apache Doris? {#why}
-Most data warehouses give you `LIKE '%foo%'` and call it text search. That works for one row in ten thousand. It falls over the moment you need to find every order comment that mentions "battery" or "batteries", every log line that contains the phrase "connection refused", or every product whose description starts with "wireless".
+Apache Doris full-text search collapses real text retrieval and warehouse analytics into one engine, with token-aware operators that go far beyond `LIKE` substring matching. Most data warehouses give you `LIKE '%foo%'` and call it text search. That works for one row in ten thousand. It falls over the moment you need to find every order comment that mentions "battery" or "batteries", every log line that contains the phrase "connection refused", or every product whose description starts wi [...]
-`LIKE` reads each value end to end, byte by byte. It does not know that "Running" and "running" are the same word, or that "search engine optimization" is three terms in order rather than a substring. The usual workaround is to bolt Elasticsearch on the side, pay for a second cluster, and write fragile sync pipelines.
+`LIKE` reads each value end to end, byte by byte. The operator does not know that "Running" and "running" are the same word, or that "search engine optimization" is three terms in order rather than a substring. The usual workaround is to bolt Elasticsearch on the side, pay for a second cluster, and write fragile sync pipelines.
-Doris collapses that into one engine. The same table that serves your aggregations also serves your text queries, and the operator names look like SQL.
+Apache Doris collapses that into one engine. The same table that serves your aggregations also serves your text queries, and the operator names look like SQL.
- Substring `LIKE` falls back to a full scan. The inverted index turns text predicates into a posting-list lookup.
- Search-as-you-type, phrase search, and regex over tokens need match semantics that `LIKE` cannot express.
- Mixed structured and text predicates (`category = 'A' AND content MATCH_ANY 'doris'`) plan and execute in one query.
-## What it is {#what}
+## What is Apache Doris full-text search? {#what}
-Full-text search in Doris is a query feature built on top of the inverted index. You declare an `INVERTED` index on a `STRING`, `TEXT`, `VARCHAR`, or `ARRAY<STRING>` column, pick an analyzer, and the `MATCH_*` operators in your `WHERE` clauses get pushed down to the index. The base table stays a normal Doris table, the same one your dashboards already query.
+Apache Doris full-text search is a query feature built on top of the inverted index. You declare an `INVERTED` index on a `STRING`, `TEXT`, `VARCHAR`, or `ARRAY<STRING>` column, pick an analyzer, and the `MATCH_*` operators in your `WHERE` clauses get pushed down to the index. The base table stays a normal Apache Doris table, the same one your dashboards already query.
**Key terms**
@@ -46,7 +46,9 @@ Full-text search in Doris is a query feature built on top of the inverted index.
- **Phrase query**: matches tokens in order and adjacent. Requires `"support_phrase" = "true"` on the index for non-tokenizing parsers.
- **`score()`**: BM25 relevance score, available when you have a tokenized index, a `MATCH_*` predicate, and an `ORDER BY score() LIMIT N`.
-## How it works {#how}
+## How does Apache Doris full-text search work? {#how}
+
+Apache Doris full-text search works by tokenizing text at write time, recognizing each `MATCH_*` predicate in the Nereids optimizer, and pushing it down to the inverted index so non-matching rows are never decoded. The five-step lifecycle below covers parse, push-down, scoring, and combination with structured predicates.
1. **Tokenize at write time.** When a row lands in a tablet, the analyzer splits each text value into terms. The inverted index records, for every term, which rows contain it and where.
2. **Parse the query.** A `MATCH_*` operator is recognized in Nereids as one of `Match`, `MatchAny`, `MatchAll`, `MatchPhrase`, `MatchPhrasePrefix`, `MatchPhraseEdge`, or `MatchRegexp`. Each carries the analyzer hint that controls how the query string itself is tokenized.
@@ -103,7 +105,9 @@ ORDER BY rel DESC LIMIT 3;
When several inverted indexes with different analyzers exist on one column, append `USING ANALYZER <name>` to pick one at query time.
-## When to use it {#when}
+## When should you use Apache Doris full-text search? {#when}
+
+Use Apache Doris full-text search for log/trace search, e-commerce catalog search, mixed CJK and Latin content, search-as-you-type, and RAG pipelines that pair text matches with vector similarity. Reach for `LIKE` or n-gram indexes when you need intra-token substring search, fuzzy edit-distance matching, or result highlighting.
**Good fit**
diff --git a/docs-next/key-features/group-commit.mdx b/docs-next/key-features/group-commit.mdx
index 9f8d02a2239..b6786e2d35e 100644
--- a/docs-next/key-features/group-commit.mdx
+++ b/docs-next/key-features/group-commit.mdx
@@ -19,12 +19,12 @@ featureCard:
- ingest
---
-> **TL;DR** Group Commit lets Doris batch many small `INSERT VALUES` statements or [Stream Loads](./stream-load) into one server-side [transaction](./load-transaction), so high-frequency clients stop drowning the FE in parse and plan work and the BE in tiny rowsets. Pick `sync_mode` if the load should wait for visibility, `async_mode` if it should return as soon as the row is durable in the write-ahead log. The same workload that hits `-235` backpressure errors goes to six figures of row [...]
+> **TL;DR** Apache Doris Group Commit batches many small `INSERT VALUES` statements or [Stream Loads](./stream-load) into one server-side [transaction](./load-transaction), so high-frequency clients stop drowning the FE in parse and plan work and the BE in tiny rowsets. Pick `sync_mode` if the load should wait for visibility, `async_mode` if it should return as soon as the row is durable in the write-ahead log. Group Commit lifts the same workload from `-235` backpressure errors to over [...]
-## Why it matters {#why}
+## Why use group commit in Apache Doris? {#why}
-Per-row `INSERT INTO ... VALUES` and tiny-batch Stream Loads are how a lot of real systems write to Doris. A microservice calls `INSERT` over JDBC every time a user clicks something. A [Kafka consumer](./kafka-cdc-integration) flushes whatever it has every few hundred milliseconds. An IoT collector pushes each event the moment it arrives. None of those clients can batch larger because the events don't exist yet. And without batching, every load opens its own transaction, takes its own tr [...]
+Apache Doris Group Commit eliminates the per-write overhead that high-frequency clients impose on the FE planner and BE storage engine. Per-row `INSERT INTO ... VALUES` and tiny-batch Stream Loads are how a lot of real systems write to Doris. A microservice calls `INSERT` over JDBC every time a user clicks something. A [Kafka consumer](./kafka-cdc-integration) flushes whatever it has every few hundred milliseconds. An IoT collector pushes each event the moment it arrives. None of those c [...]
- High-frequency `INSERT VALUES` saturates the FE's parser and plan cache before it saturates the BE's CPU.
- Small Stream Loads from streaming pipelines pile up tablet versions until background compaction can't keep up. The classic symptom is the `-235` "too many segments" error.
@@ -32,9 +32,9 @@ Per-row `INSERT INTO ... VALUES` and tiny-batch Stream Loads are how a lot of re
Group Commit folds many of those small writes into one transaction, on the server, without the client knowing or caring.
-## What it is {#what}
+## What is Apache Doris group commit? {#what}
-Group Commit is a server-side buffer for small writes. The BE collects rows from many concurrent loads and commits them as one transaction every few seconds, or when the buffer fills. It layers on top of `INSERT INTO ... VALUES`, Stream Load, and JDBC prepared statements without changing their public API. You turn it on with one knob: a session variable for INSERT, an HTTP header for Stream Load, or a table property as the default. The mode you pick decides how the load returns.
+Apache Doris Group Commit is a server-side buffer for small writes. The BE collects rows from many concurrent loads and commits them as one transaction every few seconds, or when the buffer fills. Group Commit layers on top of `INSERT INTO ... VALUES`, Stream Load, and JDBC prepared statements without changing their public API. You turn it on with one knob: a session variable for INSERT, an HTTP header for Stream Load, or a table property as the default. The mode you pick decides how the [...]
**Key terms**
@@ -44,7 +44,9 @@ Group Commit is a server-side buffer for small writes. The BE collects rows from
- **`WAL`**: per-BE write-ahead log used by `async_mode`. Single-replica, lives on the receiving BE.
- **`group_commit_interval_ms`** and **`group_commit_data_bytes`**: per-table flush triggers. Defaults: 10s and 128 MB.
-## How it works {#how}
+## How does Apache Doris group commit work? {#how}
+
+Apache Doris Group Commit funnels concurrent small loads into one shared BE queue per table, flushes the queue on a timer or size threshold, and commits the merged batch as a single FE transaction.
1. **Open or join a queue.** A request lands on a BE. Instead of starting a fresh transaction, the BE drops the rows into a `LoadBlockQueue` keyed by table and the current open group-commit transaction.
2. **(Async only) write to the WAL.** In `async_mode`, the rows are also persisted to the local WAL so the request can return as soon as they are durable. In `sync_mode`, this step is skipped.
@@ -79,7 +81,9 @@ Query OK, 1 row affected (0.01 sec)
Both inserts return the same `label` and `txnId`, which is how you know they merged. The rows are not visible until the BE flushes (here, one second later, because the table set `group_commit_interval_ms` to `1000`). For Stream Load, swap the session variable for a header: `-H "group_commit:async_mode"`.
-## When to use it {#when}
+## When should you use Apache Doris group commit? {#when}
+
+Use Apache Doris Group Commit when many concurrent clients write small batches and you cannot batch larger on the client side.
**Good fit**
@@ -114,3 +118,4 @@ Source: [Apache Doris 2.1.0 release notes](https://doris.apache.org/blog/release
- [Load Transactions: when you need explicit `BEGIN ... COMMIT` or 2PC](./load-transaction)
- [Stream Load](./stream-load): the synchronous HTTP load path that Group Commit batches on the server side.
- [Apache Doris 2.1.0: Group Commit benchmarks](https://doris.apache.org/blog/release-note-2.1.0/)
+- [Unique Key](./unique-key): the table model whose tight single-row UPDATE loops Group Commit is built to batch.
diff --git a/docs-next/key-features/high-concurrency-point-query.mdx b/docs-next/key-features/high-concurrency-point-query.mdx
index 04cfc0f53dc..24a45a253dc 100644
--- a/docs-next/key-features/high-concurrency-point-query.mdx
+++ b/docs-next/key-features/high-concurrency-point-query.mdx
@@ -19,37 +19,39 @@ featureCard:
- performance
---
-> **TL;DR** Doris is a columnar OLAP engine, but for primary-key equality lookups it switches lanes. Combine the [Unique Key model](./data-model) with Merge-on-Write, row-format storage (`store_row_column`), the short-circuit query path, and a server-side [`PreparedStatement`](./prepared-statement), and a `SELECT * FROM t WHERE pk = ?` query stops behaving like an OLAP query at all. One round-trip to the BE, one row back, no plan parsing, no fragment scheduling. That is how you serve a d [...]
+> **TL;DR** Apache Doris is a columnar OLAP engine, but for primary-key equality lookups it switches lanes. Combine the [Unique Key model](./data-model) with Merge-on-Write, row-format storage (`store_row_column`), the short-circuit query path, and a server-side [`PreparedStatement`](./prepared-statement), and a `SELECT * FROM t WHERE pk = ?` query stops behaving like an OLAP query at all. One round-trip to the BE, one row back, no plan parsing, no fragment scheduling. That is how the sa [...]
-## Why it matters {#why}
+## Why use high-concurrency point query in Apache Doris? {#why}
-Most analytics workloads have a long tail of "look up one row by ID" queries hiding inside them. A user-facing dashboard fetches the row for the customer who just clicked. A risk service reads a feature vector by entity ID on every API call. A recommendation backend pulls the latest profile per user. None of these are analytical, but the data lives next to the analytical data, and nobody wants to run a separate KV store just to serve them.
+The Apache Doris high-concurrency point query path exists for the long tail of "look up one row by ID" queries that hide inside most analytics workloads. A user-facing dashboard fetches the row for the customer who just clicked. A risk service reads a feature vector by entity ID on every API call. A recommendation backend pulls the latest profile per user. None of these are analytical, but the data lives next to the analytical data, and nobody wants to run a separate KV store just to ser [...]
-Default Doris is a poor fit for that traffic, and the reasons are structural:
+Default Apache Doris is a poor fit for that traffic, and the reasons are structural:
- Columnar storage reads many small column files to reconstruct one row, amplifying random IO on wide tables.
- The Frontend planner runs the same parse, analyze, optimize, and fragment pipeline whether the query reads one row or one billion. At thousands of QPS the FE CPU saturates before the BE notices.
- Every query opens its own coordinator, allocates fragments, and ships RPCs. That is overhead a `WHERE id = 42` cannot afford.
- Page Cache is column-oriented, and large analytical scans evict it constantly, so point queries miss cache exactly when they need it.
-High-concurrency point query is Doris's answer to that traffic. It reshapes storage, planning, execution, and caching so a primary-key lookup costs one round trip and a few microseconds of FE CPU.
+The Apache Doris high-concurrency point query is the answer to that traffic. It reshapes storage, planning, execution, and caching so a primary-key lookup costs one round trip and a few microseconds of FE CPU.
-## What it is {#what}
+## What is the Apache Doris high-concurrency point query? {#what}
-High-concurrency point query is a four-layer optimization that activates automatically when the table and the query both fit a strict shape: a Unique Key table with Merge-on-Write and row-store enabled, queried with equality predicates that cover the full primary key. When the conditions hold, the planner takes the short-circuit path and the BE serves the row from a row-format column instead of stitching it from per-column files.
+The Apache Doris high-concurrency point query is a four-layer optimization that activates automatically when the table and the query both fit a strict shape: a Unique Key table with Merge-on-Write and row-store enabled, queried with equality predicates that cover the full primary key. When the conditions hold, the planner takes the short-circuit path and the BE serves the row from a row-format column instead of stitching it from per-column files.
**Key terms**
-- **`Unique Key model`**: a Doris table model where rows are addressed by a primary key. With `enable_unique_key_merge_on_write=true`, the latest version of each row is materialized at write time, so reads do not merge versions on the fly.
+- **`Unique Key model`**: an Apache Doris table model where rows are addressed by a primary key. With `enable_unique_key_merge_on_write=true`, the latest version of each row is materialized at write time, so reads do not merge versions on the fly.
- **`store_row_column`**: a table property that adds a hidden column holding each row in a row-encoded blob, so the BE reads one column instead of N to return a whole row.
- **`SHORT-CIRCUIT`**: a marker in `EXPLAIN` output that appears once the planner has skipped the normal distributed plan and routed the query through a single-tablet, single-RPC path.
-- **`PreparedStatement`**: server-side statement caching over the MySQL protocol. Doris caches the parsed statement, output expressions, and descriptor table per session, then reuses them across `EXECUTE` calls.
+- **`PreparedStatement`**: server-side statement caching over the MySQL protocol. Apache Doris caches the parsed statement, output expressions, and descriptor table per session, then reuses them across `EXECUTE` calls.
- **`Row Cache`**: a separate LRU cache that holds whole rows. It survives the eviction pressure that the columnar Page Cache suffers under mixed analytical and point-query workloads.
-## How it works {#how}
+## How does the Apache Doris high-concurrency point query work? {#how}
-1. **Storage layout.** When you set `store_row_column=true` at table creation, the BE writes each row both into the columnar segments and into a row-encoded hidden column. From Doris 3.0 onward you can scope this to a subset with `row_store_columns="k1,v1,v2"` to limit the storage overhead.
+The Apache Doris high-concurrency point query path threads five layers together: storage layout, FE plan rewrite, short-circuit dispatch, server-side `PreparedStatement` reuse, and a BE row lookup with optional Row Cache.
+
+1. **Storage layout.** When you set `store_row_column=true` at table creation, the BE writes each row both into the columnar segments and into a row-encoded hidden column. From Apache Doris 3.0 onward you can scope this to a subset with `row_store_columns="k1,v1,v2"` to limit the storage overhead.
2. **Plan rewrite (FE).** The Nereids rule `LogicalResultSinkToShortCircuitPointQuery` checks the query shape: single Unique table, equality conjuncts on every key column, no joins or subqueries, no aggregations. If everything matches, it flips the `isShortCircuitQuery` flag in the `StatementContext`.
3. **Short-circuit dispatch.** Instead of building fragments and shipping them, the FE resolves the bucket through `PartitionPruneV2ForShortCircuitPlan`, picks the one tablet that can hold the key, and sends a single RPC to the BE that owns it.
4. **Server-side `PreparedStatement` reuse.** With `useServerPrepStmts=true`, the FE caches the `ShortCircuitQueryContext` per session UUID. Subsequent `EXECUTE` calls skip parsing and planning entirely; only the parameter values change.
@@ -86,7 +88,9 @@ EXPLAIN SELECT * FROM tbl_point_query WHERE k1 = 42;
The `SHORT-CIRCUIT` line is the one that matters. Without it, the same DDL plus a query that cannot satisfy the conditions (a join, a missing key column, an inequality) falls back to the regular distributed plan. To collect the FE CPU savings on top, connect the client with `jdbc:mysql://host:9030/db?useServerPrepStmts=true&cachePrepStmts=true` and use `PreparedStatement` with `?` placeholders. Confirm in `fe.audit.log` that repeat queries log `Stmt=EXECUTE(...)` rather than the raw SQL.
-## When to use it {#when}
+## When should you use the Apache Doris high-concurrency point query? {#when}
+
+The Apache Doris high-concurrency point query fits any online service that reads a row by primary key per request, especially when the data already lives in a Unique Key table alongside the analytical workload.
**Good fit**
@@ -105,7 +109,7 @@ The `SHORT-CIRCUIT` line is the one that matters. Without it, the same DDL plus
## Performance and verification {#performance}
-Doris reports that enabling server-side `PreparedStatement` on top of the short-circuit path delivers more than a 4x throughput improvement when FE CPU is the bottleneck. This is the same 4x cited from the prepared-statement angle in the [Prepared Statement card](./prepared-statement) — one combined gain, not two stackable ones. Numbers depend on row width, row cache hit rate, and how many FE Observers absorb traffic, so treat that as a planning floor, not a benchmark.
+Apache Doris reports that enabling server-side `PreparedStatement` on top of the short-circuit path delivers more than a 4x throughput improvement when FE CPU is the bottleneck. This is the same 4x cited from the prepared-statement angle in the [Prepared Statement card](./prepared-statement) — one combined gain, not two stackable ones. Numbers depend on row width, row cache hit rate, and how many FE Observers absorb traffic, so treat that as a planning floor, not a benchmark.
Two checks tell you the path is live:
@@ -120,3 +124,5 @@ If the FE CPU is still the ceiling, scale Observers and use JDBC load balancing
- [Row Store storage layout](../table-design/row-store)
- [Index overview: secondary, bloom, inverted](../table-design/index/index-overview)
- [Data pruning: making non-point queries fast](./data-pruning)
+- [MPP Architecture](./mpp): the full distributed execution path the point-query short-circuit is built to skip.
+- [Unique Key](./unique-key): the table model point query pairs with via `store_row_column` to act as a KV store.
diff --git a/docs-next/key-features/hybrid-search.mdx b/docs-next/key-features/hybrid-search.mdx
index 8ca39a905fd..e7fc4d2ea58 100644
--- a/docs-next/key-features/hybrid-search.mdx
+++ b/docs-next/key-features/hybrid-search.mdx
@@ -19,36 +19,38 @@ featureCard:
- ai
---
-> **TL;DR** Doris lets you combine [full-text matching](./full-text-search) and vector similarity in one SQL query, on one table. You declare an [inverted index](./inverted-index) and an [ANN index](./vector-index) on the columns you'd normally split between Elasticsearch and a vector database, then mix `MATCH_ANY` with `l2_distance_approximate` in the same `WHERE` clause.
+> **TL;DR** Apache Doris hybrid search combines [full-text matching](./full-text-search) and vector similarity in one SQL query on one table. The table carries both an [inverted index](./inverted-index) and an [ANN index](./vector-index) on the columns you would normally split between Elasticsearch and a vector database. A single `WHERE` clause mixes `MATCH_ANY` with `l2_distance_approximate` to filter and rank in one shot.
-## Why it matters {#why}
+## Why use hybrid search in Apache Doris? {#why}
-Search workloads rarely pick just one retrieval mode. A user searching "lightweight running shoes for trail" wants exact matches on "running" and "trail", but also semantic matches against listings that talk about "off-road" or "rugged terrain". The typical fix, splitting work across two systems, comes with the typical costs:
+Apache Doris hybrid search collapses two retrieval modes into one SQL query so you do not run a separate search cluster next to your warehouse. Search workloads rarely pick just one retrieval mode. A user searching "lightweight running shoes for trail" wants exact matches on "running" and "trail", but also semantic matches against listings that talk about "off-road" or "rugged terrain". The typical fix, splitting work across two systems, comes with the typical costs:
- Two ingest pipelines that have to stay in sync, or recall drifts.
- Two query languages, glued together by hand-written fusion logic.
- Two clusters to size, monitor, and pay for.
-Doris collapses that into one table and one SQL statement, served by the same MPP engine that runs the rest of your queries.
+Apache Doris collapses that into one table and one SQL statement, served by the same MPP engine that runs the rest of your queries.
-## What it is {#what}
+## What is Apache Doris hybrid search? {#what}
-Hybrid search in Doris is a query pattern, not a separate engine. A single table can carry both an **inverted index** (which powers `MATCH_*` operators and BM25 scoring via `score()`) and an **ANN index** (which powers approximate vector distance functions like `l2_distance_approximate` and `inner_product_approximate`). The Doris planner reads from both indexes in one query and returns one ranked result set.
+Apache Doris hybrid search is a query pattern, not a separate engine. A single table can carry both an **inverted index** (which powers `MATCH_*` operators and BM25 scoring via `score()`) and an **ANN index** (which powers approximate vector distance functions like `l2_distance_approximate` and `inner_product_approximate`). The Apache Doris planner reads from both indexes in one query and returns one ranked result set.
**Key terms**
- **Inverted index**: a full-text index on a `STRING` column. Tokenizers include `english`, `standard`, and Chinese.
- **ANN index**: an approximate-nearest-neighbor index on an `ARRAY<FLOAT>` column. Backends are HNSW, IVF, and IVF on-disk.
-- **Pre-filter**: Doris's default strategy. Apply scalar and text predicates first, then take TopN by vector distance over the rows that survive.
+- **Pre-filter**: the Apache Doris default strategy. Apply scalar and text predicates first, then take TopN by vector distance over the rows that survive.
-## How it works {#how}
+## How does Apache Doris hybrid search work? {#how}
+
+Apache Doris hybrid search builds both indexes at load time, then runs them in a single MPP plan that filters with the inverted index and ranks with the ANN index.
1. **Index build at load time.** When you ingest data, the inverted index tokenizes each text column; the ANN index builds an HNSW graph or IVF codebook over each vector. Both live alongside the base table, in the same storage tier.
2. **Plan in one shot.** When a query mixes `MATCH_*` operators and a vector distance function, the planner routes the text predicate to the inverted index and the distance function to the ANN index. There is no second query, no client-side join.
-3. **Filter, then rank.** Doris pre-filters with the inverted index first, then runs ANN TopN over the surviving row set. This is the inverse of the common "ANN first, filter after" pattern, and it keeps recall high when filters are selective.
-4. **Adaptive fallback.** If the filter is so selective that ANN traversal would skip too many candidates, Doris falls back to brute-force vector scoring on the filtered set. The same SQL keeps working; only the execution path changes.
+3. **Filter, then rank.** Apache Doris pre-filters with the inverted index first, then runs ANN TopN over the surviving row set. This is the inverse of the common "ANN first, filter after" pattern, and it keeps recall high when filters are selective.
+4. **Adaptive fallback.** If the filter is so selective that ANN traversal would skip too many candidates, Apache Doris falls back to brute-force vector scoring on the filtered set. The same SQL keeps working; only the execution path changes.
5. **Return.** One sorted, limited result set. No external fusion service, no application-side ranking glue.
## Quick start {#quick-start}
@@ -84,7 +86,9 @@ LIMIT 2;
`MATCH_ANY 'music'` shrinks the candidate set to rows whose `body` contains "music"; the ANN index then ranks those rows by distance to the query vector and returns the closest two.
-## When to use it {#when}
+## When should you use Apache Doris hybrid search? {#when}
+
+Use Apache Doris hybrid search when you need keyword filtering and vector similarity over the same data, in one SQL query, without running a separate vector database.
**Good fit**
diff --git a/docs-next/key-features/iceberg.mdx b/docs-next/key-features/iceberg.mdx
new file mode 100644
index 00000000000..dfc21564634
--- /dev/null
+++ b/docs-next/key-features/iceberg.mdx
@@ -0,0 +1,129 @@
+---
+title: Iceberg
+description: Doris is a full Apache Iceberg engine. Read and write Iceberg tables through seven catalog backends, with V2 and V3 deletes, branches, tags, and time travel.
+keywords:
+ - Apache Iceberg
+ - Iceberg catalog
+ - Iceberg REST
+ - Iceberg V3 deletion vector
+ - Iceberg branches and tags
+ - time travel
+ - Apache Doris lakehouse
+slug: /key-features/iceberg
+image: /images/next/key-features/iceberg.jpg
+last_update:
+ date: 2026-05-11
+ author: Apache Doris
+featureCard:
+ tags:
+ - lakehouse
+ - iceberg
+ - table-format
+---
+
+> **TL;DR** Apache Doris is a full Apache Iceberg engine, not just a reader. One `CREATE CATALOG` connects Apache Doris to any of seven Iceberg backends (REST, HMS, Glue, S3 Tables, JDBC, DLF, Hadoop). From there you can query, `INSERT`, `UPDATE`, `DELETE`, `MERGE INTO`, branch and tag, time-travel, evolve schema and partitions, and run maintenance (`expire_snapshots`, `rewrite_data_files`, `rewrite_manifests`) without leaving SQL. Apache Doris reads and writes V2 position and equality d [...]
+
+
+
+## Why use the Apache Doris Iceberg catalog? {#why}
+
+The Apache Doris Iceberg catalog is a full read-and-write engine, so the same cluster that serves dashboards also lands rows, runs maintenance, and manages branches in Iceberg without handing off to Spark or Flink. Iceberg's appeal is engine neutrality. Your data sits in Parquet on object storage, the table layout lives in an open spec, and any engine that speaks the spec can read and write the same table without copying it. The promise breaks the moment your engine treats Iceberg as a r [...]
+
+Apache Doris does both halves. The cluster that serves your dashboards is the same one that maintains the Iceberg tables behind them.
+
+- Federated queries that join Iceberg facts with the Doris warehouse run as one MPP plan, not a copy job.
+- ELT pipelines write back into Iceberg through standard SQL, so other engines see the new snapshot immediately.
+- Maintenance (compaction, snapshot expiry, manifest rewrite) is `ALTER TABLE ... EXECUTE`, not a separate Spark job.
+
+This card covers the Iceberg-specific surface. For federation across mixed catalog types see [Multi Catalog](./multi-catalog.mdx); for the write surface that spans Iceberg, Hive, and Paimon together, see [Managing Lake Tables](./managing-lake-table.mdx).
+
+## What is the Apache Doris Iceberg catalog? {#what}
+
+The Apache Doris Iceberg catalog is a native connector built on top of the Iceberg Java library. The connector speaks Iceberg's catalog API directly, so commits go through the same code path Spark and Trino use and produce snapshots those engines can read without translation.
+
+**Key terms**
+
+- **Catalog type**: the metastore backend that maps table names to current metadata pointers. Doris supports seven: `rest`, `hms`, `glue`, `s3tables`, `dlf`, `jdbc`, `hadoop`.
+- **Format version**: V1 (append-only), V2 (merge-on-read with position and equality deletes), V3 (deletion vectors stored as Puffin files, row lineage, default values). Doris reads V1 through V3 and writes V2 by default. V3 writes need format version 4.1+.
+- **Snapshot**: an immutable version of an Iceberg table. Every commit produces a new one.
+- **Branch and tag**: named references to snapshots. Branches move with new commits, tags are fixed. Doris supports both for read (`@branch(name)`, `@tag(name)`) and DDL (`ALTER TABLE ... CREATE BRANCH`).
+- **Position delete, equality delete, deletion vector**: three ways Iceberg records row-level deletes. Position deletes mark a row by its file path and ordinal. Equality deletes match by column values. Deletion vectors store a Roaring bitmap per data file in a Puffin sidecar.
+
+## How does the Apache Doris Iceberg catalog work? {#how}
+
+The Apache Doris Iceberg catalog connects through one of seven metastore backends, plans queries against the current snapshot (or a branch/tag/time-travel target), reconciles row-level deletes inline, and commits writes through Iceberg's catalog API.
+
+1. **Connect once.** `CREATE CATALOG ... PROPERTIES ('type' = 'iceberg', 'iceberg.catalog.type' = '<backend>', ...)` registers a metastore and a storage system. The catalog handle persists across FE restarts.
+2. **Plan against snapshots.** When a query touches an Iceberg table, the FE resolves it to the current snapshot (or a branch, tag, or time-travel target), reads the manifest list, prunes partitions and data files using min/max stats, and pushes filters down. BE workers stream the surviving Parquet or ORC files.
+3. **Reconcile deletes on the fly.** Position deletes are applied as a left anti-join on `<file_path, row_position>`. Equality deletes match on the delete schema. V3 deletion vectors fold into the scan as a bitmap lookup, which is cheaper than a join for high-cardinality deletes.
+4. **Commit atomically on write.** `INSERT`, `INSERT OVERWRITE`, `CTAS`, `UPDATE`, `DELETE`, and `MERGE INTO` go through Doris's `IcebergTransaction`. BE writers stage Parquet (or, for new tables on V3, deletion vectors when rewriting). The FE assembles the manifest list and asks the catalog to swap the table pointer. On conflict, the statement fails with a clear snapshot-mismatch error instead of producing torn writes.
+5. **Maintain in place.** Snapshot expiry, data-file compaction, manifest rewrite, branch fast-forward, and snapshot cherrypick run as `ALTER TABLE <t> EXECUTE <action>(...)` and commit through the same catalog API. No external scheduler, no Spark job on the side.
+
+## Quick start {#quick-start}
+
+```sql
+CREATE CATALOG iceberg_rest PROPERTIES (
+ 'type' = 'iceberg',
+ 'iceberg.catalog.type' = 'rest',
+ 'uri' = 'http://rest:8181',
+ 'warehouse' = 's3://lake/wh',
+ 's3.endpoint' = 'http://minio:9000',
+ 's3.access_key' = 'admin', 's3.secret_key' = 'password'
+);
+
+SWITCH iceberg_rest;
+CREATE DATABASE IF NOT EXISTS sales;
+CREATE TABLE sales.orders (id BIGINT, region STRING, amount DECIMAL(10,2), ts DATETIME)
+ PARTITION BY LIST (region, day(ts)) ()
+ PROPERTIES ('format-version' = '2');
+
+INSERT INTO sales.orders VALUES (1, 'bj', 99.50, '2026-05-08 10:00:00');
+ALTER TABLE sales.orders CREATE BRANCH dev;
+INSERT INTO sales.orders@branch(dev) VALUES (2, 'sh', 12.00, '2026-05-08 11:00:00');
+ALTER TABLE sales.orders EXECUTE expire_snapshots('retain_last' = '5');
+```
+
+**Expected result**
+
+```
++----------+
+| count(*) |
++----------+
+| 1 | -- main branch
+| 2 | -- dev branch (SELECT ... FROM sales.orders@branch(dev))
++----------+
+```
+
+Trino or Spark pointed at the same REST endpoint see the same snapshots Doris just wrote.
+
+## When should you use the Apache Doris Iceberg catalog? {#when}
+
+The Apache Doris Iceberg catalog fits federated lakehouse analytics, SQL-only ELT into Iceberg, branch-based workflows, and in-place maintenance; it is not the right tool for high-frequency single-row updates or Hudi writes.
+
+**Good fit**
+
+- Federated analytics across Iceberg and the Apache Doris warehouse, with no copy job into Apache Doris first.
+- ELT pipelines that land curated data into Iceberg from JDBC, Kafka, or Hive sources, all in SQL.
+- Team workflows that need git-style isolation: write into a dev branch, validate, then `fast_forward` the main branch.
+- Reproducible backfills and audits using `FOR VERSION AS OF` or named tags.
+- Streaming jobs that need scheduled compaction. `rewrite_data_files` and `expire_snapshots` keep the small-files problem in check.
+- Migrations off Hive. Create the Iceberg table from Doris, dual-write while you cut over, then retire the Hive copy.
+
+**Not a good fit**
+
+- High-frequency row-level updates. Iceberg commits are snapshot-based, so per-statement overhead is far higher than a Doris [Unique Key table](./unique-key.mdx). Use Doris internal tables for CDC sinks that need single-row latency.
+- DELETE or UPDATE on V1 Iceberg tables. Row-level DML needs format version 2 or 3. Either set `'format-version' = '2'` at create time, or upgrade the table.
+- Hudi writes. Doris is a Hudi reader today. Use Spark or Flink for Hudi writes.
+- Cross-catalog transactions. A statement that writes to two Iceberg catalogs (or to an Iceberg table and a Doris table) is not a single ACID unit. Stage in one and replicate.
+- Skipping maintenance. Without `expire_snapshots`, storage and metadata grow unboundedly and time-travel reads slow down. Schedule it.
+- Filesystem (`hadoop`) catalogs for concurrent writers. File-based catalogs have no lock service. Use REST, HMS, Glue, JDBC, or DLF for any multi-writer setup.
+
+## Further reading {#further-reading}
+
+- [Iceberg catalog reference](../lakehouse/catalogs/iceberg-catalog.mdx): the full DDL, DML, branch/tag, time-travel, and table-action syntax, with version notes for every capability.
+- [Managing Lake Tables](./managing-lake-table.mdx): the write and lifecycle surface Doris exposes across Iceberg, Hive, and Paimon.
+- [Multi Catalog](./multi-catalog.mdx): how Doris federates Iceberg alongside Hive, JDBC, ES, and the internal warehouse under three-part names.
+- [Catalog Integrations](./catalog-integrations.mdx): per-backend connection details for HMS, Glue, REST, S3 Tables, DLF, JDBC, and Hadoop.
+- [Lakehouse overview](../lakehouse/lakehouse-overview.md): where Iceberg fits in Doris's broader federated-analytics story.
+- [Doris + Iceberg best practices](../lakehouse/best-practices/doris-iceberg.md): a working REST + MinIO demo with partition transforms, time travel, V2 DML, and PyIceberg interop.
+- [iceberg_meta() table function](../sql-manual/sql-functions/table-valued-functions/iceberg-meta.md): query snapshots, manifests, files, refs, and history without dropping into the Iceberg Java API.
diff --git a/docs-next/key-features/incremental-materialized-view.mdx b/docs-next/key-features/incremental-materialized-view.mdx
index 281acd33ea8..9ed7a8c2d59 100644
--- a/docs-next/key-features/incremental-materialized-view.mdx
+++ b/docs-next/key-features/incremental-materialized-view.mdx
@@ -19,22 +19,22 @@ featureCard:
- performance
---
-> **TL;DR** Mark an async materialized view with [`PARTITION BY`](./partitioning-and-bucketing) and `REFRESH AUTO` and Doris tracks staleness per partition. Today's load only touches today's base partition, so the next refresh re-runs the aggregation for that one day and leaves the rest of the view alone. Queries that span fresh and stale days still hit the view: the optimizer reads valid partitions from the MV and `UNION ALL`s the stale ones from the base table.
+> **TL;DR** An Apache Doris incremental materialized view is a partitioned async MTMV with [`PARTITION BY`](./partitioning-and-bucketing) and `REFRESH AUTO`. Apache Doris tracks staleness per partition, so today's load triggers a refresh of only today's view partition while the other 29 days stay untouched. Queries that span fresh and stale days still hit the view: the optimizer reads valid partitions from the MV and `UNION ALL`s the stale ones from the base table.
-## Why it matters {#why}
+## Why use incremental materialized views in Apache Doris? {#why}
-You have a fact table partitioned by day and a materialized view that aggregates it for tomorrow's dashboard. Today's load only writes into today's partition, but a naive refresh recomputes the whole view anyway. It re-aggregates 29 days that did not change, runs for an hour, and finishes after the dashboard is already stale.
+Apache Doris incremental materialized views keep partitioned dashboards fresh by recomputing only the partitions whose base data changed, instead of re-aggregating the whole view on every load. You have a fact table partitioned by day and a materialized view that aggregates it for tomorrow's dashboard. Today's load only writes into today's partition, but a naive refresh recomputes the whole view anyway. The refresh re-aggregates 29 days that did not change, runs for an hour, and finishes [...]
- A 30-day partitioned fact table where each load only touches the latest day. A full refresh re-aggregates 29 days that did not change.
- BI dashboards that fall behind the load schedule because the MV refresh window is longer than the load interval.
- Hive- or Iceberg-backed [lakehouse tables](./managing-lake-table) where rereading every partition through object storage is slow and expensive.
-`REFRESH AUTO` on a partitioned async MV cuts that down to what actually changed. Doris tracks staleness per partition, runs `INSERT OVERWRITE` only on the partitions whose base data moved, and leaves the rest of the view exactly as it was.
+`REFRESH AUTO` on a partitioned async MV cuts that down to what actually changed. Apache Doris tracks staleness per partition, runs `INSERT OVERWRITE` only on the partitions whose base data moved, and leaves the rest of the view exactly as it was.
-## What it is {#what}
+## What is an Apache Doris incremental materialized view? {#what}
-An incremental materialized view in Doris is a partitioned async materialized view (an MTMV) created with `REFRESH AUTO`. Doris derives the view's partitions from one base table you point at in `PARTITION BY`, with an optional `DATE_TRUNC` that rolls daily base partitions up to monthly view partitions. When a base partition's data changes, only the view partition mapped to it is flagged invalid. The next refresh runs `INSERT OVERWRITE` for the invalid partitions, one statement at a time.
+An Apache Doris incremental materialized view is a partitioned async materialized view (an MTMV) created with `REFRESH AUTO`. Apache Doris derives the view's partitions from one base table you point at in `PARTITION BY`, with an optional `DATE_TRUNC` that rolls daily base partitions up to monthly view partitions. When a base partition's data changes, only the view partition mapped to it is flagged invalid. The next refresh runs `INSERT OVERWRITE` for the invalid partitions, one statement [...]
**Key terms**
@@ -44,10 +44,12 @@ An incremental materialized view in Doris is a partitioned async materialized vi
- **`Partition invalidation`**: the per-partition stale flag set when base data backing that partition changes. Visible via `SHOW PARTITIONS FROM <mv>`.
- **`Partition-granularity transparent rewrite`**: the rewriter can read valid MV partitions and `UNION ALL` the rest from the base table, so a single stale partition no longer disqualifies the whole MV.
-## How it works {#how}
+## How does an Apache Doris incremental materialized view work? {#how}
-1. **You declare the partition mapping.** `PARTITION BY (DATE_TRUNC(o_orderdate, 'MONTH'))` tells Doris to derive the MV's partitions from `orders.o_orderdate`. Internal tables and Hive tables both work; Iceberg, Paimon, and Hudi gained partitioned refresh in 3.1, with Iceberg and Paimon also supporting automatic detection of base data changes — Hudi requires manual refresh.
-2. **Doris materializes a partition layout.** At create time, the FE walks the base partitions, applies the roll-up function, and emits matching MV partitions. New base partitions trigger new MV partitions on the next refresh.
+An Apache Doris incremental materialized view works by mapping each MV partition back to a base partition at create time, flagging only the affected MV partitions when a load lands, and rebuilding just those partitions through `INSERT OVERWRITE` on the next refresh. The five-step flow below covers declaration, materialization, invalidation, refresh, and partition-granularity rewrite.
+
+1. **You declare the partition mapping.** `PARTITION BY (DATE_TRUNC(o_orderdate, 'MONTH'))` tells Apache Doris to derive the MV's partitions from `orders.o_orderdate`. Internal tables and Hive tables both work; Iceberg, Paimon, and Hudi gained partitioned refresh in 3.1, with Iceberg and Paimon also supporting automatic detection of base data changes — Hudi requires manual refresh.
+2. **Apache Doris materializes a partition layout.** At create time, the FE walks the base partitions, applies the roll-up function, and emits matching MV partitions. New base partitions trigger new MV partitions on the next refresh.
3. **Loads invalidate partitions, not the whole view.** A load that lands in `orders` partition `p_2026_05_08` only flags the MV partitions whose key set covers that day. Loads on referenced non-partitioned tables still invalidate everything by default; list those tables in `excluded_trigger_tables` if you know they only insert and never update.
4. **The refresh job rebuilds invalid partitions only.** Each invalid partition becomes one `INSERT OVERWRITE`. The `refresh_partition_num` property controls how many partitions ride per statement (default 1), trading transactional granularity for throughput. A failure stops the task without rolling back partitions that already landed.
5. **The optimizer rewrites at partition granularity.** A query that touches both fresh and stale days reads the fresh days from the MV and the stale day from the base table through `UNION ALL`. Set `grace_period` if you want the rewriter to accept partitions that are stale by less than N seconds.
@@ -87,7 +89,9 @@ After loading new rows into the `2026-05-08` base partition only, `SHOW PARTITIO
The earlier partitions kept their old `VisibleVersionTime` because nothing recomputed them. Only `p_20260508` rebuilt, in one `INSERT OVERWRITE` statement against one day's data.
-## When to use it {#when}
+## When should you use an Apache Doris incremental materialized view? {#when}
+
+Use Apache Doris incremental materialized views for append-mostly fact tables partitioned by day or hour, multi-layer DWD/DWS/ADS modeling, and Hive/Iceberg/Paimon lakehouse tables you want to materialize locally. Skip them for sub-second freshness (use a sync MV), unpartitioned base tables, and Hudi tables where Apache Doris cannot detect base data changes automatically.
**Good fit**
@@ -100,9 +104,9 @@ The earlier partitions kept their old `VisibleVersionTime` because nothing recom
- Real-time freshness in the seconds. Use a [sync materialized view](../query-acceleration/materialized-view/sync-materialized-view) instead; sync MVs are merged into the base table's write path.
- Tables that don't have a partition column. Without a partition mapping the MV degrades to a full refresh, and you get none of the incremental benefits.
-- MV definitions that can't be partition-derived. Doris rejects the `PARTITION BY` clause at create time when any of the eight derivation rules fails: aggregating the partition column (`min(o_orderdate)`), placing it on the NULL-generating side of an OUTER JOIN, dropping it from `GROUP BY` or a window's `PARTITION BY`, using a roll-up function other than `DATE_TRUNC`, and so on.
-- Hudi base tables. Doris cannot detect base data changes on Hudi today, so only manual refresh is meaningful, and the incremental decision can't be made automatically.
-- Tables you also write to from Spark or Flink without going through Doris. The metadata cache won't see those writes, so the MV will think it's still in sync. Force a refresh manually after each external write, or pick a different acceleration path.
+- MV definitions that can't be partition-derived. Apache Doris rejects the `PARTITION BY` clause at create time when any of the eight derivation rules fails: aggregating the partition column (`min(o_orderdate)`), placing it on the NULL-generating side of an OUTER JOIN, dropping it from `GROUP BY` or a window's `PARTITION BY`, using a roll-up function other than `DATE_TRUNC`, and so on.
+- Hudi base tables. Apache Doris cannot detect base data changes on Hudi today, so only manual refresh is meaningful, and the incremental decision can't be made automatically.
+- Tables you also write to from Spark or Flink without going through Apache Doris. The metadata cache won't see those writes, so the MV will think it's still in sync. Force a refresh manually after each external write, or pick a different acceleration path.
## Further reading {#further-reading}
@@ -113,3 +117,4 @@ The earlier partitions kept their old `VisibleVersionTime` because nothing recom
- [Preaggregation and Rollup](./preaggregation-and-rollup): the per-load Aggregate Key path and sync MV / ROLLUP companions to async MVs.
- [Query Cache](./query-cache): another preaggregation path; precompute via MV or cache on first read.
- [Data Lineage](./data-lineage): how the audit log's `chosen_m_views` column surfaces which MVs actually serve traffic.
+- [Iceberg](./iceberg): a common base-table source for incremental MVs — snapshot diffs make change capture cheap.
diff --git a/docs-next/key-features/inverted-index.mdx b/docs-next/key-features/inverted-index.mdx
index 549b81d6777..d1a480a2425 100644
--- a/docs-next/key-features/inverted-index.mdx
+++ b/docs-next/key-features/inverted-index.mdx
@@ -20,12 +20,12 @@ featureCard:
- search
---
-> **TL;DR** Doris's inverted index is a CLucene-backed secondary index that you declare on any column with `INDEX ... USING INVERTED`. It accelerates equality, range, `IN`, `IS NULL`, array membership, and [full-text `MATCH_*`](./full-text-search) predicates by reading a posting list instead of scanning rows. Available since 2.0.
+> **TL;DR** The Apache Doris inverted index is a CLucene-backed secondary index declared on any column with `INDEX ... USING INVERTED`. The index accelerates equality, range, `IN`, `IS NULL`, array membership, and [full-text `MATCH_*`](./full-text-search) predicates by reading a posting list instead of scanning rows. Available since Apache Doris 2.0.
-## Why it matters {#why}
+## Why use the inverted index in Apache Doris? {#why}
-OLAP queries spend most of their time on `WHERE`. A 2 TB events table with a predicate on `user_id`, a range on `request_time`, and a keyword search on `url` will, by default, decode every column for every row in every surviving segment. The usual workaround, an [incremental materialized view](./incremental-materialized-view) or a separate search cluster, doubles storage and operational cost.
+The Apache Doris inverted index turns the dominant cost of OLAP queries, evaluating `WHERE` predicates, from a full scan into a posting-list lookup. A 2 TB events table with a predicate on `user_id`, a range on `request_time`, and a keyword search on `url` would otherwise decode every column for every row in every surviving segment. The usual workaround, an [incremental materialized view](./incremental-materialized-view) or a separate search cluster, doubles storage and operational cost.
The cases that hurt:
@@ -33,13 +33,13 @@ The cases that hurt:
- Range filters on numeric or datetime columns (`WHERE event_time BETWEEN ...`) are bounded only by ZoneMap min/max and still touch every page in range.
- Keyword search on text columns has no answer in the base SQL engine. `LIKE '%term%'` falls back to a scan, and shipping the data to Elasticsearch means a second pipeline to keep in sync.
-The inverted index handles all three with the same DDL.
+The Apache Doris inverted index handles all three with the same DDL.
-## What it is {#what}
+## What is the Apache Doris inverted index? {#what}
-The inverted index in Doris is a column-level secondary index that maps each value, or each tokenized term for text columns, to the set of row IDs that contain it. It is built on top of [CLucene](https://github.com/apache/doris-thirdparty/tree/clucene), a C++ port of Lucene, and is fully integrated with Doris's columnar storage and vectorized execution.
+The Apache Doris inverted index is a column-level secondary index that maps each value, or each tokenized term for text columns, to the set of row IDs that contain it. It is built on top of [CLucene](https://github.com/apache/doris-thirdparty/tree/clucene), a C++ port of Lucene, and is fully integrated with Apache Doris columnar storage and vectorized execution.
-You declare it in DDL with `INDEX <name>(<col>) USING INVERTED [PROPERTIES(...)]`. Supported column types include the integer family, `DECIMAL`, `DATE`, `DATETIME`, `IPV4`, `IPV6`, `CHAR`/`VARCHAR`/`STRING`, and `ARRAY<T>` of those. For text columns you can attach a `parser` to control tokenization.
+An inverted index is declared in DDL with `INDEX <name>(<col>) USING INVERTED [PROPERTIES(...)]`. Supported column types include the integer family, `DECIMAL`, `DATE`, `DATETIME`, `IPV4`, `IPV6`, `CHAR`/`VARCHAR`/`STRING`, and `ARRAY<T>` of those. For text columns you can attach a `parser` to control tokenization.
**Key terms**
@@ -48,7 +48,9 @@ You declare it in DDL with `INDEX <name>(<col>) USING INVERTED [PROPERTIES(...)]
- **`MATCH operators`**: `MATCH_ANY` (any term), `MATCH_ALL` (all terms), `MATCH_PHRASE` (adjacent terms), `MATCH_PHRASE_PREFIX`, `MATCH_REGEXP`. They require an inverted index with a parser; ordinary `=` and `LIKE` do not.
- **`.idx file`**: the on-disk index payload. One `.idx` file lives next to each segment file under the tablet directory, so the index is segment-bound and follows the rowset through compaction.
-## How it works {#how}
+## How does the Apache Doris inverted index work? {#how}
+
+The Apache Doris inverted index follows a build-once, probe-many flow that keeps the index co-located with the segment it indexes.
1. **Build at write time.** When a rowset is flushed, the BE writes a CLucene-format index to a sibling `.idx` file. Each row in the segment maps 1:1 to a CLucene `DocID`, so the index never has to maintain a separate row-to-doc table.
2. **Choose a path at query time.** For a predicate the planner pushes down, the BE consults the inverted index first: it tokenizes the literal with the same parser used at build time, walks the term dictionary, and reads the posting list.
@@ -90,7 +92,9 @@ WHERE user_id = 42 AND status >= 500 AND url MATCH_ANY 'orders';
Each predicate is served by an index probe. `user_id = 42` and `status >= 500` use the inverted index for equality and range. `url MATCH_ANY 'orders'` tokenizes the column with the `english` parser and looks up the posting list for `orders`. The scanner only reads the one row that survives the AND of the three posting lists.
-## When to use it {#when}
+## When should you use the Apache Doris inverted index? {#when}
+
+The Apache Doris inverted index pays off on high-cardinality predicates, text and array search, and any column where ZoneMap pruning is too coarse.
**Good fit**
diff --git a/docs-next/key-features/kafka-cdc-integration.mdx b/docs-next/key-features/kafka-cdc-integration.mdx
index c518beebd47..2ed146c84db 100644
--- a/docs-next/key-features/kafka-cdc-integration.mdx
+++ b/docs-next/key-features/kafka-cdc-integration.mdx
@@ -20,12 +20,12 @@ featureCard:
- cdc
---
-> **TL;DR** Doris ingests Kafka topics and CDC change streams natively. Use **Routine Load** when you want a Kafka consumer that lives inside the FE and needs no external service. Use the **Flink Doris Connector** with Flink CDC when you want whole-database MySQL/Postgres/Oracle sync with exactly-once via [Stream Load](./stream-load) 2PC. Use the **Doris Kafka Connector** when Kafka Connect is already running and your payloads are Avro, Protobuf, or Debezium. All three paths land in the [...]
+> **TL;DR** Apache Doris ingests Kafka topics and CDC change streams natively through three paths. Use **Routine Load** when you want a Kafka consumer that lives inside the FE and needs no external service. Use the **Flink Doris Connector** with Flink CDC when you want whole-database MySQL/Postgres/Oracle sync with exactly-once via [Stream Load](./stream-load) 2PC. Use the **Doris Kafka Connector** when Kafka Connect is already running and your payloads are Avro, Protobuf, or Debezium. A [...]
-## Why it matters {#why}
+## Why use Kafka and CDC integration in Apache Doris? {#why}
-Most warehouses make you bolt on streaming ingest. You stand up a Kafka consumer, re-implement offset bookkeeping, retry, schema mapping, and deletes, and hope the consumer dies cleanly when the warehouse falls over. Or you run Flink in front and pay for an extra cluster. Or you give up on "real-time" and ship five-minute batches via [bulk load](./batch-load).
+Apache Doris Kafka and CDC integration puts streaming ingestion, exactly-once delivery, and out-of-order handling inside the engine, so the path from a topic or binlog to a queryable row stays short. Most warehouses make you bolt on streaming ingest. You stand up a Kafka consumer, re-implement offset bookkeeping, retry, schema mapping, and deletes, and hope the consumer dies cleanly when the warehouse falls over. Or you run Flink in front and pay for an extra cluster. Or you give up on " [...]
- A CDC pipeline replays its last batch after a checkpoint failure and doubles every row it had buffered.
- An out-of-order update on a Unique Key table loses to the older version because nothing told the engine which write is fresher.
@@ -33,9 +33,9 @@ Most warehouses make you bolt on streaming ingest. You stand up a Kafka consumer
Doris pushes most of this into the engine, so the path from a Kafka topic or a MySQL binlog to a queryable row is short and the failure modes are ones you can name.
-## What it is {#what}
+## What is the Apache Doris Kafka and CDC integration? {#what}
-Doris ships four ingestion paths for streaming and CDC traffic. They share the same backbone: transactional loads identified by a `Label`, Unique Key Merge-on-Write tables for upserts and deletes, and a sequence column that pins the winning version when events arrive out of order. They differ in who runs the consumer.
+Apache Doris Kafka and CDC integration is a set of streaming and CDC ingestion paths that share one backbone: transactional loads identified by a `Label`, Unique Key Merge-on-Write tables for upserts and deletes, and a sequence column that pins the winning version when events arrive out of order. The paths differ in who runs the consumer.
**Key terms**
@@ -45,9 +45,9 @@ Doris ships four ingestion paths for streaming and CDC traffic. They share the s
- **`Doris Kafka Connector`**: a Kafka Connect sink plugin. Inherits Kafka Connect's converter ecosystem, so Avro, Protobuf, and Debezium payloads work without a custom serializer.
- **`function_column.sequence_col`**: a per-table property on Unique Key MOW tables that selects the column whose value decides which version of a row wins. The safety net for out-of-order CDC.
-## How it works {#how}
+## How does the Apache Doris Kafka and CDC integration work? {#how}
-The two paths most teams pick share the same lifecycle once data hits the FE.
+The Apache Doris Kafka and CDC integration paths converge on one lifecycle once data hits the FE: pull a batch, wrap it in a labeled transaction, commit on confirmation, then publish to all replicas.
1. **The consumer pulls a batch.** Routine Load schedules a task on a BE that pulls from Kafka up to `max_batch_rows` (default 20M), `max_batch_size` (default 1 GB), or `max_batch_interval` (default 60 s), whichever hits first. Flink CDC streams rows over a chunked HTTP Stream Load, which stays open across the checkpoint window.
2. **The batch becomes a transaction.** The BE opens a transaction with a `Label`. Routine Load generates the label; Flink Doris Connector tags it with the checkpoint id. Either way, resubmitting the same label deduplicates.
@@ -101,7 +101,9 @@ PROPERTIES ("function_column.sequence_col" = "updated");
The `sequence_col` line is what protects you when a delayed CDC event arrives after a fresher one: the older `updated` loses, the newer wins, and the result is the same regardless of arrival order.
-## When to use it {#when}
+## When should you use the Apache Doris Kafka and CDC integration? {#when}
+
+Use the Apache Doris Kafka and CDC integration when streaming events or upstream database changes need to land in Doris with exactly-once delivery and correct ordering, not on a five-minute batch cadence.
**Good fit**
@@ -130,3 +132,5 @@ From "How Flink's real-time writes to Apache Doris stay both fast and exactly-on
- [Load Transactions: the model behind 2PC](./load-transaction)
- [Data Update and Delete: sequence column and Merge-on-Write](./data-update-delete)
- [Binlog Table Stream](./binlog-table-stream): the change feed that runs the other direction, exposing Doris row changes to downstream consumers.
+- [Iceberg](./iceberg): a common destination for curated CDC pipelines — write-back via SQL keeps other engines' readers in sync.
+- [Unique Key](./unique-key): the table model CDC sinks land into — pair with a sequence column on the source LSN/commit timestamp.
diff --git a/docs-next/key-features/llm-sql-functions.mdx b/docs-next/key-features/llm-sql-functions.mdx
index 99b7bddc729..1505c2d81ee 100644
--- a/docs-next/key-features/llm-sql-functions.mdx
+++ b/docs-next/key-features/llm-sql-functions.mdx
@@ -20,20 +20,20 @@ featureCard:
- analytics
---
-> **TL;DR** Doris ships a family of `AI_*` SQL functions that send a row's column value to an external LLM and return the result inline. You configure the provider once as an AI `RESOURCE`, then write `AI_CLASSIFY(...)`, `AI_EXTRACT(...)`, or `AI_GENERATE(...)` anywhere a value is allowed. Snowflake's Cortex AISQL and Databricks' AI Functions cover the same ground; this is the Doris equivalent.
+> **TL;DR** Apache Doris ships a family of `AI_*` SQL functions that send a row's column value to an external LLM and return the result inline. You configure the provider once as an AI `RESOURCE`, then write `AI_CLASSIFY(...)`, `AI_EXTRACT(...)`, or `AI_GENERATE(...)` anywhere a value is allowed. Snowflake's Cortex AISQL and Databricks' AI Functions cover the same ground; the Apache Doris `AI_*` family is the open-source equivalent.
-## Why it matters {#why}
+## Why use AI_* SQL functions in Apache Doris? {#why}
-Plenty of analytics work on text columns is not analytics at all. Someone has to classify support tickets by topic, score reviews for sentiment, pull product names out of free-form complaints, summarize long call notes, or redact PII before exporting. The work is repetitive, the columns are already in the warehouse, and the only piece missing is a model. Doris already serves embeddings through [EMBED()](./embedding) and an [MCP server](./mcp-server) for AI clients, so this card rounds ou [...]
+Apache Doris `AI_*` SQL functions remove the need for an external Python service when text columns require classification, extraction, summarization, or PII redaction. Plenty of analytics work on text columns is not analytics at all. Someone has to classify support tickets by topic, score reviews for sentiment, pull product names out of free-form complaints, summarize long call notes, or redact PII before exporting. The work is repetitive, the columns are already in the warehouse, and th [...]
The usual workaround is a Python service. It pulls rows from Doris, calls OpenAI or Anthropic, parses the response, and writes the result back. That service drifts out of sync with the source table, owns its own retries, and lives outside the warehouse's permissions and observability.
`AI_*` functions move that loop into SQL. The model is a registered resource. The call is a function. The result is a column you can persist with `INSERT`, filter on with `WHERE`, or compute on the fly. One language, one transaction boundary, one place to read the audit log.
-## What it is {#what}
+## What are the Apache Doris AI_* SQL functions? {#what}
-The `AI_*` family is a set of SQL-callable functions that wrap an HTTP call to an LLM provider. Each function passes the input columns through a fixed prompt template and returns a typed result, so the planner treats them like any other scalar function. Doris reaches the provider through an AI resource you create once with `CREATE RESOURCE ... TYPE='ai'`. Eleven scalar functions cover the most common tasks, and one aggregate, `AI_AGG`, rolls multiple rows into a single LLM-generated answer.
+The Apache Doris `AI_*` family is a set of SQL-callable functions that wrap an HTTP call to an LLM provider. Each function passes the input columns through a fixed prompt template and returns a typed result, so the planner treats them like any other scalar function. Apache Doris reaches the provider through an AI resource you create once with `CREATE RESOURCE ... TYPE='ai'`. Eleven scalar functions cover the most common tasks, and one aggregate, `AI_AGG`, rolls multiple rows into a singl [...]
**Key terms**
@@ -42,7 +42,9 @@ The `AI_*` family is a set of SQL-callable functions that wrap an HTTP call to a
- **`AI_AGG`**: an aggregate function that joins all rows in a group, sends them to the LLM with a custom prompt, and returns one summary per group. The group equivalent of `AI_SUMMARIZE`.
- **`default_ai_resource`**: a session variable that selects a resource for the rest of the session, so you can drop the explicit first argument.
-## How it works {#how}
+## How do the Apache Doris AI_* SQL functions work? {#how}
+
+The Apache Doris `AI_*` SQL functions run as scalar UDFs on the BE: the planner routes each call, the BE batches rows, and one HTTP request per batch reaches the provider before returning a typed value to SQL.
1. **Register the model.** `CREATE RESOURCE` validates the endpoint with a probe call (skipped for `provider_type = 'local'` or when `ai.validity_check = 'false'`), masks the API key in `SHOW RESOURCES` output, and persists the configuration in the FE's metadata.
2. **Plan the call.** When the planner sees an `AI_*` function, it routes the work to the BE the same way it would for any scalar UDF. The function's first argument must be a literal resource name; the planner rejects an expression there at analysis time.
@@ -84,7 +86,9 @@ FROM reviews;
The query runs two LLM round trips, one per function, with both rows batched into each. The output columns are typed strings, so a downstream `GROUP BY topic` or a join works without parsing.
-## When to use it {#when}
+## When should you use the Apache Doris AI_* SQL functions? {#when}
+
+The Apache Doris `AI_*` functions fit batch enrichment, ETL backfills, and cross-row roll-ups, but not interactive sub-second queries or workloads that demand strict reproducibility.
**Good fit**
diff --git a/docs-next/key-features/load-transaction.mdx b/docs-next/key-features/load-transaction.mdx
index 36341172a5d..976f9a3accd 100644
--- a/docs-next/key-features/load-transaction.mdx
+++ b/docs-next/key-features/load-transaction.mdx
@@ -19,23 +19,23 @@ featureCard:
- streaming
---
-> **TL;DR** Doris wraps every data load in a transaction. You can group several inserts under one `BEGIN ... COMMIT`, dedupe retries with a load `Label`, and drive a two-phase commit from headers in a [Stream Load](./stream-load) HTTP request. That last part is what lets Flink, Spark, and [CDC pipelines](./kafka-cdc-integration) write to Doris exactly once without keeping their own bookkeeping table.
+> **TL;DR** Apache Doris wraps every data load in a transaction. You can group several inserts under one `BEGIN ... COMMIT`, dedupe retries with a load `Label`, and drive a two-phase commit from headers in a [Stream Load](./stream-load) HTTP request. That last part is what lets Flink, Spark, and [CDC pipelines](./kafka-cdc-integration) write to Doris exactly once without keeping their own bookkeeping table.
-## Why it matters {#why}
+## Why use load transactions in Apache Doris? {#why}
-Loading data into a warehouse goes wrong in predictable ways. A Flink job restarts mid-checkpoint and replays the last batch, doubling rows. A CDC pipeline [updates a fact table](./data-update-delete) but its companion dimension write fails, leaving the two out of sync until the next backfill. A retry script re-uploads the same file because it has no way to know the previous attempt actually landed. Each of these forces ugly bookkeeping in the application.
+Apache Doris load transactions push duplicate-row and partial-write problems off the application and into the warehouse, so a Flink replay, a half-completed multi-table write, or a re-uploaded file all behave correctly. Loading data into a warehouse goes wrong in predictable ways. A Flink job restarts mid-checkpoint and replays the last batch, doubling rows. A CDC pipeline [updates a fact table](./data-update-delete) but its companion dimension write fails, leaving the two out of sync un [...]
- Replays from a stalled streaming job duplicate rows.
- Multi-table writes that should move together can stop halfway.
- Resubmissions of the same file land twice unless you track them yourself.
-Doris solves all three on the server, so the load client can stay thin.
+Apache Doris solves all three on the server, so the load client can stay thin.
-## What it is {#what}
+## What are Apache Doris load transactions? {#what}
-Every write into Doris runs inside a transaction. That covers `INSERT INTO`, Stream Load, Routine Load, and Broker Load. Each one gets a unique Label and walks through a tracked set of lifecycle states managed by the FE. You can let Doris open and commit that transaction implicitly per load, take control with explicit `BEGIN`/`COMMIT`/`ROLLBACK`, or split the commit into two phases over HTTP for streaming sinks.
+Apache Doris load transactions are the atomic write unit underneath every ingestion path. Every write into Doris runs inside a transaction. That covers `INSERT INTO`, Stream Load, Routine Load, and Broker Load. Each one gets a unique Label and walks through a tracked set of lifecycle states managed by the FE. You can let Apache Doris open and commit that transaction implicitly per load, take control with explicit `BEGIN`/`COMMIT`/`ROLLBACK`, or split the commit into two phases over HTTP [...]
**Key terms**
@@ -45,7 +45,9 @@ Every write into Doris runs inside a transaction. That covers `INSERT INTO`, Str
- **`Stream Load 2PC`**: a Stream Load that stops at PRECOMMITTED until a follow-up HTTP call commits or aborts it.
- **`Lifecycle states`**: `PREPARE` → `PRECOMMITTED` → `COMMITTED` → `VISIBLE` (or `ABORTED`). Rows are queryable only after `VISIBLE`.
-## How it works {#how}
+## How do Apache Doris load transactions work? {#how}
+
+Apache Doris load transactions walk every write through five FE-managed states (`PREPARE` → `PRECOMMITTED` → `COMMITTED` → `VISIBLE`, or `ABORTED` on failure), with a `Label` that deduplicates retries and a final replica-level publish that makes rows queryable at one instant.
1. **Open.** A load opens a transaction in `PREPARE`. The FE assigns a `TxnId` and ties it to the user-supplied or generated Label. A second submission with the same Label short-circuits to the original transaction's result.
2. **Write.** BEs receive the data, build segments, and report success. The transaction moves to `PRECOMMITTED` once all expected loads acknowledge.
@@ -81,7 +83,9 @@ Query OK, 2 rows affected
Both tables become readable at the same instant. If the second `INSERT` had failed or the connection had dropped before `COMMIT`, neither table would carry the new rows.
-## When to use it {#when}
+## When should you use Apache Doris load transactions? {#when}
+
+Reach for Apache Doris load transactions when several inserts must move together, when a streaming sink needs exactly-once semantics, or when retried loads must dedupe on a Label rather than in the application.
**Good fit**
@@ -107,3 +111,4 @@ Both tables become readable at the same instant. If the second `INSERT` had fail
- [Transactions, isolation, and Label retention](https://doris.apache.org/docs/data-operate/transaction/)
- [Stream Load 2PC walkthrough](https://doris.apache.org/docs/data-operate/import/import-way/stream-load-manual/)
- [How Flink's real-time writes to Doris stay both fast and exactly-once](https://doris.apache.org/blog/Flink-realtime-write/)
+- [Unique Key](./unique-key): the table model whose Merge-on-Write upserts commit through the same load transaction protocol.
diff --git a/docs-next/key-features/managing-lake-table.mdx b/docs-next/key-features/managing-lake-table.mdx
index 844a5d822f0..7bb4624ae1e 100644
--- a/docs-next/key-features/managing-lake-table.mdx
+++ b/docs-next/key-features/managing-lake-table.mdx
@@ -21,42 +21,44 @@ featureCard:
- data-management
---
-> **TL;DR** Once a lake catalog is connected, Doris treats Iceberg, Hive, and Paimon tables like first-class objects. You can `CREATE TABLE`, `INSERT`, `INSERT OVERWRITE`, `UPDATE`, `DELETE`, evolve schema and partitions, branch and tag Iceberg history, time-travel by snapshot, and run maintenance like `expire_snapshots` and `rewrite_data_files`. The metadata commits go through the catalog (HMS, Glue, REST, JDBC), so other engines see the same writes.
+> **TL;DR** Once a lake catalog is connected, Apache Doris treats Iceberg, Hive, and Paimon tables like first-class objects. Supported operations include `CREATE TABLE`, `INSERT`, `INSERT OVERWRITE`, `UPDATE`, `DELETE`, schema and partition evolution, Iceberg branches and tags, snapshot time travel, and maintenance like `expire_snapshots` and `rewrite_data_files`. All metadata commits go through the catalog (HMS, Glue, REST, JDBC), so Trino, Spark, and Flink see the same writes.
-## Why it matters {#why}
+## Why use Apache Doris lake table management? {#why}
-Most lakehouse engines start as readers. You connect them to an Iceberg catalog, query a few tables, and then run into the same wall: any write has to go back through Spark or Flink. So you end up running two clusters, learning two SQL dialects, and reconciling two sets of permissions for the same data. [Multi-catalog](./multi-catalog) federation built on top of [catalog integrations](./catalog-integrations) is the read half of the story; this card is the write half.
+Apache Doris lake table management lets one cluster handle both federated reads and lake writes through SQL, so teams stop running a separate Spark or Flink cluster just to mutate Iceberg, Hive, or Paimon tables. Most lakehouse engines start as readers. You connect them to an Iceberg catalog, query a few tables, and then run into the same wall: any write has to go back through Spark or Flink. So you end up running two clusters, learning two SQL dialects, and reconciling two sets of permi [...]
-Doris does both sides. The same SQL that queries a Doris internal table can create an Iceberg table, append rows, overwrite a partition, evolve the schema, fork a branch, expire old snapshots, and compact small files. The cluster that powers the dashboards also maintains the lake.
+Apache Doris does both sides. The same SQL that queries a Doris internal table can create an Iceberg table, append rows, overwrite a partition, evolve the schema, fork a branch, expire old snapshots, and compact small files. The cluster that powers the dashboards also maintains the lake.
- One engine for federated reads and lake writes, instead of pinning a Spark job to every ETL step.
- Standard SQL DDL and DML against Iceberg, Hive, and Paimon, so the team is not learning a new dialect per format.
-- Catalog-native commits. A Doris `INSERT` into Iceberg shows up in Trino, Spark, and Flink with no extra sync step.
+- Catalog-native commits. An Apache Doris `INSERT` into Iceberg shows up in Trino, Spark, and Flink with no extra sync step.
-## What it is {#what}
+## What is Apache Doris lake table management? {#what}
-Managing Lake Tables is the write and lifecycle surface Doris exposes on top of an external catalog. After `CREATE CATALOG` connects Doris to HMS, Glue, REST, JDBC, S3 Tables, or DLF, the catalog becomes a writable namespace. Doris translates `CREATE TABLE`, `ALTER TABLE`, `INSERT`, `UPDATE`, `DELETE`, and `MERGE INTO` into the right manifest writes, snapshot commits, and metastore calls for each format.
+Apache Doris lake table management is the write and lifecycle surface the engine exposes on top of an external catalog, covering DDL, DML, schema evolution, branches, tags, and table-action maintenance. After `CREATE CATALOG` connects Apache Doris to HMS, Glue, REST, JDBC, S3 Tables, or DLF, the catalog becomes a writable namespace. Apache Doris translates `CREATE TABLE`, `ALTER TABLE`, `INSERT`, `UPDATE`, `DELETE`, and `MERGE INTO` into the right manifest writes, snapshot commits, and m [...]
For the connection layer itself, see [Multi Catalog](./multi-catalog) and the [Lakehouse overview](../lakehouse/lakehouse-overview). This card is about what you do *after* the catalog is connected.
**Key terms**
-- **`Catalog`**: the connection object that points Doris at a metastore and a storage system. Writes commit through the catalog's API.
+- **`Catalog`**: the connection object that points Apache Doris at a metastore and a storage system. Writes commit through the catalog's API.
- **`Snapshot`**: an immutable version of an Iceberg or Paimon table. Every write produces a new one; reads can target a specific snapshot.
- **`Branch` and `Tag`**: named references to snapshots in Iceberg and Paimon. A branch moves with new commits, a tag is fixed.
- **`Schema evolution`**: adding, dropping, renaming, or promoting columns without rewriting data files.
- **`Table action`**: a maintenance operation invoked through `ALTER TABLE ... EXECUTE`, such as `expire_snapshots`, `rewrite_data_files`, or `rewrite_manifests`.
-## How it works {#how}
+## How does Apache Doris lake table management work? {#how}
-1. **Connect a catalog.** `CREATE CATALOG iceberg_ctl PROPERTIES (...)` registers a metastore and a storage backend. Doris caches metadata, but writes always commit through the catalog's API so other engines stay in sync.
+Apache Doris lake table management runs in five stages: connect a catalog, plan against the right format, stage data files on BEs, commit atomically through the catalog API, and maintain tables in place with SQL.
+
+1. **Connect a catalog.** `CREATE CATALOG iceberg_ctl PROPERTIES (...)` registers a metastore and a storage backend. Apache Doris caches metadata, but writes always commit through the catalog's API so other engines stay in sync.
2. **Plan against the right format.** The Nereids planner sees an external table and routes the statement through the format's writer: Iceberg manifest commits, Paimon delta commits, or Hive partition rewrites.
3. **Stage data files.** BE writers produce Parquet (or ORC for Hive) data files in the table location. Iceberg V2 deletes go into Position Delete files, V3 deletes go into Puffin Deletion Vectors, Paimon writes LSM-tree segments.
-4. **Commit atomically.** The FE assembles the manifest list and asks the catalog to swap the table pointer. If the commit conflicts with a concurrent write, Doris detects the snapshot mismatch and the statement fails cleanly.
+4. **Commit atomically.** The FE assembles the manifest list and asks the catalog to swap the table pointer. If the commit conflicts with a concurrent write, Apache Doris detects the snapshot mismatch and the statement fails cleanly.
5. **Maintain in place.** Maintenance runs as SQL on the same cluster: `EXECUTE expire_snapshots`, `EXECUTE rewrite_data_files`, `EXECUTE rewrite_manifests`, `ALTER TABLE ... CREATE BRANCH`, and so on.
-What each format supports is uneven. Iceberg is the most complete: full DDL and DML, V2 and V3 delete strategies, branches, tags, partition evolution, and table actions. Paimon supports reads, time travel, branches, and tags through Doris; writes for primary-key Paimon tables still go through Flink. Hive supports `CREATE`, `INSERT`, `INSERT OVERWRITE`, and CTAS, but partition-targeted writes and concurrent `INSERT OVERWRITE` on the same partition need the same care you would take from Sp [...]
+What each format supports is uneven. Iceberg is the most complete: full DDL and DML, V2 and V3 delete strategies, branches, tags, partition evolution, and table actions. Paimon supports reads, time travel, branches, and tags through Apache Doris; writes for primary-key Paimon tables still go through Flink. Hive supports `CREATE`, `INSERT`, `INSERT OVERWRITE`, and CTAS, but partition-targeted writes and concurrent `INSERT OVERWRITE` on the same partition need the same care you would take [...]
## Quick start {#quick-start}
@@ -89,11 +91,13 @@ ALTER TABLE sales.orders EXECUTE expire_snapshots ("retain_last" = "5");
The main branch keeps the production row, the dev branch carries an extra row, and `expire_snapshots` trims old metadata. Trino or Spark pointed at the same REST or HMS catalog will read the same snapshots Doris just produced.
-## When to use it {#when}
+## When should you use Apache Doris lake table management? {#when}
+
+Apache Doris lake table management fits federated ELT into Iceberg or Hive, git-style Iceberg branches for dev workflows, reproducible time-travel reads, scheduled maintenance like `expire_snapshots` and `rewrite_data_files`, and SQL-driven schema and partition evolution. It is not a fit for high-concurrency single-row OLTP updates, real-time CDC into plain Hive, or any write into Hudi (read-only today).
**Good fit**
-- Federated ELT pipelines that read from JDBC or Kafka catalogs and land curated data into Iceberg or Hive without leaving Doris. See the [Lakehouse overview](../lakehouse/lakehouse-overview) for the write-back pattern.
+- Federated ELT pipelines that read from JDBC or Kafka catalogs and land curated data into Iceberg or Hive without leaving Apache Doris. See the [Lakehouse overview](../lakehouse/lakehouse-overview) for the write-back pattern.
- Team workflows that need git-style isolation on Iceberg: `ALTER TABLE ... CREATE BRANCH dev`, validate against the dev branch, then `EXECUTE fast_forward` to publish.
- Time-travel reads and reproducible backfills with `FOR VERSION AS OF` or `FOR TIME AS OF` on Iceberg, Paimon, and Hudi.
- Maintenance on a schedule: nightly `expire_snapshots`, hourly `rewrite_data_files` to compact small files from streaming jobs, periodic `rewrite_manifests` after big partition reshuffles.
@@ -101,11 +105,11 @@ The main branch keeps the production row, the dev branch carries an extra row, a
**Not a good fit**
-- High-concurrency single-row OLTP-style updates. Lake commits are snapshot-based, so the per-statement overhead is too high. Use a Doris [Unique Key table](./data-update-delete) for that workload.
+- High-concurrency single-row OLTP-style updates. Lake commits are snapshot-based, so the per-statement overhead is too high. Use an Apache Doris [Unique Key table](./data-update-delete) for that workload.
- Real-time row-level upserts on Hive. Hive lacks delete files or merge-on-read. Use Iceberg V2 or V3, or Paimon primary-key tables, when CDC needs to land row by row.
-- Partition-targeted `INSERT INTO` on Hive. Doris writes to the partition based on column values, not on a `PARTITION (...)` clause; if you need explicit partition routing on Hive, use `INSERT OVERWRITE` or do the load through Spark.
+- Partition-targeted `INSERT INTO` on Hive. Apache Doris writes to the partition based on column values, not on a `PARTITION (...)` clause; if you need explicit partition routing on Hive, use `INSERT OVERWRITE` or do the load through Spark.
- Concurrent `INSERT OVERWRITE` on the same Hive partition from multiple writers. Hive has no atomic swap for partition data, so the result can be partial; serialize the writes or move that table to Iceberg.
-- Writing to Hudi. Doris is a reader for Hudi today; use Spark or Flink for Hudi writes.
+- Writing to Hudi. Apache Doris is a reader for Hudi today; use Spark or Flink for Hudi writes.
## Further reading {#further-reading}
@@ -115,3 +119,4 @@ The main branch keeps the production row, the dev branch carries an extra row, a
- [Paimon catalog reference](../lakehouse/catalogs/paimon-catalog): read, time travel, batch incremental, branch and tag, system tables.
- [Lakehouse overview](../lakehouse/lakehouse-overview): where managed lake tables fit in the broader federated-analytics story.
- [Data Update and Delete](./data-update-delete): the internal-table counterpart for high-frequency row-level changes.
+- [Iceberg](./iceberg): the Iceberg-specific surface: seven catalog backends, V2/V3 deletes, branches, tags, and time travel.
diff --git a/docs-next/key-features/mcp-server.mdx b/docs-next/key-features/mcp-server.mdx
index ae30141284e..95fd3c6bec6 100644
--- a/docs-next/key-features/mcp-server.mdx
+++ b/docs-next/key-features/mcp-server.mdx
@@ -20,32 +20,34 @@ featureCard:
- integration
---
-> **TL;DR** The Doris MCP Server is a Python service that speaks the [Model Context Protocol](https://modelcontextprotocol.io/specification/2025-11-25) and exposes Doris as a set of tools an AI assistant can call. Point Claude Desktop, Cursor, or any MCP client at it, and the model can list tables, run SQL, fetch schemas, and read audit logs against a real cluster, without you writing a custom integration.
+> **TL;DR** The Apache Doris MCP Server is a Python service that speaks the [Model Context Protocol](https://modelcontextprotocol.io/specification/2025-11-25). The server exposes Apache Doris as a set of tools an AI assistant can call, including listing tables, running SQL, fetching schemas, and reading audit logs. Point Claude Desktop, Cursor, or any MCP client at the server and the model can work against a real cluster without a custom integration.
-## Why it matters {#why}
+## Why use the Apache Doris MCP Server? {#why}
-Wiring a chat assistant to a database is the same project people repeat at every company. You stand up a small Python service. You wrap a few SQL helpers. You handle credentials, timeouts, result truncation, and read-only enforcement. You write client-specific glue for Claude Desktop, Cursor, and whatever shows up next month. The work is mostly boilerplate, but the bug surface is large: an over-eager `DELETE` from an LLM is a real outage. The in-database AI surface, [LLM SQL functions](. [...]
+The Apache Doris MCP Server replaces the same chat-assistant-to-database integration that every company would otherwise build from scratch. You would otherwise stand up a small Python service, wrap a few SQL helpers, handle credentials, timeouts, result truncation, and read-only enforcement, and then rewrite client-specific glue for Claude Desktop, Cursor, and whatever shows up next month. The work is mostly boilerplate, but the bug surface is large: an over-eager `DELETE` from an LLM is [...]
[Anthropic released MCP](https://www.anthropic.com/news/model-context-protocol) in November 2024 to standardize that work. Servers expose tools with typed inputs and outputs; clients (Claude Desktop, Cursor, Cline, Continue, Zed, and others) speak the same protocol; the model decides when to call. Database vendors have followed: ClickHouse, Snowflake, MotherDuck, BigQuery, and Supabase all ship official servers.
-The Doris MCP Server is the equivalent for Apache Doris. It is a separate repo (`apache/doris-mcp-server`), it ships under Apache 2.0, and you launch it from any MCP client config in a few lines.
+The Apache Doris MCP Server is the equivalent for Apache Doris. It lives in a separate repo (`apache/doris-mcp-server`), it ships under Apache 2.0, and you launch it from any MCP client config in a few lines.
-## What it is {#what}
+## What is the Apache Doris MCP Server? {#what}
-The server is a Python 3.12 service built on FastAPI. It connects to Doris over the MySQL protocol, registers a fixed set of MCP tools, and serves them over `stdio`, Server-Sent Events, or the streamable HTTP transport. AI assistants treat each tool as a function call. The server logs every call, applies a SQL security filter, and returns JSON.
+The Apache Doris MCP Server is a Python 3.12 service built on FastAPI. It connects to Apache Doris over the MySQL protocol, registers a fixed set of MCP tools, and serves them over `stdio`, Server-Sent Events, or the streamable HTTP transport. AI assistants treat each tool as a function call. The server logs every call, applies a SQL security filter, and returns JSON.
**Key terms**
- **MCP (Model Context Protocol)**: an open JSON-RPC 2.0 protocol for connecting LLM clients to external tools and data. Tools are typed functions; resources are read-only data; prompts are reusable templates.
-- **Tool**: one Python function decorated with `@mcp.tool()`. The Doris server ships eight: `exec_query`, `get_db_list`, `get_db_table_list`, `get_table_schema`, `get_table_comment`, `get_table_column_comments`, `get_table_indexes`, and `get_recent_audit_logs`.
+- **Tool**: one Python function decorated with `@mcp.tool()`. The Apache Doris server ships eight: `exec_query`, `get_db_list`, `get_db_table_list`, `get_table_schema`, `get_table_comment`, `get_table_column_comments`, `get_table_indexes`, and `get_recent_audit_logs`.
- **Transport**: how the client and server talk. `stdio` runs the server as a subprocess (the default for Claude Desktop). SSE and Streamable HTTP are for remote deployments.
- **SQL security filter**: a server-side guard, on by default, that blocks `DROP`, `DELETE`, `INSERT`, `UPDATE`, `ALTER`, and `CREATE`, and adds an automatic `LIMIT` to bare `SELECT` statements.
-## How it works {#how}
+## How does the Apache Doris MCP Server work? {#how}
+
+The Apache Doris MCP Server runs through a five-step loop: the client launches the server, the server connects to the cluster, the model picks a tool, the server filters and runs the SQL, and results return as JSON.
1. **The client launches the server.** In `stdio` mode, Claude Desktop or Cursor spawns the server as a subprocess and communicates over stdin/stdout. In SSE or HTTP mode, you run the server long-lived and the client connects over the network.
-2. **The server connects to Doris.** It reads `DB_HOST`, `DB_PORT`, `DB_USER`, `DB_PASSWORD`, and `DB_DATABASE` from environment variables, then opens a MySQL-protocol connection on port 9030. No JDBC URL, no driver setup.
+2. **The server connects to Apache Doris.** It reads `DB_HOST`, `DB_PORT`, `DB_USER`, `DB_PASSWORD`, and `DB_DATABASE` from environment variables, then opens a MySQL-protocol connection on port 9030. No JDBC URL, no driver setup.
3. **The model calls a tool.** The assistant decides, given the user's prompt, which tool to invoke. For "what tables hold order data?", that is `get_db_table_list`. For "summarize yesterday's slow queries," that is `get_recent_audit_logs`. The user typically approves each call before it runs.
4. **The server filters and runs the query.** `exec_query` parses the statement, rejects anything that mutates data when `ENABLE_SQL_SECURITY_CHECK=true`, and appends a `LIMIT` if the query has none. A 30-second timeout (configurable per call) caps runtime.
5. **Results return as JSON.** The client renders them inline in the chat. Large result sets are truncated by `max_rows`, default 100, so a careless `SELECT *` does not blow up the model's context window.
@@ -80,7 +82,9 @@ information_schema, mysql, ssb, tpch_100
The assistant can now compose follow-up calls: `get_db_table_list('ssb')`, then `get_table_schema('lineorder', 'ssb')`, then a plain `exec_query` once it has the column names.
-## When to use it {#when}
+## When should you use the Apache Doris MCP Server? {#when}
+
+The Apache Doris MCP Server fits read-mostly AI assistant scenarios, especially schema discovery, ad-hoc analysis, and on-call investigation against a real cluster.
**Good fit**
@@ -95,12 +99,12 @@ The assistant can now compose follow-up calls: `get_db_table_list('ssb')`, then
- Untrusted data. An attacker who can put text into a row your assistant later reads can attempt prompt injection. The community has documented real incidents on Postgres MCP servers; treat anything the model fetches as data, not instructions, and review tool calls before running them. See [MCP security best practices](https://modelcontextprotocol.io/docs/tutorials/security/security_best_practices).
- Browsing multi-million-row tables. Tool results land in the model's context window, and the per-token bill scales accordingly. Cap `max_rows`, ask the model to write aggregations, and reach for a notebook for anything beyond a sample.
- Multi-tenant clusters with no row-level scoping. The server connects with one MySQL account; whatever that account can see, the model can see. Create a dedicated read-only user, restrict its database grants, and never reuse a power-user account.
-- Workloads that need fine-grained, programmable tool access. The eight tools cover schema and read paths well, but anything beyond that (custom workflows, batch jobs, NL2SQL with user-defined prompts) belongs in a custom integration that calls Doris directly.
+- Workloads that need fine-grained, programmable tool access. The eight tools cover schema and read paths well, but anything beyond that (custom workflows, batch jobs, NL2SQL with user-defined prompts) belongs in a custom integration that calls Apache Doris directly.
## Further reading {#further-reading}
- [AI on Doris](../ai/ai-overview): how the MCP server fits alongside `AI_*` SQL functions, embeddings, and vector search.
- [LLM SQL Functions](./llm-sql-functions): the in-database side of the AI story, for batch enrichment of text columns.
- [Doris MCP Server repository](https://github.com/apache/doris-mcp-server): source, install instructions, and the full property list for each tool.
-- [Model Context Protocol specification](https://modelcontextprotocol.io/specification/2025-11-25): the protocol Doris speaks, including transport details and security guidance.
+- [Model Context Protocol specification](https://modelcontextprotocol.io/specification/2025-11-25): the protocol Apache Doris speaks, including transport details and security guidance.
- [Anthropic's MCP announcement](https://www.anthropic.com/news/model-context-protocol): the original problem statement and the design choices behind tools, resources, and prompts.
diff --git a/docs-next/key-features/metadata-cache.mdx b/docs-next/key-features/metadata-cache.mdx
index 2a9cc7bfd48..c154f5b73c1 100644
--- a/docs-next/key-features/metadata-cache.mdx
+++ b/docs-next/key-features/metadata-cache.mdx
@@ -20,12 +20,12 @@ featureCard:
- performance
---
-> **TL;DR** When Doris queries [Hive, Iceberg, Hudi, or Paimon](./multi-catalog), most of the wall clock can be spent listing partitions, parsing manifests, and asking the Hive Metastore questions. Doris caches all of it: schemas, partition values, file listings, Iceberg manifest entries, Paimon snapshots. Caches are TTL-bounded, can refresh from Hive Metastore events for seconds-level lag, and can be invalidated explicitly with `REFRESH CATALOG`. Iceberg and Paimon get format-aware inva [...]
+> **TL;DR** When Apache Doris queries [Hive, Iceberg, Hudi, or Paimon](./multi-catalog), most of the wall clock can be spent listing partitions, parsing manifests, and asking the Hive Metastore questions. The Apache Doris metadata cache holds all of it: schemas, partition values, file listings, Iceberg manifest entries, Paimon snapshots. Caches are TTL-bounded (defaults: 24 hours for name lists, 8 hours for partitions and files), can refresh from Hive Metastore events for seconds-level l [...]
-## Why it matters {#why}
+## Why use the Apache Doris metadata cache? {#why}
-Lakehouse query planning is dominated by metadata, not by data. A query against a [partitioned](./partitioning-and-bucketing) Hive table needs to:
+The Apache Doris metadata cache cuts lakehouse query planning from seconds to milliseconds by absorbing the HMS round-trips, S3/HDFS LISTs, and Iceberg manifest reads that otherwise dominate the wall clock. Lakehouse query planning is dominated by metadata, not by data. A query against a [partitioned](./partitioning-and-bucketing) Hive table needs to:
- Round-trip the Hive Metastore for the database list, table list, column schema, and partition list.
- Issue an S3 or HDFS LIST against each surviving partition path to discover the data files.
@@ -40,9 +40,9 @@ A cold catalog with 10K partitions can spend longer in planning than in executio
Metadata Cache absorbs all of these, with refresh paths that recover correctness whenever the source actually changes.
-## What it is {#what}
+## What is the Apache Doris metadata cache? {#what}
-Metadata Cache is an in-memory cache on the Doris FE for everything an external catalog can tell you, short of the file bytes themselves. Each backend (Hive, Iceberg, Hudi, Paimon, JDBC) has its own implementation of a shared framework, so the user-facing model is consistent: per-cache TTL, per-cache size limit, and a small set of refresh primitives.
+The Apache Doris metadata cache is an in-memory cache on the FE for everything an external catalog can tell you, short of the file bytes themselves. Each backend (Hive, Iceberg, Hudi, Paimon, JDBC) has its own implementation of a shared framework, so the user-facing model is consistent: per-cache TTL, per-cache size limit, and a small set of refresh primitives.
**Key terms**
@@ -52,16 +52,16 @@ Metadata Cache is an in-memory cache on the Doris FE for everything an external
- **`Manifest cache`**: parsed Iceberg manifest entries (lists of data and delete files for a manifest). Bounded by entry count, not TTL.
- **`HMS event listener`**: a polling client that consumes Hive Metastore notification events and incrementally invalidates affected entries.
-## How it works {#how}
+## How does the Apache Doris metadata cache work? {#how}
-A query traverses the cache before issuing any external RPC.
+Every Apache Doris query traverses the cache before issuing any external RPC: lookup, TTL eviction, size eviction, format-aware invalidation, HMS event polling, and explicit `REFRESH` together keep the hot metadata in memory and stale entries out.
1. **Lookup.** When the planner asks for a table's schema, partitions, or files, the cache is checked first using a Caffeine-backed loader. A miss triggers a fetch from the underlying catalog (HMS Thrift call, Iceberg metadata read, Hudi timeline scan) and the result is stored.
2. **TTL eviction.** Each cache module has its own TTL. After 3.0.7, the timer resets on every access, so a hot table stays cached as long as it's queried. Defaults are conservative: name lists at 24 hours with a 10-minute refresh, partition and file caches at 8 hours.
3. **Size eviction.** Caches use LRU under a size cap (`max_external_schema_cache_num`, `max_external_file_cache_num`, etc.) so a long tail of cold tables can't displace the hot ones forever.
4. **Format-aware invalidation.** Iceberg manifest cache entries are keyed by manifest path and live without a TTL: once a manifest is parsed, it's correct as long as some snapshot still references it. Paimon caches the latest snapshot reference and reloads only when the table moves forward. Hive does not have this luxury, so HMS events fill the gap.
5. **HMS event-based refresh.** When `enable_hms_events_incremental_sync` is on, the FE polls the Hive Metastore notification log every `hms_events_polling_interval_ms` and replays CREATE / ALTER / DROP / INSERT events. Affected entries are invalidated within seconds, without a global flush.
-6. **Explicit refresh.** `REFRESH CATALOG`, `REFRESH DATABASE`, and `REFRESH TABLE` provide the escape hatch for writes that don't go through HMS (direct file uploads, Iceberg commits from outside Doris) and for testing.
+6. **Explicit refresh.** `REFRESH CATALOG`, `REFRESH DATABASE`, and `REFRESH TABLE` provide the escape hatch for writes that don't go through HMS (direct file uploads, Iceberg commits from outside Apache Doris) and for testing.
The common path serves metadata at memory latency. The recovery path catches up in seconds when the source moves.
@@ -100,14 +100,14 @@ ALTER CATALOG hive_ctl SET PROPERTIES (
After warmup, hit rates are above 99% on the schema and partition caches. Misses correspond to the first access of new tables or to entries that aged out.
-## When to use it {#when}
+## When should you use the Apache Doris metadata cache? {#when}
-The Metadata Cache is on by default for every external catalog; the question is how to tune it.
+The Apache Doris metadata cache is on by default for every external catalog; the practical question is how to tune it for each workload shape, and where to disable it entirely.
**Good fit**
- Hive tables with stable schemas and a moderate write rate. Pair the default TTLs with the HMS event listener for seconds-level freshness.
-- Iceberg, Paimon, and Hudi tables of any size. Snapshot immutability lets the cache hold manifest entries indefinitely; the only thing that needs invalidating is the current snapshot pointer, which Doris does on every query.
+- Iceberg, Paimon, and Hudi tables of any size. Snapshot immutability lets the cache hold manifest entries indefinitely; the only thing that needs invalidating is the current snapshot pointer, which Apache Doris does on every query.
- Multi-tenant lakehouse with thousands of catalogs and tables. Schema and partition caches keep planning fast even when the long tail of objects exceeds memory: hot ones stay, cold ones get evicted.
- High-concurrency dashboards. Without a cache, every concurrent query would re-list the same partitions; with one, the listing is computed once and shared.
@@ -127,3 +127,4 @@ The Metadata Cache is on by default for every external catalog; the question is
- [Data Cache & Page Cache](./data-cache-page-cache): the byte-level storage caches that sit beneath the metadata cache.
- [Parquet Reader Optimization](./parquet-reader-optimization): once metadata says which files to read, this is what reads them efficiently.
- [Building the next-generation data lakehouse (Apache Doris blog)](https://doris.apache.org/blog/Building-the-Next-Generation-Data-Lakehouse-10X-Performance/): the architectural overview that introduces the Meta Cache and HMS event listener.
+- [Iceberg](./iceberg): the catalog whose manifest list and snapshot metadata benefit most from the metadata cache on remote object stores.
diff --git a/docs-next/key-features/mpp.mdx b/docs-next/key-features/mpp.mdx
new file mode 100644
index 00000000000..202b93bfcf1
--- /dev/null
+++ b/docs-next/key-features/mpp.mdx
@@ -0,0 +1,120 @@
+---
+title: MPP Architecture
+description: Doris's massively parallel processing layer that turns one SQL query into a DAG of fragment instances running across every BE in a shared-nothing cluster.
+keywords:
+ - MPP
+ - massively parallel processing
+ - PlanFragment
+ - Coordinator
+ - ExchangeNode
+ - shared-nothing
+ - Apache Doris architecture
+ - distributed query execution
+slug: /key-features/mpp
+image: /images/next/key-features/mpp.jpg
+last_update:
+ date: 2026-05-11
+ author: Apache Doris
+featureCard:
+ tags:
+ - query-acceleration
+ - performance
+---
+
+> **TL;DR** Apache Doris MPP turns one SQL query into a DAG of `PlanFragment`s that run in parallel on every BE in the cluster. The FE-side `Coordinator` plans the DAG with the Nereids optimizer, ships each fragment to its BEs over BRPC, and stitches fragments together with `ExchangeNode` + `DataStreamSink` shuffles. Shared-nothing storage and a per-query memory budget (`exec_mem_limit`, default 2 GB) let the cluster grow horizontally: every BE you add brings more CPU, memory, and local [...]
+
+
+
+## Why use MPP in Apache Doris? {#why}
+
+Apache Doris MPP exists so a single analytical query can use every CPU in the cluster instead of one machine's. A 10-billion-row aggregation, a five-way join across fact tables, a wide GROUP BY on a year of logs: none of these finish in human time on one node. The MPP layer is what turns "buy a bigger box" into "add more BEs."
+
+The pain it solves is concrete:
+
+- **Single-node planners run out of headroom.** A query that touches a terabyte of columnar data needs the scan, filter, join, and aggregate work spread across machines instead of stacked on one.
+- **Manual sharding is a tax on the data team.** Hand-written union queries over per-shard tables work, until the join keys move or a shard rebalances.
+- **Storage-bound bottlenecks waste compute.** Pulling shards into one box to compute means the network does the work the cluster could have done in place.
+
+Apache Doris MPP gives the planner permission to split the query, send compute to the data, and shuffle only the rows that actually need to cross the wire.
+
+## What is the Apache Doris MPP architecture? {#what}
+
+Apache Doris MPP is a shared-nothing distributed execution model where the Frontend (FE) plans the query as a DAG of `PlanFragment`s and the Backends (BEs) run those fragments in parallel against their local tablets. The Nereids optimizer compiles the SQL into a logical plan, splits it on shuffle boundaries, and the FE `Coordinator` schedules each `PlanFragment` to a set of BEs. Each BE materializes the fragment into one or more **fragment instances**, which the BE-internal pipeline sche [...]
+
+**Key terms**
+
+- **`PlanFragment`**: the smallest unit of distributed work the FE ships to a BE. One query is one DAG of fragments.
+- **Fragment instance**: a runtime copy of a `PlanFragment` running on one BE. Multiple instances of the same fragment on different BEs give you horizontal parallelism.
+- **`Coordinator`**: the FE-side object that builds the fragment DAG, assigns BEs, ships fragments, collects results, and handles failures. Implemented in `fe/fe-core/src/main/java/org/apache/doris/qe/Coordinator.java`.
+- **`ExchangeNode` / `DataStreamSink`**: the receiver / sender pair that moves data between fragments over the network. The planner inserts them on every shuffle boundary.
+- **Distribution mode**: how rows are routed across the network on a shuffle: `UNPARTITIONED`, `RANDOM`, `HASH_PARTITIONED`, or `BUCKET_SHFFULE_HASH_PARTITIONED`.
+- **Nereids planner**: the Cascades-style cost-based optimizer that picks the join order, the shuffle method, and the parallelism. The legacy planner has been deleted; the `enable_nereids_planner` switch is marked `REMOVED`.
+
+## How does the Apache Doris MPP architecture work? {#how}
+
+Apache Doris MPP works in five stages: the FE plans, splits, and ships fragments; each BE turns its fragment into pipeline tasks; tablets are scanned locally; data is shuffled through `ExchangeNode`s only where the plan requires it; and a single root fragment returns the result to the client.
+
+1. **Plan and split.** The Nereids optimizer rewrites the SQL into a physical plan and the FE walks the tree to cut it into `PlanFragment`s at every point where data has to redistribute (an aggregation that needs all keys on one machine, a hash join that needs both sides hash-partitioned the same way, the final result sink).
+2. **Pick a shuffle method.** For each fragment boundary the planner picks one of the four distribution modes, and for hash joins picks one of four strategies: **Broadcast Join** (copy the right side to every left instance), **Shuffle Join** (hash-partition both sides on the join key), **Bucket Shuffle Join** (reuse the left table's bucket layout to ship only the right side), or **Colocate Join** (skip the shuffle entirely because both tables already live on the same BEs). Cost determine [...]
+3. **Ship and instantiate.** The `Coordinator` sends each fragment to its assigned BEs over BRPC. Each BE creates fragment instances, one per parallelism slot, and hands them to the Pipeline Execution Engine for scheduling.
+4. **Scan local, shuffle when needed.** Scan operators read tablets that live on the same BE, so the bulk data never crosses the network. Only the rows the next fragment needs, serialized as columnar `Block`s, cross the wire through `DataStreamSink` → `ExchangeNode`.
+5. **Return.** A root fragment with an `UNPARTITIONED` sink runs on one BE, collects the merged result, and sends it back to the FE, which streams it to the client over the MySQL protocol.
+
+## Quick start {#quick-start}
+
+There is nothing to enable: MPP is how every query runs. The most useful operator-facing tool is `EXPLAIN`, which shows the fragment DAG, distribution modes, and shuffle strategies the optimizer picked.
+
+```sql
+EXPLAIN
+SELECT o.o_orderpriority, COUNT(*) AS orders, SUM(l.l_extendedprice) AS revenue
+FROM orders o
+JOIN lineitem l ON o.o_orderkey = l.l_orderkey
+WHERE o.o_orderdate >= '2026-01-01'
+GROUP BY o.o_orderpriority
+ORDER BY revenue DESC
+LIMIT 10;
+```
+
+**Expected result (abridged)**
+
+```
+PLAN FRAGMENT 2 (BUCKET_SHFFULE_HASH_PARTITIONED on o_orderkey)
+ HASH JOIN join op: INNER JOIN ... (Bucket Shuffle)
+ SCAN orders (partitions=12, tablets=120, ...)
+ EXCHANGE HASH_PARTITIONED <-- from FRAGMENT 1
+PLAN FRAGMENT 1 (RANDOM)
+ SCAN lineitem (partitions=48, tablets=480, ...)
+PLAN FRAGMENT 0 (UNPARTITIONED)
+ RESULT SINK
+ AGGREGATE (merge finalize)
+ EXCHANGE HASH_PARTITIONED <-- from FRAGMENT 2
+```
+
+Three fragments. Fragment 1 scans `lineitem` on every BE that holds a tablet, hashes rows on `l_orderkey`, and pushes them into Fragment 2 over an exchange. Fragment 2 joins them against `orders` using Bucket Shuffle (`orders` is bucketed on `o_orderkey`, so its tablets don't move), pre-aggregates on `o_orderpriority`, and pushes the partial groups into Fragment 0. Fragment 0 finalizes the aggregation, sorts, and writes the result.
+
+## When should you use the Apache Doris MPP architecture? {#when}
+
+Apache Doris MPP is the default execution mode for every analytical query: there is no other path for general SELECTs. Tune around it, not against it.
+
+**Good fit**
+
+- Analytical scans across tens of millions to billions of rows. Wide GROUP BYs, large joins, time-window aggregations.
+- Star-schema and snowflake joins. Picking Bucket Shuffle or Colocate Join over Broadcast is exactly what the Nereids optimizer is built for.
+- Multi-tenant clusters where queries should land on every BE. Pair MPP with [Workload Group](./workload-group.mdx) for CPU and memory isolation per tenant.
+- Long-running queries on skewed data. The MPP planner's shuffle decisions feed the BE's Local Shuffle, which smooths skew inside each BE.
+
+**Not a good fit**
+
+- Single-row primary-key lookups at thousands of QPS. The cost of planning, shipping fragments, and round-tripping over BRPC dwarfs the work; use [High-Concurrency Point Query](./high-concurrency-point-query.mdx) instead.
+- Treating MPP as a row-by-row OLTP engine. There is no per-row transaction layer; updates go through Unique Key tables and bulk load paths.
+- Confusing MPP with Pipeline or vectorized execution. The three are stacked, not interchangeable. [Pipeline Execution Engine](./pipeline-execution-engine.mdx) runs operators inside one BE; [Vectorized Execution](./vectorized-execution.mdx) runs the inner loop over column batches; MPP coordinates them across BEs.
+- Tiny tables on a tiny cluster. If the whole table fits in one fragment's memory and one BE's CPU finishes the work in under a second, a Broadcast Join on a single BE is what the planner will pick, and that is the right answer.
+
+## Further reading {#further-reading}
+
+- [Product concepts: Fragment, Instance, Exchange](../features-architecture/product-concepts.md): the canonical user-facing definitions the rest of the docs build on.
+- [System architecture](../features-architecture/system-architecture.md): FE/BE roles, BDBJE for FE consensus, and where the Coordinator sits in the topology.
+- [Join optimization and shuffle strategies](../query-data/join.md): the cost trade-offs between Broadcast, Shuffle, Bucket Shuffle, and Colocate Join, with examples.
+- [Pipeline Execution Engine](./pipeline-execution-engine.mdx): the BE-internal scheduler MPP hands fragments to. Read this for the per-BE story.
+- [Parallelism tuning](../query-acceleration/tuning/parallelism-tuning.md): when and how to override `parallel_pipeline_task_num` per workload.
+- [Evolution of the Apache Doris execution engine](/blog/evolution-of-the-apache-doris-execution-engine/): the design history from volcano to Pipeline to PipelineX, with the MPP planner shifts along the way.
diff --git a/docs-next/key-features/multi-catalog.mdx b/docs-next/key-features/multi-catalog.mdx
index dcc6c4ec187..c1fcc3b4410 100644
--- a/docs-next/key-features/multi-catalog.mdx
+++ b/docs-next/key-features/multi-catalog.mdx
@@ -20,37 +20,39 @@ featureCard:
- external-catalog
---
-> **TL;DR** Multi Catalog turns one Doris cluster into a federation. You register external systems (Hive, Iceberg, Hudi, Paimon, Delta Lake, MySQL, PostgreSQL, Oracle, Elasticsearch, BigQuery, MaxCompute, and more) as named catalogs, address every table as `<catalog>.<db>.<table>`, and join across them in a single SQL statement. The internal warehouse is just another catalog called `internal`. No copy job, no second query engine, one permission model.
+> **TL;DR** Apache Doris Multi Catalog turns one cluster into a federation that registers external systems (Hive, Iceberg, Hudi, Paimon, Delta Lake, MySQL, PostgreSQL, Oracle, Elasticsearch, BigQuery, MaxCompute, and more) as named catalogs. Every table is addressed as `<catalog>.<db>.<table>` and joined across catalogs in a single SQL statement. The internal warehouse is just another catalog called `internal`, so one permission model covers all sources with no copy job and no second que [...]
-## Why it matters {#why}
+## Why use Apache Doris Multi Catalog? {#why}
-Most analytics teams already run more than one storage system. Fact tables sit in Hive or Iceberg on object storage. Dimensions and operational state live in MySQL or PostgreSQL. Text-heavy workloads land in Elasticsearch. The warehouse holds the curated marts. Asking a question that crosses two of these usually means a copy job, a CDC pipeline, or a second query engine bolted on the side. Each of those costs you freshness or money, often both. Plugging in a new system through [catalog i [...]
+Apache Doris Multi Catalog removes the copy job, CDC pipeline, or second query engine that teams usually stand up to query across more than one storage system. Most analytics teams already run more than one storage system. Fact tables sit in Hive or Iceberg on object storage. Dimensions and operational state live in MySQL or PostgreSQL. Text-heavy workloads land in Elasticsearch. The warehouse holds the curated marts. Asking a question that crosses two of these usually costs you freshnes [...]
-Multi Catalog removes the wrapper. Doris connects to each system once, learns its catalog of databases and tables, and exposes them under a name you choose. From then on, the table lives at `<catalog>.<db>.<table>` and behaves like any other table in your queries.
+Apache Doris connects to each system once, learns its catalog of databases and tables, and exposes them under a name you choose. From then on, the table lives at `<catalog>.<db>.<table>` and behaves like any other table in your queries.
- Joining a fact table on Iceberg with a dimension table on MySQL is one SELECT, not a pipeline.
- The warehouse and the lake share one SQL surface, one user, one privilege model.
- Adding a new source means a `CREATE CATALOG`, not a deploy.
-## What it is {#what}
+## What is the Apache Doris Multi Catalog? {#what}
-Multi Catalog is a federation layer at the top of Doris's metadata stack. Doris organizes everything in a three-level namespace: catalog, database, table. The built-in catalog `internal` holds Doris's own tables. Every other catalog is an external catalog backed by a connector that knows how to talk to one kind of source. The connector handles metadata discovery, type mapping, partition pruning, and read (and increasingly write) operations. The query optimizer plans across all of them at [...]
+The Apache Doris Multi Catalog is a federation layer at the top of the metadata stack that organizes everything in a three-level namespace: catalog, database, table. The built-in catalog `internal` holds Apache Doris's own tables. Every other catalog is an external catalog backed by a connector that knows how to talk to one kind of source. The connector handles metadata discovery, type mapping, partition pruning, and read (and increasingly write) operations. The query optimizer plans acr [...]
**Key terms**
-- **`internal`**: the built-in catalog that holds Doris's native tables. Cannot be created, renamed, or dropped.
-- **External catalog**: any catalog created with `CREATE CATALOG ... PROPERTIES ("type" = "...")`. Doris ships connectors for `hms` (Hive and Hive-compatible metastores), `iceberg`, `hudi`, `paimon`, `jdbc`, `es`, `max_compute`, `bigquery`, `kafka`, `trino-connector`, and more.
+- **`internal`**: the built-in catalog that holds Apache Doris's native tables. Cannot be created, renamed, or dropped.
+- **External catalog**: any catalog created with `CREATE CATALOG ... PROPERTIES ("type" = "...")`. Apache Doris ships connectors for `hms` (Hive and Hive-compatible metastores), `iceberg`, `hudi`, `paimon`, `jdbc`, `es`, `max_compute`, `bigquery`, `kafka`, `trino-connector`, and more.
- **Three-part name**: `catalog.db.table`. Always resolves the same way, whether referenced inside a single catalog session or across catalogs.
- **`SWITCH` and `USE`**: `SWITCH iceberg_ctl` flips the session's current catalog (similar to `USE` for databases). `USE iceberg_ctl.iceberg_db` switches catalog and database in one statement.
- **Metadata cache**: the FE-side cache that holds schemas, partition lists, and file listings for external catalogs so planning stays fast. Refreshed by TTL, by source-side events, or by `REFRESH CATALOG`.
-## How it works {#how}
+## How does the Apache Doris Multi Catalog work? {#how}
-1. **Register a catalog.** `CREATE CATALOG <name> PROPERTIES ("type" = "<type>", ...)` records the connection details on the FE. The properties are persisted in the FE edit log; nothing else moves. Doris does not copy schemas or data.
+The Apache Doris Multi Catalog records each external source as a named connector on the FE, then resolves every table reference as a three-part name that the optimizer plans across in one MPP query.
+
+1. **Register a catalog.** `CREATE CATALOG <name> PROPERTIES ("type" = "<type>", ...)` records the connection details on the FE. The properties are persisted in the FE edit log; nothing else moves. Apache Doris does not copy schemas or data.
2. **Discover lazily.** The first time a session touches the catalog, the connector lists its databases and tables, fetches schemas, and populates the metadata cache. See [Metadata Cache](./metadata-cache) for what gets cached and for how long.
3. **Resolve three-part names.** Every table reference is parsed as `catalog.db.table`. Unqualified names use the session's current catalog and database. `SWITCH`, `USE`, and the user property `default_init_catalog` control where unqualified names land.
-4. **Plan one query.** The optimizer pulls statistics from each catalog's connector, picks join order and predicate pushdown, and produces a single distributed plan. The plan can mix scans against `internal` Doris tablets, Iceberg manifests, MySQL JDBC fetches, and ES queries inside the same fragment graph.
+4. **Plan one query.** The optimizer pulls statistics from each catalog's connector, picks join order and predicate pushdown, and produces a single distributed plan. The plan can mix scans against `internal` Apache Doris tablets, Iceberg manifests, MySQL JDBC fetches, and ES queries inside the same fragment graph.
5. **Stay current.** TTLs and source-side events (Hive Metastore events, Iceberg snapshot pointers, Paimon snapshot reloads) keep the cache aligned with the source. `REFRESH CATALOG`, `REFRESH DATABASE`, and `REFRESH TABLE` cover the rest.
## Quick start {#quick-start}
@@ -92,21 +94,23 @@ WHERE f.dt = CURRENT_DATE() GROUP BY 1, 3;
`SHOW CATALOGS` lists `internal`, `hive_ctl`, and `mysql_ctl`. The query plans and runs as one MPP job; no staging table needed.
-## When to use it {#when}
+## When should you use the Apache Doris Multi Catalog? {#when}
+
+Use the Apache Doris Multi Catalog whenever a query needs to touch the warehouse and one or more external sources without an intermediate ETL stage.
**Good fit**
-- Federated analytics across a lake (Hive, Iceberg, Hudi, Paimon, Delta Lake) and the Doris warehouse, without ETL into Doris first.
+- Federated analytics across a lake (Hive, Iceberg, Hudi, Paimon, Delta Lake) and the Apache Doris warehouse, without ETL into Apache Doris first.
- Joining warehouse facts with operational dimensions in MySQL, PostgreSQL, Oracle, SQL Server, or other JDBC sources.
- Pulling text matches from Elasticsearch into a SQL pipeline alongside structured filters.
- Migrations and dual-running. Point Doris at the existing Hive or external warehouse, query through Doris, and move workloads over piece by piece.
-- ZeroETL data integration. `INSERT INTO internal.x SELECT ... FROM hive_ctl.y` ingests into Doris without a separate loader.
+- ZeroETL data integration. `INSERT INTO internal.x SELECT ... FROM hive_ctl.y` ingests into Apache Doris without a separate loader.
**Not a good fit**
-- Treating an external catalog as a write-heavy OLTP target. Write-back exists for Hive, Iceberg, and JDBC catalogs, but throughput is bound by the source and is not on par with writes into `internal` Doris tables. Land hot writes in `internal` and write back in batches.
-- Transactional updates against external sources. Doris does not offer cross-catalog transactions, and most external catalogs do not support row-level updates the way Doris does. Use the source's own DML for that.
-- Cross-catalog DDL. `CREATE TABLE ... AS SELECT` works across catalogs as a copy, but creating the *same* table in two catalogs is your responsibility, not Doris's.
+- Treating an external catalog as a write-heavy OLTP target. Write-back exists for Hive, Iceberg, and JDBC catalogs, but throughput is bound by the source and is not on par with writes into `internal` Apache Doris tables. Land hot writes in `internal` and write back in batches.
+- Transactional updates against external sources. Apache Doris does not offer cross-catalog transactions, and most external catalogs do not support row-level updates the way the internal catalog does. Use the source's own DML for that.
+- Cross-catalog DDL. `CREATE TABLE ... AS SELECT` works across catalogs as a copy, but creating the *same* table in two catalogs is your responsibility, not Apache Doris's.
- Stitching two compute pools across a single query. Federation is about data sources, not compute. For compute isolation see [Compute Group](./compute-group).
- Hand-tuning every connector here. For source-specific configuration (AWS Glue, Iceberg REST, individual JDBC dialects, S3-compatible storages, Kerberos, IAM), see the dedicated [Catalog Integrations](./catalog-integrations) card and the per-catalog reference docs.
@@ -119,3 +123,4 @@ WHERE f.dt = CURRENT_DATE() GROUP BY 1, 3;
- [Lakehouse Overview](../lakehouse/lakehouse-overview): where Multi Catalog sits in Doris's lakehouse story.
- [CREATE CATALOG reference](../sql-manual/sql-statements/catalog/CREATE-CATALOG): the full DDL syntax, supported types, and property reference.
- [SWITCH CATALOG reference](../sql-manual/sql-statements/session/context/SWITCH-CATALOG): session-level switching semantics.
+- [Iceberg](./iceberg): the deepest first-class connector in Doris's federated catalog stack — full read and write engine, not just a reader.
diff --git a/docs-next/key-features/parquet-reader-optimization.mdx b/docs-next/key-features/parquet-reader-optimization.mdx
index 89a1388acde..67f5905cc05 100644
--- a/docs-next/key-features/parquet-reader-optimization.mdx
+++ b/docs-next/key-features/parquet-reader-optimization.mdx
@@ -19,12 +19,12 @@ featureCard:
- query-acceleration
---
-> **TL;DR** When Doris queries Parquet files in an [external catalog](./multi-catalog) or through a TVF, it uses its own C++ [vectorized reader](./vectorized-execution) instead of going through Java or Arrow. The reader prunes whole row groups and individual pages from the file's own statistics, evaluates predicates on dictionary IDs, reads filter columns before payload columns, and coalesces remote reads into fewer requests. Most queries finish without ever decoding the columns they don [...]
+> **TL;DR** Apache Doris ships a native C++ [vectorized Parquet reader](./vectorized-execution) for [external catalogs](./multi-catalog) and TVFs, avoiding any Java or Arrow round trip. The reader prunes whole row groups and individual pages from the file's own statistics, evaluates predicates on dictionary IDs, and reads filter columns before payload columns. Remote reads are coalesced into fewer requests, so most queries finish without ever decoding the columns they don't need.
-## Why it matters {#why}
+## Why use the Parquet reader in Apache Doris? {#why}
-A Hive or Iceberg table on S3 is, in the end, a pile of Parquet files. Reading them is dominated by three costs: [fetching bytes from object storage](./data-cache-page-cache), decompressing and decoding pages, and converting values into the engine's runtime representation. A naive reader pays all three on every query.
+The Apache Doris native Parquet reader exists to cut the three dominant lakehouse scan costs: [fetching bytes from object storage](./data-cache-page-cache), decompressing and decoding pages, and converting values into the engine's runtime representation. A Hive or Iceberg table on S3 is, in the end, a pile of Parquet files, and a naive reader pays all three costs on every query.
That cost shows up in shapes anyone who has tuned a lakehouse query has hit:
@@ -33,11 +33,11 @@ That cost shows up in shapes anyone who has tuned a lakehouse query has hit:
- A `SELECT *` over a wide event table on S3 issues hundreds of small HTTP GETs per file, one per column chunk.
- A predicate like `country = 'JP'` decompresses and decodes UTF-8 across millions of rows, even though the column is dictionary-encoded and the answer could be one integer comparison per row.
-Doris's reader is built to avoid each of these before any page leaves storage.
+The Apache Doris Parquet reader is built to avoid each of these before any page leaves storage.
-## What it is {#what}
+## What is the Apache Doris Parquet reader? {#what}
-Doris ships its own native C++ Parquet reader. Instead of going through a Java library or an Arrow round trip, it reads Parquet metadata, prunes at every available granularity (row group, page, dictionary, column), and decodes values directly into the same `Block` format the vectorized executor consumes.
+The Apache Doris Parquet reader is a native C++ component that reads Parquet metadata, prunes at every available granularity (row group, page, dictionary, column), and decodes values directly into the same `Block` format the vectorized executor consumes. There is no Java library or Arrow round trip in the path.
**Key terms**
@@ -46,9 +46,9 @@ Doris ships its own native C++ Parquet reader. Instead of going through a Java l
- **`Lazy materialization`**: a two-pass read that decodes filter columns first, then fetches payload columns only for the rows that survived.
- **`Dictionary filtering`**: evaluating a predicate against a column chunk's dictionary page once, then comparing rows against the resulting set of dictionary IDs.
-## How it works {#how}
+## How does the Apache Doris Parquet reader work? {#how}
-Each step shrinks what the next step has to look at.
+The Apache Doris Parquet reader runs a five-stage pipeline where each step shrinks what the next step has to look at.
1. **Row-group pruning.** When the file is opened, the reader walks each row group's column statistics and drops the whole group if the predicate cannot match its min/max range. The same step probes the Parquet bloom filter when the file has one. Both checks run on metadata only, before any column page is fetched. Controlled by `enable_parquet_filter_by_min_max` and `enable_parquet_filter_by_bloom_filter`.
2. **Page pruning via PageIndex.** For each surviving row group, the reader parses ColumnIndex and OffsetIndex to drop individual pages whose min/max cannot match. OffsetIndex gives the byte range of each surviving page, so the reader skips straight to the next one without scanning the rejected bytes. Toggle with the BE config `enable_parquet_page_index`.
@@ -85,9 +85,9 @@ VFileScanNode
The reader dropped 18 of 24 row groups on the date predicate, then dropped 210 pages inside the survivors using PageIndex on `status`. Lazy materialization skipped decoding `region` and `amount` for the 92.1% of rows that didn't match. Ninety-six column chunks were merged into 14 ranged GETs against S3.
-## When to use it {#when}
+## When should you use the Apache Doris Parquet reader? {#when}
-This path is used automatically whenever Doris reads Parquet, so the question is how to make sure it can do its job.
+The Apache Doris Parquet reader is used automatically whenever a query reads Parquet, so the question is how to make sure it can do its job.
**Good fit**
@@ -115,3 +115,4 @@ This path is used automatically whenever Doris reads Parquet, so the question is
- [Multi-Catalog](./multi-catalog): how Parquet tables on S3, HDFS, Hive, Iceberg, and Paimon plug into Doris in the first place.
- [Catalog Integrations](./catalog-integrations): the per-source connector reference for the catalogs that hold Parquet files.
- [Building the next-generation data lakehouse (Apache Doris blog)](https://doris.apache.org/blog/Building-the-Next-Generation-Data-Lakehouse-10X-Performance/): the original public writeup of the native Parquet reader.
+- [Iceberg](./iceberg): the lakehouse table format whose Parquet scans this optimization accelerates most directly.
diff --git a/docs-next/key-features/partitioning-and-bucketing.mdx b/docs-next/key-features/partitioning-and-bucketing.mdx
index 9c9f4b76b4e..7363d021490 100644
--- a/docs-next/key-features/partitioning-and-bucketing.mdx
+++ b/docs-next/key-features/partitioning-and-bucketing.mdx
@@ -20,12 +20,12 @@ featureCard:
- performance
---
-> **TL;DR** Every Doris table is split twice: a `PARTITION BY` clause carves the rows by a column value (usually a date), and a `DISTRIBUTED BY` clause shards each partition into N tablets. Partitions give the planner something coarse to drop and the operator something to archive; buckets give the cluster something [parallel to scan and write](./pipeline-execution-engine). Pick the date column queries filter on for the partition key, pick a [high-cardinality equality column](./high-concu [...]
+> **TL;DR** Every Apache Doris table is split twice: a `PARTITION BY` clause carves rows by a column value (usually a date), and a `DISTRIBUTED BY` clause shards each partition into N tablets. Partitions give the planner something coarse to drop and the operator something to archive; buckets give the cluster something [parallel to scan and write](./pipeline-execution-engine). Pick the date column queries filter on for the partition key, pick a [high-cardinality equality column](./high-co [...]
-## Why it matters {#why}
+## Why use partitioning and bucketing in Apache Doris? {#why}
-Once a table outgrows one machine, three problems show up.
+Apache Doris partitioning and bucketing solve the three problems that show up once a table outgrows one machine: queries that scan everything, single-tablet tables that pin one BE, and deletions that drag on for hours.
- A `WHERE date BETWEEN '2026-04-01' AND '2026-04-07'` query on a 365-day events table reads every row of every day until the planner is told it can skip the other 358.
- A 4 TB table loaded into one tablet pins one BE to 100% CPU and leaves the other nine idle. The cluster has parallelism; the table does not expose it.
@@ -33,9 +33,9 @@ Once a table outgrows one machine, three problems show up.
Partitioning solves the first and third. Bucketing solves the second. Between them, they decide where each row lives and which BEs touch it at query time.
-## What it is {#what}
+## What are Apache Doris partitioning and bucketing? {#what}
-Partitioning and bucketing form Doris's two-level physical layout. The `PARTITION BY` clause divides the table into disjoint subsets by a column value (Range or List), and `DISTRIBUTED BY` further shards each partition into a fixed number of tablets, the unit of replication and parallelism. A row first finds its partition by value, then its tablet by hash (or random assignment).
+Apache Doris partitioning and bucketing are the two-level physical layout for every table. The `PARTITION BY` clause divides the table into disjoint subsets by a column value (Range or List), and `DISTRIBUTED BY` further shards each partition into a fixed number of tablets, the unit of replication and parallelism. A row first finds its partition by value, then its tablet by hash (or random assignment).
**Key terms**
@@ -48,9 +48,9 @@ Partitioning and bucketing form Doris's two-level physical layout. The `PARTITIO
- **`DISTRIBUTED BY RANDOM BUCKETS N`**: rows scattered across buckets without a key. Avoids skew, but no bucket pruning. Duplicate-Key tables only.
- **`BUCKETS AUTO`**: the FE picks the bucket count from `estimate_partition_size`, BE count, and disk count. Set per partition.
-## How it works {#how}
+## How do Apache Doris partitioning and bucketing work? {#how}
-A row's path from `INSERT` to disk runs through both layers.
+Apache Doris partitioning and bucketing together route a row's path from `INSERT` to disk through two layers, partition selection and tablet selection.
1. **Map the row to a partition.** The planner evaluates the partition expression. Range partitions binary-search a sorted interval list; List partitions look up the value in a hash map. Auto Partition creates the partition on the fly if none matches.
2. **Map the row to a bucket.** Hash distribution computes `crc32(bucket_cols) % bucket_num`; random distribution picks a tablet round-robin (or sticks the whole batch on one tablet when `load_to_single_tablet = true`).
@@ -93,7 +93,9 @@ WHERE order_date >= '2026-05-01' AND user_id = 200;
Two months of data produced two partitions automatically. The planner kept one of those partitions on the date predicate, and one tablet out of sixteen on the `user_id` equality, so the query reads 1/32 of the table. Writes spread across all sixteen tablets, so a single load uses sixteen BE workers in parallel.
-## When to use it {#when}
+## When should you use Apache Doris partitioning and bucketing? {#when}
+
+Apache Doris partitioning fits time-series and enumerated-dimension tables that need pruning or archival; bucketing fits any table large enough that one tablet cannot keep up with scans or writes.
**Good fit**
@@ -120,3 +122,5 @@ Two months of data produced two partitions automatically. The planner kept one o
- [Data Bucketing: choosing the method, the key, and the bucket count](../table-design/data-partitioning/data-bucketing)
- [Data Pruning: the layered pruning chain that runs after partition and tablet selection](./data-pruning)
- [`ALTER TABLE PARTITION` reference](../sql-manual/sql-statements/table-and-view/table/ALTER-TABLE-PARTITION)
+- [MPP Architecture](./mpp): Bucket Shuffle and Colocate Join read the bucket layout you choose here to avoid network shuffles.
+- [Unique Key](./unique-key): the table model where partition columns must be a subset of the unique key — otherwise dedup breaks.
diff --git a/docs-next/key-features/pipeline-execution-engine.mdx b/docs-next/key-features/pipeline-execution-engine.mdx
index 42a8dad7706..861a155c5e5 100644
--- a/docs-next/key-features/pipeline-execution-engine.mdx
+++ b/docs-next/key-features/pipeline-execution-engine.mdx
@@ -19,20 +19,20 @@ featureCard:
- performance
---
-> **TL;DR** Pipeline is Doris's only execution engine since v3.0. Each query fragment is split into a DAG of pipelines; each pipeline starts at a source, ends at a sink, and runs as several PipelineTasks on a fixed-size thread pool whose width is bounded by the BE's CPU budget. Operators yield instead of blocking, dependencies coordinate the DAG, and Local Shuffle re-balances skewed data inside the BE. The volcano model and its one-thread-per-fragment-instance bill are gone.
+> **TL;DR** The Apache Doris pipeline execution engine has been the only execution engine since v3.0, replacing the legacy volcano model. Each query fragment is split into a DAG of pipelines that start at a source, end at a sink, and run as PipelineTasks on a fixed-size BE thread pool bounded by the BE's CPU budget. Operators yield instead of blocking, dependencies coordinate the DAG, and Local Shuffle re-balances skewed data inside the BE.
-## Why it matters {#why}
+## Why use the pipeline execution engine in Apache Doris? {#why}
-The volcano model that Doris used to ship gave every fragment instance its own OS thread. With a few concurrent queries this is fine. With a hundred it becomes a problem: thousands of threads stack up, most of them blocked on IO or on each other, and the BE eventually thrashes the kernel scheduler and runs out of memory before it runs out of work.
+The Apache Doris pipeline execution engine fixes three problems that the legacy volcano model could not: unbounded thread counts under concurrency, idle cores on skewed joins and aggregations, and per-fragment-instance tuning that was a guessing game. The volcano model that Apache Doris used to ship gave every fragment instance its own OS thread. With a few concurrent queries this is fine. With a hundred it becomes a problem: thousands of threads stack up, most of them blocked on IO or o [...]
There are two more wounds the old engine left open. Wide-table aggregations and joins on skewed data underused the cores: when one bucket holds 7 rows and the others hold 1, the 7-row task is the long pole and the rest of the cluster sits idle waiting for it. And tuning concurrency at the fragment-instance level was a guessing game with too many knobs.
Pipeline closes all three. The thread count is bounded, not query-count-bounded. Operators yield when they cannot make progress, freeing threads instead of holding them. And Local Shuffle redistributes skewed data inside the BE so the long pole is short.
-## What it is {#what}
+## What is the Apache Doris pipeline execution engine? {#what}
-Pipeline is a push-based, async query execution model. The FE compiles the query into a Plan, splits it into PlanFragments (one per BE), and each BE breaks each fragment into a DAG of Pipelines. A Pipeline is a chain that begins with one SourceOperator (a scan or an exchange), ends with one SinkOperator (a network shuffle or a hash-table writer), and may have any number of operators in between. Pipelines that have to wait on each other are coordinated by Dependencies. Each Pipeline mater [...]
+The Apache Doris pipeline execution engine is a push-based, async query execution model that turns each PlanFragment into a DAG of pipelines scheduled on a bounded BE thread pool. The FE compiles the query into a Plan, splits it into PlanFragments (one per BE), and each BE breaks each fragment into a DAG of Pipelines. A Pipeline is a chain that begins with one SourceOperator (a scan or an exchange), ends with one SinkOperator (a network shuffle or a hash-table writer), and may have any n [...]
**Key terms**
@@ -44,7 +44,9 @@ Pipeline is a push-based, async query execution model. The FE compiles the query
- **Local Shuffle (Local Exchange)**: a Pipeline Breaker that re-shards data locally inside the BE using HASH or Round Robin, so a skewed scan does not turn into a skewed join.
- **`parallel_pipeline_task_num`**: the session knob for per-fragment parallelism. Default `0` means half the CPU cores; usually best left alone.
-## How it works {#how}
+## How does the Apache Doris pipeline execution engine work? {#how}
+
+The Apache Doris pipeline execution engine works in six stages: the FE plans and ships fragments, each BE turns a fragment into a pipeline DAG, PipelineTasks are spawned, they run on a bounded thread pool, Local Shuffle rebalances skewed data, and parallel scan keeps slow buckets from stalling the query.
1. **FE plans, splits, ships.** The FE compiles the query, inserts `ExchangeNode` and `DataSink` for inter-BE shuffle, and sends each PlanFragment to a BE.
2. **BE turns each fragment into a pipeline DAG.** Blocking operators (hash-table build, sort, aggregation) become a Sink that ends one pipeline plus a Source that starts the next one. Dependencies wire the DAG together; a Join's probe waits on its build's `set_ready`.
@@ -87,11 +89,13 @@ EXPLAIN SELECT /*+ SET_VAR(parallel_pipeline_task_num=16) */
The hint applies only to this query. `EXPLAIN` shows the pipeline IDs, the Local Exchange (if planned), and the operator chain. With profiling enabled, the runtime profile reports per-PipelineTask `ExecTime`, `YieldCounts`, and `MemoryReserveFailedTimes`, which are the levers you actually use to tune real workloads.
-## When to use it {#when}
+## When should you use the Apache Doris pipeline execution engine? {#when}
+
+The Apache Doris pipeline execution engine runs every query on v3.0+ by default, so the practical question is when to tune it rather than whether to use it. Use the default for mixed concurrent workloads, join-heavy or skewed queries, and long-running scans; only override `parallel_pipeline_task_num` when profiling shows a real bottleneck.
**Good fit**
-- Every Doris workload on v3.0+. There is no other engine.
+- Every Apache Doris workload on v3.0+. There is no other engine.
- Mixed concurrent workloads on a shared BE: the bounded thread pool keeps queries from stepping on each other.
- Join-heavy or aggregation-heavy queries on skewed input: Local Shuffle smooths the imbalance instead of letting the long pole drag the wall clock.
- Long-running scans on uneven buckets: parallel scan keeps the slow bucket from stalling the rest.
@@ -111,3 +115,4 @@ The hint applies only to this query. `EXPLAIN` shows the pipeline IDs, the Local
- [Vectorized Execution](./vectorized-execution): the operator layer Pipeline schedules. Often confused with Pipeline; the two are orthogonal.
- [Workload Group](./workload-group): the layer that binds pipeline threads to per-group cgroups for CPU and concurrency limits.
- [Evolution of the Apache Doris execution engine](/blog/evolution-of-the-apache-doris-execution-engine/): the design history of how Doris went from volcano to Pipeline to PipelineX.
+- [MPP Architecture](./mpp): the cross-BE coordinator above the Pipeline Engine — splits one SQL into a fragment DAG and assigns each fragment to BEs.
diff --git a/docs-next/key-features/pluggable-auth.mdx b/docs-next/key-features/pluggable-auth.mdx
index b04ebc55435..28f24bb65e9 100644
--- a/docs-next/key-features/pluggable-auth.mdx
+++ b/docs-next/key-features/pluggable-auth.mdx
@@ -20,22 +20,22 @@ featureCard:
- authorization
---
-> **TL;DR** Doris splits identity (who you are) from policy (what you can do), and you can swap the backend for either side without forking the FE. The MySQL handshake hits an `Authenticator` chosen by `authentication_type`: built-in password, LDAP, or any plugin discovered through Java SPI. Authorization runs through a `CatalogAccessController` chosen by `access_controller_type`: the built-in RBAC engine, or Apache Ranger for centralized policy. Set the config, drop the plugin jar, rest [...]
+> **TL;DR** Apache Doris splits identity (who you are) from policy (what you can do), and you can swap the backend for either side without forking the FE. The MySQL handshake hits an `Authenticator` chosen by `authentication_type`: built-in password, LDAP, or any plugin discovered through Java SPI. Authorization runs through a `CatalogAccessController` chosen by `access_controller_type`: the built-in RBAC engine, or Apache Ranger for centralized policy. Set the config, drop the plugin ja [...]
-## Why it matters {#why}
+## Why use pluggable authentication and authorization in Apache Doris? {#why}
-Most teams arrive at Doris with an IAM stack already in place. There is an LDAP directory, a Kerberos realm, an OIDC provider for SSO, and a Ranger or Sentry deployment that owns Hive permissions. A warehouse that ignores all of it forces the platform team to maintain a parallel user list, sync passwords by script, and explain to auditors why query history shows different identities than the rest of the lakehouse. Pair the same identity with [multi-catalog](./multi-catalog) federation an [...]
+Apache Doris pluggable authentication and authorization let an existing IAM stack (LDAP, Kerberos, OIDC, Ranger) own identity and policy for the warehouse, so the platform team does not maintain a parallel user list. Most teams arrive at Doris with an IAM stack already in place. There is an LDAP directory, a Kerberos realm, an OIDC provider for SSO, and a Ranger or Sentry deployment that owns Hive permissions. A warehouse that ignores all of it forces the platform team to maintain a para [...]
-Doris treats both authentication and authorization as plug-in points. Authentication backends discover themselves through Java SPI; so do authorization backends. Switching from local passwords to LDAP is one config line. Switching from built-in RBAC to Ranger is a different config line. The two decisions are independent: an OIDC plugin in front of Ranger is fine, and so is LDAP in front of built-in RBAC.
+Apache Doris treats both authentication and authorization as plug-in points. Authentication backends discover themselves through Java SPI; so do authorization backends. Switching from local passwords to LDAP is one config line. Switching from built-in RBAC to Ranger is a different config line. The two decisions are independent: an OIDC plugin in front of Ranger is fine, and so is LDAP in front of built-in RBAC.
- One IdP for the whole data platform instead of a separate Doris user list.
- Centralized policy in Ranger that already governs Hive, HDFS, and Iceberg, with Doris hooked into the same service.
- Custom auth (proprietary token formats, tenant-aware password verification) without patching FE source.
-## What it is {#what}
+## What is Apache Doris pluggable authentication and authorization? {#what}
-Pluggable auth in Doris is two side-by-side extension points on the FE. The first is the `Authenticator` interface, called once per MySQL handshake to turn the credentials in the auth packet into a `UserIdentity`. The second is the `CatalogAccessController` interface, called on every query to check whether that identity can touch the catalog, database, table, or column. Both load through `java.util.ServiceLoader`, so a third party jar dropped on the classpath registers itself.
+Apache Doris pluggable authentication and authorization is two side-by-side extension points on the FE. The first is the `Authenticator` interface, called once per MySQL handshake to turn the credentials in the auth packet into a `UserIdentity`. The second is the `CatalogAccessController` interface, called on every query to check whether that identity can touch the catalog, database, table, or column. Both load through `java.util.ServiceLoader`, so a third party jar dropped on the classp [...]
**Key terms**
@@ -45,7 +45,9 @@ Pluggable auth in Doris is two side-by-side extension points on the FE. The firs
- **`AuthenticatorFactory` / `AccessControllerFactory`**: the SPI hooks listed in `META-INF/services`. Implement one to add a new backend.
- **External catalog authorizer**: each external catalog (Hive, Iceberg) can pick its own controller, so a Hive catalog can defer to its existing Ranger Hive policies while the internal catalog uses Doris RBAC.
-## How it works {#how}
+## How does Apache Doris pluggable authentication and authorization work? {#how}
+
+Apache Doris pluggable authentication and authorization runs two independent SPI chains: an `Authenticator` resolves the MySQL handshake into a `UserIdentity`, then a per-catalog `CatalogAccessController` evaluates every statement, including row filters and column masks.
1. **Pick the authenticator at startup.** `AuthenticatorManager` reads `authentication_type`, normalizes the value, and resolves it through `ServiceLoader<AuthenticatorFactory>`. Built-in factories (`DefaultAuthenticatorFactory`, `LdapAuthenticatorFactory`) ship in the FE jar; custom factories live in `fe/plugins/`. If no legacy factory matches, the manager falls back to `AuthenticationPluginAuthenticator`, which handles OIDC, OAuth2, and JWT.
2. **Authenticate the connection.** When a client opens a MySQL connection, the chosen authenticator inspects the handshake. Native password compares the scrambled hash, LDAP binds against the directory, OIDC validates a JWT and pulls claims from it. Success returns a `UserIdentity`; failure returns a typed reason.
@@ -85,7 +87,9 @@ mysql> SELECT current_user(), user();
`jdoe` logged in with the LDAP password, picked up the `doris_analyst` role through LDAP group mapping, and any subsequent `SELECT` is checked against the Ranger policy bound to the `doris` service.
-## When to use it {#when}
+## When should you use Apache Doris pluggable authentication and authorization? {#when}
+
+Use Apache Doris pluggable authentication and authorization when an existing IdP or policy service should own warehouse identity, and the FE just consumes credentials and policy decisions.
**Good fit**
diff --git a/docs-next/key-features/preaggregation-and-rollup.mdx b/docs-next/key-features/preaggregation-and-rollup.mdx
index 83bc9ad6aff..bfc954dd11e 100644
--- a/docs-next/key-features/preaggregation-and-rollup.mdx
+++ b/docs-next/key-features/preaggregation-and-rollup.mdx
@@ -19,12 +19,12 @@ featureCard:
- performance
---
-> **TL;DR** Doris's [Aggregate Key model](./data-model) merges rows that share a key while you load them, using per-column functions like `SUM`, `MAX`, `REPLACE`, and `BITMAP_UNION`. A `ROLLUP` does the same thing for query shapes the base table can't serve well: you keep an extra materialized index with fewer columns or a different sort order, and the planner picks it for you. You read precomputed numbers instead of grouping raw rows on every query.
+> **TL;DR** Apache Doris preaggregation merges rows that share a key at load time using the [Aggregate Key model](./data-model) and per-column functions like `SUM`, `MAX`, `REPLACE`, and `BITMAP_UNION`. A `ROLLUP` is a second materialized index — with fewer columns or a different sort order — that the planner transparently picks for queries the base table cannot serve well. Queries read precomputed numbers instead of grouping raw rows on every request.
-## Why it matters {#why}
+## Why use preaggregation and rollup in Apache Doris? {#why}
-A typical fact table is wide and granular: per-event ad clicks, per-request API logs, per-second device readings. The same dashboard then asks for daily totals, top-N users, or distinct viewers across a month. Without preaggregation, every query rescans millions of rows and re-runs the same `GROUP BY` you ran yesterday.
+Apache Doris preaggregation and rollup move repeated `GROUP BY` work from query time to load time, so dashboards read precomputed totals instead of rescanning the raw fact table on every refresh. A typical fact table is wide and granular: per-event ad clicks, per-request API logs, per-second device readings. The same dashboard then asks for daily totals, top-N users, or distinct viewers across a month. Without preaggregation, every query rescans millions of rows and re-runs the same `GRO [...]
- Repeated work. Every dashboard refresh redoes the same aggregation over the same raw data.
- Wide scans. A coarse query reads every value column on the base table even when it only needs two.
@@ -32,9 +32,9 @@ A typical fact table is wide and granular: per-event ad clicks, per-request API
Preaggregation moves the cost from query time to load time. A `ROLLUP` adds a second precomputed view for the queries the base table doesn't fit by itself; for cross-table or partition-mapped rewrites, reach for an [incremental materialized view](./incremental-materialized-view).
-## What it is {#what}
+## What is Apache Doris preaggregation and rollup? {#what}
-The Aggregate Key model is a table type. You declare which columns are keys and stamp each value column with an aggregation function. Doris then merges rows that share a key both at load and during compaction, so the table never stores duplicates that have a defined merge rule.
+Apache Doris preaggregation is the Aggregate Key table model that merges rows sharing a key using per-column aggregation functions, and a rollup is an extra materialized index over that table that the planner picks transparently. The Aggregate Key model is a table type. You declare which columns are keys and stamp each value column with an aggregation function. Apache Doris then merges rows that share a key both at load and during compaction, so the table never stores duplicates that hav [...]
A `ROLLUP` is a second materialized index over the same base table. It holds a column subset, possibly in a different sort order, with the same per-column functions applied. The query planner scores all available indexes for an incoming query and reads from whichever one matches best.
@@ -46,13 +46,15 @@ A `ROLLUP` is a second materialized index over the same base table. It holds a c
- **Rollup index**: an extra materialized layout the planner can read from instead of the base index.
- **Sync materialized view**: the `CREATE MATERIALIZED VIEW` syntax that supersedes `ALTER TABLE ADD ROLLUP`. Same machinery, richer expressions, recommended for new designs.
-## How it works {#how}
+## How does Apache Doris preaggregation and rollup work? {#how}
-1. **Define the aggregation contract.** In `CREATE TABLE`, the `AGGREGATE KEY(...)` clause names the key columns. Every other column gets a function: `clicks BIGINT SUM`, `last_seen DATETIME REPLACE`, `unique_users BITMAP BITMAP_UNION`. Doris uses these functions to merge any two rows that share the keys.
+Apache Doris preaggregation works in five stages: declaring the aggregation contract in DDL, merging rows on load and compaction, adding rollups for skewed query shapes, picking the best index per query, and preferring sync materialized views for new code.
+
+1. **Define the aggregation contract.** In `CREATE TABLE`, the `AGGREGATE KEY(...)` clause names the key columns. Every other column gets a function: `clicks BIGINT SUM`, `last_seen DATETIME REPLACE`, `unique_users BITMAP BITMAP_UNION`. Apache Doris uses these functions to merge any two rows that share the keys.
2. **Merge on load.** When data arrives, Backend nodes fold incoming rows against existing rows that share the keys, then write a new immutable rowset. Background compaction merges those rowsets later using the same rules. Queries see the merged view, not the raw inputs.
-3. **Add a rollup for skew query patterns.** `ALTER TABLE ad_events ADD ROLLUP rollup_ad (dt, ad_id, clicks, cost)` creates a second materialized index sorted by `(dt, ad_id)`. Doris rebuilds it as an asynchronous schema change, then keeps it in sync on every subsequent load inside the same transaction as the base write.
+3. **Add a rollup for skew query patterns.** `ALTER TABLE ad_events ADD ROLLUP rollup_ad (dt, ad_id, clicks, cost)` creates a second materialized index sorted by `(dt, ad_id)`. Apache Doris rebuilds it as an asynchronous schema change, then keeps it in sync on every subsequent load inside the same transaction as the base write.
4. **Pick an index per query.** The Nereids optimizer scores every index against the query's predicates and group keys. It prefers indexes where the predicate columns sit at the prefix and where fewer columns need scanning. The chosen index shows up in `EXPLAIN` and in the BE profile.
-5. **Use sync MV for new code.** `CREATE MATERIALIZED VIEW` covers everything `ADD ROLLUP` does plus expressions like `bitmap_union(to_bitmap(user_id))`. The Doris source comment is blunt: *"In function level, the mv completely covers the rollup in the future."* Existing rollups keep working.
+5. **Use sync MV for new code.** `CREATE MATERIALIZED VIEW` covers everything `ADD ROLLUP` does plus expressions like `bitmap_union(to_bitmap(user_id))`. The Apache Doris source comment is blunt: *"In function level, the mv completely covers the rollup in the future."* Existing rollups keep working.
## Quick start {#quick-start}
@@ -82,7 +84,9 @@ PREAGGREGATION: ON
The rollup drops `user_id`, so the planner picks it for any aggregate query that groups only by `(dt, ad_id)`. Inserting the same `(dt, ad_id, user_id)` twice produces one row with summed `clicks` and `cost`, and the rollup ends up with one row per `(dt, ad_id)`.
-## When to use it {#when}
+## When should you use Apache Doris preaggregation and rollup? {#when}
+
+Apache Doris preaggregation fits append-mostly fact tables with stable aggregation patterns, distinct-count and quantile rollups via `BITMAP_UNION` and `HLL_UNION`, and dashboards that run the same `GROUP BY` thousands of times a day. It is not a fit for raw per-row lookups, multi-table joins, or frequent point updates by primary key.
**Good fit**
diff --git a/docs-next/key-features/prepared-statement.mdx b/docs-next/key-features/prepared-statement.mdx
index 4f4195946ab..d7476837106 100644
--- a/docs-next/key-features/prepared-statement.mdx
+++ b/docs-next/key-features/prepared-statement.mdx
@@ -19,12 +19,12 @@ featureCard:
- performance
---
-> **TL;DR** Doris speaks the MySQL wire protocol, including server-side prepared statements. Set `useServerPrepStmts=true` on the JDBC URL and Doris parses and plans your statement once per session, then reuses that plan on every `EXECUTE`. For high-QPS [point-query](./high-concurrency-point-query) traffic on the short-circuit path, that cuts FE CPU enough to lift throughput by roughly 4x. For batch INSERTs and any other shape that repeats, you stop paying parse and plan cost on every row.
+> **TL;DR** Apache Doris supports server-side prepared statements over the MySQL wire protocol. Setting `useServerPrepStmts=true` on the JDBC URL makes the FE parse and plan a statement once per session and reuse that plan on every `EXECUTE`. On high-QPS [point-query](./high-concurrency-point-query) traffic the short-circuit path delivers roughly 4x more throughput; batch INSERTs and any repeated query shape stop paying parse and plan cost per row.
-## Why it matters {#why}
+## Why use prepared statements in Apache Doris? {#why}
-Most analytical engines pay parse and plan cost on every query. That cost is invisible in a daily report job. It is the whole story once a service starts asking the same query thousands of times per second. A user-facing dashboard reads a row by primary key on every click. A feature store reads a [vector](./vector-index) by entity ID on every prediction. A bulk loader fires the same `INSERT INTO t VALUES (?, ?, ?)` over and over.
+Apache Doris prepared statements cache the parsed plan per session and reuse it on every `EXECUTE`, removing the FE-side parse, analyze, and optimize cost that dominates high-QPS workloads. Most analytical engines pay parse and plan cost on every query. That cost is invisible in a daily report job. It is the whole story once a service starts asking the same query thousands of times per second. A user-facing dashboard reads a row by primary key on every click. A feature store reads a [vec [...]
In all three cases the SQL text is identical; only the parameters change. Re-parsing and re-planning that statement on every call burns FE CPU for no reason. At a few hundred QPS the FE saturates before the BE is doing any real work, and adding BE nodes does nothing for you. PreparedStatement is what you reach for there.
@@ -32,9 +32,9 @@ In all three cases the SQL text is identical; only the parameters change. Re-par
- Repeating queries pay parse, analyze, optimize, and serialization cost on every call.
- A separate KV store is a heavy answer to "make my point queries cheap."
-## What it is {#what}
+## What is the Apache Doris prepared statement? {#what}
-PreparedStatement in Doris is the MySQL wire-protocol pair `COM_STMT_PREPARE` / `COM_STMT_EXECUTE` (with `COM_STMT_CLOSE` to evict), served by the FE. When the client opens a connection with `useServerPrepStmts=true`, the JDBC driver issues `COM_STMT_PREPARE` once per distinct SQL text. The FE parses, analyzes, and stores the plan in a per-session cache keyed by statement ID. Every subsequent `EXECUTE` ships only the binary parameter bytes. The FE binds the values to the cached plan and [...]
+The Apache Doris prepared statement is a server-side implementation of the MySQL wire-protocol pair `COM_STMT_PREPARE` / `COM_STMT_EXECUTE` (with `COM_STMT_CLOSE` to evict), served by the FE and backed by a per-session plan cache. When the client opens a connection with `useServerPrepStmts=true`, the JDBC driver issues `COM_STMT_PREPARE` once per distinct SQL text. The FE parses, analyzes, and stores the plan in a per-session cache keyed by statement ID. Every subsequent `EXECUTE` ships [...]
**Key terms**
@@ -44,7 +44,9 @@ PreparedStatement in Doris is the MySQL wire-protocol pair `COM_STMT_PREPARE` /
- **Placeholder (`?`)**: a positional parameter bound at execute time. A single statement can carry up to 65,536 of them.
- **`ShortCircuitQueryContext`**: an extra cache that lives alongside the prepared plan when the query qualifies for the short-circuit point-query path. It holds the serialized descriptor table, output expressions, and a UUID the BE uses to look up its own pre-built structures.
-## How it works {#how}
+## How does the Apache Doris prepared statement work? {#how}
+
+The Apache Doris prepared statement runs in five steps: prepare the SQL on the FE, cache the short-circuit plan when eligible, execute with binary parameter bytes, optionally audit, and close to evict.
1. **Prepare.** The client sends `COM_STMT_PREPARE "SELECT ... WHERE k1 = ?"`. The FE parses the SQL, runs Nereids analysis, stores a `PreparedStatementContext` in the connection's map, and returns the statement ID and parameter count.
2. **Cache the short-circuit path (if eligible).** When the query is a single Unique Key table with equality predicates on the full key, the rewrite rule `LogicalResultSinkToShortCircuitPointQuery` flips a flag, and the FE serializes the descriptor table and output expressions into a `ShortCircuitQueryContext` keyed by a UUID.
@@ -91,19 +93,21 @@ Stmt=EXECUTE(-2147481418)
instead of the raw SQL. That is the signal the cached plan is being reused. Pair it with `EXPLAIN`, which should show `SHORT-CIRCUIT` on the scan node, to confirm both layers are active.
-## When to use it {#when}
+## When should you use Apache Doris prepared statements? {#when}
+
+Apache Doris prepared statements fit high-QPS point queries on Unique Key tables, repeated batch INSERTs, and any client-side workload that fires the same statement thousands of times per second. They are not a fit for one-off ad-hoc queries, drivers without clean server-side support, or statements with non-deterministic functions on the short-circuit path.
**Good fit**
- High-QPS point queries on Unique Key tables, where parse cost dominates.
- Batch INSERTs over [Group Commit](./group-commit), where a loader fires the same `INSERT INTO t VALUES (?, ?, ?)` thousands of times per second.
- Any query the same client repeats at meaningful rates: feature lookups, status checks, allow-listing, online-service reads.
-- Workloads where you would rather not stand up a separate KV store next to Doris.
+- Workloads where you would rather not stand up a separate KV store next to Apache Doris.
**Not a good fit**
- One-off ad-hoc queries. The `PREPARE` round trip costs more than the parse it saves.
-- Drivers without clean server-side prepared-statement support. The Rust `sqlx` driver, for example, has [reported](https://github.com/launchbadge/sqlx/issues/3335) issues with Doris's implementation. Use a MySQL Connector that talks the binary protocol cleanly.
+- Drivers without clean server-side prepared-statement support. The Rust `sqlx` driver, for example, has [reported](https://github.com/launchbadge/sqlx/issues/3335) issues with Apache Doris's implementation. Use a MySQL Connector that talks the binary protocol cleanly.
- Connections that set `useServerPrepStmts=true` without `cachePrepStmts=true`. Every call re-issues `COM_STMT_PREPARE`, which is slower than the default client-side mode.
- Statements with non-deterministic functions (`NOW()`, `RAND()`). The BE refuses to reuse the cached short-circuit context, so you only save parse cost.
- Range and aggregate queries, if you are hoping for the short-circuit speedup. PreparedStatement still saves parse cost, but the BE runs a normal distributed plan. For those workloads, lean on [data pruning](./data-pruning) instead.
@@ -125,3 +129,4 @@ If FE CPU is still the ceiling, scale Observers and use JDBC load balancing: `jd
- [High-Concurrency Point Query feature card](./high-concurrency-point-query)
- [Group Commit for high-throughput INSERTs](./group-commit)
- [Data pruning for non-point queries](./data-pruning)
+- [Unique Key](./unique-key): the partner table model for KV-style point lookups when combined with `store_row_column`.
diff --git a/docs-next/key-features/query-cache.mdx b/docs-next/key-features/query-cache.mdx
index 36e55e5a85f..a542031aec1 100644
--- a/docs-next/key-features/query-cache.mdx
+++ b/docs-next/key-features/query-cache.mdx
@@ -18,12 +18,12 @@ featureCard:
- caching
---
-> **TL;DR** Query Cache is Doris's [pipeline-engine](./pipeline-execution-engine) cache for aggregation queries on internal OLAP tables. It stores partial aggregation results at tablet granularity, keyed by a digest of the plan plus the tablet ID and version. Two queries that share a tablet share the cache for that tablet, even when their date ranges only partly overlap. The dashboard that re-runs `SELECT region, SUM(revenue) FROM orders WHERE dt = '...' GROUP BY region` every fifteen se [...]
+> **TL;DR** The Apache Doris Query Cache is a [pipeline-engine](./pipeline-execution-engine) cache for aggregation queries on internal OLAP tables. The Query Cache stores partial aggregation results at tablet granularity, keyed by a digest of the plan plus the tablet ID and version. Two queries that share a tablet share the cache for that tablet, even when their date ranges only partly overlap — so a dashboard that polls the same `GROUP BY` every 15 seconds stops re-scanning the same tab [...]
-## Why it matters {#why}
+## Why use the Apache Doris Query Cache? {#why}
-Most BI traffic is repetitive aggregation. A dashboard tile re-runs the same `SUM/COUNT/AVG` every refresh. A T+1 report aggregates last night's [partition](./partitioning-and-bucketing) once, then re-aggregates it every time someone opens the page today. A trader's panel polls the same `GROUP BY symbol` every five seconds against an order book that only changes a few tablets per minute.
+The Apache Doris Query Cache eliminates the redundant tablet-level work that BI dashboards generate when the same aggregation runs against overlapping partitions every few seconds. Most BI traffic is repetitive aggregation. A dashboard tile re-runs the same `SUM/COUNT/AVG` every refresh. A T+1 report aggregates last night's [partition](./partitioning-and-bucketing) once, then re-aggregates it every time someone opens the page today. A trader's panel polls the same `GROUP BY symbol` every [...]
The same tablet contributes the same partial aggregate again and again, and the cluster pays the full scan and aggregation cost every time.
@@ -31,9 +31,9 @@ SQL Cache helps when the SQL text is byte-identical, but it gives up as soon as
Query Cache works at a smaller grain than either of those: the contribution of one tablet to one aggregation. That is the level at which dashboard refreshes actually overlap, where filter ranges slide a day forward and only one or two partitions are new per refresh.
-## What it is {#what}
+## What is the Apache Doris Query Cache? {#what}
-Query Cache is a pipeline-engine optimization that activates when a fragment matches the shape `AggregationNode -> OlapScanNode`, or `AggregationNode -> AggregationNode -> OlapScanNode` for two-stage aggregation. Filter and project nodes are allowed in between. Joins, sorts, unions, window functions, and exchange nodes are not. When a fragment qualifies, the BE checks the cache before scanning. On a hit, the scan range is skipped and the cached blocks are emitted directly. On a miss, the [...]
+The Apache Doris Query Cache is a pipeline-engine optimization that activates when a fragment matches the shape `AggregationNode -> OlapScanNode`, or `AggregationNode -> AggregationNode -> OlapScanNode` for two-stage aggregation. Filter and project nodes are allowed in between. Joins, sorts, unions, window functions, and exchange nodes are not. When a fragment qualifies, the BE checks the cache before scanning. On a hit, the scan range is skipped and the cached blocks are emitted directl [...]
**Key terms**
@@ -43,7 +43,9 @@ Query Cache is a pipeline-engine optimization that activates when a fragment mat
- **`Tablet range`**: derived from partition predicates. Included in the cache key so two queries can reuse the cache for partitions they have in common.
- **`LRU-K (K=2)`**: admission policy. A new entry has to be touched at least twice before it earns a slot, so a one-off ad-hoc query cannot evict the entries that the dashboards depend on.
-## How it works {#how}
+## How does the Apache Doris Query Cache work? {#how}
+
+The Apache Doris Query Cache works by detecting the agg-over-scan pattern in the FE, building a per-tablet cache key from the plan digest plus the tablet ID and version, and probing the cache before each BE scans. The five-step flow below covers plan check, key construction, probe, hit, and miss.
1. **Plan check (FE).** Nereids walks each fragment looking for the agg-over-scan pattern. Joins, sorts, runtime-filter targets, external table scans, and non-deterministic expressions disqualify the fragment.
2. **Build the cache key (FE).** For single-column RANGE partitioned tables, the planner extracts partition predicates and computes the intersection of the predicate range with each scanned partition. The plan digest covers the rest of the query. The per-tablet key is `(digest, tablet_id, tablet_range)`.
@@ -80,7 +82,9 @@ CacheSourceOperator
The first run misses every tablet and writes per-tablet partial aggregates. The second run hits the cache for the four overlapping days (`2024-01-05` through `2024-01-07`) and only scans the new days (`2024-01-08` through `2024-01-11`). To confirm a hit, look at `CacheSourceOperator` in the profile: a per-tablet `HitCache: true` means that tablet was served from memory.
-## When to use it {#when}
+## When should you use the Apache Doris Query Cache? {#when}
+
+Use the Apache Doris Query Cache for BI dashboards, T+1 reports, and rolling-window aggregations on internal single-column `RANGE` partitioned tables. Avoid it for JOIN/ORDER/UNION/window queries, external lakehouse tables (use SQL Cache instead), high-write hot tables, and primary-key lookups.
**Good fit**
@@ -117,3 +121,4 @@ The first run misses every tablet and writes per-tablet partial aggregates. The
- [Caching in Doris: choosing the right strategy](../query-acceleration/caching-intro)
- [Data Cache & Page Cache](./data-cache-page-cache): the storage-tier caches that sit below Query Cache for repeated scans.
- [Metadata Cache](./metadata-cache): the external-catalog metadata cache for federated lakehouse queries.
+- [MPP Architecture](./mpp): the execution model whose fragment DAG the query cache short-circuits when result is reusable.
diff --git a/docs-next/key-features/reciprocal-rank-fusion.mdx b/docs-next/key-features/reciprocal-rank-fusion.mdx
index 396cc2e1d75..e6e528a4e48 100644
--- a/docs-next/key-features/reciprocal-rank-fusion.mdx
+++ b/docs-next/key-features/reciprocal-rank-fusion.mdx
@@ -19,12 +19,12 @@ featureCard:
- ai
---
-> **TL;DR** Doris does not ship a built-in `rrf()` function, but you can implement Reciprocal Rank Fusion in plain SQL using [`ROW_NUMBER()`](./analytic-functions) over a [BM25](./bm25) ranking and a [vector ranking](./vector-index), then summing `1.0 / (60 + rank)` across both. The result is a single fused top-K list whose ordering does not depend on calibrating BM25 scores against cosine distance.
+> **TL;DR** Apache Doris does not ship a built-in `rrf()` function, but you can implement Reciprocal Rank Fusion in plain SQL. The pattern uses [`ROW_NUMBER()`](./analytic-functions) over a [BM25](./bm25) ranking and a [vector ranking](./vector-index), then sums `1.0 / (60 + rank)` across both lists with `k = 60` as the standard smoothing constant. The result is a single fused top-K list whose ordering does not depend on calibrating BM25 scores against cosine distance.
-## Why it matters {#why}
+## Why use Reciprocal Rank Fusion in Apache Doris? {#why}
-Hybrid search returns two ranked lists for the same query: one from full-text matching (BM25 scores via `score()`) and one from vector similarity (`l2_distance_approximate` or `inner_product_approximate`). The two scores live on incompatible scales. BM25 is an unbounded positive number whose value depends on corpus statistics; cosine distance sits between 0 and 2; L2 distance has no upper bound at all. Adding or averaging these scores gives a number with no defensible meaning, and per-qu [...]
+Reciprocal Rank Fusion gives Apache Doris hybrid search a defensible default for merging BM25 and vector results without calibrating two incompatible score scales. Hybrid search returns two ranked lists for the same query: one from full-text matching (BM25 scores via `score()`) and one from vector similarity (`l2_distance_approximate` or `inner_product_approximate`). The two scores live on incompatible scales. BM25 is an unbounded positive number whose value depends on corpus statistics; [...]
Reciprocal Rank Fusion sidesteps the problem by throwing the scores away and keeping only the ranks. It has one parameter to tune and has held up well on retrieval benchmarks since Cormack, Clarke, and Buettcher published it in 2009.
@@ -32,9 +32,9 @@ Reciprocal Rank Fusion sidesteps the problem by throwing the scores away and kee
- Min-max normalization breaks when one query produces a long-tailed score distribution.
- An application-side fusion service is one more thing to deploy and keep in sync with the database.
-## What it is {#what}
+## What is Reciprocal Rank Fusion in Apache Doris? {#what}
-RRF is a rank-based fusion algorithm. Given a document `d` that appears in ranked lists `R_1, R_2, ..., R_n`, its fused score is the sum of reciprocal ranks across the lists it appears in:
+Reciprocal Rank Fusion in Apache Doris is a SQL pattern, not a built-in function: a rank-based fusion algorithm expressed with `ROW_NUMBER`, `FULL OUTER JOIN`, and the standard `1/(k+rank)` formula. Given a document `d` that appears in ranked lists `R_1, R_2, ..., R_n`, its fused score is the sum of reciprocal ranks across the lists it appears in:
```
score(d) = Σ 1 / (k + rank_i(d))
@@ -42,7 +42,7 @@ score(d) = Σ 1 / (k + rank_i(d))
`rank_i(d)` is `d`'s 1-based position in list `i`. `k` is a smoothing constant; the value Cormack et al. recommend, and that most production systems use, is `60`. Documents missing from a list contribute zero from that list. The fused score is then sorted descending to produce the final ranking.
-Doris exposes this as a SQL pattern, not a function. You write two ranked subqueries, one for BM25 and one for ANN, give each row a `ROW_NUMBER()`, then `FULL OUTER JOIN` and sum the reciprocals.
+In Apache Doris the implementation is a SQL pattern, not a function. You write two ranked subqueries, one for BM25 and one for ANN, give each row a `ROW_NUMBER()`, then `FULL OUTER JOIN` and sum the reciprocals.
**Key terms**
@@ -51,7 +51,9 @@ Doris exposes this as a SQL pattern, not a function. You write two ranked subque
- **`ROW_NUMBER() OVER (ORDER BY ...)`**: assigns a 1-based rank to each row within a result set.
- **`k`**: the RRF smoothing constant. Lower values (20–40) sharpen the influence of top-ranked rows; higher values (80+) flatten it.
-## How it works {#how}
+## How does Reciprocal Rank Fusion work in Apache Doris? {#how}
+
+Apache Doris implements Reciprocal Rank Fusion by running two ranked retrievals as CTEs, assigning a `ROW_NUMBER()` to each, `FULL OUTER JOIN`-ing the two lists on document id, and summing `1/(k+rank)` to produce the final ordering. The five-step flow below covers retrieval, ranking, join, fusion, and trim.
1. **Run two ranked retrievals.** A BM25 subquery filters with `MATCH_ANY` and orders by `score()` descending. An ANN subquery orders by vector distance ascending. Each is capped with `LIMIT N`, where `N` is the candidate pool size (typically 50–200).
2. **Assign per-list ranks.** `ROW_NUMBER() OVER (ORDER BY ...)` numbers the rows 1, 2, 3 within each subquery. The numbers are integers, so they are safe to mix across runs.
@@ -94,7 +96,9 @@ ORDER BY rrf DESC LIMIT 5;
Document `1` ranks first in both lists, so its fused score is `1/61 + 1/61 ≈ 0.0328`. Documents `3` and `7` appear in only one list each, so they contribute a single reciprocal term and fall in behind the row that placed in both.
-## When to use it {#when}
+## When should you use Reciprocal Rank Fusion in Apache Doris? {#when}
+
+Use Apache Doris Reciprocal Rank Fusion for RAG pipelines, catalog/document search, and any workload that combines BM25 and vector retrieval where calibrating absolute scores is impractical. Skip it for single-retriever workloads, very small candidate pools, calibrated-probability use cases, and hot-path queries where the extra join hurts latency.
**Good fit**
diff --git a/docs-next/key-features/resource-group.mdx b/docs-next/key-features/resource-group.mdx
index 59b10de041b..a216376f492 100644
--- a/docs-next/key-features/resource-group.mdx
+++ b/docs-next/key-features/resource-group.mdx
@@ -19,20 +19,20 @@ featureCard:
- multi-tenancy
---
-> **TL;DR** Resource Group tags BE nodes into named pools, places table replicas across those pools, and binds each user to one or more pools. A query runs only on BEs in the user's pools and only reads replicas placed there. Because the pools are different BE processes, a crash or OOM in one group cannot take down another. The cost is storage: every pool that needs to read a table needs its own replica.
+> **TL;DR** Apache Doris Resource Group tags BE nodes into named pools, places table replicas across those pools, and binds each user to one or more pools. A query runs only on BEs in the user's pools and only reads replicas placed there. Because the pools are different BE processes, a crash or OOM in one group cannot take down another. The cost is storage: every pool that needs to read a table needs its own replica.
-## Why it matters {#why}
+## Why use Resource Group in Apache Doris? {#why}
-A real Doris cluster usually serves more than one workload, and the workloads do not trust each other. The 2 a.m. ETL job rewrites yesterday's partitions on the same BEs that an executive dashboard polls every fifteen seconds during the day. A finance batch reruns a heavy report at 9 a.m. while an analyst kicks off an unbounded `SELECT *`.
+Apache Doris Resource Group exists because a single cluster typically serves more than one workload, and the workloads do not trust each other. The 2 a.m. ETL job rewrites yesterday's partitions on the same BEs that an executive dashboard polls every fifteen seconds during the day. A finance batch reruns a heavy report at 9 a.m. while an analyst kicks off an unbounded `SELECT *`.
[Workload Group](./workload-group) handles a lot of this in-process: cgroups for CPU, a per-group memory tracker, a per-group queue. It is cheap and flexible. What it cannot do is survive a BE crash. If the kernel OOM-kills a BE, every group on that BE goes with it. If the BE segfaults inside a query, the dashboard and the ETL job both reconnect. For workloads that need to be insulated from each other at the process layer, in-process tools are not enough.
-Resource Group fixes that without standing up a second cluster. You tag each BE with a label, you tell each table how many replicas belong to each tag, and you bind each user to the tags they are allowed to use. The cluster keeps one FE quorum, one schema catalog, one set of credentials, but the BEs are now divided into pools that share nothing at runtime.
+Apache Doris Resource Group fixes that without standing up a second cluster. Each BE gets a label, each table declares how many replicas belong to each tag, and each user is bound to the tags they are allowed to use. The cluster keeps one FE quorum, one schema catalog, one set of credentials, but the BEs are now divided into pools that share nothing at runtime.
-## What it is {#what}
+## What is the Apache Doris Resource Group? {#what}
-Resource Group is a placement-and-routing feature built on **resource tags**. Every BE belongs to exactly one location tag. Every replica of every table belongs to exactly one tag, decided at create time by `replication_allocation`. Every user has a set of tags they are allowed to use, decided by the `resource_tags.location` property. The query planner intersects them: it picks BEs in the user's tag set and reads only the replicas placed on those BEs.
+The Apache Doris Resource Group is a placement-and-routing feature built on **resource tags**. Every BE belongs to exactly one location tag. Every replica of every table belongs to exactly one tag, decided at create time by `replication_allocation`. Every user has a set of tags they are allowed to use, decided by the `resource_tags.location` property. The query planner intersects them: it picks BEs in the user's tag set and reads only the replicas placed on those BEs.
**Key terms**
@@ -41,7 +41,9 @@ Resource Group is a placement-and-routing feature built on **resource tags**. Ev
- **`resource_tags.location`**: a user property listing the tags the user is allowed to access. Comma-separate to grant several. Empty means the default behavior described below.
- **Default tag**: the built-in `default` tag. Every BE starts there, and it cannot be dropped. From 2.0.3 onward, ordinary users with no `resource_tags.location` set can only see `default`; `root` and `admin` can see everything.
-## How it works {#how}
+## How does the Apache Doris Resource Group work? {#how}
+
+The Apache Doris Resource Group works in five steps: tag the BEs, allocate replicas across tags, bind users to tags, route the query, and split the load path.
1. **Tag the BEs.** `ALTER SYSTEM MODIFY BACKEND "host:9050" SET ("tag.location" = "online")`. The FE writes the change to its edit log so the assignment survives restarts. `SHOW BACKENDS` reports the tag in its `Tag` column.
2. **Allocate replicas across tags.** When a table is created with `replication_allocation = "tag.location.online:2, tag.location.offline:1"`, the FE checks that enough BEs carry each tag, then asks the tablet scheduler to place one replica per slot. A failed check reports "Failed to find enough host with tag(...)" and the DDL aborts. You can also set the property at the database level so new tables inherit it.
@@ -81,7 +83,9 @@ SET PROPERTY FOR 'etl_user' 'resource_tags.location' = 'offline';
`SHOW BACKENDS` confirms the assignment. After `bi_user` reconnects, their queries plan against the two `online` replicas and never touch `be3`. `etl_user` runs on `be3` and reads the single `offline` replica. A BE crash on `be3` is invisible to `bi_user`.
-## When to use it {#when}
+## When should you use the Apache Doris Resource Group? {#when}
+
+The Apache Doris Resource Group fits any deployment that needs BE-process-level isolation between workloads on a single cluster, and accepts the extra storage cost of one replica per pool.
**Good fit**
diff --git a/docs-next/key-features/spill-to-disk.mdx b/docs-next/key-features/spill-to-disk.mdx
index 016531051d5..dc42fcb6c96 100644
--- a/docs-next/key-features/spill-to-disk.mdx
+++ b/docs-next/key-features/spill-to-disk.mdx
@@ -20,23 +20,23 @@ featureCard:
- reliability
---
-> **TL;DR** Spill to Disk lets HashJoin, Aggregation, Sort, and CTE operators write intermediate state to local disk when a query is about to blow its memory budget. The query slows down, but it finishes. You turn it on with `set enable_spill = true` (per session) or at the [Workload Group](./workload-group) level, point BE at a spill directory, and read spill counters back from the query profile.
+> **TL;DR** Apache Doris Spill to Disk lets HashJoin, Aggregation, Sort, and CTE operators write intermediate state to local disk when a query is about to exceed its memory budget. The query slows down but finishes instead of OOM-ing. Enable it with `set enable_spill = true` per session or at the [Workload Group](./workload-group) level, point BE at a spill directory, and read spill counters back from the query profile.
-## Why it matters {#why}
+## Why use spill to disk in Apache Doris? {#why}
-Doris is an MPP engine. By default the join hash tables, the aggregation state, the sort buffer all live in BE memory. That is great for latency and bad the moment a query asks for more memory than the node, the query, or the workload group is allowed to use. The classic failure mode is a nightly ETL or an ad-hoc TPC-DS style query that touches a wide fact table, builds a multi-gigabyte hash table, then dies with `Memory limit exceeded`. The user retries, hits the same wall, and either t [...]
+Spill to Disk turns out-of-memory failures in Apache Doris into slower but successful queries. Apache Doris is an MPP engine: by default the join hash tables, the aggregation state, and the sort buffer all live in BE memory. That is great for latency and bad the moment a query asks for more memory than the node, the query, or the workload group is allowed to use. The classic failure mode is a nightly ETL or an ad-hoc TPC-DS style query that touches a wide fact table, builds a multi-gigab [...]
- A single oversized HashJoin or Aggregation kills the whole query.
- ETL on tables much larger than RAM is fragile or simply impossible.
- Bumping `exec_mem_limit` per query starves everyone else.
- Concurrent heavy queries push the BE close to the global memory ceiling and trigger emergency cancellations.
-Spill to Disk turns those hard failures into slower, but successful, queries. The same node that used to OOM at 10x dataset size now finishes at 50x, paying disk IO instead of crashing.
+Apache Doris Spill to Disk turns those hard failures into slower, but successful, queries. The same node that used to OOM at 10x dataset size now finishes at 50x, paying disk IO instead of crashing.
-## What it is {#what}
+## What is Apache Doris Spill to Disk? {#what}
-Spill to Disk is a runtime mechanism in the [Pipeline Execution Engine](./pipeline-execution-engine) that lets memory-intensive operators evict their intermediate state to local disk under memory pressure, then read it back to finish the query. It is operator-level, not query-level: only the operators that are actually large get spilled, and a query can keep running on a mix of in-memory and on-disk state.
+Apache Doris Spill to Disk is a runtime mechanism in the [Pipeline Execution Engine](./pipeline-execution-engine) that lets memory-intensive operators evict their intermediate state to local disk under memory pressure, then read it back to finish the query. The mechanism is operator-level, not query-level: only the operators that are actually large get spilled, and a query can keep running on a mix of in-memory and on-disk state.
**Key terms**
@@ -46,7 +46,9 @@ Spill to Disk is a runtime mechanism in the [Pipeline Execution Engine](./pipeli
- **`Force spill`**: a debug mode (`enable_force_spill`) that spills even when there is plenty of memory. Use it to validate query correctness on the spill path before you trust it in production.
- **Spill storage path**: the local directories the BE uses for spill files, configured separately from the table storage path.
-## How it works {#how}
+## How does Apache Doris Spill to Disk work? {#how}
+
+Apache Doris Spill to Disk works by reserving memory before allocation, detecting pressure, picking the operator with the most revocable memory, partitioning its state to disk, then resuming the query once memory frees up. The six-step flow below covers the full lifecycle.
1. **Reserve before allocate.** Each pipeline task estimates the memory it needs for the next block and asks the per-query and per-process memory tracker for a reservation. Cheap and accurate beats panic-cancelling later.
2. **Detect pressure.** A reservation fails when the query, the workload group, or the BE process would cross its limit. The task gets paused instead of cancelled.
@@ -77,7 +79,9 @@ FROM information_schema.backend_active_tasks;
The query returns its top-100 rows. In the [query profile](../query-acceleration/query-profile), look for `Spilled: true` on the `PARTITIONED_HASH_JOIN_SINK_OPERATOR` or `AGGREGATION_SINK_OPERATOR`, plus `SpillWriteBlockBytes`, `SpillWriteFileBytes`, and `SpillWriteTime`. Audit logs add `SpillWriteBytesToLocalStorage` and `SpillReadBytesFromLocalStorage` for the same query. If those counters are zero, the query fit in memory and never spilled, which is the desired outcome.
-## When to use it {#when}
+## When should you use Apache Doris Spill to Disk? {#when}
+
+Use Apache Doris Spill to Disk for heavy ETL, batch analytics, and multi-tenant workloads where finishing the query matters more than latency. Skip it for sub-second serving paths and runaway queries.
**Good fit**
@@ -101,3 +105,4 @@ The query returns its top-100 rows. In the [query profile](../query-acceleration
- [Pipeline Execution Engine](./pipeline-execution-engine): the runtime that owns the reserve-before-allocate accounting and the revocable-memory bookkeeping.
- [Query profile](../query-acceleration/query-profile): how to read the per-operator `Spill*` counters that confirm spill happened.
- [`backend_active_tasks` system table](../admin-manual/system-tables/information_schema/workload_groups): live per-query spill bytes for monitoring and alerting.
+- [MPP Architecture](./mpp): the distributed execution layer that allocates `exec_mem_limit` per BE, the budget spill-to-disk respects.
diff --git a/docs-next/key-features/stream-load.mdx b/docs-next/key-features/stream-load.mdx
index ff8e2a202ad..aa00966411d 100644
--- a/docs-next/key-features/stream-load.mdx
+++ b/docs-next/key-features/stream-load.mdx
@@ -20,22 +20,22 @@ featureCard:
- real-time-ingest
---
-> **TL;DR** Stream Load lets a client `curl` (or any HTTP library) push a file or a stream of bytes into a Doris table over a single PUT request. The FE redirects the request to a BE, the BE acts as the coordinator, parses CSV/JSON/Parquet/ORC, writes to the target tablets, and returns a JSON body that tells the caller whether the whole batch committed. One label dedupes retries, one [transaction](./load-transaction) makes the batch atomic, and the response carries the row counts and per [...]
+> **TL;DR** Apache Doris Stream Load lets a client `curl` (or any HTTP library) push a file or a stream of bytes into a table over a single HTTP PUT request. The FE redirects the request to a BE, the chosen BE acts as the coordinator, parses CSV/JSON/Parquet/ORC, writes to the target tablets, and returns a JSON body that tells the caller whether the whole batch committed. One label dedupes retries, one [transaction](./load-transaction) makes the batch atomic, and the JSON response carrie [...]
-## Why it matters {#why}
+## Why use Apache Doris Stream Load? {#why}
-A lot of real-time writes into a warehouse start out as a script that pushes a file. The script lives inside a Flink sink, a [Kafka consumer](./kafka-cdc-integration), an `rsync` cron, or a microservice that just produced a few thousand events. Whoever wrote the script wants to send the bytes, get an answer on the same connection, and know that a retry is safe.
+Apache Doris Stream Load is the path for clients that already speak HTTP and want a synchronous answer per batch, with a label-keyed retry guarantee and no extra transaction manager. A lot of real-time writes into a warehouse start out as a script that pushes a file. The script lives inside a Flink sink, a [Kafka consumer](./kafka-cdc-integration), an `rsync` cron, or a microservice that just produced a few thousand events. Whoever wrote the script wants to send the bytes, get an answer [...]
-- A Flink job needs to flush its current checkpoint to Doris and learn whether the commit succeeded before it advances offsets.
+- A Flink job needs to flush its current checkpoint to Apache Doris and learn whether the commit succeeded before it advances offsets.
- An ETL script has a 200 MB CSV from yesterday's export and wants the rows visible before the morning report runs.
- A side service emits a JSON batch every few seconds and needs at-most-once semantics without bringing in a transaction manager.
-Stream Load covers that shape of write directly. One HTTP PUT, one label, one transaction, one synchronous response on the same socket.
+Apache Doris Stream Load covers that shape of write directly. One HTTP PUT, one label, one transaction, one synchronous response on the same socket.
-## What it is {#what}
+## What is Apache Doris Stream Load? {#what}
-Stream Load is a synchronous HTTP load endpoint exposed by every Doris node. The client sends an HTTP PUT to either an FE or a BE; the FE redirects to a BE in round-robin order, and the chosen BE becomes the coordinator for that load. The coordinator parses the request body, distributes rows to the BEs that own each tablet, opens one transaction with the FE, commits when every BE acks, and returns a JSON body on the same socket. Doris has supported CSV since the beginning and now also ac [...]
+Apache Doris Stream Load is a synchronous HTTP load endpoint exposed by every node in the cluster. The client sends an HTTP PUT to either an FE or a BE; the FE redirects to a BE in round-robin order, and the chosen BE becomes the coordinator for that load. The coordinator parses the request body, distributes rows to the BEs that own each tablet, opens one transaction with the FE, commits when every BE acks, and returns a JSON body on the same socket. Apache Doris has supported CSV since [...]
**Key terms**
@@ -45,7 +45,9 @@ Stream Load is a synchronous HTTP load endpoint exposed by every Doris node. The
- **`Expect: 100-continue`**: required header. The client posts the headers first, lets the server check auth and start a transaction, then sends the body.
- **`max_filter_ratio`**: the per-load tolerance for malformed rows. Defaults to 0, so one bad row fails the batch.
-## How it works {#how}
+## How does Apache Doris Stream Load work? {#how}
+
+Apache Doris Stream Load runs the entire load as one HTTP request: the FE redirects to a coordinator BE, the BE parses the body, opens a transaction, distributes rows to tablet owners, and answers on the same socket.
1. **Client sends PUT to FE (or BE).** The headers carry the label, format, column mapping, filters, and timeouts. The body is the file or stream. `Expect: 100-continue` lets the server reject the request before the body is uploaded.
2. **FE picks a coordinator BE and redirects.** The FE returns an HTTP 307 to a BE chosen in round-robin order. A client that already knows a healthy BE can skip this hop and PUT to the BE directly.
@@ -85,7 +87,9 @@ curl --location-trusted -u root: \
`Status: Success` means the batch committed and the rows are queryable. Resending the same `curl` returns `Status: Label Already Exists` and writes nothing, so a client that retries on a network blip will not double-load.
-## When to use it {#when}
+## When should you use Apache Doris Stream Load? {#when}
+
+Use Apache Doris Stream Load when one HTTP PUT per batch, with a label-keyed retry guarantee and a synchronous JSON response, is exactly the contract the client wants.
**Good fit**
@@ -97,10 +101,10 @@ curl --location-trusted -u root: \
**Not a good fit**
- Tens of thousands of tiny PUTs per second from many clients. Each load opens its own transaction and produces its own rowset, which overwhelms the FE planner and triggers `-235 Too many segments` on the BE. Use [Group Commit](./group-commit) so the BE merges small writes into one transaction, or move to [Routine Load](../data-operate/import/import-way/routine-load-manual) for Kafka.
-- Single files much larger than 10 GB. Stream Load runs as one synchronous request; if the connection drops you start over. Split the file or use [Batch Load](./batch-load), which runs server-side and survives the client.
+- Single files much larger than 10 GB. Apache Doris Stream Load runs as one synchronous request; if the connection drops you start over. Split the file or use [Batch Load](./batch-load), which runs server-side and survives the client.
- Continuous tailing of a Kafka topic. Stream Load knows nothing about offsets. Use [Routine Load](../data-operate/import/import-way/routine-load-manual), which manages offsets, restart, and pause for you.
- Stream Load 2PC against Merge-on-Write Unique Key tables **in cloud / storage-compute-separated deployments**. The BE rejects this combination with `Status::NotSupported`. On-prem (shared-nothing) clusters allow it, and the regression suite covers it.
-- Loads that need server-side recording out of the box. By default, Doris does not log Stream Load history; turn on `enable_stream_load_record` in `be.conf` if `SHOW STREAM LOAD` should return anything.
+- Loads that need server-side recording out of the box. By default, Apache Doris does not log Stream Load history; turn on `enable_stream_load_record` in `be.conf` if `SHOW STREAM LOAD` should return anything.
## Further reading {#further-reading}
@@ -110,3 +114,4 @@ curl --location-trusted -u root: \
- [Load transactions and Stream Load 2PC: when checkpoints need exactly-once across two systems](./load-transaction)
- [Routine Load: managed Kafka consumer with offset and pause control](../data-operate/import/import-way/routine-load-manual)
- [Doris Streamloader: a multi-concurrency client wrapper around the Stream Load API](../connection-integration/data-integration/doris-streamloader)
+- [Unique Key](./unique-key): the primary destination for partial-column upserts (`partial_columns: true`) from CDC pipelines.
diff --git a/docs-next/key-features/unique-key.mdx b/docs-next/key-features/unique-key.mdx
new file mode 100644
index 00000000000..777999a5fc1
--- /dev/null
+++ b/docs-next/key-features/unique-key.mdx
@@ -0,0 +1,133 @@
+---
+title: Unique Key
+description: Doris's primary key table model. One row per primary key, with SQL UPDATE/DELETE, partial-column upserts, and a sequence column for out-of-order writes.
+keywords:
+ - primary key
+ - primary key table
+ - Unique Key model
+ - upsert
+ - Merge-on-Write
+ - sequence column
+ - partial column update
+ - Apache Doris CDC sink
+slug: /key-features/unique-key
+image: /images/next/key-features/unique-key.jpg
+last_update:
+ date: 2026-05-11
+ author: Apache Doris
+featureCard:
+ tags:
+ - table-design
+ - upsert
+---
+
+> **TL;DR** The Apache Doris Unique Key model is the primary key table model — one row per primary key, with SQL `UPDATE` and `DELETE` like a transactional database. Partial-column upserts go straight from [Stream Load](./stream-load.mdx) into the table. Since version 2.1 the Unique Key model runs on Merge-on-Write by default, so duplicates are resolved at write time and queries skip version merging at read time.
+
+
+
+## Why use the Apache Doris Unique Key model? {#why}
+
+The Apache Doris Unique Key model gives an OLAP engine real primary-key semantics: one row per key, immediate uniqueness on write, and SQL `UPDATE`/`DELETE` that work without partition rewrites. Most analytical engines treat tables as append-only. Then CDC happens. A MySQL row gets updated, a user requests deletion under GDPR, a late event needs to overwrite an older one. With an append-only table model you can record those changes, but you cannot make the warehouse forget the old versio [...]
+
+- A Flink CDC job watches MySQL and needs every primary-key update to land in seconds.
+- An order status table flips through `created`, `paid`, `shipped`, `delivered`, and queries should only see the latest state.
+- A user profile table feeds real-time joins, and the join should never see yesterday's email address.
+
+The Unique Key model is the table model for this. A new write with an existing key replaces the old row, `UPDATE` and `DELETE` work as SQL, and analytical queries still run at columnar speed. ClickHouse's `ReplacingMergeTree` is the closest analog in other OLAP engines, but it does not give you immediate uniqueness: duplicate rows stay visible until a background merge runs, and even after that the engine only guarantees eventual deduplication. A query right after the write can still see [...]
+
+## What is the Apache Doris Unique Key model? {#what}
+
+The Apache Doris Unique Key model is a table model that keeps exactly one row per primary key, enforced at write time by the Merge-on-Write engine. The `UNIQUE KEY(...)` clause in `CREATE TABLE` names the key columns; everything else is a value column. Two rows with the same key cannot coexist. A new write with an existing key replaces the row.
+
+```sql
+... UNIQUE KEY(order_id) ...
+... UNIQUE KEY(tenant_id, user_id) ...
+```
+
+The FE catalog records this as `KeysType.UNIQUE_KEYS`, the contract the storage engine enforces from then on. The model is fixed at create time and cannot be altered later.
+
+**Key terms**
+
+- **Primary key**: the columns named in `UNIQUE KEY(...)`. They both sort the data on disk and enforce uniqueness.
+- **Value columns**: every other column. Stored as written, with no aggregation.
+- **Merge-on-Write (MoW)**: the default backend since 2.1. New writes look up the existing row, flip its row ID in a per-rowset delete bitmap, and append the new row to a fresh rowset.
+- **Merge-on-Read (MoR)**: the older backend. Writes append; reads merge multiple versions per key on the fly. Still selectable via `enable_unique_key_merge_on_write = false`.
+- **Delete bitmap**: a Roaring bitmap keyed by `(rowset_id, segment_id, version)` listing row IDs that queries should skip. The MoW artifact that replaces read-time merging.
+- **Sequence column** (`function_column.sequence_col`): a value column that decides which write wins when two updates carry the same key. Out-of-order CDC events use this to keep the newer state.
+- **Partial column update**: an upsert that writes only some columns. Apache Doris reads the rest from the existing row and writes a complete row back.
+- **`__DORIS_DELETE_SIGN__`**: a hidden tinyint column. Set it to `1` through any load path to soft-delete the row.
+
+## How does the Apache Doris Unique Key model work? {#how}
+
+The Apache Doris Unique Key model runs each upsert in five stages: buffer the batch, look up keys in the per-segment primary-key index, flip delete-bitmap bits on old rows, resolve order with the sequence column, and publish the new rowset.
+
+Take an upsert against a Unique Key MoW table.
+
+1. **Buffer the batch.** BEs collect the incoming rows in a memtable and sort them by key.
+2. **Look up each key.** For every key in the batch, the BE consults the per-segment primary-key index (one short read per key) to find any existing rowset and row ID. The index is a sorted, paginated structure built when each segment flushed, conceptually similar to a RocksDB partitioned index.
+3. **Mark the old rows.** Each affected rowset gets the old row IDs flipped on in its delete bitmap. The bitmap is per `(rowset_id, segment_id, version)` and the old data stays on disk until compaction reclaims it.
+4. **Resolve order with the sequence column.** If `function_column.sequence_col` is set, MoW compares the incoming row's sequence value against the current row's `__DORIS_SEQUENCE_COL__`. The larger value wins. Equal values fall back to load order. The sequence type must be an integer or `DATE`/`DATETIME`.
+5. **Publish the new rowset.** The transaction commits, the new version becomes visible, and queries from that point on filter through the merged delete bitmap. No version-merging step at read time.
+
+`DELETE FROM ... WHERE` follows the same flow without the new rowset. Partial updates add one step: before writing, the BE reads the unmodified columns for each key so it can store a complete row.
+
+## Quick start {#quick-start}
+
+```sql
+CREATE TABLE orders (
+ order_id BIGINT,
+ status VARCHAR(20),
+ amount DECIMAL(10, 2),
+ updated DATETIME
+)
+UNIQUE KEY(order_id)
+DISTRIBUTED BY HASH(order_id) BUCKETS 4
+PROPERTIES ("function_column.sequence_col" = "updated");
+
+INSERT INTO orders VALUES
+ (1, 'created', 99.50, '2026-05-08 10:00:00'),
+ (1, 'paid', 99.50, '2026-05-08 10:05:00'),
+ (1, 'created', 99.50, '2026-05-08 09:00:00'); -- late event
+```
+
+**Expected result**
+
+```
++----------+--------+--------+---------------------+
+| order_id | status | amount | updated |
++----------+--------+--------+---------------------+
+| 1 | paid | 99.50 | 2026-05-08 10:05:00 |
++----------+--------+--------+---------------------+
+```
+
+Three inserts, one row. The second `INSERT` overwrote the first because the keys matched. The third tried to overwrite again, but its `updated` value was older than the current sequence, so MoW kept the existing row. Drop the `function_column.sequence_col` property and the third row would have won purely on load order, which is what you do not want for CDC.
+
+## When should you use the Apache Doris Unique Key model? {#when}
+
+The Apache Doris Unique Key model fits CDC sinks, real-time dimension and lookup tables, frequently changing order/user/balance tables, KV-style point lookups paired with `store_row_column`, GDPR row-level deletes, and wide-table assembly from multiple upstreams. It is not a fit for append-only event streams, tight single-row update loops, random or very wide primary keys, or `count(*)`-heavy workloads on the older Merge-on-Read backend.
+
+**Good fit**
+
+- CDC sinks from MySQL, Postgres, or any source with a primary key. Pair with a sequence column on the source's commit timestamp or LSN. See [Kafka CDC Integration](./kafka-cdc-integration.mdx).
+- Real-time dimension and lookup tables joined into fact-table queries.
+- Order, user, balance, or session-state tables that change frequently and are queried analytically.
+- KV-style point lookups, when you combine MoW with `store_row_column` and a prepared statement. See [High-Concurrency Point Query](./high-concurrency-point-query.mdx).
+- GDPR row-level deletes by user ID, including soft-deletes through `__DORIS_DELETE_SIGN__` for any load path.
+- Wide-table assembly where each upstream owns a column group and writes only its columns via `partial_columns: true` Stream Load. See [Partial Column Update](../data-operate/update/partial-column-update.md).
+
+**Not a good fit**
+
+- Append-only event streams (logs, clickstreams, IoT readings). You pay the per-write key-index lookup for capability you do not use. Pick the Duplicate Key model instead. See [Data Model](./data-model.mdx).
+- Tight loops of single-row `UPDATE`s. Each statement is its own transaction, and commit overhead dominates. Batch through Stream Load with `partial_columns: true`, or use [Group Commit](./group-commit.mdx).
+- Random or very wide primary keys (UUIDs, long composite keys). Every load batch walks the PK index, and random keys defeat its cache locality. Prefer integers, keep the key narrow, and use the smallest type that fits. Floats, doubles, and complex types like `ARRAY`, `MAP`, `STRUCT`, `JSON`, and `VARIANT` are rejected as key columns outright.
+- Using the partition key as a superset of the primary key. Partition columns must be a subset of the unique key, never the other way around. Otherwise the same key can land in two partitions and dedup breaks.
+- Heavy `count(*)` workloads on the MoR variant. MoR rebuilds versions at query time and counting is expensive. Either enable Merge-on-Write (default since 2.1) or, if the data is genuinely append-only, switch to Duplicate.
+
+## Further reading {#further-reading}
+
+- [Unique Key Model](../table-design/data-model/unique.md): SQL syntax, Merge-on-Write vs. Merge-on-Read, partition rules, and creation properties in detail.
+- [Data Update and Delete](./data-update-delete.mdx): the SQL surface (`UPDATE`, `DELETE`, partial-column upserts, atomic partition swaps) and the Merge-on-Write internals that power it.
+- [Data Model](./data-model.mdx): how Unique Key compares to Duplicate and Aggregate, and which workloads each model fits.
+- [Concurrency Control with the Sequence Column](../data-operate/update/unique-update-concurrent-control.md): allowed types, null handling, and how out-of-order writes are resolved.
+- [High-Concurrency Point Query](./high-concurrency-point-query.mdx): the short-circuit plan that turns a Unique Key table into a KV store.
+- [Apache Doris 2.1.0: Merge-on-Write becomes the Unique Key default](/blog/release-note-2.1.0/)
diff --git a/docs-next/key-features/variant-data-type.mdx b/docs-next/key-features/variant-data-type.mdx
index 99fa5181168..494ddb2c971 100644
--- a/docs-next/key-features/variant-data-type.mdx
+++ b/docs-next/key-features/variant-data-type.mdx
@@ -19,21 +19,21 @@ featureCard:
- semi-structured
---
-> **TL;DR** `VARIANT` is a column type for JSON whose shape changes over time. On write, Doris parses each document, infers a type per JSON path, and persists hot paths as their own [columnar subcolumns](./columnar-storage). Filters and aggregations on `v['user_id']` then read one column instead of parsing the whole document. The type was inspired by Snowflake and ClickHouse VARIANT, and went GA in Apache Doris 2.1.0 (March 2024).
+> **TL;DR** Apache Doris `VARIANT` is a column type for JSON whose shape changes over time. On write, Apache Doris parses each document, infers a type per JSON path, and persists hot paths as their own [columnar subcolumns](./columnar-storage), defaulting to 2048 subcolumns per VARIANT column. Filters and aggregations on `v['user_id']` then read one column instead of parsing the whole document. VARIANT went GA in Apache Doris 2.1.0 (March 2024), inspired by Snowflake and ClickHouse.
-## Why it matters {#why}
+## Why use the VARIANT data type in Apache Doris? {#why}
-Storing JSON in an OLAP system has been a long compromise.
+The Apache Doris VARIANT data type ends the old trade-off between flexible JSON storage and column-store query speed. Storing JSON in an OLAP system has been a long compromise.
- A `STRING` or `JSONB` column keeps you flexible, but every query that touches a path parses the whole document. `WHERE payload['action'] = 'opened'` over a billion rows reads every byte of every payload, even if `action` is one short field in a fat document.
- Static columns are fast, but only if the schema is stable. When producers add fields next sprint, you are back to `ALTER TABLE` migrations or a JSON sidecar column.
-`VARIANT` removes the choice. You declare one column, write JSON into it, and Doris quietly turns the hot paths into real columnar storage. Queries stay in SQL; on disk the table looks a lot like one you would have designed by hand.
+Apache Doris VARIANT removes the choice. You declare one column, write JSON into it, and Apache Doris quietly turns the hot paths into real columnar storage. Queries stay in SQL; on disk the table looks a lot like one you would have designed by hand.
-## What it is {#what}
+## What is the Apache Doris VARIANT data type? {#what}
-`VARIANT` is a semi-structured column type backed by Doris's `DataTypeVariant` and `ColumnVariant` (formerly `ColumnObject`, modeled on ClickHouse's design). It accepts any valid JSON value: scalars, one-dimensional arrays, and nested objects. As rows arrive, the writer builds a prefix tree of paths, infers a type per path, and writes each path as its own page-encoded, page-indexed column inside the segment. You query JSON; on disk you get one subcolumn per path.
+The Apache Doris VARIANT data type is a semi-structured column type backed by `DataTypeVariant` and `ColumnVariant` (formerly `ColumnObject`, modeled on ClickHouse's design). VARIANT accepts any valid JSON value: scalars, one-dimensional arrays, and nested objects. As rows arrive, the writer builds a prefix tree of paths, infers a type per path, and writes each path as its own page-encoded, page-indexed column inside the segment. You query JSON; on disk you get one subcolumn per path.
**Key terms**
@@ -43,9 +43,9 @@ Storing JSON in an OLAP system has been a long compromise.
- **`Sparse column`**: a shared fallback column where long-tail paths land when the path count exceeds the subcolumn budget. One physical column, many logical paths.
- **`Type promotion`**: the rule that decides what to do when the same path arrives as `INT` in one row and `STRING` in another. Compatible types widen (`TINYINT` to `BIGINT`); incompatible types fall back to `JSONB`.
-## How it works {#how}
+## How does the Apache Doris VARIANT data type work? {#how}
-For one path, the lifecycle on disk:
+Apache Doris VARIANT works by parsing each JSON document at write time, inferring a type per JSON path, and persisting frequent paths as independent columnar subcolumns while long-tail paths land in a shared sparse column. For one path, the lifecycle on disk:
1. **Parse and infer.** As JSON enters the memtable, the writer adds every leaf path to a prefix tree and picks the narrowest type that fits the values: `BIGINT`, `DOUBLE`, `STRING`, one-dimensional `ARRAY<T>`, or a nested `VARIANT`.
2. **Promote or fall back.** If a new value conflicts with the current type, Doris widens where it can (`TINYINT` plus `DOUBLE` becomes `DOUBLE`). If nothing widens, the path falls back to `JSONB` and queries lose pushdown on that path until the schema is corrected.
@@ -91,7 +91,9 @@ GROUP BY action;
The filter reads one subcolumn, hits the inverted index, and never touches the `commits` array. Run `SET describe_extend_variant_column = true; DESC github_events;` to see the inferred subcolumns (`payload.action`, `payload.commits`, `payload.issue.number`, ...) as if you had declared them by hand.
-## When to use it {#when}
+## When should you use the Apache Doris VARIANT data type? {#when}
+
+Use Apache Doris VARIANT when the JSON schema evolves but most queries hit a handful of familiar paths. Stick to static columns or `STRUCT` when the shape is fixed, and avoid VARIANT for primary, sort, or join keys.
**Good fit**
diff --git a/docs-next/key-features/vector-index.mdx b/docs-next/key-features/vector-index.mdx
index 707bb7f243a..fdfc17c67ca 100644
--- a/docs-next/key-features/vector-index.mdx
+++ b/docs-next/key-features/vector-index.mdx
@@ -19,36 +19,38 @@ featureCard:
- ai
---
-> **TL;DR** Doris 4.0 added a native ANN index on `ARRAY<FLOAT>` columns. You declare it inline in `CREATE TABLE` (or with `CREATE INDEX ... USING ANN`), pick HNSW or IVF, pick L2 or inner product, and then call `l2_distance_approximate` or `inner_product_approximate` in `ORDER BY ... LIMIT k`. The planner does the rest. Built on Faiss, it pre-filters before the TopN, so `WHERE` predicates stay correct.
+> **TL;DR** Apache Doris 4.0 added a native ANN index on `ARRAY<FLOAT>` columns. The index is declared inline in `CREATE TABLE` (or with `CREATE INDEX ... USING ANN`); pick HNSW or IVF, pick L2 or inner product, and then call `l2_distance_approximate` or `inner_product_approximate` in `ORDER BY ... LIMIT k`. Built on Faiss, the vector index pre-filters before the TopN, so `WHERE` predicates stay correct.
-## Why it matters {#why}
+## Why use the vector index in Apache Doris? {#why}
-The default move when you need vector search is to bolt on a separate vector database. That works, but you pay for it twice. Once in dollars, and once in the operational tax of running two systems where the keyword filters live on one side and the [embeddings](./embedding) live on the other.
+The Apache Doris vector index removes the need for a separate vector database alongside your analytics warehouse. The default move when you need vector search is to bolt on a second cluster. That works, but you pay for it twice. Once in dollars, and once in the operational tax of running two systems where the keyword filters live on one side and the [embeddings](./embedding) live on the other.
- A separate cluster to size, monitor, and pay for, just to answer "find the closest 10 vectors."
- Drift between the analytics table and the vector store, when a row gets updated in one and not the other.
- Glue code in the application that fans out the query and merges the results, because the SQL engine cannot see the vectors.
-Doris pulls the vector index into the same table as the rest of your data. One pipeline in, one SQL query out.
+Apache Doris pulls the vector index into the same table as the rest of your data. One pipeline in, one SQL query out.
-## What it is {#what}
+## What is the Apache Doris vector index? {#what}
-Vector Index in Doris is an ANN (Approximate Nearest Neighbor) secondary index on a vector-typed column. There is no new column type: vectors are stored as `ARRAY<FLOAT>` of fixed dimension. The index sits next to the data, builds at segment granularity, and gets used whenever the planner sees an approximate distance function in `ORDER BY`.
+The Apache Doris vector index is an ANN (Approximate Nearest Neighbor) secondary index on a vector-typed column. There is no new column type: vectors are stored as `ARRAY<FLOAT>` of fixed dimension. The index sits next to the data, builds at segment granularity, and gets used whenever the planner sees an approximate distance function in `ORDER BY`.
**Key terms**
-- **`ARRAY<FLOAT>`**: how Doris stores a vector. The column must be `NOT NULL`, and every row must have the same length as the index `dim`.
+- **`ARRAY<FLOAT>`**: how Apache Doris stores a vector. The column must be `NOT NULL`, and every row must have the same length as the index `dim`.
- **`USING ANN`**: the index clause. Properties pick the algorithm (`hnsw`, `ivf`, `ivf_on_disk`), the metric (`l2_distance` or `inner_product`), and the dimension.
- **HNSW**: the default graph-based algorithm. Fast, high recall, must stay in memory.
- **IVF / IVF on-disk**: inverted-file algorithms for larger sets where memory is tight.
- **`l2_distance_approximate` / `inner_product_approximate`**: the SQL functions that trigger the index. The non-`_approximate` versions exist too, but they fall through to a brute-force scan.
-## How it works {#how}
+## How does the Apache Doris vector index work? {#how}
-1. **Build at load time.** When a segment is written, Doris hands the column to Faiss and builds the index file alongside the data. One index per segment, no separate build job.
+The Apache Doris vector index runs through a five-step lifecycle: build at load time, recognize the query pattern, pre-filter, merge per-segment TopN, and let session knobs tune recall.
+
+1. **Build at load time.** When a segment is written, Apache Doris hands the column to Faiss and builds the index file alongside the data. One index per segment, no separate build job.
2. **Plan recognizes the pattern.** A query that reads `ORDER BY l2_distance_approximate(col, [...]) LIMIT k` is rewritten into an `AnnTopN` operator on the storage side. The literal vector is pushed down with the LIMIT.
-3. **Pre-filter, then TopN.** If there is a `WHERE` clause, Doris evaluates the predicate first, then runs ANN TopN on what survives. This keeps recall honest under selective filters. The predicate columns must have their own secondary index (such as inverted), or the engine falls back to brute force on those rows to keep results correct.
+3. **Pre-filter, then TopN.** If there is a `WHERE` clause, Apache Doris evaluates the predicate first, then runs ANN TopN on what survives. This keeps recall honest under selective filters. The predicate columns must have their own secondary index (such as inverted), or the engine falls back to brute force on those rows to keep results correct.
4. **Per-segment local TopN, then merge.** Each segment returns its own k closest. The TopN operator merges across segments and tablets to produce the global k.
5. **Tuneable at query time.** Session variables like `hnsw_ef_search` widen the candidate queue for higher recall, at the cost of latency. The defaults are conservative; nudge them when you have a recall target.
@@ -88,7 +90,9 @@ FROM docs ORDER BY dist LIMIT 3;
The query against `id=1` is exact (distance 0). The ANN index ranks the two close neighbors first and the outlier last. The same query shape against the SIFT-1M dataset (1 million 128-dim vectors) runs in about 20 ms with the index, versus about 290 ms without; both numbers are from the [Vector Search overview](../table-design/index/vector-index/overview).
-## When to use it {#when}
+## When should you use the Apache Doris vector index? {#when}
+
+The Apache Doris vector index fits any workload where the embeddings live in the same table as the structured columns you already filter and aggregate on.
**Good fit**
@@ -103,7 +107,7 @@ The query against `id=1` is exact (distance 0). The ANN index ranks the two clos
- `AGGREGATE KEY` tables, or `UNIQUE KEY` without merge-on-write. ANN indexes today require `DUPLICATE KEY` or MoW `UNIQUE KEY`.
- Nullable vector columns or rows with mismatched length. The column must be `NOT NULL ARRAY<FLOAT>`, and every row must match the declared `dim` exactly. Loads with the wrong shape fail.
- `score()` plus a distance function in the same `ORDER BY`. The TopN push-down accepts one ordering expression; if you need both BM25 and ANN scores fused, do the fusion above the SQL layer.
-- Cosine similarity called as `cosine_distance`. Doris has no cosine metric; normalize the vectors before insert and use `inner_product`. On unit-length vectors, cosine equals inner product.
+- Cosine similarity called as `cosine_distance`. Apache Doris has no cosine metric; normalize the vectors before insert and use `inner_product`. On unit-length vectors, cosine equals inner product.
## Performance / numbers {#performance}
diff --git a/docs-next/key-features/vectorized-execution.mdx b/docs-next/key-features/vectorized-execution.mdx
index a8ecfcab761..0aaa0bcf326 100644
--- a/docs-next/key-features/vectorized-execution.mdx
+++ b/docs-next/key-features/vectorized-execution.mdx
@@ -18,12 +18,12 @@ featureCard:
- performance
---
-> **TL;DR** Every Doris query runs on the vectorized engine. Operators move data in `Block`s of up to 4,096 column-oriented rows, the inner loops are tight enough for the compiler and hand-written intrinsics to use SIMD, and the row-based engine was deleted entirely in v2.0. There is nothing to enable. The only knob most users will ever touch is `batch_size`.
+> **TL;DR** Every Apache Doris query runs on the vectorized engine. Operators move data in `Block`s of up to 4,096 column-oriented rows, and the inner loops are tight enough for the compiler and hand-written intrinsics to use SIMD. The row-based engine was deleted entirely in v2.0, so there is nothing to enable; the only knob most users will ever touch is `batch_size`.
-## Why it matters {#why}
+## Why use vectorized execution in Apache Doris? {#why}
-A classic Volcano-style executor calls `next()` once per row, walking through a tree of virtual function dispatches to produce a single tuple. On a modern CPU, the actual arithmetic is a rounding error on top of the dispatch, branch mispredictions, and cache misses. Aggregating a billion rows row-by-row turns a fast machine into a slow one.
+Apache Doris vectorized execution replaces per-row virtual dispatch with column-batch inner loops so the CPU spends its time on arithmetic instead of indirection. A classic Volcano-style executor calls `next()` once per row, walking through a tree of virtual function dispatches to produce a single tuple. On a modern CPU, the actual arithmetic is a rounding error on top of the dispatch, branch mispredictions, and cache misses. Aggregating a billion rows row-by-row turns a fast machine int [...]
- One virtual call per row, per operator, multiplies into millions of indirect jumps the branch predictor cannot help with.
- Random row layout in memory keeps the CPU's L1/L2 caches mostly empty of useful data.
@@ -31,9 +31,9 @@ A classic Volcano-style executor calls `next()` once per row, walking through a
Vectorized execution flips the inner loop. An operator receives a batch of column values, runs a straight loop over that batch, and the hot path ends up looking like the kind of code AVX2 was built for.
-## What it is {#what}
+## What is Apache Doris vectorized execution? {#what}
-Vectorized execution in Doris is a column-batch query engine: every operator's `get_block()` consumes one `Block` and produces another, where a `Block` is a small set of columnar arrays describing 1 to `batch_size` rows. SIMD intrinsics accelerate the parts that benefit most: filter selection, decimal arithmetic, string comparison, JSON parsing. The model traces back to MonetDB/X100 and Vectorwise. ClickHouse, StarRocks, and DuckDB use the same design.
+Apache Doris vectorized execution is a column-batch query engine: every operator's `get_block()` consumes one `Block` and produces another, where a `Block` is a small set of columnar arrays describing 1 to `batch_size` rows. SIMD intrinsics accelerate the parts that benefit most: filter selection, decimal arithmetic, string comparison, JSON parsing. The model traces back to MonetDB/X100 and Vectorwise. ClickHouse, StarRocks, and DuckDB use the same design.
**Key terms**
@@ -42,7 +42,9 @@ Vectorized execution in Doris is a column-batch query engine: every operator's `
- **`batch_size`**: maximum rows in a Block. Default 4,096, configurable per session. Larger batches amortize per-call overhead; smaller ones reduce memory footprint and tail latency.
- **SIMD path**: hand-written code in `be/src/util/simd/` plus compiler-auto-vectorized loops, dispatched at runtime to AVX2 on x86 or NEON on ARM.
-## How it works {#how}
+## How does Apache Doris vectorized execution work? {#how}
+
+Apache Doris vectorized execution chains operators that pass columnar `Block`s of up to 4,096 rows, runs straight column-at-a-time loops with SIMD where it pays off, and lets the Pipeline scheduler interleave those batches across cores.
1. **Plan to operator chain.** The optimizer produces a plan, the FE rewrites it as a pipeline of operators, and each operator implements `get_block(state, block, eos)`.
2. **Pull a batch.** A scan operator reads up to `batch_size` rows from a tablet's columnar storage, fills a Block in place, and returns it. Rows never get materialized into structs.
@@ -84,7 +86,9 @@ LIMIT 10;
The aggregation runs column-at-a-time over 8,192-row Blocks. On a wide-table scan with millions of matching rows, the larger batch trades a bit more BE memory for fewer scheduling round-trips, which usually wins on long scans and loses on tiny point queries.
-## When to use it {#when}
+## When should you use Apache Doris vectorized execution? {#when}
+
+Apache Doris vectorized execution is the only execution mode since v2.0, so the real question is when to tune `batch_size` or rely on a different query path entirely.
**Good fit**
@@ -112,3 +116,4 @@ The aggregation runs column-at-a-time over 8,192-row Blocks. On a wide-table sca
- [Columnar Storage](./columnar-storage): the on-disk format whose pages and encodings feed directly into the vectorized columns.
- [Parquet Reader Optimization](./parquet-reader-optimization): how external Parquet decoding lands straight in the vectorized engine without an Arrow round trip.
- [Apache Doris 2.0 release notes (row-based engine removed)](https://doris.apache.org/blog/release-note-2.0.0/)
+- [MPP Architecture](./mpp): the layer that schedules vectorized operators across BEs, not just within one.
diff --git a/docs-next/key-features/vertical-compaction.mdx b/docs-next/key-features/vertical-compaction.mdx
index 06d9e1ea150..8b954d1a6d2 100644
--- a/docs-next/key-features/vertical-compaction.mdx
+++ b/docs-next/key-features/vertical-compaction.mdx
@@ -18,12 +18,12 @@ featureCard:
- performance
---
-> **TL;DR** Doris's classic [compaction](./data-compaction) merges rowsets a row at a time, which means every column sits in memory at once. On a 200-column table that gets expensive fast. Vertical compaction splits the schema into small column groups, merges one group at a time, and reuses a sequence marker built from the key columns. It cuts compaction memory to roughly 1/10 and runs about 15% faster on wide tables. On by default since Doris 1.2.2.
+> **TL;DR** Apache Doris vertical compaction splits a wide table's schema into small column groups and merges one group at a time, instead of holding every column in memory like the classic row-wise [compaction](./data-compaction). The algorithm cuts compaction memory to roughly 1/10 and runs about 15% faster on wide tables, using a per-row sequence marker built from the key columns to keep the output identical. Vertical compaction has been enabled by default since Apache Doris 1.2.2.
-## Why it matters {#why}
+## Why use vertical compaction in Apache Doris? {#why}
-Compaction is the background job that merges a tablet's rowsets into fewer, bigger ones so queries don't have to walk across hundreds of versions. The classic algorithm reads one row from each input rowset, picks a winner, writes it out, and repeats. That works fine on a 5-column fact table. It falls over on a 200-column user table — especially when those columns hold sprawling [VARIANT JSON payloads](./variant-data-type):
+Vertical compaction keeps Apache Doris compaction memory bounded on wide tables, where row-wise merging would otherwise pull every column into memory at once and OOM the BE. Compaction itself is the background job that merges a tablet's rowsets into fewer, bigger ones so queries don't have to walk across hundreds of versions. The classic algorithm reads one row from each input rowset, picks a winner, writes it out, and repeats. That works fine on a 5-column fact table. It falls over on a [...]
- A merging block has to hold every column for every row in the batch. Memory scales with column count and with batch size at the same time.
- The Backend (BE) runs many compaction tasks in parallel. Even a few wide-table tasks can push a node into OOM territory.
@@ -31,9 +31,9 @@ Compaction is the background job that merges a tablet's rowsets into fewer, bigg
Vertical compaction is the fix. It changes the unit of merging from "one row across all columns" to "one column group at a time across all rows," and pays for that with one extra structure: a per-row sequence marker.
-## What it is {#what}
+## What is Apache Doris vertical compaction? {#what}
-Vertical compaction merges columns in groups rather than merging rows. The Backend splits the table schema into a key-column group plus several value-column groups (5 columns per group by default), merges the key group first to fix a global ordering, then merges each value group by replaying that ordering. The output rowset matches what row-wise compaction would produce; only the memory profile changes.
+Apache Doris vertical compaction is a compaction algorithm that merges columns in groups rather than merging rows. The Backend splits the table schema into a key-column group plus several value-column groups (5 columns per group by default), merges the key group first to fix a global ordering, then merges each value group by replaying that ordering. The output rowset matches what row-wise compaction would produce; only the memory profile changes.
**Key terms**
@@ -43,7 +43,9 @@ Vertical compaction merges columns in groups rather than merging rows. The Backe
- **`Vertical merge iterator`**: the BE component that scans a single column group from all input rowsets and emits the merged column.
- **`enable_vertical_compaction`**: BE config flag, `true` by default. Disabling it falls back to row-wise compaction.
-## How it works {#how}
+## How does Apache Doris vertical compaction work? {#how}
+
+Apache Doris vertical compaction works by splitting the schema into column groups, merging the key group first to fix the row order, then replaying that order for every value group so non-group columns never sit in memory. The five-step flow below covers split, merge, replay, stitch, and spill.
1. **Split the schema into groups.** The BE walks the tablet schema, puts the key columns into group 0, and chunks the remaining columns into groups of `vertical_compaction_num_columns_per_group` (default 5). When `enable_vertical_compact_variant_subcolumns` is on, the BE folds variant subcolumns into the same flow.
2. **Merge the key group and record the order.** The Merger heap-sorts the key columns across all input rowsets. As each row leaves the heap, it writes one entry into the RowSourcesBuffer recording where the row came from. The output of this pass is a sorted key column plus a global RowSources tape.
@@ -86,7 +88,9 @@ SHOW CREATE TABLE user_profile;
A group size of 3 trades a bit more IO for tighter memory use, which is what you want on tables with many large `STRING` or `JSONB` columns. Verify it took effect with `SHOW CREATE TABLE`. To roll the change back to defaults, drop the property with `ALTER TABLE`.
-## When to use it {#when}
+## When should you use Apache Doris vertical compaction? {#when}
+
+Use Apache Doris vertical compaction on wide schemas, on tablets that OOM under row-wise compaction, and on tables with sparse VARIANT subcolumns. Skip the tuning on narrow tables and on disk-bound clusters where extra read passes hurt more than the memory savings help.
**Good fit**
@@ -115,3 +119,4 @@ A group size of 3 trades a bit more IO for tighter memory use, which is what you
- [Compaction principles, types, and scheduling](../admin-manual/trouble-shooting/compaction-principles): the scoring and scheduling logic that picks which tablets to compact next.
- [BE config reference](../admin-manual/config/be-config): every `vertical_compaction_*` flag with its default and unit.
- [Understanding Data Compaction in 3 Minutes (blog)](/blog/Understanding-Data-Compaction-in-3-Minutes/): the original write-up that introduces vertical compaction with diagrams.
+- [Unique Key](./unique-key): the table model that produces the per-rowset delete bitmaps vertical compaction has to reconcile.
diff --git a/docs-next/key-features/workload-group.mdx b/docs-next/key-features/workload-group.mdx
index d4c1305919f..4f7fe88ad25 100644
--- a/docs-next/key-features/workload-group.mdx
+++ b/docs-next/key-features/workload-group.mdx
@@ -19,20 +19,20 @@ featureCard:
- multi-tenancy
---
-> **TL;DR** Workload Group slices a single Doris cluster into named groups, each with its own CPU share, memory ceiling, query concurrency cap, and IO budget. Heavy ETL goes in one group, dashboards in another, ad-hoc analysts in a third. A runaway query in one group no longer takes the others down with it. Linux cgroups handle CPU isolation; the BE process handles memory, IO, and concurrency.
+> **TL;DR** Apache Doris Workload Group slices a single cluster into named groups, each with its own CPU share, memory ceiling, query concurrency cap, and IO budget. Heavy ETL goes in one group, dashboards in another, ad-hoc analysts in a third. A runaway query in one group no longer takes the others down with it. Linux cgroups handle CPU isolation; the BE process handles memory, IO, and concurrency.
-## Why it matters {#why}
+## Why use workload group in Apache Doris? {#why}
-A production Doris cluster almost never runs one workload. A typical day looks like this: a 2 a.m. ETL pipeline rewrites yesterday's partitions, a finance team reruns their daily report at 9 a.m., an executive dashboard polls the same `GROUP BY region` every fifteen seconds during business hours, and an analyst kicks off a `SELECT *` because they forgot the `WHERE` clause.
+Apache Doris workload group prevents mixed workloads from starving each other on a shared cluster. A production Doris cluster almost never runs one workload. A typical day looks like this: a 2 a.m. ETL pipeline rewrites yesterday's partitions, a finance team reruns their daily report at 9 a.m., an executive dashboard polls the same `GROUP BY region` every fifteen seconds during business hours, and an analyst kicks off a `SELECT *` because they forgot the `WHERE` clause.
Without resource controls, those four workloads share one global pool. The ETL job runs on the same threads that serve the dashboard. A bad ad-hoc query fills the BE's memory and triggers a kill on the report. Concurrency is unbounded, so a sudden spike of dashboard refreshes piles up against the ETL load and degrades everyone.
Workload Group fixes this without adding hardware. You define one group per workload class, set the limits each group is allowed, and bind users to the right group. The cluster keeps running on the same set of BEs, but the BE schedulers, memory tracker, and cgroup controllers now divide capacity along the lines you drew.
-## What it is {#what}
+## What is the Apache Doris workload group? {#what}
-Workload Group is an in-process isolation feature: every BE process serves every group, but the BE knows which threads, scan slots, and memory belong to which group. CPU isolation routes BE worker threads into per-group cgroups. Memory isolation tracks per-group usage in the BE memory tracker and triggers spilling or query cancellation when a group exceeds its ceiling. Concurrency control sits at the FE: each group has a queue, and queries past the limit wait or fail fast.
+Apache Doris workload group is an in-process isolation feature: every BE process serves every group, but the BE knows which threads, scan slots, and memory belong to which group. CPU isolation routes BE worker threads into per-group cgroups. Memory isolation tracks per-group usage in the BE memory tracker and triggers spilling or query cancellation when a group exceeds its ceiling. Concurrency control sits at the FE: each group has a queue, and queries past the limit wait or fail fast.
**Key terms**
@@ -42,7 +42,9 @@ Workload Group is an in-process isolation feature: every BE process serves every
- **`read_bytes_per_second` / `remote_read_bytes_per_second`**: per-group IO throttles, applied per data directory on local tables and per BE for external sources.
- **`normal` group**: the default group, auto-created on first start, used for any session that has not been bound elsewhere. You cannot drop it.
-## How it works {#how}
+## How does the Apache Doris workload group work? {#how}
+
+The Apache Doris workload group resolves the target group at the FE, admits or queues the query, then enforces CPU, memory, and IO limits at the BE.
1. **Resolve the group (FE).** The FE picks a group in this order: an explicit `/*+ SET_VAR(workload_group='...') */` hint, the session variable `workload_group`, the user property `default_workload_group`, then `normal`. The FE checks USAGE privilege on the chosen group.
2. **Admit or queue (FE).** The group's queue checks how many queries are already running in this group on this FE. If under `max_concurrency`, the query gets a slot and proceeds. If over, it waits, capped by `max_queue_size` and timed out by `queue_timeout`. A full queue rejects fast.
@@ -86,7 +88,9 @@ FROM orders WHERE dt = CURRENT_DATE() GROUP BY region;
`SHOW WORKLOAD GROUPS` (or `SELECT ... FROM information_schema.workload_groups`) reports live counts. The 12 queries waiting in `bi` are queued behind the 47 already running. The four ETL queries get a guaranteed 30% of CPU even when BI is busy, and they cannot push past 70%.
-## When to use it {#when}
+## When should you use the Apache Doris workload group? {#when}
+
+Use the Apache Doris workload group when a single cluster runs more than one workload class and you need predictable behavior under load without buying more hardware.
**Good fit**
@@ -113,3 +117,4 @@ FROM orders WHERE dt = CURRENT_DATE() GROUP BY region;
- [Resource Group](./resource-group): physical BE-node isolation for stronger SLA boundaries.
- [Compute Group](./compute-group): the compute-storage-decoupled equivalent that adds isolation without copying data.
- [Multi-tenant workload isolation in Apache Doris](https://doris.apache.org/blog/multi-tenant-workload-isolation-in-apache-doris/): the original design blog with internal benchmarks for soft- and hard-limit CPU on a 16-core BE.
+- [MPP Architecture](./mpp): the execution model whose fragment instances Workload Group caps per query.
diff --git a/docusaurus.config.js b/docusaurus.config.js
index 4992cfb699f..df8ce856012 100644
--- a/docusaurus.config.js
+++ b/docusaurus.config.js
@@ -326,10 +326,10 @@ const config = {
const filteredItems = items.filter(item => {
const pathname = new URL(item.url).pathname.replace(/\/+$/, '');
if (['/search', '/ja/search', '/zh-CN/search'].includes(pathname)) return false;
- // 灰度期 docs-next 不进 sitemap
- if (pathname.startsWith('/docs-next') || pathname.startsWith('/zh-CN/docs-next')) {
- return false;
- }
+ // Exclude docs-next during graceful rollout (kept in sync with
+ // static/robots.txt and .github/workflows/docs-next-build.yml).
+ if (pathname === '/docs-next' || pathname.startsWith('/docs-next/')) return false;
+ if (pathname === '/zh-CN/docs-next' || pathname.startsWith('/zh-CN/docs-next/')) return false;
return true;
});
for (let item of filteredItems) {
diff --git a/src/components/use-cases-next/CustomerFacingAnalyticsNext.tsx b/src/components/use-cases-next/CustomerFacingAnalyticsNext.tsx
index d527161b59c..f77b00daa31 100644
--- a/src/components/use-cases-next/CustomerFacingAnalyticsNext.tsx
+++ b/src/components/use-cases-next/CustomerFacingAnalyticsNext.tsx
@@ -429,6 +429,7 @@ const capabilities: Capability[] = [
{ label: 'Group Commit', href: '/docs-next/dev/key-features/group-commit' },
{ label: 'Kafka/CDC Integration', href: '/docs-next/dev/key-features/kafka-cdc-integration' },
{ label: 'Incremental Materialized View', href: '/docs-next/dev/key-features/incremental-materialized-view' },
+ { label: 'Unique Key', href: '/key-features/unique-key' },
],
},
{
@@ -486,6 +487,7 @@ const capabilities: Capability[] = [
{ label: 'Parquet Reader Optimization', href: '/docs-next/dev/key-features/parquet-reader-optimization' },
{ label: 'Data Cache & Page Cache', href: '/docs-next/dev/key-features/data-cache-page-cache' },
{ label: 'Metadata Cache', href: '/docs-next/dev/key-features/metadata-cache' },
+ { label: 'Iceberg', href: '/key-features/iceberg' },
],
},
];
diff --git a/src/components/use-cases-next/DataWarehousingNext.tsx b/src/components/use-cases-next/DataWarehousingNext.tsx
index 2a02dd5b711..9704fbba5c4 100644
--- a/src/components/use-cases-next/DataWarehousingNext.tsx
+++ b/src/components/use-cases-next/DataWarehousingNext.tsx
@@ -423,6 +423,7 @@ const capabilities: Capability[] = [
{ label: 'Rollup', href: '#' },
{ label: 'Analytic Functions', href: '#' },
{ label: 'Partitioning and Bucketing', href: '#' },
+ { label: 'Unique Key', href: '/key-features/unique-key' },
],
},
{
@@ -480,6 +481,7 @@ const capabilities: Capability[] = [
poweredBy: [
{ label: 'Multi Catalog', href: '#' },
{ label: 'Managing Lake Table', href: '#' },
+ { label: 'Iceberg', href: '/key-features/iceberg' },
],
},
{
diff --git a/src/theme/DocItem/Layout/pathTransfer.ts b/src/theme/DocItem/Layout/pathTransfer.ts
index be79736db7d..277c8fe5355 100644
--- a/src/theme/DocItem/Layout/pathTransfer.ts
+++ b/src/theme/DocItem/Layout/pathTransfer.ts
@@ -1,4 +1,7 @@
const transformPathWithoutZhCN = (pathname: string): string => {
+ if (pathname.startsWith('/docs-next/')) {
+ return `/docs-next${pathname.replace('/docs-next/dev', '')}.md`;
+ }
if(pathname.startsWith('/docs')){
const pathWithoutDocs = pathname.replace('/docs', '');
if(pathname.includes('/4.x')){
@@ -32,6 +35,9 @@ const stripLocalePrefix = (pathname: string): string => {
};
const transformPathWithZhCN = (pathname: string): string => {
+ if (pathname.startsWith('/zh-CN/docs-next/')) {
+ return `/i18n/zh-CN/docusaurus-plugin-content-docs-next/current${pathname.replace('/zh-CN/docs-next/dev', '')}.md`;
+ }
if (pathname.startsWith('/zh-CN/docs')) {
if(pathname.includes('/4.x')){
return `/i18n/zh-CN/docusaurus-plugin-content-docs/version-4.x${pathname.replace('/zh-CN/docs/4.x', '')}.md`;
diff --git a/static/images/next/key-features/iceberg.jpg b/static/images/next/key-features/iceberg.jpg
new file mode 100644
index 00000000000..26987a072ab
Binary files /dev/null and b/static/images/next/key-features/iceberg.jpg differ
diff --git a/static/images/next/key-features/mpp.jpg b/static/images/next/key-features/mpp.jpg
new file mode 100644
index 00000000000..23e62b3752a
Binary files /dev/null and b/static/images/next/key-features/mpp.jpg differ
diff --git a/static/images/next/key-features/unique-key.jpg b/static/images/next/key-features/unique-key.jpg
new file mode 100644
index 00000000000..de6edab2d09
Binary files /dev/null and b/static/images/next/key-features/unique-key.jpg differ
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org