You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@paimon.apache.org by lz...@apache.org on 2026/08/13 13:56:14 UTC
(paimon) branch master updated: [docs] Clarify snapshot record count semantics (#9202)
This is an automated email from the ASF dual-hosted git repository.
JingsongLi pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/paimon.git
The following commit(s) were added to refs/heads/master by this push:
new c2aea2bac1 [docs] Clarify snapshot record count semantics (#9202)
c2aea2bac1 is described below
commit c2aea2bac1820e25744e7f85e9944a2130dd211f
Author: wangwj <ho...@gmail.com>
AuthorDate: Thu Aug 13 21:56:08 2026 +0800
[docs] Clarify snapshot record count semantics (#9202)
---
docs/docs/concepts/spec/snapshot.md | 9 +++++++--
docs/docs/concepts/system-tables.mdx | 6 ++++++
docs/docs/pypaimon/system-tables.md | 9 +++++++--
paimon-api/src/main/java/org/apache/paimon/Snapshot.java | 4 ++--
.../src/test/java/org/apache/paimon/append/BlobTableTest.java | 5 +++++
.../java/org/apache/paimon/append/MultipleBlobTableTest.java | 5 +++++
6 files changed, 32 insertions(+), 6 deletions(-)
diff --git a/docs/docs/concepts/spec/snapshot.md b/docs/docs/concepts/spec/snapshot.md
index d09db56a51..a9ae2586b0 100644
--- a/docs/docs/concepts/spec/snapshot.md
+++ b/docs/docs/concepts/spec/snapshot.md
@@ -58,11 +58,16 @@ Snapshot File is JSON, it includes:
11. commitKind: type of changes in this snapshot, including append, compact, overwrite and analyze.
12. timeMillis: commit time millis.
13. logOffsets: commit log offsets.
-14. totalRecordCount: record count of all changes occurred in this snapshot.
-15. deltaRecordCount: record count of all new changes occurred in this snapshot.
+14. totalRecordCount: unmerged record count of all live data files in this snapshot.
+15. deltaRecordCount: net change of the unmerged record count from data files added and deleted in this snapshot.
16. changelogRecordCount: record count of all changelog produced in this snapshot.
17. watermark: watermark for input records, from Flink watermark mechanism, Long.MIN_VALUE if there is no watermark.
18. statistics: stats file name for statistics of this table.
19. properties: additional key-value properties of this snapshot.
20. nextRowId: next row id for row tracking.
21. operation: logical operation type, e.g. WRITE, DELETE, UPDATE, MERGE. Null if not set.
+
+The record counts are calculated per data file and are not logical row counts. For example, a
+Dedicated Format table stores the regular columns and each dedicated BLOB column in separate data
+files. Appending `N` logical rows to a table with one dedicated BLOB column therefore increases the
+`deltaRecordCount` by `2 * N`. Use `COUNT(*)` when you need the logical row count.
diff --git a/docs/docs/concepts/system-tables.mdx b/docs/docs/concepts/system-tables.mdx
index 67fb73a541..0bb6d2629c 100644
--- a/docs/docs/concepts/system-tables.mdx
+++ b/docs/docs/concepts/system-tables.mdx
@@ -64,6 +64,12 @@ SELECT * FROM my_table$snapshots;
By querying the snapshots table, you can know the commit and expiration information about that table and time travel through the data.
+`total_record_count` and `delta_record_count` are unmerged counts calculated from data files, not
+logical row counts. All dedicated data files contribute to these values. For example, appending `N`
+logical rows to a table with one [dedicated BLOB column](../multimodal-table/blob#storage-layout)
+adds `N` records to the regular data files and `N` records to the BLOB files, so
+`delta_record_count` increases by `2 * N`. Use `COUNT(*)` when you need the logical row count.
+
### Schemas Table
You can query the historical schemas of the table through schemas table.
diff --git a/docs/docs/pypaimon/system-tables.md b/docs/docs/pypaimon/system-tables.md
index fdbf287745..35f2532afd 100644
--- a/docs/docs/pypaimon/system-tables.md
+++ b/docs/docs/pypaimon/system-tables.md
@@ -86,12 +86,17 @@ One row per persisted snapshot.
| `base_manifest_list` | STRING NOT NULL | |
| `delta_manifest_list` | STRING NOT NULL | |
| `changelog_manifest_list` | STRING | |
-| `total_record_count` | BIGINT | |
-| `delta_record_count` | BIGINT | |
+| `total_record_count` | BIGINT | Unmerged record count of all live data files; not a logical row count |
+| `delta_record_count` | BIGINT | Net change of the unmerged record count from added and deleted data files |
| `changelog_record_count` | BIGINT | |
| `watermark` | BIGINT | |
| `next_row_id` | BIGINT | |
+Dedicated Format columns are stored in separate data files, and each of those files contributes to
+the snapshot record counts. For example, appending `N` logical rows to a table with one dedicated
+BLOB column increases `delta_record_count` by `2 * N`. Use `COUNT(*)` when you need the logical row
+count.
+
### `$schemas`
Every committed schema version, with `fields` / `partition_keys` /
diff --git a/paimon-api/src/main/java/org/apache/paimon/Snapshot.java b/paimon-api/src/main/java/org/apache/paimon/Snapshot.java
index d34b061b78..2f98783ecc 100644
--- a/paimon-api/src/main/java/org/apache/paimon/Snapshot.java
+++ b/paimon-api/src/main/java/org/apache/paimon/Snapshot.java
@@ -148,11 +148,11 @@ public class Snapshot implements Serializable {
@JsonProperty(FIELD_TIME_MILLIS)
protected final long timeMillis;
- // record count of all changes occurred in this snapshot
+ // unmerged record count of all live data files in this snapshot
@JsonProperty(FIELD_TOTAL_RECORD_COUNT)
protected final long totalRecordCount;
- // record count of all new changes occurred in this snapshot
+ // net change of the unmerged record count from data files added and deleted in this snapshot
@JsonProperty(FIELD_DELTA_RECORD_COUNT)
protected final long deltaRecordCount;
diff --git a/paimon-core/src/test/java/org/apache/paimon/append/BlobTableTest.java b/paimon-core/src/test/java/org/apache/paimon/append/BlobTableTest.java
index 0c6833ce28..caa268e0ca 100644
--- a/paimon-core/src/test/java/org/apache/paimon/append/BlobTableTest.java
+++ b/paimon-core/src/test/java/org/apache/paimon/append/BlobTableTest.java
@@ -134,6 +134,11 @@ public class BlobTableTest extends TableTestBase {
commitDefault(writeDataDefault(1000, 1));
+ assertThat(getTableDefault().snapshotManager().latestSnapshot().totalRecordCount())
+ .isEqualTo(2000L);
+ assertThat(getTableDefault().snapshotManager().latestSnapshot().deltaRecordCount())
+ .isEqualTo(2000L);
+
AtomicInteger integer = new AtomicInteger(0);
List<DataFileMeta> filesMetas =
diff --git a/paimon-core/src/test/java/org/apache/paimon/append/MultipleBlobTableTest.java b/paimon-core/src/test/java/org/apache/paimon/append/MultipleBlobTableTest.java
index 8275558526..a8fd2cc328 100644
--- a/paimon-core/src/test/java/org/apache/paimon/append/MultipleBlobTableTest.java
+++ b/paimon-core/src/test/java/org/apache/paimon/append/MultipleBlobTableTest.java
@@ -62,6 +62,11 @@ public class MultipleBlobTableTest extends TableTestBase {
commitDefault(writeDataDefault(1000, 1));
+ assertThat(getTableDefault().snapshotManager().latestSnapshot().totalRecordCount())
+ .isEqualTo(3000L);
+ assertThat(getTableDefault().snapshotManager().latestSnapshot().deltaRecordCount())
+ .isEqualTo(3000L);
+
AtomicInteger integer = new AtomicInteger(0);
FileStoreTable table = getTableDefault();