You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@paimon.apache.org by lz...@apache.org on 2024/07/09 04:24:26 UTC
(paimon) branch master updated: [doc] Split spec to multiple pages
This is an automated email from the ASF dual-hosted git repository.
lzljs3620320 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/paimon.git
The following commit(s) were added to refs/heads/master by this push:
new 2b0a2a9b7 [doc] Split spec to multiple pages
2b0a2a9b7 is described below
commit 2b0a2a9b782188a35f257c05fea607e437842ded
Author: Jingsong <ji...@gmail.com>
AuthorDate: Tue Jul 9 12:23:22 2024 +0800
[doc] Split spec to multiple pages
---
docs/content/concepts/spec/_index.md | 23 +++
docs/content/concepts/spec/datafile.md | 95 ++++++++++++
docs/content/concepts/spec/indexfile.md | 42 ++++++
docs/content/concepts/spec/manifest.md | 73 +++++++++
docs/content/concepts/spec/overview.md | 84 +++++++++++
docs/content/concepts/spec/schema.md | 106 +++++++++++++
docs/content/concepts/spec/snapshot.md | 65 ++++++++
docs/content/concepts/specification.md | 253 --------------------------------
8 files changed, 488 insertions(+), 253 deletions(-)
diff --git a/docs/content/concepts/spec/_index.md b/docs/content/concepts/spec/_index.md
new file mode 100644
index 000000000..3bd8e657f
--- /dev/null
+++ b/docs/content/concepts/spec/_index.md
@@ -0,0 +1,23 @@
+---
+title: Specification
+bookCollapseSection: true
+weight: 4
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
diff --git a/docs/content/concepts/spec/datafile.md b/docs/content/concepts/spec/datafile.md
new file mode 100644
index 000000000..6ba43a421
--- /dev/null
+++ b/docs/content/concepts/spec/datafile.md
@@ -0,0 +1,95 @@
+---
+title: "DataFile"
+weight: 5
+type: docs
+aliases:
+- /concepts/spec/datafile.html
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# DataFile
+
+## Partition
+
+Consider a Partition table via Flink SQL:
+
+```sql
+CREATE TABLE part_t (
+ f0 INT,
+ f1 STRING,
+ dt STRING
+) PARTITIONED BY (dt);
+
+INSERT INTO part_t VALUES (1, '11', '20240514');
+```
+
+The file system will be:
+
+```shell
+part_t
+├── dt=20240514
+│ └── bucket-0
+│ └── data-ca1c3c38-dc8d-4533-949b-82e195b41bd4-0.orc
+├── manifest
+│ ├── manifest-08995fe5-c2ac-4f54-9a5f-d3af1fcde41d-0
+│ ├── manifest-list-51c16f7b-421c-4bc0-80a0-17677f343358-0
+│ └── manifest-list-51c16f7b-421c-4bc0-80a0-17677f343358-1
+├── schema
+│ └── schema-0
+└── snapshot
+ ├── EARLIEST
+ ├── LATEST
+ └── snapshot-1
+```
+
+Paimon adopts the same partitioning concept as Apache Hive to separate data. The files of the partition will be placed
+in a separate partition directory.
+
+## Bucket
+
+The storage of all Paimon tables relies on buckets, and data files are stored in the bucket directory. The
+relationship between various table types and buckets in Paimon:
+
+1. Primary Key Table:
+ 1. bucket = -1: Default mode, the dynamic bucket mode records which bucket the key corresponds to through the index
+ files. The index records the correspondence between the hash value of the primary-key and the bucket.
+ 2. bucket = 10: The data is distributed to the corresponding buckets according to the hash value of bucket key (
+ default is primary key).
+2. Append Table:
+ 1. bucket = -1: Default mode, ignoring bucket concept, although all data is written to bucket-0, the parallelism of
+ reads and writes is unrestricted.
+ 2. bucket = 10: You need to define bucket-key too, the data is distributed to the corresponding buckets according to
+ the hash value of bucket key.
+
+## Data File
+
+The name of data file is `data-${uuid}-${id}.${format}`. For the append table, the file stores the data of the table
+without adding any new columns. But for the primary key table, each row of data stores additional system columns:
+
+1. `_VALUE_KIND`: row is deleted or added. Similar to RocksDB, each row of data can be deleted or added, which will be
+ used for updating the primary key table.
+2. `_SEQUENCE_NUMBER`: this number is used for comparison during updates, determining which data came first and which
+ data came later.
+3. `_KEY_` prefix to key columns, this is to avoid conflicts with columns of the table.
+
+## Changelog File
+
+Changelog file and Data file are exactly the same, it only takes effect on the primary key table. It is similar to the
+Binlog in a database, recording changes to the data in the table.
diff --git a/docs/content/concepts/spec/indexfile.md b/docs/content/concepts/spec/indexfile.md
new file mode 100644
index 000000000..cfcbcade9
--- /dev/null
+++ b/docs/content/concepts/spec/indexfile.md
@@ -0,0 +1,42 @@
+---
+title: "IndexFile"
+weight: 6
+type: docs
+aliases:
+- /concepts/spec/indexfile.html
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# IndexFile
+
+## Global Index
+
+Global Index is in the index directory, currently, only two places will use global index:
+
+1. bucket = -1 + primary key table: in dynamic bucket mode, the index records the correspondence between the hash value
+ of the primary-key and the bucket, each bucket has an index file.
+2. Deletion Vectors: index stores the deletion file, and each bucket has a deletion file.
+
+## Data File Index
+
+Define `file-index.bloom-filter.columns`, Paimon will create its corresponding index file for each file. If the index
+file is too small, it will be stored directly in the manifest, or in the directory of the data file. Each data file
+corresponds to an index file, which has a separate file definition and can contain different types of indexes with
+multiple columns.
diff --git a/docs/content/concepts/spec/manifest.md b/docs/content/concepts/spec/manifest.md
new file mode 100644
index 000000000..6f91a5a7a
--- /dev/null
+++ b/docs/content/concepts/spec/manifest.md
@@ -0,0 +1,73 @@
+---
+title: "Manifest"
+weight: 4
+type: docs
+aliases:
+- /concepts/spec/manifest.html
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Manifest
+
+## Manifest List
+
+Manifest List includes meta of several manifest files. Its name contains UUID, it is a avro file, the schema is:
+
+1. fileName: manifest file name.
+2. fileSize: manifest file size.
+3. numAddedFiles: number added files in manifest.
+4. numDeletedFiles: number deleted files in manifest.
+5. partitionStats: partition stats, the minimum and maximum values of partition fields in this manifest are beneficial
+ for skipping certain manifest files during queries, it is a SimpleStats.
+6. schemaId: schema id when writing this manifest file.
+
+## Manifest
+
+Manifest includes meta of several data files or changelog files. Its name contains UUID, it is a avro file.
+
+The changes of the file are saved in the manifest, and the file can be added or deleted. Manifests should be in
+an orderly manner, and the same file may be added or deleted multiple times. The last version should be read. This
+design can make commit lighter to support file deletion generated by compaction.
+
+The schema is:
+
+1. kind: ADD or DELETE,
+2. partition: partition spec, a BinaryRow.
+3. bucket: bucket of this file.
+4. totalBuckets: total buckets when write this file, it is used for verification after bucket changes.
+5. file: data file meta.
+
+The data file meta is:
+
+1. fileName: file name.
+2. fileSize: file size.
+3. rowCount: total number of rows (including add & delete) in this file.
+4. minKey: the minimum key of this file.
+5. maxKey: the maximum key of this file.
+6. keyStats: the statistics of the key.
+7. valueStats: the statistics of the value.
+8. minSequenceNumber: the minimum sequence number.
+9. maxSequenceNumber: the maximum sequence number.
+10. schemaId: schema id when write this file.
+11. level: level of this file, in LSM.
+12. extraFiles: extra files for this file, for example, data file index file.
+13. creationTime: creation time of this file.
+14. deleteRowCount: rowCount = addRowCount + deleteRowCount.
+15. embeddedIndex: if data file index is too small, store the index in manifest.
diff --git a/docs/content/concepts/spec/overview.md b/docs/content/concepts/spec/overview.md
new file mode 100644
index 000000000..a48e8451e
--- /dev/null
+++ b/docs/content/concepts/spec/overview.md
@@ -0,0 +1,84 @@
+---
+title: "Overview"
+weight: 1
+type: docs
+aliases:
+- /concepts/spec/overview.html
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Spec Overview
+
+This is the specification for the Paimon table format, this document standardizes the underlying file structure and
+design of Paimon.
+
+{{< img src="/img/file-layout.png">}}
+
+## Terms
+
+- Schema: fields, primary keys definition, partition keys definition and options.
+- Snapshot: the entrance to all data committed at some specific time point.
+- Manifest list: includes several manifest files.
+- Manifest: includes several data files or changelog files.
+- Data File: contains incremental records.
+- Changelog File: contains records produced by changelog-producer.
+- Global Index: index for a bucket or partition.
+- Data File Index: index for a data file.
+
+Run Flink SQL with Paimon:
+
+```sql
+CREATE CATALOG my_catalog WITH (
+ 'type' = 'paimon',
+ 'warehouse' = '/your/path'
+);
+USE CATALOG my_catalog;
+
+CREATE TABLE my_table (
+ k INT PRIMARY KEY NOT ENFORCED,
+ f0 INT,
+ f1 STRING
+);
+
+INSERT INTO my_table VALUES (1, 11, '111');
+```
+
+Take a look to the disk:
+
+```shell
+warehouse
+└── default.db
+ └── my_table
+ ├── bucket-0
+ │ └── data-59f60cb9-44af-48cc-b5ad-59e85c663c8f-0.orc
+ ├── index
+ │ └── index-5625e6d9-dd44-403b-a738-2b6ea92e20f1-0
+ ├── manifest
+ │ ├── index-manifest-5d670043-da25-4265-9a26-e31affc98039-0
+ │ ├── manifest-6758823b-2010-4d06-aef0-3b1b597723d6-0
+ │ ├── manifest-list-9f856d52-5b33-4c10-8933-a0eddfaa25bf-0
+ │ └── manifest-list-9f856d52-5b33-4c10-8933-a0eddfaa25bf-1
+ ├── schema
+ │ └── schema-0
+ └── snapshot
+ ├── EARLIEST
+ ├── LATEST
+ └── snapshot-1
+```
diff --git a/docs/content/concepts/spec/schema.md b/docs/content/concepts/spec/schema.md
new file mode 100644
index 000000000..f9915e9f4
--- /dev/null
+++ b/docs/content/concepts/spec/schema.md
@@ -0,0 +1,106 @@
+---
+title: "Schema"
+weight: 2
+type: docs
+aliases:
+- /concepts/spec/schema.html
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Schema
+
+The version of the schema file starts from 0 and currently retains all versions of the schema. There may be old files
+that rely on the old schema version, so its deletion should be done with caution.
+
+Schema File is JSON, it includes:
+
+1. fields: data field list, data field contains `id`, `name`, `type`, field id is used to support schema evolution.
+2. partitionKeys: field name list, partition definition of the table, it cannot be modified.
+3. primaryKeys: field name list, primary key definition of the table, it cannot be modified.
+4. options: map<string, string>, no ordered, options of the table, including a lot of capabilities and optimizations.
+
+## Example
+
+```shell
+{
+ "version" : 3,
+ "id" : 0,
+ "fields" : [ {
+ "id" : 0,
+ "name" : "order_id",
+ "type" : "BIGINT NOT NULL"
+ }, {
+ "id" : 1,
+ "name" : "order_name",
+ "type" : "STRING"
+ }, {
+ "id" : 2,
+ "name" : "order_user_id",
+ "type" : "BIGINT"
+ }, {
+ "id" : 3,
+ "name" : "order_shop_id",
+ "type" : "BIGINT"
+ } ],
+ "highestFieldId" : 3,
+ "partitionKeys" : [ ],
+ "primaryKeys" : [ "order_id" ],
+ "options" : {
+ "bucket" : "5"
+ },
+ "comment" : "",
+ "timeMillis" : 1720496663041
+}
+```
+
+## Compatibility
+
+For old versions:
+- version 1: should put `bucket -> 1` to options if there is no `bucket` key.
+- version 1 & 2: should put `file.format -> orc` to options if there is no `file.format` key.
+
+## DataField
+
+DataField represents a column of the table.
+
+1. id: int, column id, automatic increment, it is used for schema evolution.
+2. name: string, column name.
+3. type: data type, it is very similar to SQL type string.
+4. description: string.
+
+## Update Schema
+
+Updating the schema should generate a new schema file.
+
+```shell
+warehouse
+└── default.db
+ └── my_table
+ ├── schema
+ ├── schema-0
+ ├── schema-1
+ └── schema-2
+```
+
+There is a reference to schema in the snapshot. The schema file with the highest numerical value is usually the latest
+schema file.
+
+Old schema files cannot be directly deleted because there may be old data files that reference old schema files. When
+reading table, it is necessary to rely on them for schema evolution reading.
diff --git a/docs/content/concepts/spec/snapshot.md b/docs/content/concepts/spec/snapshot.md
new file mode 100644
index 000000000..5c0f58ac4
--- /dev/null
+++ b/docs/content/concepts/spec/snapshot.md
@@ -0,0 +1,65 @@
+---
+title: "Snapshot"
+weight: 3
+type: docs
+aliases:
+- /concepts/spec/snapshot.html
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Snapshot
+
+Each commit generates a snapshot file, and the version of the snapshot file starts from 1 and must be continuous.
+`EARLIEST` and `LATEST` are hint files at the beginning and end of the snapshot list, and they can be inaccurate.
+When hint files are inaccurate, the read will scan all snapshot files to determine the beginning and end.
+
+```shell
+warehouse
+└── default.db
+ └── my_table
+ ├── snapshot
+ ├── EARLIEST
+ ├── LATEST
+ ├── snapshot-1
+ ├── snapshot-2
+ └── snapshot-3
+```
+
+Writing commit will preempt the next snapshot id, and once the snapshot file is successfully written, this commit will
+be visible.
+
+Snapshot File is JSON, it includes:
+
+1. version: Snapshot file version, current is 3.
+2. id: snapshot id, same to file name.
+3. schemaId: the corresponding schema version for this commit.
+4. baseManifestList: a manifest list recording all changes from the previous snapshots.
+5. deltaManifestList: a manifest list recording all new changes occurred in this snapshot.
+6. changelogManifestList: a manifest list recording all changelog produced in this snapshot, null if no changelog is produced.
+7. indexManifest: a manifest recording all index files of this table, null if no index file.
+8. commitUser: usually generated by UUID, it is used for recovery of streaming writes, one stream write job with one user.
+9. commitIdentifier: transaction id corresponding to streaming write, each transaction may result in multiple commits for different commitKinds.
+10. commitKind: type of changes in this snapshot, including append, compact, overwrite and analyze.
+11. timeMillis: commit time millis.
+12. totalRecordCount: record count of all changes occurred in this snapshot.
+13. deltaRecordCount: record count of all new changes occurred in this snapshot.
+14. changelogRecordCount: record count of all changelog produced in this snapshot.
+15. watermark: watermark for input records, from Flink watermark mechanism, null if there is no watermark.
+16. statistics: stats file name for statistics of this table.
diff --git a/docs/content/concepts/specification.md b/docs/content/concepts/specification.md
deleted file mode 100644
index edbe74e74..000000000
--- a/docs/content/concepts/specification.md
+++ /dev/null
@@ -1,253 +0,0 @@
----
-title: "Specification"
-weight: 4
-type: docs
-aliases:
-- /concepts/spec.html
----
-<!--
-Licensed to the Apache Software Foundation (ASF) under one
-or more contributor license agreements. See the NOTICE file
-distributed with this work for additional information
-regarding copyright ownership. The ASF licenses this file
-to you under the Apache License, Version 2.0 (the
-"License"); you may not use this file except in compliance
-with the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing,
-software distributed under the License is distributed on an
-"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-KIND, either express or implied. See the License for the
-specific language governing permissions and limitations
-under the License.
--->
-
-# Specification
-
-This is the specification for the Paimon table format, this document standardizes the underlying file structure and
-design of Paimon.
-
-{{< img src="/img/file-layout.png">}}
-
-## Terms
-
-- Schema: fields, primary keys definition, partition keys definition and options.
-- Snapshot: the entrance to all data committed at some specific time point.
-- Manifest list: includes several manifest files.
-- Manifest: includes several data files or changelog files.
-- Data File: contains incremental records.
-- Changelog File: contains records produced by changelog-producer.
-- Global Index: index for a bucket or partition.
-- Data File Index: index for a data file.
-
-Run Flink SQL with Paimon:
-
-```sql
-CREATE CATALOG my_catalog WITH (
- 'type' = 'paimon',
- 'warehouse' = '/your/path'
-);
-USE CATALOG my_catalog;
-
-CREATE TABLE my_table (
- k INT PRIMARY KEY NOT ENFORCED,
- f0 INT,
- f1 STRING
-);
-
-INSERT INTO my_table VALUES (1, 11, '111');
-```
-
-Take a look to the disk:
-
-```shell
-warehouse
-└── default.db
- └── my_table
- ├── bucket-0
- │ └── data-59f60cb9-44af-48cc-b5ad-59e85c663c8f-0.orc
- ├── index
- │ └── index-5625e6d9-dd44-403b-a738-2b6ea92e20f1-0
- ├── manifest
- │ ├── index-manifest-5d670043-da25-4265-9a26-e31affc98039-0
- │ ├── manifest-6758823b-2010-4d06-aef0-3b1b597723d6-0
- │ ├── manifest-list-9f856d52-5b33-4c10-8933-a0eddfaa25bf-0
- │ └── manifest-list-9f856d52-5b33-4c10-8933-a0eddfaa25bf-1
- ├── schema
- │ └── schema-0
- └── snapshot
- ├── EARLIEST
- ├── LATEST
- └── snapshot-1
-```
-
-## Schema
-
-The version of the schema file starts from 0 and currently retains all versions of the schema. There may be old files
-that rely on the old schema version, so its deletion should be done with caution.
-
-Schema File is JSON, it includes:
-
-1. fields: data field list, data field contains `id`, `name`, `type`, field id is used to support schema evolution.
-2. partitionKeys: partition definition of the table, it cannot be modified.
-3. primaryKeys: primary key definition of the table, it cannot be modified.
-4. options: options of the table, including a lot of capabilities and optimizations.
-
-## Snapshot
-
-Each commit generates a snapshot file, and the version of the snapshot file starts from 1 and must be continuous.
-`EARLIEST` and `LATEST` are hint files at the beginning and end of the snapshot list, and they can be inaccurate.
-When hint files are inaccurate, the read will scan all snapshot files to determine the beginning and end.
-
-Writing commit will preempt the next snapshot id, and once the snapshot file is successfully written, this commit will
-be visible.
-
-Snapshot File is JSON, it includes:
-
-1. version: Snapshot file version, current is 3.
-2. id: snapshot id, same to file name.
-3. schemaId: the corresponding schema version for this commit.
-4. baseManifestList: a manifest list recording all changes from the previous snapshots.
-5. deltaManifestList: a manifest list recording all new changes occurred in this snapshot.
-6. changelogManifestList: a manifest list recording all changelog produced in this snapshot, null if no changelog is produced.
-7. indexManifest: a manifest recording all index files of this table, null if no index file.
-8. commitUser: usually generated by UUID, it is used for recovery of streaming writes, one stream write job with one user.
-9. commitIdentifier: transaction id corresponding to streaming write, each transaction may result in multiple commits for different commitKinds.
-10. commitKind: type of changes in this snapshot, including append, compact, overwrite and analyze.
-11. timeMillis: commit time millis.
-12. totalRecordCount: record count of all changes occurred in this snapshot.
-13. deltaRecordCount: record count of all new changes occurred in this snapshot.
-14. changelogRecordCount: record count of all changelog produced in this snapshot.
-15. watermark: watermark for input records, from Flink watermark mechanism, null if there is no watermark.
-16. statistics: stats file name for statistics of this table.
-
-## Manifest List
-
-Manifest List includes meta of several manifest files. Its name contains UUID, it is a avro file, the schema is:
-
-1. fileName: manifest file name.
-2. fileSize: manifest file size.
-3. numAddedFiles: number added files in manifest.
-4. numDeletedFiles: number deleted files in manifest.
-5. partitionStats: partition stats, the minimum and maximum values of partition fields in this manifest are beneficial
- for skipping certain manifest files during queries, it is a SimpleStats.
-6. schemaId: schema id when writing this manifest file.
-
-## Manifest
-
-Manifest includes meta of several data files or changelog files. Its name contains UUID, it is a avro file.
-
-The changes of the file are saved in the manifest, and the file can be added or deleted. Manifests should be in
-an orderly manner, and the same file may be added or deleted multiple times. The last version should be read. This
-design can make commit lighter to support file deletion generated by compaction.
-
-The schema is:
-
-1. kind: ADD or DELETE,
-2. partition: partition spec, a BinaryRow.
-3. bucket: bucket of this file.
-4. totalBuckets: total buckets when write this file, it is used for verification after bucket changes.
-5. file: data file meta.
-
-The data file meta is:
-
-1. fileName: file name.
-2. fileSize: file size.
-3. rowCount: total number of rows (including add & delete) in this file.
-4. minKey: the minimum key of this file.
-5. maxKey: the maximum key of this file.
-6. keyStats: the statistics of the key.
-7. valueStats: the statistics of the value.
-8. minSequenceNumber: the minimum sequence number.
-9. maxSequenceNumber: the maximum sequence number.
-10. schemaId: schema id when write this file.
-11. level: level of this file, in LSM.
-12. extraFiles: extra files for this file, for example, data file index file.
-13. creationTime: creation time of this file.
-14. deleteRowCount: rowCount = addRowCount + deleteRowCount.
-15. embeddedIndex: if data file index is too small, store the index in manifest.
-
-## Partition
-
-Consider a Partition table via Flink SQL:
-
-```sql
-CREATE TABLE part_t (
- f0 INT,
- f1 STRING,
- dt STRING
-) PARTITIONED BY (dt);
-
-INSERT INTO part_t VALUES (1, '11', '20240514');
-```
-
-The file system will be:
-
-```shell
-part_t
-├── dt=20240514
-│ └── bucket-0
-│ └── data-ca1c3c38-dc8d-4533-949b-82e195b41bd4-0.orc
-├── manifest
-│ ├── manifest-08995fe5-c2ac-4f54-9a5f-d3af1fcde41d-0
-│ ├── manifest-list-51c16f7b-421c-4bc0-80a0-17677f343358-0
-│ └── manifest-list-51c16f7b-421c-4bc0-80a0-17677f343358-1
-├── schema
-│ └── schema-0
-└── snapshot
- ├── EARLIEST
- ├── LATEST
- └── snapshot-1
-```
-
-Paimon adopts the same partitioning concept as Apache Hive to separate data. The files of the partition will be placed
-in a separate partition directory.
-
-## Bucket
-
-The storage of all Paimon tables relies on buckets, and data files are stored in the bucket directory. The
-relationship between various table types and buckets in Paimon:
-
-1. Primary Key Table:
- 1. bucket = -1: Default mode, the dynamic bucket mode records which bucket the key corresponds to through the index
- files. The index records the correspondence between the hash value of the primary-key and the bucket.
- 2. bucket = 10: The data is distributed to the corresponding buckets according to the hash value of bucket key (
- default is primary key).
-2. Append Table:
- 1. bucket = -1: Default mode, ignoring bucket concept, although all data is written to bucket-0, the parallelism of
- reads and writes is unrestricted.
- 2. bucket = 10: You need to define bucket-key too, the data is distributed to the corresponding buckets according to
- the hash value of bucket key.
-
-## Data File
-
-The name of data file is `data-${uuid}-${id}.${format}`. For the append table, the file stores the data of the table
-without adding any new columns. But for the primary key table, each row of data stores additional system columns:
-
-1. `_VALUE_KIND`: row is deleted or added. Similar to RocksDB, each row of data can be deleted or added, which will be
- used for updating the primary key table.
-2. `_SEQUENCE_NUMBER`: this number is used for comparison during updates, determining which data came first and which
- data came later.
-3. `_KEY_` prefix to key columns, this is to avoid conflicts with columns of the table.
-
-## Changelog File
-
-Changelog file and Data file are exactly the same, it only takes effect on the primary key table. It is similar to the
-Binlog in a database, recording changes to the data in the table.
-
-## Global Index
-
-Global Index is in the index directory, currently, only two places will use global index:
-
-1. bucket = -1 + primary key table: in dynamic bucket mode, the index records the correspondence between the hash value
- of the primary-key and the bucket, each bucket has an index file.
-2. Deletion Vectors: index stores the deletion file, and each bucket has a deletion file.
-
-## Data File Index
-
-Define `file-index.bloom-filter.columns`, Paimon will create its corresponding index file for each file. If the index
-file is too small, it will be stored directly in the manifest, or in the directory of the data file. Each data file
-corresponds to an index file, which has a separate file definition and can contain different types of indexes with
-multiple columns.