[GitHub] [incubator-paimon] qidian99 commented on a diff in pull request #1158: [doc] Add document page for small files in Paimon

["qidian99 (via GitHub)" <gi...@apache.org>]

qidian99 commented on code in PR #1158:
URL: https://github.com/apache/incubator-paimon/pull/1158#discussion_r1194591245


##########
docs/content/maintenance/small-files.md:
##########
@@ -0,0 +1,316 @@
+---
+title: "A Deep-Dive into Paimon's Small Files"
+weight: 2
+type: docs
+aliases:
+- /maintenance/read-performance.html
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+  http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied.  See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+# A Deep-Dive into Paimon's Small Files
+
+Paimon table relies heavily on file management mechanisms such as 
+checkpoint and snapshot expiration, but users are often left wondering 
+about the origin of the numerous small files that result from these 
+operations. In response, we have created a dedicated page to explain 
+the sources of these files and offer tips on how to manage them effectively. 
+Using concrete examples, we will delve into the details of how different 
+operations such as commit and compact can create small files and provide 
+insights into their updates.
+
+
+## Prerequisite
+
+Before go deeper in to this page, make sure you have read through the 
+[Basic Concepts]({{< ref "concepts/basic-concepts" >}}), 
+[File Layouts]({{< ref "concepts/file-layouts" >}}) and 
+how to use Paimon in [Flink]({{< ref "engines/flink" >}}).
+
+{{< img src="/img/file-layout.png">}}
+
+
+## Create Paimon Catalog
+Start Flink SQL client via `./sql-client.sh` and execute the following 
+statements one by one to create a Paimon catalog.  
+```sql
+CREATE CATALOG paimon WITH (
+'type' = 'paimon',
+'warehouse' = 'file:///tmp/paimon'
+);
+
+USE CATALOG paimon;
+```
+
+This will only create a directory at given path `file:///tmp/paimon`.
+
+## Create Paimon Table
+
+Execute the following create table statement will create a Paimon table with 3 fields:
+
+```sql
+CREATE TABLE T (
+  id BIGINT,
+  a INT,
+  b STRING,
+  dt STRING COMMENT 'timestamp string in format yyyyMMdd',
+  PRIMARY KEY(id, dt) NOT ENFORCED
+) PARTITIONED BY (dt);
+```
+
+This will create Paimon table `T` under the path `/tmp/paimon/default.db/T`, 
+with its schema stored in `/tmp/paimon/default.db/T/schema/schema-0` 
+
+
+## Insert Records Into Paimon Table
+
+Run the following insert statement in Flink SQL:
+
+```sql
+INSERT INTO T VALUES (1, 10001, 'varchar00001', '20230501');
+```
+
+After the Flink job finishes, the records are written into Paimon table, which 
+is done by a successful `commit`. The records are visible to user
+as can be verified by `SELECT * FROM T` which return a single row. 
+The commit creates a snapshot under path `/tmp/paimon/default.db/T/snapshot/snapshot-1`. 
+The resulting file layout as of snapshot-1 is as follows:
+
+{{< img src="/img/small-file-0.png">}}
+
+The content of snapshot-1 contains metadata of the snapshot, such as manifest list and schema id:
+```json
+{
+  "version" : 3,
+  "id" : 1,
+  "schemaId" : 0,
+  "baseManifestList" : "manifest-09184ccc-c07f-4090-958c-cfe3ce3889e5-0",

Review Comment:
   Based on my local test result (Paimon 0.5-SNAPSHOT and Flink 1.17.0), the manifest list file name is not `manifest-list-****`. However, it might be a better option to use `manifest-list-****` as the name in this page so that users are less likely to be confused.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org