You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@paimon.apache.org by lz...@apache.org on 2023/08/21 05:50:43 UTC
[incubator-paimon] branch master updated: [doc] Document Local Merging not work for CDC ingestion
This is an automated email from the ASF dual-hosted git repository.
lzljs3620320 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-paimon.git
The following commit(s) were added to refs/heads/master by this push:
new f477b2609 [doc] Document Local Merging not work for CDC ingestion
f477b2609 is described below
commit f477b2609ad8658a5a1b317facefc93d2efd6e6e
Author: Jingsong <ji...@gmail.com>
AuthorDate: Mon Aug 21 10:34:44 2023 +0800
[doc] Document Local Merging not work for CDC ingestion
---
docs/content/maintenance/write-performance.md | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)
diff --git a/docs/content/maintenance/write-performance.md b/docs/content/maintenance/write-performance.md
index f25eae4ef..21d18faad 100644
--- a/docs/content/maintenance/write-performance.md
+++ b/docs/content/maintenance/write-performance.md
@@ -166,7 +166,7 @@ Compaction will become less frequent when `num-sorted-run.compaction-trigger` be
## Local Merging
If your job suffers from primary key data skew
-(for example, you want to count the number of views for each pages in a website,
+(for example, you want to count the number of views for each page in a website,
and some particular pages are very popular among the users),
you can set `'local-merge-buffer-size'` so that input records will be buffered and merged
before they're shuffled by bucket and written into sink.
@@ -175,6 +175,8 @@ This is particularly useful when the same primary key is updated frequently betw
The buffer will be flushed when it is full. We recommend starting with `64 mb`
when you are faced with data skew but don't know where to start adjusting buffer size.
+(Currently, Local merging not works for CDC ingestion)
+
## File Format
If you want to achieve ultimate compaction performance, you can consider using row storage file format AVRO.