You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@paimon.apache.org by lz...@apache.org on 2023/12/01 07:03:00 UTC
(incubator-paimon) 26/46: [doc] Document Asynchronous for lookup changelog-producer
This is an automated email from the ASF dual-hosted git repository.
lzljs3620320 pushed a commit to branch release-0.6
in repository https://gitbox.apache.org/repos/asf/incubator-paimon.git
commit e320085722677f819fcb120cc01aab363ffb84fa
Author: Jingsong <ji...@gmail.com>
AuthorDate: Wed Nov 29 11:33:39 2023 +0800
[doc] Document Asynchronous for lookup changelog-producer
---
docs/content/maintenance/write-performance.md | 7 +++++++
1 file changed, 7 insertions(+)
diff --git a/docs/content/maintenance/write-performance.md b/docs/content/maintenance/write-performance.md
index 59bdc9267..6eb503539 100644
--- a/docs/content/maintenance/write-performance.md
+++ b/docs/content/maintenance/write-performance.md
@@ -39,6 +39,10 @@ Option `'changelog-producer' = 'lookup' or 'full-compaction'`, and option `'full
large impact on write performance, if it is a snapshot / full synchronization phase you can unset these options and
then enable them again in the incremental phase.
+If you find that the input of the job shows a jagged pattern in the case of backpressure, it may be imbalanced work
+nodes. You can consider turning on [Asynchronous Compaction]({{< ref "#asynchronous-compaction" >}}) to observe if the
+throughput is increased.
+
## Parallelism
It is recommended that the parallelism of sink should be less than or equal to the number of buckets, preferably equal. You can control the parallelism of the sink with the `sink.parallelism` table property.
@@ -80,6 +84,9 @@ sort-spill-threshold = 10
This configuration will generate more files during peak write periods and gradually merge into optimal read
performance during low write periods.
+In the case of `'changelog-producer' = 'lookup'`, by default, the lookup will be completed at checkpointing, which
+will block the checkpoint. If you want an asynchronous lookup, you can enable `'changelog-producer.lookup-wait' = 'true'`.
+
### Number of Sorted Runs to Pause Writing
When the number of sorted runs is small, Paimon writers will perform compaction asynchronously in separated threads, so