You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@celeborn.apache.org by zh...@apache.org on 2023/07/05 10:30:43 UTC
[incubator-celeborn] branch branch-0.3 updated: [CELEBORN-765][DOC] Disable partitionSplit in Flink engine related co…
This is an automated email from the ASF dual-hosted git repository.
zhongqiangchen pushed a commit to branch branch-0.3
in repository https://gitbox.apache.org/repos/asf/incubator-celeborn.git
The following commit(s) were added to refs/heads/branch-0.3 by this push:
new d464a3521 [CELEBORN-765][DOC] Disable partitionSplit in Flink engine related co…
d464a3521 is described below
commit d464a3521f8d6376791c7d56179668db1165a77a
Author: zhongqiang.czq <zh...@alibaba-inc.com>
AuthorDate: Wed Jul 5 18:04:10 2023 +0800
[CELEBORN-765][DOC] Disable partitionSplit in Flink engine related co…
…nfigurations
### What changes were proposed in this pull request?
In Doc Readme, setting partitionSplit to false should be added in Flink engine related configurations.
### Why are the changes needed?
Currently, Mappartition split is not supported, but shuffle partition split is enabled by default, so error will be thrown when flink task's shuffle data size exceeds 1G(by Default).
### Does this PR introduce _any_ user-facing change?
no
### How was this patch tested?
manually
Closes #1679 from zhongqiangczq/readme.
Authored-by: zhongqiang.czq <zh...@alibaba-inc.com>
Signed-off-by: zhongqiang.czq <zh...@alibaba-inc.com>
---
README.md | 2 ++
1 file changed, 2 insertions(+)
diff --git a/README.md b/README.md
index d6fbd88a4..76174d0e6 100644
--- a/README.md
+++ b/README.md
@@ -193,6 +193,8 @@ celeborn.worker.directMemoryRatioToResume 0.6
celeborn.worker.partition.initial.readBuffersMin 512
celeborn.worker.partition.initial.readBuffersMax 1024
celeborn.worker.readBuffer.allocationWait 10ms
+# Currently, shuffle partitionSplit is not supported, so you should disable split in celeborn worker side or set `celeborn.client.shuffle.partitionSplit.threshold` to a high value in flink client side.
+celeborn.worker.shuffle.partitionSplit.enabled false
```
4. Copy Celeborn and configurations to all nodes