You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@celeborn.apache.org by ch...@apache.org on 2022/11/27 14:59:22 UTC
[incubator-celeborn-website] branch main updated: Remove unused configuration docs
This is an automated email from the ASF dual-hosted git repository.
chengpan pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/incubator-celeborn-website.git
The following commit(s) were added to refs/heads/main by this push:
new 92d26aa Remove unused configuration docs
92d26aa is described below
commit 92d26aad3cb57e34289300aea9e39084b3cc66ac
Author: Cheng Pan <ch...@apache.org>
AuthorDate: Sun Nov 27 22:59:14 2022 +0800
Remove unused configuration docs
---
docs/configuration/.gitkeep | 0
docs/configuration/client.md | 65 --------------
docs/configuration/columnar-shuffle.md | 28 ------
docs/configuration/index.md | 153 ---------------------------------
docs/configuration/master.md | 49 -----------
docs/configuration/metrics.md | 31 -------
docs/configuration/network.md | 44 ----------
docs/configuration/quota.md | 26 ------
docs/configuration/worker.md | 83 ------------------
9 files changed, 479 deletions(-)
diff --git a/docs/configuration/.gitkeep b/docs/configuration/.gitkeep
new file mode 100644
index 0000000..e69de29
diff --git a/docs/configuration/client.md b/docs/configuration/client.md
deleted file mode 100644
index 9a5127f..0000000
--- a/docs/configuration/client.md
+++ /dev/null
@@ -1,65 +0,0 @@
----
-license: |
- Licensed to the Apache Software Foundation (ASF) under one or more
- contributor license agreements. See the NOTICE file distributed with
- this work for additional information regarding copyright ownership.
- The ASF licenses this file to You under the Apache License, Version 2.0
- (the "License"); you may not use this file except in compliance with
- the License. You may obtain a copy of the License at
-
- https://www.apache.org/licenses/LICENSE-2.0
-
- Unless required by applicable law or agreed to in writing, software
- distributed under the License is distributed on an "AS IS" BASIS,
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- See the License for the specific language governing permissions and
- limitations under the License.
----
-
-<!--begin-include-->
-| Key | Default | Description | Since |
-| --- | ------- | ----------- | ----- |
-| celeborn.application.heartbeatInterval | 10s | Interval for client to send heartbeat message to master. | 0.2.0 |
-| celeborn.client.maxRetries | 15 | Max retry times for client to connect master endpoint | 0.2.0 |
-| celeborn.client.rpc.askTimeout | <value of celeborn.network.timeout> | Timeout for client RPC ask operations. | 0.2.0 |
-| celeborn.fetch.maxReqsInFlight | 3 | Amount of in-flight chunk fetch request. | 0.2.0 |
-| celeborn.fetch.timeout | 120s | Timeout for a task to fetch chunk. | 0.2.0 |
-| celeborn.master.endpoints | <localhost>:9097 | Endpoints of master nodes for celeborn client to connect, allowed pattern is: `<host1>:<port1>[,<host2>:<port2>]*`, e.g. `clb1:9097,clb2:9098,clb3:9099`. If the port is omitted, 9097 will be used. | 0.2.0 |
-| celeborn.push.buffer.initial.size | 8k | | 0.2.0 |
-| celeborn.push.buffer.max.size | 64k | Max size of reducer partition buffer memory for shuffle hash writer. The pushed data will be buffered in memory before sending to Celeborn worker. For performance consideration keep this buffer size higher than 32K. Example: If reducer amount is 2000, buffer size is 64K, then each task will consume up to `64KiB * 2000 = 125MiB` heap memory. | 0.2.0 |
-| celeborn.push.limit.inFlight.sleepInterval | 50ms | Sleep interval when check netty in-flight requests to be done. | 0.2.0 |
-| celeborn.push.limit.inFlight.timeout | 240s | Timeout for netty in-flight requests to be done. | 0.2.0 |
-| celeborn.push.maxReqsInFlight | 32 | Amount of Netty in-flight requests. The maximum memory is `celeborn.push.maxReqsInFlight` * `celeborn.push.buffer.max.size` * compression ratio(1 in worst case), default: 64Kib * 32 = 2Mib | 0.2.0 |
-| celeborn.push.queue.capacity | 512 | Push buffer queue size for a task. The maximum memory is `celeborn.push.buffer.max.size` * `celeborn.push.queue.capacity`, default: 64KiB * 512 = 32MiB | 0.2.0 |
-| celeborn.push.replicate.enabled | true | When true, Celeborn worker will replicate shuffle data to another Celeborn worker asynchronously to ensure the pushed shuffle data won't be lost after the node failure. | 0.2.0 |
-| celeborn.push.retry.threads | 8 | Thread number to process shuffle re-send push data requests. | 0.2.0 |
-| celeborn.push.sortMemory.threshold | 64m | When SortBasedPusher use memory over the threshold, will trigger push data. | 0.2.0 |
-| celeborn.push.splitPartition.threads | 8 | Thread number to process shuffle split request in shuffle client. | 0.2.0 |
-| celeborn.push.stageEnd.timeout | 240s | Timeout for StageEnd. | 0.2.0 |
-| celeborn.rpc.cache.concurrencyLevel | 32 | The number of write locks to update rpc cache. | 0.2.0 |
-| celeborn.rpc.cache.expireTime | 15s | The time before a cache item is removed. | 0.2.0 |
-| celeborn.rpc.cache.size | 256 | The max cache items count for rpc cache. | 0.2.0 |
-| celeborn.rpc.maxParallelism | 1024 | Max parallelism of client on sending RPC requests. | 0.2.0 |
-| celeborn.shuffle.batchHandleChangePartition.enabled | false | When true, LifecycleManager will handle change partition request in batch. Otherwise, LifecycleManager will process the requests one by one | 0.2.0 |
-| celeborn.shuffle.batchHandleChangePartition.interval | 100ms | Interval for LifecycleManager to schedule handling change partition requests in batch. | 0.2.0 |
-| celeborn.shuffle.batchHandleChangePartition.threads | 8 | Threads number for LifecycleManager to handle change partition request in batch. | 0.2.0 |
-| celeborn.shuffle.chuck.size | 8m | Max chunk size of reducer's merged shuffle data. For example, if a reducer's shuffle data is 128M and the data will need 16 fetch chunk requests to fetch. | 0.2.0 |
-| celeborn.shuffle.compression.codec | LZ4 | The codec used to compress shuffle data. By default, Celeborn provides two codecs: `lz4` and `zstd`. | 0.2.0 |
-| celeborn.shuffle.compression.zstd.level | 1 | Compression level for Zstd compression codec, its value should be an integer between -5 and 22. Increasing the compression level will result in better compression at the expense of more CPU and memory. | 0.2.0 |
-| celeborn.shuffle.expired.checkInterval | 60s | Interval for client to check expired shuffles. | 0.2.0 |
-| celeborn.shuffle.forceFallback.enabled | false | Whether force fallback shuffle to Spark's default. | 0.2.0 |
-| celeborn.shuffle.forceFallback.numPartitionsThreshold | 500000 | Celeborn will only accept shuffle of partition number lower than this configuration value. | 0.2.0 |
-| celeborn.shuffle.manager.port | 0 | Port used by the LifecycleManager on the Driver. | 0.2.0 |
-| celeborn.shuffle.partition.type | REDUCE | Type of shuffle's partition. | 0.2.0 |
-| celeborn.shuffle.partitionSplit.mode | SOFT | soft: the shuffle file size might be larger than split threshold. hard: the shuffle file size will be limited to split threshold. | 0.2.0 |
-| celeborn.shuffle.partitionSplit.threshold | 1G | Shuffle file size threshold, if file size exceeds this, trigger split. | 0.2.0 |
-| celeborn.shuffle.rangeReadFilter.enabled | false | If a spark application have skewed partition, this value can set to true to improve performance. | 0.2.0 |
-| celeborn.shuffle.register.maxRetries | 3 | Max retry times for client to register shuffle. | 0.2.0 |
-| celeborn.shuffle.register.retryWait | 3s | Wait time before next retry if register shuffle failed. | 0.2.0 |
-| celeborn.shuffle.writer | HASH | Celeborn supports the following kind of shuffle writers. 1. hash: hash-based shuffle writer works fine when shuffle partition count is normal; 2. sort: sort-based shuffle writer works fine when memory pressure is high or shuffle partition count it huge. | 0.2.0 |
-| celeborn.slots.reserve.maxRetries | 3 | Max retry times for client to reserve slots. | 0.2.0 |
-| celeborn.slots.reserve.retryWait | 3s | Wait time before next retry if reserve slots failed. | 0.2.0 |
-| celeborn.storage.hdfs.dir | <undefined> | HDFS dir configuration for Celeborn to access HDFS. | 0.2.0 |
-| celeborn.worker.excluded.checkInterval | 30s | Interval for client to refresh excluded worker list. | 0.2.0 |
-| celeborn.worker.excluded.expireTimeout | 600s | Timeout time for LifecycleManager to clear reserved excluded worker. | 0.2.0 |
-<!--end-include-->
diff --git a/docs/configuration/columnar-shuffle.md b/docs/configuration/columnar-shuffle.md
deleted file mode 100644
index f27d0f5..0000000
--- a/docs/configuration/columnar-shuffle.md
+++ /dev/null
@@ -1,28 +0,0 @@
----
-license: |
- Licensed to the Apache Software Foundation (ASF) under one or more
- contributor license agreements. See the NOTICE file distributed with
- this work for additional information regarding copyright ownership.
- The ASF licenses this file to You under the Apache License, Version 2.0
- (the "License"); you may not use this file except in compliance with
- the License. You may obtain a copy of the License at
-
- https://www.apache.org/licenses/LICENSE-2.0
-
- Unless required by applicable law or agreed to in writing, software
- distributed under the License is distributed on an "AS IS" BASIS,
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- See the License for the specific language governing permissions and
- limitations under the License.
----
-
-<!--begin-include-->
-| Key | Default | Description | Since |
-| --- | ------- | ----------- | ----- |
-| celeborn.columnar.offHeap.enabled | false | Whether to use off heap columnar vector. | 0.2.0 |
-| celeborn.columnar.shuffle.batch.size | 10000 | Vector batch size for columnar shuffle. | 0.2.0 |
-| celeborn.columnar.shuffle.codegen.enabled | false | Whether to use codegen for columnar-based shuffle. | 0.2.0 |
-| celeborn.columnar.shuffle.enabled | false | Whether to enable columnar-based shuffle. | 0.2.0 |
-| celeborn.columnar.shuffle.encoding.dictionary.enabled | false | Whether to use dictionary encoding for columnar-based shuffle data. | 0.2.0 |
-| celeborn.columnar.shuffle.encoding.dictionary.maxFactor | 0.3 | Max factor for dictionary size. The max dictionary size is `min(32.0 KB, celeborn.columnar.shuffle.batch.size * celeborn.columnar.shuffle.encoding.dictionary.maxFactor)`. | 0.2.0 |
-<!--end-include-->
diff --git a/docs/configuration/index.md b/docs/configuration/index.md
deleted file mode 100644
index 9a5042d..0000000
--- a/docs/configuration/index.md
+++ /dev/null
@@ -1,153 +0,0 @@
----
-hide:
- - navigation
-
-license: |
- Licensed under the Apache License, Version 2.0 (the "License");
- you may not use this file except in compliance with the License.
- You may obtain a copy of the License at
-
- https://www.apache.org/licenses/LICENSE-2.0
-
- Unless required by applicable law or agreed to in writing, software
- distributed under the License is distributed on an "AS IS" BASIS,
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- See the License for the specific language governing permissions and
- limitations under the License.
----
-
-Configuration Guide
-===
-This documentation contains Celeborn configuration details and a tuning guide.
-
-## Important Configurations
-
-### Environment Variables
-
-- `CELEBORN_WORKER_MEMORY=4g`
-- `CELEBORN_WORKER_OFFHEAP_MEMORY=24g`
-
-Celeborn workers tend to improve performance by using off-heap buffers.
-Off-heap memory requirement can be estimated as below:
-
-```
-numDirs = `celeborn.worker.storage.dirs` # the amount of directory will be used by Celeborn storage
-bufferSize = `celeborn.worker.flusher.buffer.size` # the amount of memory will be used by a single flush buffer
-off-heap-memory = bufferSize * estimatedTasks * 2 + network memory
-```
-
-For example, if an Celeborn worker has 10 storage directories or disks and the buffer size is set to 256 KiB.
-The necessary off-heap memory is 10 GiB.
-
-Network memory will be consumed when netty reads from a TPC channel, there will need some extra
-memory. Empirically, Celeborn worker off-heap memory should be set to `(numDirs * bufferSize * 1.2)`.
-
-## All Configurations
-
-### Client
-
-{!
-include-markdown "./client.md"
-start="<!--begin-include-->"
-end="<!--end-include-->"
-!}
-
-### Columnar Shuffle
-
-{!
-include-markdown "./columnar-shuffle.md"
-start="<!--begin-include-->"
-end="<!--end-include-->"
-!}
-
-### Master
-
-{!
-include-markdown "./master.md"
-start="<!--begin-include-->"
-end="<!--end-include-->"
-!}
-
-### Worker
-
-{!
-include-markdown "./worker.md"
-start="<!--begin-include-->"
-end="<!--end-include-->"
-!}
-
-### Quota
-
-{!
-include-markdown "./quota.md"
-start="<!--begin-include-->"
-end="<!--end-include-->"
-!}
-
-### Network
-
-{!
-include-markdown "./network.md"
-start="<!--begin-include-->"
-end="<!--end-include-->"
-!}
-
-### Metrics
-
-{!
-include-markdown "./metrics.md"
-start="<!--begin-include-->"
-end="<!--end-include-->"
-!}
-
-#### metrics.properties
-
-```properties
-*.sink.csv.class=org.apache.celeborn.common.metrics.sink.CsvSink
-*.sink.prometheusServlet.class=org.apache.celeborn.common.metrics.sink.PrometheusServlet
-```
-
-### Environment Variables
-
-Recommend configuring in `conf/celeborn-env.sh`.
-
-| Key | Default | Description |
-|----------------------------------|-------------------------------------------------|-------------|
-| `CELEBORN_HOME` | ``$(cd "`dirname "$0"`"/..; pwd)`` | |
-| `CELEBORN_CONF_DIR` | `${CELEBORN_CONF_DIR:-"${CELEBORN_HOME}/conf"}` | |
-| `CELEBORN_MASTER_MEMORY` | 1 GB | |
-| `CELEBORN_WORKER_MEMORY` | 1 GB | |
-| `CELEBORN_WORKER_OFFHEAP_MEMORY` | 1 GB | |
-| `CELEBORN_MASTER_JAVA_OPTS` | | |
-| `CELEBORN_WORKER_JAVA_OPTS` | | |
-| `CELEBORN_PID_DIR` | `${CELEBORN_HOME}/pids` | |
-| `CELEBORN_LOG_DIR` | `${CELEBORN_HOME}/logs` | |
-| `CELEBORN_SSH_OPTS` | `-o StrictHostKeyChecking=no` | |
-| `CELEBORN_SLEEP` | | |
-
-## Tuning
-
-Assume we have a cluster described as below:
-5 Celeborn Workers with 20 GB off-heap memory and 10 disks.
-As we need to reserve 20% off-heap memory for netty,
-so we could assume 16 GB off-heap memory can be used for flush buffers.
-
-If `spark.celeborn.push.buffer.size` is 64 KB, we can have in-flight requests up to 1310720.
-If you have 8192 mapper tasks, you could set `spark.celeborn.push.maxReqsInFlight=160` to gain performance improvements.
-
-If `celeborn.worker.flush.buffer.size` is 256 KB, we can have total slots up to 327680 slots.
-
-## Worker Recover Status After Restart
-
-`ShuffleClient` records the shuffle partition location's host, service port, and filename,
-to support workers recovering reading existing shuffle data after worker restart,
-during worker shutdown, workers should store the meta about reading shuffle partition files in LevelDB,
-and restore the meta after restarting workers, also workers should keep a stable service port to support
-`ShuffleClient` retry reading data. Users should set `celeborn.worker.graceful.shutdown.enabled` to `true` and
-set below service port with stable port to support worker recover status.
-```
-rss.worker.rpc.port
-rss.fetchserver.port
-rss.pushserver.port
-rss.replicateserver.port
-```
diff --git a/docs/configuration/master.md b/docs/configuration/master.md
deleted file mode 100644
index 59adaa0..0000000
--- a/docs/configuration/master.md
+++ /dev/null
@@ -1,49 +0,0 @@
----
-license: |
- Licensed to the Apache Software Foundation (ASF) under one or more
- contributor license agreements. See the NOTICE file distributed with
- this work for additional information regarding copyright ownership.
- The ASF licenses this file to You under the Apache License, Version 2.0
- (the "License"); you may not use this file except in compliance with
- the License. You may obtain a copy of the License at
-
- https://www.apache.org/licenses/LICENSE-2.0
-
- Unless required by applicable law or agreed to in writing, software
- distributed under the License is distributed on an "AS IS" BASIS,
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- See the License for the specific language governing permissions and
- limitations under the License.
----
-
-<!--begin-include-->
-| Key | Default | Description | Since |
-| --- | ------- | ----------- | ----- |
-| celeborn.application.heartbeat.timeout | 120s | Application heartbeat timeout. | 0.2.0 |
-| celeborn.ha.enabled | false | When true, master nodes run as Raft cluster mode. | 0.2.0 |
-| celeborn.ha.master.node.<id>.host | <required> | Host to bind of master node <id> in HA mode. | 0.2.0 |
-| celeborn.ha.master.node.<id>.port | 9097 | Port to bind of master node <id> in HA mode. | 0.2.0 |
-| celeborn.ha.master.node.<id>.ratis.port | 9872 | Ratis port to bind of master node <id> in HA mode. | 0.2.0 |
-| celeborn.ha.master.ratis.raft.rpc.type | netty | RPC type for Ratis, available options: netty, grpc. | 0.2.0 |
-| celeborn.ha.master.ratis.raft.server.storage.dir | /tmp/ratis | | 0.2.0 |
-| celeborn.master.host | <localhost> | Hostname for master to bind. | 0.2.0 |
-| celeborn.master.metrics.prometheus.host | 0.0.0.0 | Master's Prometheus host. | 0.2.0 |
-| celeborn.master.metrics.prometheus.port | 9098 | Master's Prometheus port. | 0.2.0 |
-| celeborn.master.port | 9097 | Port for master to bind. | 0.2.0 |
-| celeborn.metrics.app.topDiskUsage.count | 50 | Size for top items about top disk usage applications list. | 0.2.0 |
-| celeborn.metrics.app.topDiskUsage.interval | 10min | Time length for a window about top disk usage application list. | 0.2.0 |
-| celeborn.metrics.app.topDiskUsage.windowSize | 24 | Window size about top disk usage application list. | 0.2.0 |
-| celeborn.metrics.capacity | 4096 | The maximum number of metrics which a source can use to generate output strings. | 0.2.0 |
-| celeborn.metrics.collectPerfCritical.enabled | false | It controls whether to collect metrics which may affect performance. When enable, Celeborn collects them. | 0.2.0 |
-| celeborn.metrics.enabled | true | When true, enable metrics system. | 0.2.0 |
-| celeborn.metrics.sample.rate | 1.0 | It controls if Celeborn collect timer metrics for some operations. Its value should be in [0.0, 1.0]. | 0.2.0 |
-| celeborn.metrics.timer.slidingWindow.size | 4096 | The sliding window size of timer metric. | 0.2.0 |
-| celeborn.shuffle.estimatedPartitionSize.update.initialDelay | 5min | Initial delay time before start updating partition size for estimation. | 0.2.0 |
-| celeborn.shuffle.estimatedPartitionSize.update.interval | 10min | Interval of updating partition size for estimation. | 0.2.0 |
-| celeborn.shuffle.initialEstimatedPartitionSize | 64mb | Initial partition size for estimation, it will change according to runtime stats. | 0.2.0 |
-| celeborn.slots.assign.extraSlots | 2 | Extra slots number when master assign slots. | 0.2.0 |
-| celeborn.slots.assign.loadAware.diskGroupGradient | 0.1 | This value means how many more workload will be placed into a faster disk group than a slower group. | 0.2.0 |
-| celeborn.slots.assign.loadAware.numDiskGroups | 5 | This configuration is a guidance for load-aware slot allocation algorithm. This value is control how many disk groups will be created. | 0.2.0 |
-| celeborn.slots.assign.policy | ROUNDROBIN | Policy for master to assign slots, Celeborn supports two types of policy: roundrobin and loadaware. | 0.2.0 |
-| celeborn.worker.heartbeat.timeout | 120s | Worker heartbeat timeout. | 0.2.0 |
-<!--end-include-->
diff --git a/docs/configuration/metrics.md b/docs/configuration/metrics.md
deleted file mode 100644
index 2644ea0..0000000
--- a/docs/configuration/metrics.md
+++ /dev/null
@@ -1,31 +0,0 @@
----
-license: |
- Licensed to the Apache Software Foundation (ASF) under one or more
- contributor license agreements. See the NOTICE file distributed with
- this work for additional information regarding copyright ownership.
- The ASF licenses this file to You under the Apache License, Version 2.0
- (the "License"); you may not use this file except in compliance with
- the License. You may obtain a copy of the License at
-
- https://www.apache.org/licenses/LICENSE-2.0
-
- Unless required by applicable law or agreed to in writing, software
- distributed under the License is distributed on an "AS IS" BASIS,
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- See the License for the specific language governing permissions and
- limitations under the License.
----
-
-<!--begin-include-->
-| Key | Default | Description | Since |
-| --- | ------- | ----------- | ----- |
-| celeborn.master.metrics.prometheus.host | 0.0.0.0 | Master's Prometheus host. | 0.2.0 |
-| celeborn.master.metrics.prometheus.port | 9098 | Master's Prometheus port. | 0.2.0 |
-| celeborn.metrics.capacity | 4096 | The maximum number of metrics which a source can use to generate output strings. | 0.2.0 |
-| celeborn.metrics.collectPerfCritical.enabled | false | It controls whether to collect metrics which may affect performance. When enable, Celeborn collects them. | 0.2.0 |
-| celeborn.metrics.enabled | true | When true, enable metrics system. | 0.2.0 |
-| celeborn.metrics.sample.rate | 1.0 | It controls if Celeborn collect timer metrics for some operations. Its value should be in [0.0, 1.0]. | 0.2.0 |
-| celeborn.metrics.timer.slidingWindow.size | 4096 | The sliding window size of timer metric. | 0.2.0 |
-| celeborn.worker.metrics.prometheus.host | 0.0.0.0 | Worker's Prometheus host. | 0.2.0 |
-| celeborn.worker.metrics.prometheus.port | 9096 | Worker's Prometheus port. | 0.2.0 |
-<!--end-include-->
diff --git a/docs/configuration/network.md b/docs/configuration/network.md
deleted file mode 100644
index 8724de5..0000000
--- a/docs/configuration/network.md
+++ /dev/null
@@ -1,44 +0,0 @@
----
-license: |
- Licensed to the Apache Software Foundation (ASF) under one or more
- contributor license agreements. See the NOTICE file distributed with
- this work for additional information regarding copyright ownership.
- The ASF licenses this file to You under the Apache License, Version 2.0
- (the "License"); you may not use this file except in compliance with
- the License. You may obtain a copy of the License at
-
- https://www.apache.org/licenses/LICENSE-2.0
-
- Unless required by applicable law or agreed to in writing, software
- distributed under the License is distributed on an "AS IS" BASIS,
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- See the License for the specific language governing permissions and
- limitations under the License.
----
-
-<!--begin-include-->
-| Key | Default | Description | Since |
-| --- | ------- | ----------- | ----- |
-| celeborn.<module>.decoder.mode | default | Netty TransportFrameDecoder implementation, available options: default, supplier. | |
-| celeborn.<module>.io.backLog | 0 | Requested maximum length of the queue of incoming connections. Default 0 for no backlog. | |
-| celeborn.<module>.io.clientThreads | 0 | Number of threads used in the client thread pool. Default to 0, which is 2x#cores. | |
-| celeborn.<module>.io.connectTimeout | <value of celeborn.network.connect.timeout> | Socket connect timeout. | |
-| celeborn.<module>.io.connectionTimeout | <value of celeborn.network.timeout> | Connection active timeout. | |
-| celeborn.<module>.io.enableVerboseMetrics | false | Whether to track Netty memory detailed metrics. If true, the detailed metrics of Netty PoolByteBufAllocator will be gotten, otherwise only general memory usage will be tracked. | |
-| celeborn.<module>.io.lazyFD | true | Whether to initialize FileDescriptor lazily or not. If true, file descriptors are created only when data is going to be transferred. This can reduce the number of open files. | |
-| celeborn.<module>.io.maxRetries | 3 | Max number of times we will try IO exceptions (such as connection timeouts) per request. If set to 0, we will not do any retries. | |
-| celeborn.<module>.io.mode | NIO | Netty EventLoopGroup backend, available options: NIO, EPOLL. | |
-| celeborn.<module>.io.numConnectionsPerPeer | 2 | Number of concurrent connections between two nodes. | |
-| celeborn.<module>.io.preferDirectBufs | true | If true, we will prefer allocating off-heap byte buffers within Netty. | |
-| celeborn.<module>.io.receiveBuffer | 0b | Receive buffer size (SO_RCVBUF). Note: the optimal size for receive buffer and send buffer should be latency * network_bandwidth. Assuming latency = 1ms, network_bandwidth = 10Gbps buffer size should be ~ 1.25MB. | 0.2.0 |
-| celeborn.<module>.io.retryWait | 5s | Time that we will wait in order to perform a retry after an IOException. Only relevant if maxIORetries > 0. | 0.2.0 |
-| celeborn.<module>.io.sendBuffer | 0b | Send buffer size (SO_SNDBUF). | 0.2.0 |
-| celeborn.<module>.io.serverThreads | 0 | Number of threads used in the server thread pool. Default to 0, which is 2x#cores. | |
-| celeborn.network.connect.timeout | 10s | Default socket connect timeout. | 0.2.0 |
-| celeborn.network.timeout | 240s | Default timeout for network operations. | 0.2.0 |
-| celeborn.port.maxRetries | 1 | When port is occupied, we will retry for max retry times. | 0.2.0 |
-| celeborn.rpc.askTimeout | <value of celeborn.network.timeout> | Timeout for RPC ask operations. | 0.2.0 |
-| celeborn.rpc.connect.threads | 64 | | 0.2.0 |
-| celeborn.rpc.lookupTimeout | 30s | Timeout for RPC lookup operations. | 0.2.0 |
-| celeborn.shuffle.maxChunksBeingTransferred | 9223372036854775807 | The max number of chunks allowed to be transferred at the same time on shuffle service. Note that new incoming connections will be closed when the max number is hit. The client will retry according to the shuffle retry configs (see `celeborn.shuffle.io.maxRetries` and `celeborn.shuffle.io.retryWait`), if those limits are reached the task will fail with fetch failure. | 0.2.0 |
-<!--end-include-->
diff --git a/docs/configuration/quota.md b/docs/configuration/quota.md
deleted file mode 100644
index 7d124ce..0000000
--- a/docs/configuration/quota.md
+++ /dev/null
@@ -1,26 +0,0 @@
----
-license: |
- Licensed to the Apache Software Foundation (ASF) under one or more
- contributor license agreements. See the NOTICE file distributed with
- this work for additional information regarding copyright ownership.
- The ASF licenses this file to You under the Apache License, Version 2.0
- (the "License"); you may not use this file except in compliance with
- the License. You may obtain a copy of the License at
-
- https://www.apache.org/licenses/LICENSE-2.0
-
- Unless required by applicable law or agreed to in writing, software
- distributed under the License is distributed on an "AS IS" BASIS,
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- See the License for the specific language governing permissions and
- limitations under the License.
----
-
-<!--begin-include-->
-| Key | Default | Description | Since |
-| --- | ------- | ----------- | ----- |
-| celeborn.quota.configuration.path | <undefined> | Quota configuration file path. | 0.2.0 |
-| celeborn.quota.enabled | true | When true, before registering shuffle, LifecycleManager should check if current user have enough quota space, if cluster don't have enough quota space for current user, fallback to Spark's default shuffle | 0.2.0 |
-| celeborn.quota.identity.provider | org.apache.celeborn.common.identity.DefaultIdentityProvider | IdentityProvider class name. Default class is `org.apache.celeborn.common.identity.DefaultIdentityProvider`, return `org.apache.celeborn.common.identity.UserIdentifier` with default tenant id and username from `org.apache.hadoop.security.UserGroupInformation`. | 0.2.0 |
-| celeborn.quota.manager | org.apache.celeborn.common.quota.DefaultQuotaManager | QuotaManger class name. Default class is `org.apache.celeborn.common.quota.DefaultQuotaManager`. | 0.2.0 |
-<!--end-include-->
diff --git a/docs/configuration/worker.md b/docs/configuration/worker.md
deleted file mode 100644
index 03881f9..0000000
--- a/docs/configuration/worker.md
+++ /dev/null
@@ -1,83 +0,0 @@
----
-license: |
- Licensed to the Apache Software Foundation (ASF) under one or more
- contributor license agreements. See the NOTICE file distributed with
- this work for additional information regarding copyright ownership.
- The ASF licenses this file to You under the Apache License, Version 2.0
- (the "License"); you may not use this file except in compliance with
- the License. You may obtain a copy of the License at
-
- https://www.apache.org/licenses/LICENSE-2.0
-
- Unless required by applicable law or agreed to in writing, software
- distributed under the License is distributed on an "AS IS" BASIS,
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- See the License for the specific language governing permissions and
- limitations under the License.
----
-
-<!--begin-include-->
-| Key | Default | Description | Since |
-| --- | ------- | ----------- | ----- |
-| celeborn.client.maxRetries | 15 | Max retry times for client to connect master endpoint | 0.2.0 |
-| celeborn.master.endpoints | <localhost>:9097 | Endpoints of master nodes for celeborn client to connect, allowed pattern is: `<host1>:<port1>[,<host2>:<port2>]*`, e.g. `clb1:9097,clb2:9098,clb3:9099`. If the port is omitted, 9097 will be used. | 0.2.0 |
-| celeborn.metrics.capacity | 4096 | The maximum number of metrics which a source can use to generate output strings. | 0.2.0 |
-| celeborn.metrics.collectPerfCritical.enabled | false | It controls whether to collect metrics which may affect performance. When enable, Celeborn collects them. | 0.2.0 |
-| celeborn.metrics.enabled | true | When true, enable metrics system. | 0.2.0 |
-| celeborn.metrics.sample.rate | 1.0 | It controls if Celeborn collect timer metrics for some operations. Its value should be in [0.0, 1.0]. | 0.2.0 |
-| celeborn.metrics.timer.slidingWindow.size | 4096 | The sliding window size of timer metric. | 0.2.0 |
-| celeborn.shuffle.chuck.size | 8m | Max chunk size of reducer's merged shuffle data. For example, if a reducer's shuffle data is 128M and the data will need 16 fetch chunk requests to fetch. | 0.2.0 |
-| celeborn.shuffle.minPartitionSizeToEstimate | 8mb | Ignore partition size smaller than this configuration of partition size for estimation. | 0.2.0 |
-| celeborn.storage.hdfs.dir | <undefined> | HDFS dir configuration for Celeborn to access HDFS. | 0.2.0 |
-| celeborn.worker.closeIdleConnections | false | Whether worker will close idle connections. | 0.2.0 |
-| celeborn.worker.commit.threads | 32 | Thread number of worker to commit shuffle data files asynchronously. | 0.2.0 |
-| celeborn.worker.directMemoryRatioToPauseReceive | 0.85 | If direct memory usage reaches this limit, the worker will stop to receive data from Celeborn shuffle clients. | 0.2.0 |
-| celeborn.worker.directMemoryRatioToPauseReplicate | 0.95 | If direct memory usage reaches this limit, the worker will stop to receive replication data from other workers. | 0.2.0 |
-| celeborn.worker.directMemoryRatioToResume | 0.5 | If direct memory usage is less than this limit, worker will resume. | 0.2.0 |
-| celeborn.worker.disk.check.timeout | 30s | Timeout time for worker check device status. | 0.2.0 |
-| celeborn.worker.disk.checkFileClean.maxRetries | 3 | The number of retries for a worker to check if the working directory is cleaned up before registering with the master. | 0.2.0 |
-| celeborn.worker.disk.checkFileClean.timeout | 1000ms | The wait time per retry for a worker to check if the working directory is cleaned up before registering with the master. | 0.2.0 |
-| celeborn.worker.disk.reserve.size | 5G | Celeborn worker reserved space for each disk. | 0.2.0 |
-| celeborn.worker.fetch.io.threads | <undefined> | Netty IO thread number of worker to handle client fetch data. The default threads number is `size(celeborn.worker.storage.dirs)*2`. | 0.2.0 |
-| celeborn.worker.fetch.port | 0 | Server port for Worker to receive fetch data request from ShuffleClient. | 0.2.0 |
-| celeborn.worker.flusher.avgFlushTime.slidingWindow.size | 20 | The size of sliding windows used to calculate statistics about flushed time and count. | 0.2.0 |
-| celeborn.worker.flusher.buffer.size | 256k | Size of buffer used by a single flusher. | 0.2.0 |
-| celeborn.worker.flusher.hdd.threads | 1 | Flusher's thread count per disk used for write data to HDD disks. | 0.2.0 |
-| celeborn.worker.flusher.hdfs.threads | 4 | Flusher's thread count used for write data to HDFS. | 0.2.0 |
-| celeborn.worker.flusher.shutdownTimeout | 3s | Timeout for a flusher to shutdown. | 0.2.0 |
-| celeborn.worker.flusher.ssd.threads | 8 | Flusher's thread count per disk used for write data to SSD disks. | 0.2.0 |
-| celeborn.worker.graceful.shutdown.checkSlotsFinished.interval | 1s | The wait interval of checking whether all released slots to be committed or destroyed during worker graceful shutdown | 0.2.0 |
-| celeborn.worker.graceful.shutdown.checkSlotsFinished.timeout | 480s | The wait time of waiting for the released slots to be committed or destroyed during worker graceful shutdown. | 0.2.0 |
-| celeborn.worker.graceful.shutdown.enabled | false | When true, during worker shutdown, the worker will wait for all released slots to be committed or destroyed. | 0.2.0 |
-| celeborn.worker.graceful.shutdown.partitionSorter.shutdownTimeout | 120s | The wait time of waiting for sorting partition files during worker graceful shutdown. | 0.2.0 |
-| celeborn.worker.graceful.shutdown.recoverPath | <tmp>/recover | The path to store levelDB. | 0.2.0 |
-| celeborn.worker.graceful.shutdown.timeout | 600s | The worker's graceful shutdown timeout time. | 0.2.0 |
-| celeborn.worker.heartbeat.timeout | 120s | Worker heartbeat timeout. | 0.2.0 |
-| celeborn.worker.memory.checkInterval | 10ms | Interval of worker direct memory checking. | 0.2.0 |
-| celeborn.worker.memory.reportInterval | 10s | Interval of worker direct memory tracker reporting to log. | 0.2.0 |
-| celeborn.worker.metrics.prometheus.host | 0.0.0.0 | Worker's Prometheus host. | 0.2.0 |
-| celeborn.worker.metrics.prometheus.port | 9096 | Worker's Prometheus port. | 0.2.0 |
-| celeborn.worker.monitor.disk.checkInterval | 60s | Intervals between device monitor to check disk. | 0.2.0 |
-| celeborn.worker.monitor.disk.checklist | readwrite,diskusage | Monitor type for disk, available items are: iohang, readwrite and diskusage. | 0.2.0 |
-| celeborn.worker.monitor.disk.enabled | true | When true, worker will monitor device and report to master. | 0.2.0 |
-| celeborn.worker.monitor.disk.sys.block.dir | /sys/block | The directory where linux file block information is stored. | 0.2.0 |
-| celeborn.worker.noneEmptyDirExpireDuration | 1d | If a non-empty application shuffle data dir have not been operated during le duration time, will mark this application as expired. | 0.2.0 |
-| celeborn.worker.partitionSorter.directMemoryRatioThreshold | 0.1 | Max ratio of partition sorter's memory for sorting, when reserved memory is higher than max partition sorter memory, partition sorter will stop sorting. | 0.2.0 |
-| celeborn.worker.partitionSorter.reservedMemoryPerPartition | 1mb | Initial reserve memory when sorting a shuffle file off-heap. | 0.2.0 |
-| celeborn.worker.partitionSorter.sort.timeout | 220s | Timeout for a shuffle file to sort. | 0.2.0 |
-| celeborn.worker.push.io.threads | <undefined> | Netty IO thread number of worker to handle client push data. The default threads number is `size(celeborn.worker.storage.dirs)*2`. | 0.2.0 |
-| celeborn.worker.push.port | 0 | Server port for Worker to receive push data request from ShuffleClient. | 0.2.0 |
-| celeborn.worker.register.timeout | 180s | Worker register timeout. | 0.2.0 |
-| celeborn.worker.replicate.fastFail.duration | 60s | If a replicate request not replied during the duration, worker will mark the replicate data request as failed. | 0.2.0 |
-| celeborn.worker.replicate.io.threads | <undefined> | Netty IO thread number of worker to replicate shuffle data. The default threads number is `size(celeborn.worker.storage.dirs)*2`. | 0.2.0 |
-| celeborn.worker.replicate.port | 0 | Server port for Worker to receive replicate data request from other Workers. | 0.2.0 |
-| celeborn.worker.replicate.threads | 64 | Thread number of worker to replicate shuffle data. | 0.2.0 |
-| celeborn.worker.rpc.port | 0 | Server port for Worker to receive RPC request. | 0.2.0 |
-| celeborn.worker.shuffle.commit.timeout | 120s | Timeout for a Celeborn worker to commit files of a shuffle. | 0.2.0 |
-| celeborn.worker.storage.baseDir.number | 16 | How many directories will be used if `celeborn.worker.storage.dirs` is not set. The directory name is a combination of `celeborn.worker.storage.baseDir.prefix` and from one(inclusive) to `celeborn.worker.storage.baseDir.number`(inclusive) step by one. | 0.2.0 |
-| celeborn.worker.storage.baseDir.prefix | /mnt/disk | Base directory for Celeborn worker to write if `celeborn.worker.storage.dirs` is not set. | 0.2.0 |
-| celeborn.worker.storage.dirs | <undefined> | Directory list to store shuffle data. It's recommended to configure one directory on each disk. Storage size limit can be set for each directory. For the sake of performance, there should be no more than 2 flush threads on the same disk partition if you are using HDD, and should be 8 or more flush threads on the same disk partition if you are using SSD. For example: `dir1[:capacity=][:disktype=][:flushthread=],dir2[:capacity=][:disktyp [...]
-| celeborn.worker.workingDir | hadoop/rss-worker/shuffle_data | Worker's working dir path name. | 0.2.0 |
-| celeborn.worker.writer.close.timeout | 120s | Timeout for a file writer to close | 0.2.0 |
-| celeborn.worker.writer.create.maxAttempts | 3 | Retry count for a file writer to create if its creation was failed. | 0.2.0 |
-<!--end-include-->