[GitHub] [incubator-celeborn] zhongqiangczq commented on a diff in pull request #1782: [CELEBORN-864][DOC] Document on blacklist

["zhongqiangczq (via GitHub)" <gi...@apache.org>]

zhongqiangczq commented on code in PR #1782:
URL: https://github.com/apache/incubator-celeborn/pull/1782#discussion_r1280592069


##########
docs/developers/blacklist.md:
##########
@@ -0,0 +1,81 @@
+---
+license: |
+  Licensed to the Apache Software Foundation (ASF) under one or more
+  contributor license agreements.  See the NOTICE file distributed with
+  this work for additional information regarding copyright ownership.
+  The ASF licenses this file to You under the Apache License, Version 2.0
+  (the "License"); you may not use this file except in compliance with
+  the License.  You may obtain a copy of the License at
+
+      https://www.apache.org/licenses/LICENSE-2.0
+
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS,
+  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  See the License for the specific language governing permissions and
+  limitations under the License.
+---
+
+# Overview
+`Worker`s can fail, temporarily or permanently. To reduce the impact of `Worker` failure, Celeborn tries to
+figure out `Worker` status as soon as possible, and as correct as possible. This article describes detailed
+design of blacklist.
+
+## Participants
+As described [Previously](../../developers/overview#components), Celeborn has three components: `Master`, `Worker`,
+and `Client`. `Client` is further separated into `LifecycleManager` and `ShuffleClient`. `Master`/`LifecycleManager`
+/`ShuffleClient` need to know about `Worker` status, actively or re-actively.
+
+## Master Blacklist
+`Master` maintains the ground-truth status of `Worker`s, with relatively longer delay. Master maintains four
+lists of `Worker`s with different status:
+
+- Active list. `Worker`s that have successfully registered to `Master`, and heartbeat never timed out.
+- Excluded list. `Worker`s that are inside active list, but have no available disks for allocating new
+  slots. `Master` recognizes such `Worker`s through heartbeat from `Worker`s.
+- Graceful shutdown list. `Worker`s that are inside active list, but have triggered
+  [Graceful Shutdown](../../upgrading). `Master` expects these `Worker`s should re-register themselves soon.
+- Lost list. `Worker`s whose heartbeat timed out. These `Worker`s will be removed from active and excluded
+  list, but will not be removed from graceful shutdown list.
+
+Upon receiving RequestSlots, `Master` will only choose `Worker`s in active list, but not in excluded or graceful
+shutdown list. Since `Master` only exclude `Worker`s upon heartbeat, it has relative long delay.
+
+## ShuffleClient Blacklist
+`ShuffleClient`'s local blacklist is essential to performance. Say the timeout for setup network
+connection is 10s, if `ShuffleClient` blindly pushes data to a non-exist `Worker`, the task will hang forever.
+
+Waiting for `Master` to inform the blacklist is unacceptable because of the delay. Instead, `ShuffleClient`
+actively exclude `Worker`s when it encounters critical exceptions, for example:
+
+- Fail to create network connection
+- Fail to push data
+- Fail to fetch data
+- Connection exception happened
+
+In addition to exclude the `Worker`s locally, `ShuffleClient` also carries the cause of push failure with
+[Revive](../../developers/faulttolerant#handle-pushdata-failure) for `LifecycleManager` to also exclude the `Worker`s,
+see the section below.
+
+Such strategy is aggressive, meaning false negative may happen. To rectify, `ShuffleClient` removes from
+the excluded list whenever an event happens that indicates that `Worker` is available, for example:
+
+- When the `Worker` is allocated slots in register shuffle
+- When `LifecycleManager` says the `Worker` is available in response of Revive
+
+Currently, blacklist in `ShuffleClient` is optional, users can configure using the following configs:
+
+`celeborn.client.push/fetch.excludeWorkerOnFailure.enabled`
+
+## LifecycleManager Blacklist
+The accuracy and delay in `LifecycleManager`'s blacklist stands between `Master` and `Worker`. `LifecyleManager`
+excludes a `Worker` in the following scenarios:
+
+- Receives Revive request and the cause is critical
+- Fail to send RPC to a `Worker`
+- From `Master`'s excluded list, carried in the heartbeat response
+
+`LifecycleManager` will remove `Worker` from the excluded list in the following scenarios:
+
+- For critical causes, when timeout expires (defaults to 180s)
+- for non-critical causes, when it's not in `Master`'s blacklist

Review Comment:
   ```suggestion
   - For non-critical causes, when it's not in `Master`'s blacklist
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org