You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@celeborn.apache.org by "cfmcgrady (via GitHub)" <gi...@apache.org> on 2023/06/20 11:38:19 UTC
[GitHub] [incubator-celeborn] cfmcgrady opened a new pull request, #1613: [CELEBORN-703][WORKER][PERF] Avoid calling `CelebornConf.get` multi-time when `PushDataHandler` handle `PushData`/`PushMergedData`
cfmcgrady opened a new pull request, #1613:
URL: https://github.com/apache/incubator-celeborn/pull/1613
<!--
Thanks for sending a pull request! Here are some tips for you:
- Make sure the PR title start w/ a JIRA ticket, e.g. '[CELEBORN-XXXX] Your PR title ...'.
- Be sure to keep the PR description updated to reflect all changes.
- Please write your PR title to summarize what this PR proposes.
- If possible, provide a concise example to reproduce the issue for a faster review.
-->
### What changes were proposed in this pull request?
### Why are the changes needed?
### Does this PR introduce _any_ user-facing change?
### How was this patch tested?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] cfmcgrady commented on a diff in pull request #1613: [CELEBORN-703][WORKER][PERF] Avoid calling `CelebornConf.get` multi-time when `PushDataHandler` handle `PushData`/`PushMergedData`
Posted by "cfmcgrady (via GitHub)" <gi...@apache.org>.
cfmcgrady commented on code in PR #1613:
URL: https://github.com/apache/incubator-celeborn/pull/1613#discussion_r1235196674
##########
worker/src/main/scala/org/apache/celeborn/service/deploy/worker/PushDataHandler.scala:
##########
@@ -128,12 +131,12 @@ class PushDataHandler extends BaseMessageHandler with Logging {
val isMaster = mode == PartitionLocation.Mode.MASTER
// For test
- if (isMaster && conf.testPushMasterDataTimeout && !pushMasterDataTimeoutTested) {
+ if (isMaster && !pushMasterDataTimeoutTested) {
Review Comment:
make sense
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] waitinfuture closed pull request #1613: [CELEBORN-703][WORKER][PERF] Avoid calling `CelebornConf#get` multi-time when `PushDataHandler` handle `PushData`/`PushMergedData`
Posted by "waitinfuture (via GitHub)" <gi...@apache.org>.
waitinfuture closed pull request #1613: [CELEBORN-703][WORKER][PERF] Avoid calling `CelebornConf#get` multi-time when `PushDataHandler` handle `PushData`/`PushMergedData`
URL: https://github.com/apache/incubator-celeborn/pull/1613
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] pan3793 commented on a diff in pull request #1613: [CELEBORN-703][WORKER][PERF] Avoid calling `CelebornConf#get` multi-time when `PushDataHandler` handle `PushData`/`PushMergedData`
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #1613:
URL: https://github.com/apache/incubator-celeborn/pull/1613#discussion_r1236258826
##########
worker/src/main/scala/org/apache/celeborn/service/deploy/worker/PushDataHandler.scala:
##########
@@ -42,24 +42,27 @@ import org.apache.celeborn.service.deploy.worker.storage.{FileWriter, HdfsFlushe
class PushDataHandler extends BaseMessageHandler with Logging {
- var workerSource: WorkerSource = _
- var partitionLocationInfo: WorkerPartitionLocationInfo = _
- var shuffleMapperAttempts: ConcurrentHashMap[String, AtomicIntegerArray] = _
- var shufflePartitionType: ConcurrentHashMap[String, PartitionType] = _
- var shufflePushDataTimeout: ConcurrentHashMap[String, Long] = _
- var replicateThreadPool: ThreadPoolExecutor = _
- var unavailablePeers: ConcurrentHashMap[WorkerInfo, Long] = _
- var pushClientFactory: TransportClientFactory = _
- var registered: AtomicBoolean = new AtomicBoolean(false)
- var workerInfo: WorkerInfo = _
- var diskReserveSize: Long = _
- var partitionSplitMinimumSize: Long = _
- var shutdown: AtomicBoolean = _
- var storageManager: StorageManager = _
- var conf: CelebornConf = _
- @volatile var pushMasterDataTimeoutTested = false
- @volatile var pushSlaveDataTimeoutTested = false
- var workerPartitionSplitEnabled: Boolean = _
+ private var workerSource: WorkerSource = _
+ private var partitionLocationInfo: WorkerPartitionLocationInfo = _
+ private var shuffleMapperAttempts: ConcurrentHashMap[String, AtomicIntegerArray] = _
+ private var shufflePartitionType: ConcurrentHashMap[String, PartitionType] = _
+ private var shufflePushDataTimeout: ConcurrentHashMap[String, Long] = _
+ private var replicateThreadPool: ThreadPoolExecutor = _
+ private var unavailablePeers: ConcurrentHashMap[WorkerInfo, Long] = _
+ private var pushClientFactory: TransportClientFactory = _
+ private var registered: AtomicBoolean = new AtomicBoolean(false)
Review Comment:
the initial value could be eliminated then?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] waitinfuture commented on a diff in pull request #1613: [CELEBORN-703][WORKER][PERF] Avoid calling `CelebornConf#get` multi-time when `PushDataHandler` handle `PushData`/`PushMergedData`
Posted by "waitinfuture (via GitHub)" <gi...@apache.org>.
waitinfuture commented on code in PR #1613:
URL: https://github.com/apache/incubator-celeborn/pull/1613#discussion_r1236264704
##########
worker/src/main/scala/org/apache/celeborn/service/deploy/worker/PushDataHandler.scala:
##########
@@ -1068,6 +1074,6 @@ class PushDataHandler extends BaseMessageHandler with Logging {
}
object PushDataHandler {
- @volatile var pushMasterDataTimeoutTested = false
- @volatile var pushSlaveDataTimeoutTested = false
+ @volatile private var pushMasterMergeDataTimeoutTested = false
Review Comment:
I dived into it and think the reason is that, if put the variables in the object, it's process-level singleton, as a result, although we have 3 workers in the MiniCluster, only one timeout will be triggered, and only one worker will be put in the blacklist. However, when put the variables into the class, all three workers will trigger timeout and all will be blacklist, as a result it will cause Revive Exception for no slots available. Following are related logs.
When put the variables in the Object:
```
Reporting Worker Failure:
Host:30.221.101.208:RpcPort:57780:PushPort:57781:FetchPort:57783:ReplicatePort:57782 PUSH_DATA_TIMEOUT_MASTER 1687321559106
Current blacklist:
```
When put the variables in the class:
```
Reporting Worker Failure:
Host:30.221.101.208:RpcPort:57565:PushPort:57568:FetchPort:57570:ReplicatePort:57569 PUSH_DATA_TIMEOUT_MASTER 1687321430258
Current blacklist:
Host:30.221.101.208:RpcPort:57560:PushPort:57562:FetchPort:57564:ReplicatePort:57563 PUSH_DATA_TIMEOUT_MASTER 1687321430254
Host:30.221.101.208:RpcPort:57572:PushPort:57576:FetchPort:57578:ReplicatePort:57577 PUSH_DATA_TIMEOUT_MASTER 1687321430248
```
```
Revive failed, status SLOT_NOT_AVAILABLE
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] cfmcgrady commented on a diff in pull request #1613: [CELEBORN-703][WORKER][PERF] Avoid calling `CelebornConf#get` multi-time when `PushDataHandler` handle `PushData`/`PushMergedData`
Posted by "cfmcgrady (via GitHub)" <gi...@apache.org>.
cfmcgrady commented on code in PR #1613:
URL: https://github.com/apache/incubator-celeborn/pull/1613#discussion_r1236418703
##########
worker/src/main/scala/org/apache/celeborn/service/deploy/worker/PushDataHandler.scala:
##########
@@ -1068,6 +1074,6 @@ class PushDataHandler extends BaseMessageHandler with Logging {
}
object PushDataHandler {
- @volatile var pushMasterDataTimeoutTested = false
- @volatile var pushSlaveDataTimeoutTested = false
+ @volatile private var pushMasterMergeDataTimeoutTested = false
Review Comment:
thanks for the detailed explanation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] waitinfuture commented on a diff in pull request #1613: [CELEBORN-703][WORKER][PERF] Avoid calling `CelebornConf#get` multi-time when `PushDataHandler` handle `PushData`/`PushMergedData`
Posted by "waitinfuture (via GitHub)" <gi...@apache.org>.
waitinfuture commented on code in PR #1613:
URL: https://github.com/apache/incubator-celeborn/pull/1613#discussion_r1236264704
##########
worker/src/main/scala/org/apache/celeborn/service/deploy/worker/PushDataHandler.scala:
##########
@@ -1068,6 +1074,6 @@ class PushDataHandler extends BaseMessageHandler with Logging {
}
object PushDataHandler {
- @volatile var pushMasterDataTimeoutTested = false
- @volatile var pushSlaveDataTimeoutTested = false
+ @volatile private var pushMasterMergeDataTimeoutTested = false
Review Comment:
I dived into it and think the reason is that, if put the variables in the object, it's process-level singleton, as a result, although we have 3 workers in the MiniCluster, only one timeout will be triggered, and the worker will be put in the blacklist. However, when put the variables into Handler, all three workers will trigger timeout and all will be blacklist, as a result it will cause Revive Exception for no slots available. Following are related logs:
```
Reporting Worker Failure:
Host:30.221.101.208:RpcPort:57565:PushPort:57568:FetchPort:57570:ReplicatePort:57569 PUSH_DATA_TIMEOUT_MASTER 1687321430258
Current blacklist:
Host:30.221.101.208:RpcPort:57560:PushPort:57562:FetchPort:57564:ReplicatePort:57563 PUSH_DATA_TIMEOUT_MASTER 1687321430254
Host:30.221.101.208:RpcPort:57572:PushPort:57576:FetchPort:57578:ReplicatePort:57577 PUSH_DATA_TIMEOUT_MASTER 1687321430248
```
```
Revive failed, status SLOT_NOT_AVAILABLE
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] waitinfuture commented on a diff in pull request #1613: [CELEBORN-703][WORKER][PERF] Avoid calling `CelebornConf.get` multi-time when `PushDataHandler` handle `PushData`/`PushMergedData`
Posted by "waitinfuture (via GitHub)" <gi...@apache.org>.
waitinfuture commented on code in PR #1613:
URL: https://github.com/apache/incubator-celeborn/pull/1613#discussion_r1235187781
##########
worker/src/main/scala/org/apache/celeborn/service/deploy/worker/PushDataHandler.scala:
##########
@@ -128,12 +131,12 @@ class PushDataHandler extends BaseMessageHandler with Logging {
val isMaster = mode == PartitionLocation.Mode.MASTER
// For test
- if (isMaster && conf.testPushMasterDataTimeout && !pushMasterDataTimeoutTested) {
+ if (isMaster && !pushMasterDataTimeoutTested) {
Review Comment:
Hi @cfmcgrady , the purpose here is that if ```conf.testPushSlaveDataTimeout``` is enabled, go into the test branch only once. So I think we can just reorder the conditions like
```
if (isMaster && !pushMasterDataTimeoutTested && conf.testPushMasterDataTimeout)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] codecov[bot] commented on pull request #1613: [CELEBORN-703][WORKER][PERF] Avoid calling `CelebornConf.get` multi-time when `PushDataHandler` handle `PushData`/`PushMergedData`
Posted by "codecov[bot] (via GitHub)" <gi...@apache.org>.
codecov[bot] commented on PR #1613:
URL: https://github.com/apache/incubator-celeborn/pull/1613#issuecomment-1598626569
## [Codecov](https://app.codecov.io/gh/apache/incubator-celeborn/pull/1613?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
> Merging [#1613](https://app.codecov.io/gh/apache/incubator-celeborn/pull/1613?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) (38f40be) into [main](https://app.codecov.io/gh/apache/incubator-celeborn/commit/7d634db54712a0924862581a8c9630f3b22cd1cd?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) (7d634db) will **increase** coverage by `0.03%`.
> The diff coverage is `n/a`.
```diff
@@ Coverage Diff @@
## main #1613 +/- ##
==========================================
+ Coverage 45.73% 45.76% +0.03%
==========================================
Files 159 159
Lines 9944 9945 +1
Branches 972 972
==========================================
+ Hits 4547 4550 +3
+ Misses 5098 5096 -2
Partials 299 299
```
[see 3 files with indirect coverage changes](https://app.codecov.io/gh/apache/incubator-celeborn/pull/1613/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
:mega: We’re building smart automated test selection to slash your CI/CD build times. [Learn more](https://about.codecov.io/iterative-testing/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] cfmcgrady commented on a diff in pull request #1613: [CELEBORN-703][WORKER][PERF] Avoid calling `CelebornConf#get` multi-time when `PushDataHandler` handle `PushData`/`PushMergedData`
Posted by "cfmcgrady (via GitHub)" <gi...@apache.org>.
cfmcgrady commented on code in PR #1613:
URL: https://github.com/apache/incubator-celeborn/pull/1613#discussion_r1236258196
##########
worker/src/main/scala/org/apache/celeborn/service/deploy/worker/PushDataHandler.scala:
##########
@@ -42,24 +42,27 @@ import org.apache.celeborn.service.deploy.worker.storage.{FileWriter, HdfsFlushe
class PushDataHandler extends BaseMessageHandler with Logging {
- var workerSource: WorkerSource = _
- var partitionLocationInfo: WorkerPartitionLocationInfo = _
- var shuffleMapperAttempts: ConcurrentHashMap[String, AtomicIntegerArray] = _
- var shufflePartitionType: ConcurrentHashMap[String, PartitionType] = _
- var shufflePushDataTimeout: ConcurrentHashMap[String, Long] = _
- var replicateThreadPool: ThreadPoolExecutor = _
- var unavailablePeers: ConcurrentHashMap[WorkerInfo, Long] = _
- var pushClientFactory: TransportClientFactory = _
- var registered: AtomicBoolean = new AtomicBoolean(false)
- var workerInfo: WorkerInfo = _
- var diskReserveSize: Long = _
- var partitionSplitMinimumSize: Long = _
- var shutdown: AtomicBoolean = _
- var storageManager: StorageManager = _
- var conf: CelebornConf = _
- @volatile var pushMasterDataTimeoutTested = false
- @volatile var pushSlaveDataTimeoutTested = false
- var workerPartitionSplitEnabled: Boolean = _
+ private var workerSource: WorkerSource = _
+ private var partitionLocationInfo: WorkerPartitionLocationInfo = _
+ private var shuffleMapperAttempts: ConcurrentHashMap[String, AtomicIntegerArray] = _
+ private var shufflePartitionType: ConcurrentHashMap[String, PartitionType] = _
+ private var shufflePushDataTimeout: ConcurrentHashMap[String, Long] = _
+ private var replicateThreadPool: ThreadPoolExecutor = _
+ private var unavailablePeers: ConcurrentHashMap[WorkerInfo, Long] = _
+ private var pushClientFactory: TransportClientFactory = _
+ private var registered: AtomicBoolean = new AtomicBoolean(false)
Review Comment:
No, the variable `shutdown` is assigned as a reference to `worker.shutdown` during the class initialization process.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] cfmcgrady commented on a diff in pull request #1613: [CELEBORN-703][WORKER][PERF] Avoid calling `CelebornConf#get` multi-time when `PushDataHandler` handle `PushData`/`PushMergedData`
Posted by "cfmcgrady (via GitHub)" <gi...@apache.org>.
cfmcgrady commented on code in PR #1613:
URL: https://github.com/apache/incubator-celeborn/pull/1613#discussion_r1236423721
##########
worker/src/main/scala/org/apache/celeborn/service/deploy/worker/PushDataHandler.scala:
##########
@@ -1068,6 +1074,6 @@ class PushDataHandler extends BaseMessageHandler with Logging {
}
object PushDataHandler {
- @volatile var pushMasterDataTimeoutTested = false
- @volatile var pushSlaveDataTimeoutTested = false
+ @volatile private var pushMasterMergeDataTimeoutTested = false
Review Comment:
The reason behind the successful passing of the `PushDataTimeoutTest` is that the `PushDataHandler#handlePushData` is not triggered. This is due to the fact that the current test case consistently ensures that the buffer size of the hash-based shuffle writer remains below the `PUSH_BUFFER_MAX_SIZE`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] cfmcgrady commented on a diff in pull request #1613: [CELEBORN-703][WORKER][PERF] Avoid calling `CelebornConf.get` multi-time when `PushDataHandler` handle `PushData`/`PushMergedData`
Posted by "cfmcgrady (via GitHub)" <gi...@apache.org>.
cfmcgrady commented on code in PR #1613:
URL: https://github.com/apache/incubator-celeborn/pull/1613#discussion_r1236201446
##########
worker/src/main/scala/org/apache/celeborn/service/deploy/worker/PushDataHandler.scala:
##########
@@ -1068,6 +1074,6 @@ class PushDataHandler extends BaseMessageHandler with Logging {
}
object PushDataHandler {
- @volatile var pushMasterDataTimeoutTested = false
- @volatile var pushSlaveDataTimeoutTested = false
+ @volatile private var pushMasterMergeDataTimeoutTested = false
Review Comment:
I would like to inquire about the reasoning behind choosing the `pushMasterMergeDataTimeoutTested` as an object variable instead of a class variable. @waitinfuture
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] cfmcgrady commented on a diff in pull request #1613: [CELEBORN-703][WORKER][PERF] Avoid calling `CelebornConf#get` multi-time when `PushDataHandler` handle `PushData`/`PushMergedData`
Posted by "cfmcgrady (via GitHub)" <gi...@apache.org>.
cfmcgrady commented on code in PR #1613:
URL: https://github.com/apache/incubator-celeborn/pull/1613#discussion_r1236284270
##########
worker/src/main/scala/org/apache/celeborn/service/deploy/worker/PushDataHandler.scala:
##########
@@ -42,24 +42,27 @@ import org.apache.celeborn.service.deploy.worker.storage.{FileWriter, HdfsFlushe
class PushDataHandler extends BaseMessageHandler with Logging {
- var workerSource: WorkerSource = _
- var partitionLocationInfo: WorkerPartitionLocationInfo = _
- var shuffleMapperAttempts: ConcurrentHashMap[String, AtomicIntegerArray] = _
- var shufflePartitionType: ConcurrentHashMap[String, PartitionType] = _
- var shufflePushDataTimeout: ConcurrentHashMap[String, Long] = _
- var replicateThreadPool: ThreadPoolExecutor = _
- var unavailablePeers: ConcurrentHashMap[WorkerInfo, Long] = _
- var pushClientFactory: TransportClientFactory = _
- var registered: AtomicBoolean = new AtomicBoolean(false)
- var workerInfo: WorkerInfo = _
- var diskReserveSize: Long = _
- var partitionSplitMinimumSize: Long = _
- var shutdown: AtomicBoolean = _
- var storageManager: StorageManager = _
- var conf: CelebornConf = _
- @volatile var pushMasterDataTimeoutTested = false
- @volatile var pushSlaveDataTimeoutTested = false
- var workerPartitionSplitEnabled: Boolean = _
+ private var workerSource: WorkerSource = _
+ private var partitionLocationInfo: WorkerPartitionLocationInfo = _
+ private var shuffleMapperAttempts: ConcurrentHashMap[String, AtomicIntegerArray] = _
+ private var shufflePartitionType: ConcurrentHashMap[String, PartitionType] = _
+ private var shufflePushDataTimeout: ConcurrentHashMap[String, Long] = _
+ private var replicateThreadPool: ThreadPoolExecutor = _
+ private var unavailablePeers: ConcurrentHashMap[WorkerInfo, Long] = _
+ private var pushClientFactory: TransportClientFactory = _
+ private var registered: AtomicBoolean = new AtomicBoolean(false)
Review Comment:
updated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] cfmcgrady commented on a diff in pull request #1613: [CELEBORN-703][WORKER][PERF] Avoid calling `CelebornConf#get` multi-time when `PushDataHandler` handle `PushData`/`PushMergedData`
Posted by "cfmcgrady (via GitHub)" <gi...@apache.org>.
cfmcgrady commented on code in PR #1613:
URL: https://github.com/apache/incubator-celeborn/pull/1613#discussion_r1236419219
##########
worker/src/main/scala/org/apache/celeborn/service/deploy/worker/PushDataHandler.scala:
##########
@@ -1068,6 +1074,6 @@ class PushDataHandler extends BaseMessageHandler with Logging {
}
object PushDataHandler {
- @volatile var pushMasterDataTimeoutTested = false
- @volatile var pushSlaveDataTimeoutTested = false
+ @volatile private var pushMasterMergeDataTimeoutTested = false
Review Comment:
then the variable `pushMasterDataTimeoutTested` and `pushSlaveDataTimeoutTested` should also be a singleton.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] cfmcgrady commented on a diff in pull request #1613: [CELEBORN-703][WORKER][PERF] Avoid calling `CelebornConf.get` multi-time when `PushDataHandler` handle `PushData`/`PushMergedData`
Posted by "cfmcgrady (via GitHub)" <gi...@apache.org>.
cfmcgrady commented on code in PR #1613:
URL: https://github.com/apache/incubator-celeborn/pull/1613#discussion_r1235196674
##########
worker/src/main/scala/org/apache/celeborn/service/deploy/worker/PushDataHandler.scala:
##########
@@ -128,12 +131,12 @@ class PushDataHandler extends BaseMessageHandler with Logging {
val isMaster = mode == PartitionLocation.Mode.MASTER
// For test
- if (isMaster && conf.testPushMasterDataTimeout && !pushMasterDataTimeoutTested) {
+ if (isMaster && !pushMasterDataTimeoutTested) {
Review Comment:
~~make sense~~
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] waitinfuture commented on a diff in pull request #1613: [CELEBORN-703][WORKER][PERF] Avoid calling `CelebornConf.get` multi-time when `PushDataHandler` handle `PushData`/`PushMergedData`
Posted by "waitinfuture (via GitHub)" <gi...@apache.org>.
waitinfuture commented on code in PR #1613:
URL: https://github.com/apache/incubator-celeborn/pull/1613#discussion_r1236195603
##########
worker/src/main/scala/org/apache/celeborn/service/deploy/worker/PushDataHandler.scala:
##########
@@ -128,12 +131,12 @@ class PushDataHandler extends BaseMessageHandler with Logging {
val isMaster = mode == PartitionLocation.Mode.MASTER
// For test
- if (isMaster && conf.testPushMasterDataTimeout && !pushMasterDataTimeoutTested) {
+ if (isMaster && !pushMasterDataTimeoutTested) {
Review Comment:
Right, thanks for your clearance :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] pan3793 commented on a diff in pull request #1613: [CELEBORN-703][WORKER][PERF] Avoid calling `CelebornConf#get` multi-time when `PushDataHandler` handle `PushData`/`PushMergedData`
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #1613:
URL: https://github.com/apache/incubator-celeborn/pull/1613#discussion_r1236251375
##########
worker/src/main/scala/org/apache/celeborn/service/deploy/worker/PushDataHandler.scala:
##########
@@ -42,24 +42,27 @@ import org.apache.celeborn.service.deploy.worker.storage.{FileWriter, HdfsFlushe
class PushDataHandler extends BaseMessageHandler with Logging {
- var workerSource: WorkerSource = _
- var partitionLocationInfo: WorkerPartitionLocationInfo = _
- var shuffleMapperAttempts: ConcurrentHashMap[String, AtomicIntegerArray] = _
- var shufflePartitionType: ConcurrentHashMap[String, PartitionType] = _
- var shufflePushDataTimeout: ConcurrentHashMap[String, Long] = _
- var replicateThreadPool: ThreadPoolExecutor = _
- var unavailablePeers: ConcurrentHashMap[WorkerInfo, Long] = _
- var pushClientFactory: TransportClientFactory = _
- var registered: AtomicBoolean = new AtomicBoolean(false)
- var workerInfo: WorkerInfo = _
- var diskReserveSize: Long = _
- var partitionSplitMinimumSize: Long = _
- var shutdown: AtomicBoolean = _
- var storageManager: StorageManager = _
- var conf: CelebornConf = _
- @volatile var pushMasterDataTimeoutTested = false
- @volatile var pushSlaveDataTimeoutTested = false
- var workerPartitionSplitEnabled: Boolean = _
+ private var workerSource: WorkerSource = _
+ private var partitionLocationInfo: WorkerPartitionLocationInfo = _
+ private var shuffleMapperAttempts: ConcurrentHashMap[String, AtomicIntegerArray] = _
+ private var shufflePartitionType: ConcurrentHashMap[String, PartitionType] = _
+ private var shufflePushDataTimeout: ConcurrentHashMap[String, Long] = _
+ private var replicateThreadPool: ThreadPoolExecutor = _
+ private var unavailablePeers: ConcurrentHashMap[WorkerInfo, Long] = _
+ private var pushClientFactory: TransportClientFactory = _
+ private var registered: AtomicBoolean = new AtomicBoolean(false)
Review Comment:
could be `val`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] waitinfuture commented on a diff in pull request #1613: [CELEBORN-703][WORKER][PERF] Avoid calling `CelebornConf#get` multi-time when `PushDataHandler` handle `PushData`/`PushMergedData`
Posted by "waitinfuture (via GitHub)" <gi...@apache.org>.
waitinfuture commented on code in PR #1613:
URL: https://github.com/apache/incubator-celeborn/pull/1613#discussion_r1236440452
##########
worker/src/main/scala/org/apache/celeborn/service/deploy/worker/PushDataHandler.scala:
##########
@@ -1068,6 +1074,6 @@ class PushDataHandler extends BaseMessageHandler with Logging {
}
object PushDataHandler {
- @volatile var pushMasterDataTimeoutTested = false
- @volatile var pushSlaveDataTimeoutTested = false
+ @volatile private var pushMasterMergeDataTimeoutTested = false
Review Comment:
Oh, that's unexpected, thanks for your exploration. So I think we should make it consistent for ```testPushMasterDataTimeout/testPushSlaveDataTimeout``` and ```pushMasterMergeDataTimeoutTested/pushSlaveMergeDataTimeoutTested```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] cfmcgrady commented on a diff in pull request #1613: [CELEBORN-703][WORKER][PERF] Avoid calling `CelebornConf#get` multi-time when `PushDataHandler` handle `PushData`/`PushMergedData`
Posted by "cfmcgrady (via GitHub)" <gi...@apache.org>.
cfmcgrady commented on code in PR #1613:
URL: https://github.com/apache/incubator-celeborn/pull/1613#discussion_r1236258196
##########
worker/src/main/scala/org/apache/celeborn/service/deploy/worker/PushDataHandler.scala:
##########
@@ -42,24 +42,27 @@ import org.apache.celeborn.service.deploy.worker.storage.{FileWriter, HdfsFlushe
class PushDataHandler extends BaseMessageHandler with Logging {
- var workerSource: WorkerSource = _
- var partitionLocationInfo: WorkerPartitionLocationInfo = _
- var shuffleMapperAttempts: ConcurrentHashMap[String, AtomicIntegerArray] = _
- var shufflePartitionType: ConcurrentHashMap[String, PartitionType] = _
- var shufflePushDataTimeout: ConcurrentHashMap[String, Long] = _
- var replicateThreadPool: ThreadPoolExecutor = _
- var unavailablePeers: ConcurrentHashMap[WorkerInfo, Long] = _
- var pushClientFactory: TransportClientFactory = _
- var registered: AtomicBoolean = new AtomicBoolean(false)
- var workerInfo: WorkerInfo = _
- var diskReserveSize: Long = _
- var partitionSplitMinimumSize: Long = _
- var shutdown: AtomicBoolean = _
- var storageManager: StorageManager = _
- var conf: CelebornConf = _
- @volatile var pushMasterDataTimeoutTested = false
- @volatile var pushSlaveDataTimeoutTested = false
- var workerPartitionSplitEnabled: Boolean = _
+ private var workerSource: WorkerSource = _
+ private var partitionLocationInfo: WorkerPartitionLocationInfo = _
+ private var shuffleMapperAttempts: ConcurrentHashMap[String, AtomicIntegerArray] = _
+ private var shufflePartitionType: ConcurrentHashMap[String, PartitionType] = _
+ private var shufflePushDataTimeout: ConcurrentHashMap[String, Long] = _
+ private var replicateThreadPool: ThreadPoolExecutor = _
+ private var unavailablePeers: ConcurrentHashMap[WorkerInfo, Long] = _
+ private var pushClientFactory: TransportClientFactory = _
+ private var registered: AtomicBoolean = new AtomicBoolean(false)
Review Comment:
No, the variable `registered` is assigned as a reference to `worker. registered` during the class initialization process.
##########
worker/src/main/scala/org/apache/celeborn/service/deploy/worker/PushDataHandler.scala:
##########
@@ -42,24 +42,27 @@ import org.apache.celeborn.service.deploy.worker.storage.{FileWriter, HdfsFlushe
class PushDataHandler extends BaseMessageHandler with Logging {
- var workerSource: WorkerSource = _
- var partitionLocationInfo: WorkerPartitionLocationInfo = _
- var shuffleMapperAttempts: ConcurrentHashMap[String, AtomicIntegerArray] = _
- var shufflePartitionType: ConcurrentHashMap[String, PartitionType] = _
- var shufflePushDataTimeout: ConcurrentHashMap[String, Long] = _
- var replicateThreadPool: ThreadPoolExecutor = _
- var unavailablePeers: ConcurrentHashMap[WorkerInfo, Long] = _
- var pushClientFactory: TransportClientFactory = _
- var registered: AtomicBoolean = new AtomicBoolean(false)
- var workerInfo: WorkerInfo = _
- var diskReserveSize: Long = _
- var partitionSplitMinimumSize: Long = _
- var shutdown: AtomicBoolean = _
- var storageManager: StorageManager = _
- var conf: CelebornConf = _
- @volatile var pushMasterDataTimeoutTested = false
- @volatile var pushSlaveDataTimeoutTested = false
- var workerPartitionSplitEnabled: Boolean = _
+ private var workerSource: WorkerSource = _
+ private var partitionLocationInfo: WorkerPartitionLocationInfo = _
+ private var shuffleMapperAttempts: ConcurrentHashMap[String, AtomicIntegerArray] = _
+ private var shufflePartitionType: ConcurrentHashMap[String, PartitionType] = _
+ private var shufflePushDataTimeout: ConcurrentHashMap[String, Long] = _
+ private var replicateThreadPool: ThreadPoolExecutor = _
+ private var unavailablePeers: ConcurrentHashMap[WorkerInfo, Long] = _
+ private var pushClientFactory: TransportClientFactory = _
+ private var registered: AtomicBoolean = new AtomicBoolean(false)
Review Comment:
No, the variable `registered` is assigned as a reference to `worker.registered` during the class initialization process.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] cfmcgrady commented on a diff in pull request #1613: [CELEBORN-703][WORKER][PERF] Avoid calling `CelebornConf#get` multi-time when `PushDataHandler` handle `PushData`/`PushMergedData`
Posted by "cfmcgrady (via GitHub)" <gi...@apache.org>.
cfmcgrady commented on code in PR #1613:
URL: https://github.com/apache/incubator-celeborn/pull/1613#discussion_r1237138700
##########
worker/src/main/scala/org/apache/celeborn/service/deploy/worker/PushDataHandler.scala:
##########
@@ -1068,6 +1074,6 @@ class PushDataHandler extends BaseMessageHandler with Logging {
}
object PushDataHandler {
- @volatile var pushMasterDataTimeoutTested = false
- @volatile var pushSlaveDataTimeoutTested = false
+ @volatile private var pushMasterMergeDataTimeoutTested = false
Review Comment:
ok, will raise a follow-up
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] cfmcgrady commented on a diff in pull request #1613: [CELEBORN-703][WORKER][PERF] Avoid calling `CelebornConf.get` multi-time when `PushDataHandler` handle `PushData`/`PushMergedData`
Posted by "cfmcgrady (via GitHub)" <gi...@apache.org>.
cfmcgrady commented on code in PR #1613:
URL: https://github.com/apache/incubator-celeborn/pull/1613#discussion_r1236194271
##########
worker/src/main/scala/org/apache/celeborn/service/deploy/worker/PushDataHandler.scala:
##########
@@ -128,12 +131,12 @@ class PushDataHandler extends BaseMessageHandler with Logging {
val isMaster = mode == PartitionLocation.Mode.MASTER
// For test
- if (isMaster && conf.testPushMasterDataTimeout && !pushMasterDataTimeoutTested) {
+ if (isMaster && !pushMasterDataTimeoutTested) {
Review Comment:
Apologies for the confusion before. In fact, `conf.testPushMasterDataTimeout` will be invoked every time the condition is checked. This is because, in the production environment, `!pushMasterDataTimeoutTested` always returns `true`. Simply reordering the conditions will not resolve this issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] cfmcgrady commented on a diff in pull request #1613: [CELEBORN-703][WORKER][PERF] Avoid calling `CelebornConf.get` multi-time when `PushDataHandler` handle `PushData`/`PushMergedData`
Posted by "cfmcgrady (via GitHub)" <gi...@apache.org>.
cfmcgrady commented on code in PR #1613:
URL: https://github.com/apache/incubator-celeborn/pull/1613#discussion_r1236201528
##########
worker/src/main/scala/org/apache/celeborn/service/deploy/worker/PushDataHandler.scala:
##########
@@ -1068,6 +1074,6 @@ class PushDataHandler extends BaseMessageHandler with Logging {
}
object PushDataHandler {
- @volatile var pushMasterDataTimeoutTested = false
- @volatile var pushSlaveDataTimeoutTested = false
+ @volatile private var pushMasterMergeDataTimeoutTested = false
Review Comment:
If I were to move `pushMasterMergeDataTimeoutTested ` to a class variable, it would result in a failure of the CI.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org