You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@celeborn.apache.org by "kerwin-zk (via GitHub)" <gi...@apache.org> on 2023/03/24 08:18:29 UTC
[GitHub] [incubator-celeborn] kerwin-zk opened a new pull request, #1379: [CELEBORN-472] Support using Celeborn in the scenario of switching multiple SparkSessions in the same process
kerwin-zk opened a new pull request, #1379:
URL: https://github.com/apache/incubator-celeborn/pull/1379
### What changes were proposed in this pull request?
Support using Celeborn in the scenario of switching multiple SparkSessions in the same process.
### Why are the changes needed?
In the current celeborn, if you switch the SparkSession in the Spark jar package application, the following exception will occur.
```
org.apache.celeborn.shaded.io.netty.channel.AbstractChannel - Force-closing a channel whose registration task was not accepted by an event loop: [id: 0xca6cd07d]
java.util.concurrent.RejectedExecutionException: event executor terminated
at org.apache.celeborn.shaded.io.netty.util.concurrent.SingleThreadEventExecutor.reject(SingleThreadEventExecutor.java:932)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] kerwin-zk commented on pull request #1379: [CELEBORN-472] Support using Celeborn in the scenario of switching multiple SparkContexts in the same process
Posted by "kerwin-zk (via GitHub)" <gi...@apache.org>.
kerwin-zk commented on PR #1379:
URL: https://github.com/apache/incubator-celeborn/pull/1379#issuecomment-1482529952
> > Here instead of using multiple SparkContexts at the same time, but to use only one at a time, close it after use, and then create a new one.
>
> that makes sense, please clarify it in PR description
@pan3793 done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] pan3793 commented on pull request #1379: [CELEBORN-472] Support using Celeborn in the scenario of switching multiple SparkSessions in the same process
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on PR #1379:
URL: https://github.com/apache/incubator-celeborn/pull/1379#issuecomment-1482478855
> Support using Celeborn in the scenario of switching multiple SparkSessions in the same process.
I don't get your point, or you are saying "multiple `SparkContext`s in the same process." instead of "`SparkSession`s"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] waitinfuture merged pull request #1379: [CELEBORN-472] Support using Celeborn in the scenario of switching multiple SparkContexts in the same process
Posted by "waitinfuture (via GitHub)" <gi...@apache.org>.
waitinfuture merged PR #1379:
URL: https://github.com/apache/incubator-celeborn/pull/1379
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] kerwin-zk commented on pull request #1379: [CELEBORN-472] Support using Celeborn in the scenario of switching multiple SparkSessions in the same process
Posted by "kerwin-zk (via GitHub)" <gi...@apache.org>.
kerwin-zk commented on PR #1379:
URL: https://github.com/apache/incubator-celeborn/pull/1379#issuecomment-1482497228
> multiple
@pan3793 Yes. When using different SparkSessions(Essentially SparkContext) in one process, you need to close one and use another one. After closing a SparkSession, RssShuffleManager will call the stop method to close the singleton shuffleclient. When the next SparkSession uses shuffleclient again in same process. The shuffleclient was closed, causing the problem.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] kerwin-zk commented on pull request #1379: [CELEBORN-472] Support using Celeborn in the scenario of switching multiple SparkSessions in the same process
Posted by "kerwin-zk (via GitHub)" <gi...@apache.org>.
kerwin-zk commented on PR #1379:
URL: https://github.com/apache/incubator-celeborn/pull/1379#issuecomment-1482512287
> Multiple Spark contexts are discouraged and totally disallowed since Spark 3.0 [SPARK-26362](https://issues.apache.org/jira/browse/SPARK-26362).
@pan3793
Here instead of using multiple SparkContexts at the same time, but to use only one at a time, close it after use, and then create a new one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] pan3793 commented on pull request #1379: [CELEBORN-472] Support using Celeborn in the scenario of switching multiple SparkContexts in the same process
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on PR #1379:
URL: https://github.com/apache/incubator-celeborn/pull/1379#issuecomment-1486283320
This is kinda resource leak, back ported to branch-0.2
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] codecov[bot] commented on pull request #1379: [CELEBORN-472] Support using Celeborn in the scenario of switching multiple SparkContexts in the same process
Posted by "codecov[bot] (via GitHub)" <gi...@apache.org>.
codecov[bot] commented on PR #1379:
URL: https://github.com/apache/incubator-celeborn/pull/1379#issuecomment-1482533298
## [Codecov](https://codecov.io/gh/apache/incubator-celeborn/pull/1379?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#1379](https://codecov.io/gh/apache/incubator-celeborn/pull/1379?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (b55276c) into [main](https://codecov.io/gh/apache/incubator-celeborn/commit/2bfa7e896591eed0bc1021002728a01e842ed013?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (2bfa7e8) will **increase** coverage by `0.05%`.
> The diff coverage is `n/a`.
```diff
@@ Coverage Diff @@
## main #1379 +/- ##
============================================
+ Coverage 29.80% 29.84% +0.05%
- Complexity 909 912 +3
============================================
Files 222 222
Lines 18489 18487 -2
Branches 1970 1970
============================================
+ Hits 5508 5516 +8
+ Misses 12587 12581 -6
+ Partials 394 390 -4
```
[see 4 files with indirect coverage changes](https://codecov.io/gh/apache/incubator-celeborn/pull/1379/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
:mega: We’re building smart automated test selection to slash your CI/CD build times. [Learn more](https://about.codecov.io/iterative-testing/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] pan3793 commented on pull request #1379: [CELEBORN-472] Support using Celeborn in the scenario of switching multiple SparkSessions in the same process
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on PR #1379:
URL: https://github.com/apache/incubator-celeborn/pull/1379#issuecomment-1482514409
> Here instead of using multiple SparkContexts at the same time, but to use only one at a time, close it after use, and then create a new one.
that makes sense, please clarify it in PR description
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] AngersZhuuuu commented on pull request #1379: [CELEBORN-472] Support using Celeborn in the scenario of switching multiple SparkSessions in the same process
Posted by "AngersZhuuuu (via GitHub)" <gi...@apache.org>.
AngersZhuuuu commented on PR #1379:
URL: https://github.com/apache/incubator-celeborn/pull/1379#issuecomment-1482506630
> Multiple Spark contexts are discouraged and totally disallowed since Spark 3.0 [SPARK-26362](https://issues.apache.org/jira/browse/SPARK-26362).
But. this fix still looks like reasonable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] pan3793 commented on pull request #1379: [CELEBORN-472] Support using Celeborn in the scenario of switching multiple SparkSessions in the same process
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on PR #1379:
URL: https://github.com/apache/incubator-celeborn/pull/1379#issuecomment-1482508719
@AngersZhuuuu is right. The change itself LGTM, it's kinda resource cleanup.
`SparkContext` and `SparkSession` are indeed different things, please update the title and description.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] pan3793 commented on pull request #1379: [CELEBORN-472] Support using Celeborn in the scenario of switching multiple SparkContexts in the same process
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on PR #1379:
URL: https://github.com/apache/incubator-celeborn/pull/1379#issuecomment-1484431463
@kerwin-zk looks like you did not setup apache account properly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] pan3793 commented on pull request #1379: [CELEBORN-472] Support using Celeborn in the scenario of switching multiple SparkSessions in the same process
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on PR #1379:
URL: https://github.com/apache/incubator-celeborn/pull/1379#issuecomment-1482501243
Multiple Spark contexts are discouraged and totally disallowed since Spark 3.0 [SPARK-26362](https://issues.apache.org/jira/browse/SPARK-26362).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] kerwin-zk commented on pull request #1379: [CELEBORN-472] Support using Celeborn in the scenario of switching multiple SparkSessions in the same process
Posted by "kerwin-zk (via GitHub)" <gi...@apache.org>.
kerwin-zk commented on PR #1379:
URL: https://github.com/apache/incubator-celeborn/pull/1379#issuecomment-1482497709
> The stacktrace only has one line? Any more background why it occurs?
done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] AngersZhuuuu commented on a diff in pull request #1379: [CELEBORN-472] Support using Celeborn in the scenario of switching multiple SparkContexts in the same process
Posted by "AngersZhuuuu (via GitHub)" <gi...@apache.org>.
AngersZhuuuu commented on code in PR #1379:
URL: https://github.com/apache/incubator-celeborn/pull/1379#discussion_r1148720025
##########
client-spark/spark-3/src/main/java/org/apache/spark/shuffle/celeborn/RssShuffleManager.java:
##########
@@ -155,9 +155,11 @@ public ShuffleBlockResolver shuffleBlockResolver() {
public void stop() {
if (rssShuffleClient != null) {
rssShuffleClient.shutdown();
+ ShuffleClient.reset();
}
if (lifecycleManager != null) {
lifecycleManager.stop();
+ lifecycleManager = null;
}
if (sortShuffleManager() != null) {
sortShuffleManager().stop();
Review Comment:
Also assign sortShuffleManager as null>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] AngersZhuuuu commented on pull request #1379: [CELEBORN-472] Support using Celeborn in the scenario of switching multiple SparkSessions in the same process
Posted by "AngersZhuuuu (via GitHub)" <gi...@apache.org>.
AngersZhuuuu commented on PR #1379:
URL: https://github.com/apache/incubator-celeborn/pull/1379#issuecomment-1482449468
I think we can have a UT for this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] pan3793 commented on pull request #1379: [CELEBORN-472] Support using Celeborn in the scenario of switching multiple SparkSessions in the same process
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on PR #1379:
URL: https://github.com/apache/incubator-celeborn/pull/1379#issuecomment-1482453696
The stacktrace only has one line? Any more background why it occurs?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] kerwin-zk commented on pull request #1379: [CELEBORN-472] Support using Celeborn in the scenario of switching multiple SparkSessions in the same process
Posted by "kerwin-zk (via GitHub)" <gi...@apache.org>.
kerwin-zk commented on PR #1379:
URL: https://github.com/apache/incubator-celeborn/pull/1379#issuecomment-1482497502
> I think we can have a UT for this?
ok.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] kerwin-zk commented on pull request #1379: [CELEBORN-472] Support using Celeborn in the scenario of switching multiple SparkContexts in the same process
Posted by "kerwin-zk (via GitHub)" <gi...@apache.org>.
kerwin-zk commented on PR #1379:
URL: https://github.com/apache/incubator-celeborn/pull/1379#issuecomment-1484585162
> @kerwin-zk looks like you did not setup apache account properly.
@pan3793 done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org