You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@celeborn.apache.org by "kerwin-zk (via GitHub)" <gi...@apache.org> on 2023/03/24 08:18:29 UTC

[GitHub] [incubator-celeborn] kerwin-zk opened a new pull request, #1379: [CELEBORN-472] Support using Celeborn in the scenario of switching multiple SparkSessions in the same process

kerwin-zk opened a new pull request, #1379:
URL: https://github.com/apache/incubator-celeborn/pull/1379

   ### What changes were proposed in this pull request?
   Support using Celeborn in the scenario of switching multiple SparkSessions in the same process.
   
   
   ### Why are the changes needed?
   In the current celeborn, if you switch the SparkSession in the Spark jar package application, the following exception will occur.
   ```
   org.apache.celeborn.shaded.io.netty.channel.AbstractChannel - Force-closing a channel whose registration task was not accepted by an event loop: [id: 0xca6cd07d]
   java.util.concurrent.RejectedExecutionException: event executor terminated
   	at org.apache.celeborn.shaded.io.netty.util.concurrent.SingleThreadEventExecutor.reject(SingleThreadEventExecutor.java:932)
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [incubator-celeborn] kerwin-zk commented on pull request #1379: [CELEBORN-472] Support using Celeborn in the scenario of switching multiple SparkContexts in the same process

Posted by "kerwin-zk (via GitHub)" <gi...@apache.org>.
kerwin-zk commented on PR #1379:
URL: https://github.com/apache/incubator-celeborn/pull/1379#issuecomment-1482529952

   > > Here instead of using multiple SparkContexts at the same time, but to use only one at a time, close it after use, and then create a new one.
   > 
   > that makes sense, please clarify it in PR description
   
   @pan3793 done.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [incubator-celeborn] pan3793 commented on pull request #1379: [CELEBORN-472] Support using Celeborn in the scenario of switching multiple SparkSessions in the same process

Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on PR #1379:
URL: https://github.com/apache/incubator-celeborn/pull/1379#issuecomment-1482478855

   > Support using Celeborn in the scenario of switching multiple SparkSessions in the same process.
   
   I don't get your point, or you are saying "multiple `SparkContext`s in the same process." instead of "`SparkSession`s"


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [incubator-celeborn] waitinfuture merged pull request #1379: [CELEBORN-472] Support using Celeborn in the scenario of switching multiple SparkContexts in the same process

Posted by "waitinfuture (via GitHub)" <gi...@apache.org>.
waitinfuture merged PR #1379:
URL: https://github.com/apache/incubator-celeborn/pull/1379


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [incubator-celeborn] kerwin-zk commented on pull request #1379: [CELEBORN-472] Support using Celeborn in the scenario of switching multiple SparkSessions in the same process

Posted by "kerwin-zk (via GitHub)" <gi...@apache.org>.
kerwin-zk commented on PR #1379:
URL: https://github.com/apache/incubator-celeborn/pull/1379#issuecomment-1482497228

   > multiple
   
   @pan3793 Yes. When using different SparkSessions(Essentially SparkContext) in one process, you need to close one and use another one. After closing a SparkSession, RssShuffleManager will call the stop method to close the singleton shuffleclient. When the next SparkSession uses shuffleclient again in same process. The shuffleclient was closed, causing the problem.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [incubator-celeborn] kerwin-zk commented on pull request #1379: [CELEBORN-472] Support using Celeborn in the scenario of switching multiple SparkSessions in the same process

Posted by "kerwin-zk (via GitHub)" <gi...@apache.org>.
kerwin-zk commented on PR #1379:
URL: https://github.com/apache/incubator-celeborn/pull/1379#issuecomment-1482512287

   > Multiple Spark contexts are discouraged and totally disallowed since Spark 3.0 [SPARK-26362](https://issues.apache.org/jira/browse/SPARK-26362).
   
   @pan3793  
   Here instead of using multiple SparkContexts at the same time, but to use only one at a time, close it after use, and then create a new one.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [incubator-celeborn] pan3793 commented on pull request #1379: [CELEBORN-472] Support using Celeborn in the scenario of switching multiple SparkContexts in the same process

Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on PR #1379:
URL: https://github.com/apache/incubator-celeborn/pull/1379#issuecomment-1486283320

   This is kinda resource leak, back ported to branch-0.2


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [incubator-celeborn] codecov[bot] commented on pull request #1379: [CELEBORN-472] Support using Celeborn in the scenario of switching multiple SparkContexts in the same process

Posted by "codecov[bot] (via GitHub)" <gi...@apache.org>.
codecov[bot] commented on PR #1379:
URL: https://github.com/apache/incubator-celeborn/pull/1379#issuecomment-1482533298

   ## [Codecov](https://codecov.io/gh/apache/incubator-celeborn/pull/1379?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
   > Merging [#1379](https://codecov.io/gh/apache/incubator-celeborn/pull/1379?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (b55276c) into [main](https://codecov.io/gh/apache/incubator-celeborn/commit/2bfa7e896591eed0bc1021002728a01e842ed013?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (2bfa7e8) will **increase** coverage by `0.05%`.
   > The diff coverage is `n/a`.
   
   ```diff
   @@             Coverage Diff              @@
   ##               main    #1379      +/-   ##
   ============================================
   + Coverage     29.80%   29.84%   +0.05%     
   - Complexity      909      912       +3     
   ============================================
     Files           222      222              
     Lines         18489    18487       -2     
     Branches       1970     1970              
   ============================================
   + Hits           5508     5516       +8     
   + Misses        12587    12581       -6     
   + Partials        394      390       -4     
   ```
   
   
   [see 4 files with indirect coverage changes](https://codecov.io/gh/apache/incubator-celeborn/pull/1379/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   
   :mega: We’re building smart automated test selection to slash your CI/CD build times. [Learn more](https://about.codecov.io/iterative-testing/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [incubator-celeborn] pan3793 commented on pull request #1379: [CELEBORN-472] Support using Celeborn in the scenario of switching multiple SparkSessions in the same process

Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on PR #1379:
URL: https://github.com/apache/incubator-celeborn/pull/1379#issuecomment-1482514409

   > Here instead of using multiple SparkContexts at the same time, but to use only one at a time, close it after use, and then create a new one.
   
   that makes sense, please clarify it in PR description


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [incubator-celeborn] AngersZhuuuu commented on pull request #1379: [CELEBORN-472] Support using Celeborn in the scenario of switching multiple SparkSessions in the same process

Posted by "AngersZhuuuu (via GitHub)" <gi...@apache.org>.
AngersZhuuuu commented on PR #1379:
URL: https://github.com/apache/incubator-celeborn/pull/1379#issuecomment-1482506630

   > Multiple Spark contexts are discouraged and totally disallowed since Spark 3.0 [SPARK-26362](https://issues.apache.org/jira/browse/SPARK-26362).
   
   But. this fix still looks like reasonable


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [incubator-celeborn] pan3793 commented on pull request #1379: [CELEBORN-472] Support using Celeborn in the scenario of switching multiple SparkSessions in the same process

Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on PR #1379:
URL: https://github.com/apache/incubator-celeborn/pull/1379#issuecomment-1482508719

   @AngersZhuuuu is right. The change itself LGTM, it's kinda resource cleanup.
   
   `SparkContext` and `SparkSession` are indeed different things, please update the title and description.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [incubator-celeborn] pan3793 commented on pull request #1379: [CELEBORN-472] Support using Celeborn in the scenario of switching multiple SparkContexts in the same process

Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on PR #1379:
URL: https://github.com/apache/incubator-celeborn/pull/1379#issuecomment-1484431463

   @kerwin-zk looks like you did not setup apache account properly.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [incubator-celeborn] pan3793 commented on pull request #1379: [CELEBORN-472] Support using Celeborn in the scenario of switching multiple SparkSessions in the same process

Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on PR #1379:
URL: https://github.com/apache/incubator-celeborn/pull/1379#issuecomment-1482501243

   Multiple Spark contexts are discouraged and totally disallowed since Spark 3.0 [SPARK-26362](https://issues.apache.org/jira/browse/SPARK-26362).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [incubator-celeborn] kerwin-zk commented on pull request #1379: [CELEBORN-472] Support using Celeborn in the scenario of switching multiple SparkSessions in the same process

Posted by "kerwin-zk (via GitHub)" <gi...@apache.org>.
kerwin-zk commented on PR #1379:
URL: https://github.com/apache/incubator-celeborn/pull/1379#issuecomment-1482497709

   > The stacktrace only has one line? Any more background why it occurs?
   
   done.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [incubator-celeborn] AngersZhuuuu commented on a diff in pull request #1379: [CELEBORN-472] Support using Celeborn in the scenario of switching multiple SparkContexts in the same process

Posted by "AngersZhuuuu (via GitHub)" <gi...@apache.org>.
AngersZhuuuu commented on code in PR #1379:
URL: https://github.com/apache/incubator-celeborn/pull/1379#discussion_r1148720025


##########
client-spark/spark-3/src/main/java/org/apache/spark/shuffle/celeborn/RssShuffleManager.java:
##########
@@ -155,9 +155,11 @@ public ShuffleBlockResolver shuffleBlockResolver() {
   public void stop() {
     if (rssShuffleClient != null) {
       rssShuffleClient.shutdown();
+      ShuffleClient.reset();
     }
     if (lifecycleManager != null) {
       lifecycleManager.stop();
+      lifecycleManager = null;
     }
     if (sortShuffleManager() != null) {
       sortShuffleManager().stop();

Review Comment:
   Also assign sortShuffleManager as null>



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [incubator-celeborn] AngersZhuuuu commented on pull request #1379: [CELEBORN-472] Support using Celeborn in the scenario of switching multiple SparkSessions in the same process

Posted by "AngersZhuuuu (via GitHub)" <gi...@apache.org>.
AngersZhuuuu commented on PR #1379:
URL: https://github.com/apache/incubator-celeborn/pull/1379#issuecomment-1482449468

   I think we can have a UT for this?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [incubator-celeborn] pan3793 commented on pull request #1379: [CELEBORN-472] Support using Celeborn in the scenario of switching multiple SparkSessions in the same process

Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on PR #1379:
URL: https://github.com/apache/incubator-celeborn/pull/1379#issuecomment-1482453696

   The stacktrace only has one line? Any more background why it occurs?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [incubator-celeborn] kerwin-zk commented on pull request #1379: [CELEBORN-472] Support using Celeborn in the scenario of switching multiple SparkSessions in the same process

Posted by "kerwin-zk (via GitHub)" <gi...@apache.org>.
kerwin-zk commented on PR #1379:
URL: https://github.com/apache/incubator-celeborn/pull/1379#issuecomment-1482497502

   > I think we can have a UT for this?
   
   ok.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [incubator-celeborn] kerwin-zk commented on pull request #1379: [CELEBORN-472] Support using Celeborn in the scenario of switching multiple SparkContexts in the same process

Posted by "kerwin-zk (via GitHub)" <gi...@apache.org>.
kerwin-zk commented on PR #1379:
URL: https://github.com/apache/incubator-celeborn/pull/1379#issuecomment-1484585162

   > @kerwin-zk looks like you did not setup apache account properly.
   
   @pan3793  done.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org