You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@uniffle.apache.org by "rickyma (via GitHub)" <gi...@apache.org> on 2024/03/07 09:51:19 UTC
[PR] [#133][FOLLOWUP] fix(netty): Let Spark use its own NettyUtils [incubator-uniffle]
rickyma opened a new pull request, #1565:
URL: https://github.com/apache/incubator-uniffle/pull/1565
### What changes were proposed in this pull request?
Delete `org.apache.spark.network.util.NettyUtils`.
### Why are the changes needed?
It's a followup PR for https://github.com/apache/incubator-uniffle/pull/727.
When running in Spark 2.4, we will encounter exceptions as below:
24/03/07 16:34:37 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(tdwadmin); groups with view permissions: Set(); users with modify permissions: Set(tdwadmin); groups with modify permissions: Set()
Exception in thread "main" java.lang.NoSuchMethodError: org.apache.spark.network.util.NettyUtils.createEventLoop(Lorg/apache/spark/network/util/IOMode;ILjava/lang/String;)Lio/netty/channel/EventLoopGroup;
at org.apache.spark.network.client.TransportClientFactory.<init>(TransportClientFactory.java:104)
at org.apache.spark.network.TransportContext.createClientFactory(TransportContext.java:89)
at org.apache.spark.rpc.netty.NettyRpcEnv.<init>(NettyRpcEnv.scala:70)
at org.apache.spark.rpc.netty.NettyRpcEnvFactory.create(NettyRpcEnv.scala:449)
at org.apache.spark.rpc.RpcEnv$.create(RpcEnv.scala:56)
at org.apache.spark.SparkEnv$.create(SparkEnv.scala:264)
at org.apache.spark.SparkEnv$.createDriverEnv(SparkEnv.scala:193)
at org.apache.spark.SparkContext.createSparkEnv(SparkContext.scala:271)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:474)
at org.apache.spark.deploy.yarn.SQLApplicationMaster.<init>(SQLApplicationMaster.scala:96)
at org.apache.spark.deploy.yarn.SQLApplicationMaster.<init>(SQLApplicationMaster.scala:53)
at org.apache.spark.deploy.yarn.SQLApplicationMaster$$anonfun$main$1.apply$mcV$sp(SQLApplicationMaster.scala:544)
at org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:61)
at org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:60)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:2286)
at org.apache.spark.deploy.SparkHadoopUtil.runAsSparkUser(SparkHadoopUtil.scala:60)
at org.apache.spark.deploy.yarn.SQLApplicationMaster$.main(SQLApplicationMaster.scala:543)
at org.apache.spark.deploy.yarn.SQLApplicationMaster.main(SQLApplicationMaster.scala)
We should let Spark use its own NettyUtils instead of ours.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Existing UTs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [PR] [#133][FOLLOWUP] fix(netty): Let Spark use its own NettyUtils [incubator-uniffle]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #1565:
URL: https://github.com/apache/incubator-uniffle/pull/1565#issuecomment-1983184222
## Test Results
2 326 files ±0 2 326 suites ±0 4h 21m 23s :stopwatch: - 8m 3s
851 tests ±0 824 :white_check_mark: - 25 2 :zzz: ±0 0 :x: ±0 25 :fire: +25
10 018 runs ±0 9 970 :white_check_mark: - 25 23 :zzz: ±0 0 :x: ±0 25 :fire: +25
For more details on these errors, see [this check](https://github.com/apache/incubator-uniffle/runs/22384824015).
Results for commit fe157222. ± Comparison against base commit d994b272.
[test-results]:data:application/gzip;base64,H4sIAICT6WUC/1XMSw7DIAxF0a1EjDswED7uZqqIxBJqPhWBUdS9F6pA0+E7tu7ByM/Tzu6dkELfOrYnH697TGGIfluzcKWtzJTvsXxYxet67Mm5QqL/0dO/SqcBDX7OAA2mELZQXlSmkNYS5QDcnrNWEQ00OquywqX63X9Rty2Lj3kxmrgyQgg0AzqyowOpkXptRtCOqJeW0AJq9v4A/Bc06xMBAAA=
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [PR] [#133][FOLLOWUP] fix(netty): Let Spark use its own NettyUtils [incubator-uniffle]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #1565:
URL: https://github.com/apache/incubator-uniffle/pull/1565#issuecomment-1983151299
## [Codecov](https://app.codecov.io/gh/apache/incubator-uniffle/pull/1565?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
All modified and coverable lines are covered by tests :white_check_mark:
> Project coverage is 53.89%. Comparing base [(`d994b27`)](https://app.codecov.io/gh/apache/incubator-uniffle/commit/d994b272b5a8ee30155aac6d529857f044846c57?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) to head [(`fe15722`)](https://app.codecov.io/gh/apache/incubator-uniffle/pull/1565?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #1565 +/- ##
============================================
+ Coverage 53.72% 53.89% +0.17%
+ Complexity 2838 2290 -548
============================================
Files 437 359 -78
Lines 24663 16022 -8641
Branches 2094 1461 -633
============================================
- Hits 13250 8635 -4615
+ Misses 10581 6890 -3691
+ Partials 832 497 -335
```
</details>
[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/incubator-uniffle/pull/1565?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [PR] [#133][FOLLOWUP] fix(spark): Let Spark2 use its own NettyUtils [incubator-uniffle]
Posted by "jerqi (via GitHub)" <gi...@apache.org>.
jerqi commented on code in PR #1565:
URL: https://github.com/apache/incubator-uniffle/pull/1565#discussion_r1517138062
##########
client-spark/spark2-shaded/pom.xml:
##########
@@ -213,6 +213,8 @@
to="lib${rss.shade.native.packageName}_netty_transport_native_kqueue_aarch_64.jnilib"
type="glob"></mapper>
</move>
+ <!-- Delete NettyUtils, let Spark use its own NettyUtils -->
Review Comment:
Add the comment to xxplain why we do this.
##########
client-spark/spark2-shaded/pom.xml:
##########
@@ -213,6 +213,8 @@
to="lib${rss.shade.native.packageName}_netty_transport_native_kqueue_aarch_64.jnilib"
type="glob"></mapper>
</move>
+ <!-- Delete NettyUtils, let Spark use its own NettyUtils -->
Review Comment:
Add the comment to explain why we do this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [PR] [#1567] fix(spark): Let Spark use its own NettyUtils [incubator-uniffle]
Posted by "jerqi (via GitHub)" <gi...@apache.org>.
jerqi merged PR #1565:
URL: https://github.com/apache/incubator-uniffle/pull/1565
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [PR] [#133][FOLLOWUP] fix(spark): Let Spark2 use its own NettyUtils [incubator-uniffle]
Posted by "rickyma (via GitHub)" <gi...@apache.org>.
rickyma commented on code in PR #1565:
URL: https://github.com/apache/incubator-uniffle/pull/1565#discussion_r1517176801
##########
client-spark/spark2-shaded/pom.xml:
##########
@@ -213,6 +213,8 @@
to="lib${rss.shade.native.packageName}_netty_transport_native_kqueue_aarch_64.jnilib"
type="glob"></mapper>
</move>
+ <!-- Delete NettyUtils, let Spark use its own NettyUtils -->
Review Comment:
Done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [PR] [#133][FOLLOWUP] fix(netty): Let Spark2 use its own NettyUtils [incubator-uniffle]
Posted by "rickyma (via GitHub)" <gi...@apache.org>.
rickyma commented on PR #1565:
URL: https://github.com/apache/incubator-uniffle/pull/1565#issuecomment-1983825169
@jerqi @zuston PTAL.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [PR] [#133][FOLLOWUP] fix(spark): Let Spark2 use its own NettyUtils [incubator-uniffle]
Posted by "rickyma (via GitHub)" <gi...@apache.org>.
rickyma closed pull request #1565: [#133][FOLLOWUP] fix(spark): Let Spark2 use its own NettyUtils
URL: https://github.com/apache/incubator-uniffle/pull/1565
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [PR] [#133][FOLLOWUP] fix(spark): Let Spark2 use its own NettyUtils [incubator-uniffle]
Posted by "jerqi (via GitHub)" <gi...@apache.org>.
jerqi commented on code in PR #1565:
URL: https://github.com/apache/incubator-uniffle/pull/1565#discussion_r1517138062
##########
client-spark/spark2-shaded/pom.xml:
##########
@@ -213,6 +213,8 @@
to="lib${rss.shade.native.packageName}_netty_transport_native_kqueue_aarch_64.jnilib"
type="glob"></mapper>
</move>
+ <!-- Delete NettyUtils, let Spark use its own NettyUtils -->
Review Comment:
Explain why we do this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org