You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@uniffle.apache.org by "EnricoMi (via GitHub)" <gi...@apache.org> on 2023/11/15 08:54:01 UTC
[PR] [#1316] improvement: detect OutpuTracker API version via Spark version [incubator-uniffle]
EnricoMi opened a new pull request, #1317:
URL: https://github.com/apache/incubator-uniffle/pull/1317
### What changes were proposed in this pull request?
Uses the spark version to decide which OutputTracker API to call.
### Why are the changes needed?
Currently, it calls into one possible API version by reflection, and if this fails, calls into another version. Knowing the actual Spark version allows to call into the right API version.
Fix: #1316
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
No tests added.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [PR] [#1316] improvement: detect OutputTracker API version via Spark version [incubator-uniffle]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #1317:
URL: https://github.com/apache/incubator-uniffle/pull/1317#issuecomment-1949300812
## Test Results
2 429 files ±0 2 429 suites ±0 4h 40m 58s :stopwatch: + 1m 18s
820 tests +1 819 :white_check_mark: +1 1 :zzz: ±0 0 :x: ±0
9 720 runs +7 9 706 :white_check_mark: +7 14 :zzz: ±0 0 :x: ±0
Results for commit 07c05117. ± Comparison against base commit d120f4bf.
[test-results]:data:application/gzip;base64,H4sIAKzHz2UC/03MQQ6DIBCF4asY1l0AEYfpZRpBTEhVGoSV6d07NKIs3wfzH2z2i9vZs5O9xEfH9uxTu6ccx+TDRiIGrTQRvafyQ0te12vP1hYSeNPbf8rVBfPoF4L7yMUY4ikxb6WJIOuqTQQ+XHQ2+wpN87/bpA3r6hMNxsFyJQQgqMnO2oIyBiYcHaVNTzZo5FI59v0Bdgsb/hABAAA=
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [PR] [#1316] improvement: detect OutputTracker API version via Spark version [incubator-uniffle]
Posted by "EnricoMi (via GitHub)" <gi...@apache.org>.
EnricoMi commented on PR #1317:
URL: https://github.com/apache/incubator-uniffle/pull/1317#issuecomment-1812056886
Looks like `package$.MODULE$.SPARK_VERSION_SHORT` was introduced in Spark 3.0.0, so this does not work with Spark 2.x, which is EOL anyway.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [PR] [#1316] improvement: detect OutputTracker API version via Spark version [incubator-uniffle]
Posted by "EnricoMi (via GitHub)" <gi...@apache.org>.
EnricoMi commented on PR #1317:
URL: https://github.com/apache/incubator-uniffle/pull/1317#issuecomment-1817491928
> We have users who use Spark2
Sad thing, they are missing out on performance... Fixed anyway.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [PR] [#1316] improvement: detect OutputTracker API version via Spark version [incubator-uniffle]
Posted by "zuston (via GitHub)" <gi...@apache.org>.
zuston merged PR #1317:
URL: https://github.com/apache/incubator-uniffle/pull/1317
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [PR] [#1316] improvement: detect OutputTracker API version via Spark version [incubator-uniffle]
Posted by "zuston (via GitHub)" <gi...@apache.org>.
zuston commented on PR #1317:
URL: https://github.com/apache/incubator-uniffle/pull/1317#issuecomment-1953700519
Thanks for your contribution! @EnricoMi Merged/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [PR] [#1316] improvement: detect OutputTracker API version via Spark version [incubator-uniffle]
Posted by "zuston (via GitHub)" <gi...@apache.org>.
zuston commented on PR #1317:
URL: https://github.com/apache/incubator-uniffle/pull/1317#issuecomment-1813719267
Could you help review this? @summaryzb I think you will have more experience.
And the code style Ci could be fixed by running the command of `mvn spotless:apply -Pspark3 -Pspark2 -Ptez -Pmr -Phadoop2.8`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [PR] [#1316] improvement: detect OutputTracker API version via Spark version [incubator-uniffle]
Posted by "jerqi (via GitHub)" <gi...@apache.org>.
jerqi commented on PR #1317:
URL: https://github.com/apache/incubator-uniffle/pull/1317#issuecomment-1818105500
```
Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.7.0:compile (default-compile) on project rss-client-spark-common: Compilation failure
Error: /home/runner/work/incubator-uniffle/incubator-uniffle/client-spark/common/src/main/java/org/apache/spark/shuffle/SparkVersionUtils.java:[25,67] error: cannot find symbol
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [PR] [#1316] improvement: detect OutputTracker API version via Spark version [incubator-uniffle]
Posted by "EnricoMi (via GitHub)" <gi...@apache.org>.
EnricoMi commented on PR #1317:
URL: https://github.com/apache/incubator-uniffle/pull/1317#issuecomment-1816479086
> For Spark 2.3, there are errors as below
This is not compatible with Spark2. I presume, Spark2 support is about to be dropped soon.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [PR] [#1316] improvement: detect OutputTracker API version via Spark version [incubator-uniffle]
Posted by "EnricoMi (via GitHub)" <gi...@apache.org>.
EnricoMi commented on PR #1317:
URL: https://github.com/apache/incubator-uniffle/pull/1317#issuecomment-1815944413
> the code style Ci could be fixed by ...
Fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [PR] [#1316] improvement: detect OutputTracker API version via Spark version [incubator-uniffle]
Posted by "advancedxy (via GitHub)" <gi...@apache.org>.
advancedxy commented on code in PR #1317:
URL: https://github.com/apache/incubator-uniffle/pull/1317#discussion_r1398619339
##########
client-spark/common/src/main/java/org/apache/spark/shuffle/SparkVersionUtils.java:
##########
@@ -22,6 +22,7 @@
public class SparkVersionUtils {
public static final String SPARK_VERSION = package$.MODULE$.SPARK_VERSION();
+ public static final String SPARK_VERSION_SHORT = package$.MODULE$.SPARK_VERSION_SHORT();
Review Comment:
For Spark 2, I don't think we can access `SPARK_VERSION_SHORT`.
That's why the `MINOR_VERSION` exists.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [PR] [#1316] improvement: detect OutputTracker API version via Spark version [incubator-uniffle]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #1317:
URL: https://github.com/apache/incubator-uniffle/pull/1317#issuecomment-1812137053
## [Codecov](https://app.codecov.io/gh/apache/incubator-uniffle/pull/1317?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
All modified and coverable lines are covered by tests :white_check_mark:
> Comparison is base [(`19a2048`)](https://app.codecov.io/gh/apache/incubator-uniffle/commit/19a20484ffa9a6849748d76c85246de077b8b881?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) 53.65% compared to head [(`45f0a73`)](https://app.codecov.io/gh/apache/incubator-uniffle/pull/1317?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) 53.81%.
> Report is 9 commits behind head on master.
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #1317 +/- ##
============================================
+ Coverage 53.65% 53.81% +0.15%
+ Complexity 2689 2166 -523
============================================
Files 403 334 -69
Lines 23456 15088 -8368
Branches 1992 1389 -603
============================================
- Hits 12585 8119 -4466
+ Misses 10096 6516 -3580
+ Partials 775 453 -322
```
</details>
[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/incubator-uniffle/pull/1317?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [PR] [#1316] improvement: detect OutputTracker API version via Spark version [incubator-uniffle]
Posted by "jerqi (via GitHub)" <gi...@apache.org>.
jerqi commented on PR #1317:
URL: https://github.com/apache/incubator-uniffle/pull/1317#issuecomment-1816003091
For Spark 2.3, there are errors as below
```
Error: COMPILATION ERROR :
[INFO] -------------------------------------------------------------
Error: /home/runner/work/incubator-uniffle/incubator-uniffle/client-spark/common/src/main/java/org/apache/spark/shuffle/SparkVersionUtils.java:[25,67] error: cannot find symbol
symbol: method SPARK_VERSION_SHORT()
location: variable MODULE$ of type package$
```
For Spark 3.2.0, there are errors as below
```
rg.apache.uniffle.common.exception.RssException: java.lang.NoSuchMethodException: org.apache.spark.MapOutputTrackerMaster.getMapSizesByExecutorId(int, int, int, int, int)
at org.apache.spark.shuffle.RssShuffleManager.getExpectedTasksByExecutorId(RssShuffleManager.java:764)
at org.apache.spark.shuffle.RssShuffleManager.getReader(RssShuffleManager.java:549)
at org.apache.spark.shuffle.RssShuffleManager.getReader(RssShuffleManager.java:535)
at org.apach
```
The spotbugs also failed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [PR] [#1316] improvement: detect OutputTracker API version via Spark version [incubator-uniffle]
Posted by "jerqi (via GitHub)" <gi...@apache.org>.
jerqi commented on PR #1317:
URL: https://github.com/apache/incubator-uniffle/pull/1317#issuecomment-1816604512
> > For Spark 2.3, there are errors as below
>
> This is not compatible with Spark2. I presume, Spark2 support is about to be dropped soon.
We have users who use Spark2, so we can't drop them.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org