You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@uniffle.apache.org by "EnricoMi (via GitHub)" <gi...@apache.org> on 2023/11/15 08:54:01 UTC

[PR] [#1316] improvement: detect OutpuTracker API version via Spark version [incubator-uniffle]

EnricoMi opened a new pull request, #1317:
URL: https://github.com/apache/incubator-uniffle/pull/1317

   ### What changes were proposed in this pull request?
   
   Uses the spark version to decide which OutputTracker API to call.
   
   ### Why are the changes needed?
   
   Currently, it calls into one possible API version by reflection, and if this fails, calls into another version. Knowing the actual Spark version allows to call into the right API version.
   
   Fix: #1316
   
   ### Does this PR introduce _any_ user-facing change?
   
   No.
   
   ### How was this patch tested?
   
   No tests added.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


Re: [PR] [#1316] improvement: detect OutputTracker API version via Spark version [incubator-uniffle]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #1317:
URL: https://github.com/apache/incubator-uniffle/pull/1317#issuecomment-1949300812

   ## Test Results
   2 429 files  ±0  2 429 suites  ±0   4h 40m 58s :stopwatch: + 1m 18s
     820 tests +1    819 :white_check_mark: +1   1 :zzz: ±0  0 :x: ±0 
   9 720 runs  +7  9 706 :white_check_mark: +7  14 :zzz: ±0  0 :x: ±0 
   
   Results for commit 07c05117. ± Comparison against base commit d120f4bf.
   
   [test-results]:data:application/gzip;base64,H4sIAKzHz2UC/03MQQ6DIBCF4asY1l0AEYfpZRpBTEhVGoSV6d07NKIs3wfzH2z2i9vZs5O9xEfH9uxTu6ccx+TDRiIGrTQRvafyQ0te12vP1hYSeNPbf8rVBfPoF4L7yMUY4ikxb6WJIOuqTQQ+XHQ2+wpN87/bpA3r6hMNxsFyJQQgqMnO2oIyBiYcHaVNTzZo5FI59v0Bdgsb/hABAAA=
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


Re: [PR] [#1316] improvement: detect OutputTracker API version via Spark version [incubator-uniffle]

Posted by "EnricoMi (via GitHub)" <gi...@apache.org>.
EnricoMi commented on PR #1317:
URL: https://github.com/apache/incubator-uniffle/pull/1317#issuecomment-1812056886

   Looks like `package$.MODULE$.SPARK_VERSION_SHORT` was introduced in Spark 3.0.0, so this does not work with Spark 2.x, which is EOL anyway.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


Re: [PR] [#1316] improvement: detect OutputTracker API version via Spark version [incubator-uniffle]

Posted by "EnricoMi (via GitHub)" <gi...@apache.org>.
EnricoMi commented on PR #1317:
URL: https://github.com/apache/incubator-uniffle/pull/1317#issuecomment-1817491928

   > We have users who use Spark2
   
   Sad thing, they are missing out on performance... Fixed anyway.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


Re: [PR] [#1316] improvement: detect OutputTracker API version via Spark version [incubator-uniffle]

Posted by "zuston (via GitHub)" <gi...@apache.org>.
zuston merged PR #1317:
URL: https://github.com/apache/incubator-uniffle/pull/1317


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


Re: [PR] [#1316] improvement: detect OutputTracker API version via Spark version [incubator-uniffle]

Posted by "zuston (via GitHub)" <gi...@apache.org>.
zuston commented on PR #1317:
URL: https://github.com/apache/incubator-uniffle/pull/1317#issuecomment-1953700519

   Thanks for your contribution! @EnricoMi Merged/


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


Re: [PR] [#1316] improvement: detect OutputTracker API version via Spark version [incubator-uniffle]

Posted by "zuston (via GitHub)" <gi...@apache.org>.
zuston commented on PR #1317:
URL: https://github.com/apache/incubator-uniffle/pull/1317#issuecomment-1813719267

   Could you help review this? @summaryzb I think you will have more experience. 
   
   And the code style Ci could be fixed by running the command of `mvn spotless:apply -Pspark3 -Pspark2 -Ptez -Pmr -Phadoop2.8`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


Re: [PR] [#1316] improvement: detect OutputTracker API version via Spark version [incubator-uniffle]

Posted by "jerqi (via GitHub)" <gi...@apache.org>.
jerqi commented on PR #1317:
URL: https://github.com/apache/incubator-uniffle/pull/1317#issuecomment-1818105500

   ``` 
   Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.7.0:compile (default-compile) on project rss-client-spark-common: Compilation failure
   Error:  /home/runner/work/incubator-uniffle/incubator-uniffle/client-spark/common/src/main/java/org/apache/spark/shuffle/SparkVersionUtils.java:[25,67] error: cannot find symbol
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


Re: [PR] [#1316] improvement: detect OutputTracker API version via Spark version [incubator-uniffle]

Posted by "EnricoMi (via GitHub)" <gi...@apache.org>.
EnricoMi commented on PR #1317:
URL: https://github.com/apache/incubator-uniffle/pull/1317#issuecomment-1816479086

   > For Spark 2.3, there are errors as below
   
   This is not compatible with Spark2. I presume, Spark2 support is about to be dropped soon.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


Re: [PR] [#1316] improvement: detect OutputTracker API version via Spark version [incubator-uniffle]

Posted by "EnricoMi (via GitHub)" <gi...@apache.org>.
EnricoMi commented on PR #1317:
URL: https://github.com/apache/incubator-uniffle/pull/1317#issuecomment-1815944413

   > the code style Ci could be fixed by ...
   
   Fixed.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


Re: [PR] [#1316] improvement: detect OutputTracker API version via Spark version [incubator-uniffle]

Posted by "advancedxy (via GitHub)" <gi...@apache.org>.
advancedxy commented on code in PR #1317:
URL: https://github.com/apache/incubator-uniffle/pull/1317#discussion_r1398619339


##########
client-spark/common/src/main/java/org/apache/spark/shuffle/SparkVersionUtils.java:
##########
@@ -22,6 +22,7 @@
 
 public class SparkVersionUtils {
   public static final String SPARK_VERSION = package$.MODULE$.SPARK_VERSION();
+  public static final String SPARK_VERSION_SHORT = package$.MODULE$.SPARK_VERSION_SHORT();

Review Comment:
   For Spark 2, I don't think we can access `SPARK_VERSION_SHORT`.
   
   That's why the `MINOR_VERSION` exists. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


Re: [PR] [#1316] improvement: detect OutputTracker API version via Spark version [incubator-uniffle]

Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #1317:
URL: https://github.com/apache/incubator-uniffle/pull/1317#issuecomment-1812137053

   ## [Codecov](https://app.codecov.io/gh/apache/incubator-uniffle/pull/1317?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
   All modified and coverable lines are covered by tests :white_check_mark:
   > Comparison is base [(`19a2048`)](https://app.codecov.io/gh/apache/incubator-uniffle/commit/19a20484ffa9a6849748d76c85246de077b8b881?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) 53.65% compared to head [(`45f0a73`)](https://app.codecov.io/gh/apache/incubator-uniffle/pull/1317?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) 53.81%.
   > Report is 9 commits behind head on master.
   
   
   <details><summary>Additional details and impacted files</summary>
   
   
   ```diff
   @@             Coverage Diff              @@
   ##             master    #1317      +/-   ##
   ============================================
   + Coverage     53.65%   53.81%   +0.15%     
   + Complexity     2689     2166     -523     
   ============================================
     Files           403      334      -69     
     Lines         23456    15088    -8368     
     Branches       1992     1389     -603     
   ============================================
   - Hits          12585     8119    -4466     
   + Misses        10096     6516    -3580     
   + Partials        775      453     -322     
   ```
   
   
   
   </details>
   
   [:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/incubator-uniffle/pull/1317?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).   
   :loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


Re: [PR] [#1316] improvement: detect OutputTracker API version via Spark version [incubator-uniffle]

Posted by "jerqi (via GitHub)" <gi...@apache.org>.
jerqi commented on PR #1317:
URL: https://github.com/apache/incubator-uniffle/pull/1317#issuecomment-1816003091

   For Spark 2.3, there are errors as below
   ```
   Error:  COMPILATION ERROR : 
   [INFO] -------------------------------------------------------------
   Error:  /home/runner/work/incubator-uniffle/incubator-uniffle/client-spark/common/src/main/java/org/apache/spark/shuffle/SparkVersionUtils.java:[25,67] error: cannot find symbol
     symbol:   method SPARK_VERSION_SHORT()
     location: variable MODULE$ of type package$
   ```
   For Spark 3.2.0, there are errors as below
   ```
   rg.apache.uniffle.common.exception.RssException: java.lang.NoSuchMethodException: org.apache.spark.MapOutputTrackerMaster.getMapSizesByExecutorId(int, int, int, int, int)
   	at org.apache.spark.shuffle.RssShuffleManager.getExpectedTasksByExecutorId(RssShuffleManager.java:764)
   	at org.apache.spark.shuffle.RssShuffleManager.getReader(RssShuffleManager.java:549)
   	at org.apache.spark.shuffle.RssShuffleManager.getReader(RssShuffleManager.java:535)
   	at org.apach
   ```
   The spotbugs also failed.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


Re: [PR] [#1316] improvement: detect OutputTracker API version via Spark version [incubator-uniffle]

Posted by "jerqi (via GitHub)" <gi...@apache.org>.
jerqi commented on PR #1317:
URL: https://github.com/apache/incubator-uniffle/pull/1317#issuecomment-1816604512

   > > For Spark 2.3, there are errors as below
   > 
   > This is not compatible with Spark2. I presume, Spark2 support is about to be dropped soon.
   
   
   We have users who use Spark2, so we can't drop them.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org