You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "pan3793 (via GitHub)" <gi...@apache.org> on 2025/07/24 11:00:17 UTC
[PR] [SPARK-5294][YARN][BUILD] YARN External Shuffle Service jar should include `scala-library` [spark]
pan3793 opened a new pull request, #51650:
URL: https://github.com/apache/spark/pull/51650
<!--
Thanks for sending a pull request! Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://spark.apache.org/contributing.html
2. Ensure you have added or run the appropriate tests for your PR: https://spark.apache.org/developer-tools.html
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][SPARK-XXXX] Your PR title ...'.
4. Be sure to keep the PR description updated to reflect all changes.
5. Please write your PR title to summarize what this PR proposes.
6. If possible, provide a concise example to reproduce the issue for a faster review.
7. If you want to add a new configuration, please read the guideline first for naming configurations in
'core/src/main/scala/org/apache/spark/internal/config/ConfigEntry.scala'.
8. If you want to add or modify an error type or message, please read the guideline first in
'common/utils/src/main/resources/error/README.md'.
-->
### What changes were proposed in this pull request?
<!--
Please clarify what changes you are proposing. The purpose of this section is to outline the changes and how this PR fixes the issue.
If possible, please consider writing useful notes for better and faster reviews in your PR. See the examples below.
1. If you refactor some codes with changing classes, showing the class hierarchy will help reviewers.
2. If you fix some SQL features, you can provide some references of other DBMSes.
3. If there is design documentation, please add the link.
4. If there is a discussion in the mailing list, please add the link.
-->
Since SPARK-41400, the `common/network-yarn` module has started to hard depend on Scala, now it causes YARN Resource Manager to fail to start due to missing `scala-library`.
```
2025-07-24 09:55:38,369 INFO util.ApplicationClassLoader: classpath: [file:/opt/spark/yarn/spark-4.1.0-SNAPSHOT-yarn-shuffle.jar]
2025-07-24 09:55:38,369 INFO util.ApplicationClassLoader: system classes: [java., javax.accessibility., -javax.activation., javax.activity., javax.annotation., javax.annotation.processing., javax.crypto., javax.ima
geio., javax.jws., javax.lang.model., -javax.management.j2ee., javax.management., javax.naming., javax.net., javax.print., javax.rmi., javax.script., -javax.security.auth.message., javax.security.auth., javax.secur
ity.cert., javax.security.sasl., javax.sound., javax.sql., javax.swing., javax.tools., javax.transaction., -javax.xml.registry., -javax.xml.rpc., javax.xml., org.w3c.dom., org.xml.sax., org.apache.commons.logging.,
org.apache.log4j., -org.apache.hadoop.hbase., org.apache.hadoop., core-default.xml, hdfs-default.xml, mapred-default.xml, yarn-default.xml]
2025-07-24 09:55:38,538 INFO yarn.YarnShuffleService: Initializing YARN shuffle service for Spark
2025-07-24 09:55:38,539 WARN containermanager.AuxServices: The Auxiliary Service named 'spark_shuffle' in the configuration is for class org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxiliaryServiceWi
thCustomClassLoader which has a name of 'org.apache.spark.network.yarn.YarnShuffleService with custom class loader'. Because these are not the same tools trying to send ServiceData and read Service Meta Data may ha
ve issues unless the refer to the name in the config.
2025-07-24 09:55:38,808 ERROR nodemanager.NodeManager: Error starting NodeManager
java.lang.NoClassDefFoundError: scala/Product
at java.base/java.lang.ClassLoader.defineClass1(Native Method)
at java.base/java.lang.ClassLoader.defineClass(ClassLoader.java:1017)
at java.base/java.security.SecureClassLoader.defineClass(SecureClassLoader.java:150)
at java.base/java.net.URLClassLoader.defineClass(URLClassLoader.java:524)
at java.base/java.net.URLClassLoader$1.run(URLClassLoader.java:427)
at java.base/java.net.URLClassLoader$1.run(URLClassLoader.java:421)
at java.base/java.security.AccessController.doPrivileged(AccessController.java:712)
at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:420)
at org.apache.hadoop.util.ApplicationClassLoader.loadClass(ApplicationClassLoader.java:176)
at org.apache.hadoop.util.ApplicationClassLoader.loadClass(ApplicationClassLoader.java:157)
at org.apache.spark.network.yarn.YarnShuffleService.serviceInit(YarnShuffleService.java:330)
at org.apache.hadoop.service.AbstractService.init(AbstractService.java:165)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxiliaryServiceWithCustomClassLoader.serviceInit(AuxiliaryServiceWithCustomClassLoader.java:64)
at org.apache.hadoop.service.AbstractService.init(AbstractService.java:165)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices.initAuxService(AuxServices.java:475)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices.serviceInit(AuxServices.java:758)
at org.apache.hadoop.service.AbstractService.init(AbstractService.java:165)
at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:110)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.serviceInit(ContainerManagerImpl.java:336)
at org.apache.hadoop.service.AbstractService.init(AbstractService.java:165)
at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:110)
at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:501)
at org.apache.hadoop.service.AbstractService.init(AbstractService.java:165)
at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:969)
at org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:1058)
Caused by: java.lang.ClassNotFoundException: scala.Product
at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:641)
at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:188)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:525)
at org.apache.hadoop.util.ApplicationClassLoader.loadClass(ApplicationClassLoader.java:189)
at org.apache.hadoop.util.ApplicationClassLoader.loadClass(ApplicationClassLoader.java:157)
... 25 more
2025-07-24 09:55:38,815 INFO nodemanager.NodeManager: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NodeManager at hadoop-worker1.orb.local/192.168.97.6
************************************************************/
```
Note: now `spark-<version>-yarn-shuffle.jar` is ~100m, while previously it is ~10m. in `common/utils`, the Java and Scala code cross references each other, so we can not simply split it into one Java utils and one Scala utils modules, thus it's not easy to make `spark-<version>-yarn-shuffle.jar` to be scala-free as before.
### Why are the changes needed?
<!--
Please clarify why the changes are needed. For instance,
1. If you propose a new API, clarify the use case for a new API.
2. If you fix a bug, you can clarify why it is a bug.
-->
Bug fix, recover a broken feature.
### Does this PR introduce _any_ user-facing change?
<!--
Note that it means *any* user-facing change including all aspects such as new features, bug fixes, or other behavior changes. Documentation-only updates are not considered user-facing changes.
If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible.
If possible, please also clarify if this is a user-facing change compared to the released Spark versions or within the unreleased branches such as master.
If no, write 'No'.
-->
Yes, recover a broken feature.
### How was this patch tested?
<!--
If tests were added, say they were added here. Please make sure to add some test cases that check the changes thoroughly including negative and positive cases if possible.
If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future.
If tests were not added, please describe why they were not added and/or why it was difficult to add.
If benchmark tests were added, please run the benchmarks in GitHub Actions for the consistent environment, and the instructions could accord to: https://spark.apache.org/developer-tools.html#github-workflow-benchmarks.
-->
Tested on a YARN cluster, RM start successfully after patching.
Note: Hadoop does not officially support JDK 17 yet, I tested it using the 3.4.2 RC1.
### Was this patch authored or co-authored using generative AI tooling?
<!--
If generative AI tooling has been used in the process of authoring this patch, please include the
phrase: 'Generated-by: ' followed by the name of the tool and its version.
If no, write 'No'.
Please refer to the [ASF Generative Tooling Guidance](https://www.apache.org/legal/generative-tooling.html) for details.
-->
No.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-52942][YARN][BUILD] YARN External Shuffle Service jar should include `scala-library` [spark]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on PR #51650:
URL: https://github.com/apache/spark/pull/51650#issuecomment-3130428311
@dongjoon-hyun, thanks for your advice, but I'm afraid this violates the design principle of the Structured Logging Framework. I remember during the development of this feature, @gengliangwang strongly recommended replacing all variables with `MDC`. And in the `SparkLogger`, there is no such API to allow us to do:
```patch
logger.info("Registered metrics with Hadoop's DefaultMetricsSystem using namespace '{}'",
- MDC.of(LogKeys.SHUFFLE_SERVICE_METRICS_NAMESPACE$.MODULE$, metricsNamespace));
+ metricsNamespace);
```
<img width="1060" height="177" alt="image" src="https://github.com/user-attachments/assets/d3105734-6da8-402e-9df9-bd7bab1d5bc9" />
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-52942][YARN][BUILD] YARN External Shuffle Service jar should include `scala-library` [spark]
Posted by "mridulm (via GitHub)" <gi...@apache.org>.
mridulm commented on PR #51650:
URL: https://github.com/apache/spark/pull/51650#issuecomment-3162759587
Thanks for working on this @pan3793 !
I missed reviewing the rewrite for MDC/LogKeys - but it is great to see this dependency getting removed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-52942][YARN][BUILD] YARN External Shuffle Service jar should include `scala-library` [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #51650:
URL: https://github.com/apache/spark/pull/51650#issuecomment-3125017624
Thank you. Let me rephrase in my words. So, the problem is not 'Structured Logging'. Instead, the root cause is the MDC usage which is newly added based on Scala-dependency. Did I understand correctly?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-52942][YARN][BUILD] YARN External Shuffle Service jar should include `scala-library` [spark]
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang closed pull request #51650: [SPARK-52942][YARN][BUILD] YARN External Shuffle Service jar should include `scala-library`
URL: https://github.com/apache/spark/pull/51650
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-52942][YARN][BUILD] YARN External Shuffle Service jar should include `scala-library` [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #51650:
URL: https://github.com/apache/spark/pull/51650#issuecomment-3130494289
To @pan3793 , it seems that I wan't clear enough in [the previous comment](https://github.com/apache/spark/pull/51650#issuecomment-3127929767). I already walked away from this PR . That was my my idea but I'm also not insisting anything here since [the last comment](https://github.com/apache/spark/pull/51650#issuecomment-3127929767). Literally, you can ignore me (and previous my all comments) on this PR. 😄
> @dongjoon-hyun, thanks for your advice, but I'm afraid this violates the design principle of the Structured Logging Framework. I remember during the development of this feature, @gengliangwang strongly recommended replacing all variables with `MDC` for INFO and higher severity levels. And in the `SparkLogger`, there is no such API to allow us to do:
>
> ```diff
> logger.info("Registered metrics with Hadoop's DefaultMetricsSystem using namespace '{}'",
> - MDC.of(LogKeys.SHUFFLE_SERVICE_METRICS_NAMESPACE$.MODULE$, metricsNamespace));
> + metricsNamespace);
> ```
>
> <img alt="image" width="1060" height="177" src="https://private-user-images.githubusercontent.com/26535726/471787617-d3105734-6da8-402e-9df9-bd7bab1d5bc9.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3NTM3NTk0MTEsIm5iZiI6MTc1Mzc1OTExMSwicGF0aCI6Ii8yNjUzNTcyNi80NzE3ODc2MTctZDMxMDU3MzQtNmRhOC00MDJlLTlkZjktYmQ3YmFiMWQ1YmM5LnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNTA3MjklMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjUwNzI5VDAzMTgzMVomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTQ3ZTFhMGU5YmIxOTQwMzRkYTU4MmY3MDZmOGZhNmZiOWE1ZjFlMDVhZTA4ZDYxNDc4NGVkYmNmZTM1YjhkNzUmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0In0.dNN98Hf7I6mgYUTWMfqO468bQzP5Lf8My2gXGhTHBsw">
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-52942][YARN][BUILD] YARN External Shuffle Service jar should include `scala-library` [spark]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on PR #51650:
URL: https://github.com/apache/spark/pull/51650#issuecomment-3125057419
I think they are the same thing because `MDC` is part of the structured logging API.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-52942][YARN][BUILD] YARN External Shuffle Service jar should include `scala-library` [spark]
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on PR #51650:
URL: https://github.com/apache/spark/pull/51650#issuecomment-3157391414
Merged into master/branch-4.0. Thanks @pan3793 @dongjoon-hyun and @mridulm
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-52942][YARN][BUILD] YARN External Shuffle Service jar should include `scala-library` [spark]
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on PR #51650:
URL: https://github.com/apache/spark/pull/51650#issuecomment-3130642794
Actually, I'm no longer using the External Shuffle service either. But I'm wondering if it's feasible to rewrite `MDC` and `LogKeys` in Java. That should enable decoupling, right? They are both in the `org.apache.spark.internal` package, so they probably don't belong to the public API either.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-52942][YARN][BUILD] YARN External Shuffle Service jar should include `scala-library` [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #51650:
URL: https://github.com/apache/spark/pull/51650#issuecomment-3121162190
Wait, `Structured Logging Framework` is controlled by configuration. Do you mean `Structured Logging Framework`-disabled code path also requires Scala at runtime?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-52942][YARN][BUILD] YARN External Shuffle Service jar should include `scala-library` [spark]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on PR #51650:
URL: https://github.com/apache/spark/pull/51650#issuecomment-3124976107
@dongjoon-hyun `spark.log.structuredLogging.enabled=false` does not work either. The stacktrace pasted in the PR description is produced under disabling structured logging.
In the `Structured Logging Framework` API, `MDC` is a Scala `case class`, which inherits from `scala.Product`, thus always requires `scala-library` when this API is called.
https://github.com/apache/spark/blob/5a9929c32ef8aafd275a3cf4797bb0ba9a6e61e2/common/utils/src/main/java/org/apache/spark/internal/SparkLogger.java#L159-L165
https://github.com/apache/spark/blob/5a9929c32ef8aafd275a3cf4797bb0ba9a6e61e2/common/utils/src/main/scala/org/apache/spark/internal/Logging.scala#L78
https://github.com/apache/spark/blob/5a9929c32ef8aafd275a3cf4797bb0ba9a6e61e2/common/network-yarn/src/main/java/org/apache/spark/network/yarn/YarnShuffleService.java#L330-L331
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-5294][YARN][BUILD] YARN External Shuffle Service jar should include `scala-library` [spark]
Posted by "mridulm (via GitHub)" <gi...@apache.org>.
mridulm commented on PR #51650:
URL: https://github.com/apache/spark/pull/51650#issuecomment-3113841776
This is unfortunate, thanks for working on fixing it.
I am also concerned about the size escalation, and wondering if we can break the dependency on scala for ess.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-52942][YARN][BUILD] YARN External Shuffle Service jar should include `scala-library` [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #51650:
URL: https://github.com/apache/spark/pull/51650#issuecomment-3127929767
Anyway, it was just my idea. For this `YARN External Shuffle service`, I'll leave it to you guys because I'm not using this technically. So, feel free to proceed the AS-IS direction, @pan3793 and all.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-52942][YARN][BUILD] YARN External Shuffle Service jar should include `scala-library` [spark]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on PR #51650:
URL: https://github.com/apache/spark/pull/51650#issuecomment-3121061539
@dongjoon-hyun `common/network-yarn` module depends on `common/utils`, `common/network-common`, `common/network-shuffle`. If we want to make `common/network-yarn` scala-free, we should purge `Structured Logging Framework` usage in all of those modules.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-5294][YARN][BUILD] YARN External Shuffle Service jar should include `scala-library` [spark]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on PR #51650:
URL: https://github.com/apache/spark/pull/51650#issuecomment-3113795567
> Do you know why it has become so big?
@LuciferYang the jar can be produced by
```
build/mvn -pl common/network-yarn -am clean package -DskipTests -Pyarn
```
```
spark-3.3.4-yarn-shuffle.jar - 11M
spark-3.4.4-yarn-shuffle.jar - 68M -- rocksdbjni contributes the most part
spark-3.5.6-yarn-shuffle.jar - 74M -- start consuming `spark-common-utils_2.12:jar` and it's transitive deps
spark-4.0.0-yarn-shuffle.jar - 96M -- start consuming `netty-tcnative-boringssl-static:jar` broken
spark-4.1.0-SNAPSHOT-yarn-shuffle.jar - 103M -- this PR
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-52942][YARN][BUILD] YARN External Shuffle Service jar should include `scala-library` [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #51650:
URL: https://github.com/apache/spark/pull/51650#issuecomment-3119874901
Thank you so much for reporting and working on this, @pan3793 .
If then, we may want to remove `Structured Logging Framework` usage in `YARN External Shuffle Service` module? Technically, it's useless in YARN, isn't it?
> it seems we can't, because the structured logging framework API for Java depends on Scala classes.
cc @peter-toth
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-52942][YARN][BUILD] YARN External Shuffle Service jar should include `scala-library` [spark]
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on PR #51650:
URL: https://github.com/apache/spark/pull/51650#issuecomment-3142550122
fine to me
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-52942][YARN][BUILD] YARN External Shuffle Service jar should include `scala-library` [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #51650:
URL: https://github.com/apache/spark/pull/51650#issuecomment-3127879251
> I think they are the same thing because `MDC` is part of the structured logging API.
😄 What I was thinking is something like following by removing `MDC` feature in those key modules (YARN external shuffle service depends), @pan3793 . Although it means we are going to lose `MDC` feature for those modules, I thought it's a few places compared to the whole Spark modules.
```
logger.info("Registered metrics with Hadoop's DefaultMetricsSystem using namespace '{}'",
- MDC.of(LogKeys.SHUFFLE_SERVICE_METRICS_NAMESPACE$.MODULE$, metricsNamespace));
+ metricsNamespace);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-52942][YARN][BUILD] YARN External Shuffle Service jar should include `scala-library` [spark]
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on PR #51650:
URL: https://github.com/apache/spark/pull/51650#issuecomment-3157253539
After the merge of the pull request at https://github.com/apache/spark/pull/51775, the master should no longer require this patch, but the branch-4.0 should still need this fix, right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-52942][YARN][BUILD] YARN External Shuffle Service jar should include `scala-library` [spark]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on PR #51650:
URL: https://github.com/apache/spark/pull/51650#issuecomment-3157281192
After SPARK-53064, I tested that YARN ESS can successfully bootstrap and serve simple Spark queries w/o the `scala-library`, but I can NOT say that YARN ESS is scala-free, because I cannot guarantee that all the code in YARN ESS does not call Scala utility classes.
To achieve the goal, we might need to split the `common/utils` into 2 modules(one Java, one Scala), and let other common modules only depend on the pure Java utils.
For safety, we can merge this PR to both master and branch-4.0.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-5294][YARN][BUILD] YARN External Shuffle Service jar should include `scala-library` [spark]
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on PR #51650:
URL: https://github.com/apache/spark/pull/51650#issuecomment-3113445119
> Note: now spark-<version>-yarn-shuffle.jar is ~100m, while previously it is ~10m. in common/utils, the Java and Scala code cross references each other, so we can not simply split it into one Java utils and one Scala utils modules, thus it's not easy to make spark-<version>-yarn-shuffle.jar to be scala-free as before.
Do you know why it has become so big?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-5294][YARN][BUILD] YARN External Shuffle Service jar should include `scala-library` [spark]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on PR #51650:
URL: https://github.com/apache/spark/pull/51650#issuecomment-3113146400
cc @hvanhovell @LuciferYang
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-52942][YARN][BUILD] YARN External Shuffle Service jar should include `scala-library` [spark]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on PR #51650:
URL: https://github.com/apache/spark/pull/51650#issuecomment-3113861619
> ... if we can break the dependency on scala for ess.
@mridulm This is what I initially expected, but it seems we can't, because the structured logging framework API for Java depends on Scala classes.
https://github.com/apache/spark/blob/5a9929c32ef8aafd275a3cf4797bb0ba9a6e61e2/common/utils/src/main/java/org/apache/spark/internal/SparkLogger.java#L159-L165
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-52942][YARN][BUILD] YARN External Shuffle Service jar should include `scala-library` [spark]
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on PR #51650:
URL: https://github.com/apache/spark/pull/51650#issuecomment-3114392797
> > ... if we can break the dependency on scala for ess.
>
> @mridulm This is what I initially expected, but it seems we can't, because the structured logging framework API for Java depends on Scala classes.
>
> https://github.com/apache/spark/blob/5a9929c32ef8aafd275a3cf4797bb0ba9a6e61e2/common/utils/src/main/java/org/apache/spark/internal/SparkLogger.java#L159-L165
also cc @gengliangwang
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-52942][YARN][BUILD] YARN External Shuffle Service jar should include `scala-library` [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #51650:
URL: https://github.com/apache/spark/pull/51650#issuecomment-3121150549
Got it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-52942][YARN][BUILD] YARN External Shuffle Service jar should include `scala-library` [spark]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on PR #51650:
URL: https://github.com/apache/spark/pull/51650#issuecomment-3142534151
@LuciferYang, thanks for your advice. I have made some progress in rewriting `MDC` and `LogKeys` from Scala to Java locally, although it's not done yet, I'm optimistic about its feasibility. But the change is untrivial, at least not fit for branch-4.0. I suggest adopting this PR as a simple fix first, and I will open a PR following your idea and discuss later. WDYT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org