You are viewing a plain text version of this content. The canonical link for it is here.
Posted to yarn-issues@hadoop.apache.org by "tonydoen (Jira)" <ji...@apache.org> on 2022/02/17 03:25:00 UTC
[jira] [Comment Edited] (YARN-4577) Enable aux services to have their own custom classpath/jar file
[ https://issues.apache.org/jira/browse/YARN-4577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17493624#comment-17493624 ]
tonydoen edited comment on YARN-4577 at 2/17/22, 3:24 AM:
----------------------------------------------------------
Thanks [~xgong] , [xkrogen@apache.org |mailto:xkrogen@apache.org]for your contribution!
we have tried to run multiple versions of the Spark Shuffle Service according [https://github.com/apache/spark/blob/master/docs/running-on-yarn.md]
but, it wont work. any suggestion, we would appreciate it.
yarn-site.xml configuration
...
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle,spark_shuffle_1_6,spark_shuffle_3_2</value>
</property>
<property>
<name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
<value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
<property>
<name>yarn.nodemanager.aux-services.spark_shuffle_1_6.class</name>
<value>org.apache.spark.network.yarn.YarnShuffleService</value>
</property>
<property>
<name>yarn.nodemanager.aux-services.spark_shuffle_1_6.classpath</name>
<value>/home/work/app/yarn/c4prc-preview/nodemanager/spark-yarn-shuffle-lib/spark-1-6-yarn-shuffle-lib,/home/work/app/yarn/c4prc-preview/nodemanager/spark-yarn-shuffle-lib/spark-1-6-yarn-shuffle-config</value>
</property>
<property>
<name>yarn.nodemanager.aux-services.spark_shuffle_3_2.class</name>
<value>org.apache.spark.network.yarn.YarnShuffleService</value>
</property>
<property>
<name>yarn.nodemanager.aux-services.spark_shuffle_3_2.classpath</name>
<value>/home/work/app/yarn/c4prc-preview/nodemanager/spark-yarn-shuffle-lib/spark-3-2-yarn-shuffle-lib,/home/work/app/yarn/c4prc-preview/nodemanager/spark-yarn-shuffle-lib/spark-3-2-yarn-shuffle-config</value>
</property>
...
/home/work/app/yarn/c4prc-preview/nodemanager/spark-yarn-shuffle-lib/spark-3-2-yarn-shuffle-lib
is spark-yarn-shuffle lib location
/home/work/app/yarn/c4prc-preview/nodemanager/spark-yarn-shuffle-lib/spark-3-2-yarn-shuffle-config
is spark-yarn-shuffle config, just one file named [spark-shuffle-site.xml] is in the above path.
spark-shuffle-site.xml content is
<configuration>
<property>
<name>spark.shuffle.service.port</name>
<value>20811</value>
</property>
<property>
<name>spark.yarn.shuffle.service.metrics.namespace</name>
<value>sparkShuffleService1</value>
</property>
</configuration>
was (Author: JIRAUSER285351):
Thanks [~xgong] for your contribution! we have tried to run multiple versions of the Spark Shuffle Service.
> Enable aux services to have their own custom classpath/jar file
> ---------------------------------------------------------------
>
> Key: YARN-4577
> URL: https://issues.apache.org/jira/browse/YARN-4577
> Project: Hadoop YARN
> Issue Type: Improvement
> Affects Versions: 2.8.0
> Reporter: Xuan Gong
> Assignee: Xuan Gong
> Priority: Major
> Fix For: 2.9.0, 3.0.0-alpha1
>
> Attachments: YARN-4577.1.patch, YARN-4577.2.patch, YARN-4577.20160119.1.patch, YARN-4577.20160204.patch, YARN-4577.20160428.patch, YARN-4577.20160509.patch, YARN-4577.20160510.patch, YARN-4577.20160511.1.patch, YARN-4577.20160511.patch, YARN-4577.3.patch, YARN-4577.3.rebase.patch, YARN-4577.4.patch, YARN-4577.5.patch, YARN-4577.poc.patch
>
>
> Right now, users have to add their jars to the NM classpath directly, thus put them on the system classloader. But if multiple versions of the plugin are present on the classpath, there is no control over which version actually gets loaded. Or if there are any conflicts between the dependencies introduced by the auxiliary service and the NM itself, they can break the NM, the auxiliary service, or both.
> The solution could be: to instantiate aux services using a classloader that is different from the system classloader.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org