You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Sandeep Katta (Jira)" <ji...@apache.org> on 2019/09/22 06:14:00 UTC
[jira] [Updated] (SPARK-29202) --driver-java-options are not passed
to driver process in yarn client mode
[ https://issues.apache.org/jira/browse/SPARK-29202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sandeep Katta updated SPARK-29202:
----------------------------------
Description:
Run the below command
./bin/spark-sql --master yarn --driver-java-options="-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=5555"
In Spark 2.3.3
/opt/softwares/Java/jdk1.8.0_211/bin/java -cp /opt/BigdataTools/spark-2.3.3-bin-hadoop2.7/conf/:/opt/BigdataTools/spark-2.3.3-bin-hadoop2.7/jars/*:/opt/BigdataTools/hadoop-3.2.0/etc/hadoop/ -Xmx1g -agentlib:jdwp=transport=dt_socket,server=y,suspend=y,address=5555 org.apache.spark.deploy.SparkSubmit --master yarn --conf spark.driver.extraJavaOptions=-agentlib:jdwp=transport=dt_socket,server=y,suspend=y,address=5555 --class org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver spark-internal
In Spark 3.0
/opt/softwares/Java/jdk1.8.0_211/bin/java -cp /opt/apache/git/sparkSourceCode/spark/conf/:/opt/apache/git/sparkSourceCode/spark/assembly/target/scala-2.12/jars/*:/opt/BigdataTools/hadoop-3.2.0/etc/hadoop/ org.apache.spark.deploy.SparkSubmit --master yarn --conf spark.driver.extraJavaOptions=-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=5556 --class org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver spark-internal
We can see that java options are not passed to driver process in spark3
> --driver-java-options are not passed to driver process in yarn client mode
> --------------------------------------------------------------------------
>
> Key: SPARK-29202
> URL: https://issues.apache.org/jira/browse/SPARK-29202
> Project: Spark
> Issue Type: Bug
> Components: Deploy
> Affects Versions: 3.0.0
> Reporter: Sandeep Katta
> Priority: Major
>
> Run the below command
> ./bin/spark-sql --master yarn --driver-java-options="-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=5555"
>
> In Spark 2.3.3
> /opt/softwares/Java/jdk1.8.0_211/bin/java -cp /opt/BigdataTools/spark-2.3.3-bin-hadoop2.7/conf/:/opt/BigdataTools/spark-2.3.3-bin-hadoop2.7/jars/*:/opt/BigdataTools/hadoop-3.2.0/etc/hadoop/ -Xmx1g -agentlib:jdwp=transport=dt_socket,server=y,suspend=y,address=5555 org.apache.spark.deploy.SparkSubmit --master yarn --conf spark.driver.extraJavaOptions=-agentlib:jdwp=transport=dt_socket,server=y,suspend=y,address=5555 --class org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver spark-internal
>
> In Spark 3.0
> /opt/softwares/Java/jdk1.8.0_211/bin/java -cp /opt/apache/git/sparkSourceCode/spark/conf/:/opt/apache/git/sparkSourceCode/spark/assembly/target/scala-2.12/jars/*:/opt/BigdataTools/hadoop-3.2.0/etc/hadoop/ org.apache.spark.deploy.SparkSubmit --master yarn --conf spark.driver.extraJavaOptions=-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=5556 --class org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver spark-internal
> We can see that java options are not passed to driver process in spark3
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org