You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Kyle Lin <ky...@gmail.com> on 2016/01/12 09:54:06 UTC
Re: JMXSink for YARN deployment
Hello there
I run both driver and master on the same node, so I got "Port already in
use" exception.
Is there any solution to set different port for each component?
Kyle
2015-12-05 5:54 GMT+08:00 spearson23 <sp...@gmail.com>:
> Run "spark-submit --help" to see all available options.
>
> To get JMX to work you need to:
>
> spark-submit --driver-java-options "-Dcom.sun.management.jmxremote
> -Dcom.sun.management.jmxremote.authenticate=false
> -Dcom.sun.management.jmxremote.ssl=false
> -Dcom.sun.management.jmxremote.port=JMX_PORT" --conf
> spark.metrics.conf=metrics.properties --class 'CLASS_NAME' --master
> yarn-cluster --files /PATH/TO/metrics.properties /PATH/TO/JAR.FILE
>
>
> This will run JMX on the driver node on or "JMX_PORT". Note that the
> driver
> node and the YARN master node are not the same, you'll have to look where
> spark put the driver node and then connect there.
>
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/JMXSink-for-YARN-deployment-tp13958p25572.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>
Re: JMXSink for YARN deployment
Posted by Kyle Lin <ky...@gmail.com>.
Hello guys
I got a solution.
I set -Dcom.sun.management.jmxremote.port=0 to let system assign a unused
port.
Kyle
2016-01-12 16:54 GMT+08:00 Kyle Lin <ky...@gmail.com>:
> Hello there
>
>
> I run both driver and master on the same node, so I got "Port already in
> use" exception.
>
> Is there any solution to set different port for each component?
>
> Kyle
>
>
> 2015-12-05 5:54 GMT+08:00 spearson23 <sp...@gmail.com>:
>
>> Run "spark-submit --help" to see all available options.
>>
>> To get JMX to work you need to:
>>
>> spark-submit --driver-java-options "-Dcom.sun.management.jmxremote
>> -Dcom.sun.management.jmxremote.authenticate=false
>> -Dcom.sun.management.jmxremote.ssl=false
>> -Dcom.sun.management.jmxremote.port=JMX_PORT" --conf
>> spark.metrics.conf=metrics.properties --class 'CLASS_NAME' --master
>> yarn-cluster --files /PATH/TO/metrics.properties /PATH/TO/JAR.FILE
>>
>>
>> This will run JMX on the driver node on or "JMX_PORT". Note that the
>> driver
>> node and the YARN master node are not the same, you'll have to look where
>> spark put the driver node and then connect there.
>>
>>
>>
>>
>> --
>> View this message in context:
>> http://apache-spark-user-list.1001560.n3.nabble.com/JMXSink-for-YARN-deployment-tp13958p25572.html
>> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>> For additional commands, e-mail: user-help@spark.apache.org
>>
>>
>