You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Jone Zhang <jo...@gmail.com> on 2016/10/20 09:21:18 UTC

Can i display message on console when use spark on yarn?

I submit spark with "spark-submit --master yarn-cluster --deploy-mode
cluster"
How can i display message on yarn console.
I expect it to be like this:

.....
16/10/20 17:12:53 main INFO org.apache.spark.deploy.yarn.Client>SPK>
Application report for application_1453970859007_481440 (state: RUNNING)
16/10/20 17:12:58 main INFO org.apache.spark.deploy.yarn.Client>SPK>
Application report for application_1453970859007_481440 (state: RUNNING)
16/10/20 17:13:03 main INFO org.apache.spark.deploy.yarn.Client>SPK>
Application report for application_1453970859007_481440 (state: RUNNING)
16/10/20 17:13:08 main INFO org.apache.spark.deploy.yarn.Client>SPK>
Application report for application_1453970859007_481440 (state: RUNNING)
16/10/20 17:13:13 main INFO org.apache.spark.deploy.yarn.Client>SPK>
Application report for application_1453970859007_481440 (state: FINISHED)
16/10/20 17:13:13 main INFO org.apache.spark.deploy.yarn.Client>SPK>
         client token: N/A
         diagnostics: N/A
         ApplicationMaster host: 10.51.215.100
         ApplicationMaster RPC port: 0
         queue: root.default
         start time: 1476954698645
         final status: SUCCEEDED
         tracking URL:
http://10.179.20.47:8080/proxy/application_1453970859007_481440/history/application_1453970859007_481440/1
         user: mqq
===Spark Task Result is ===
===some message want to display===
16/10/20 17:13:13 Thread-3 INFO
org.apache.spark.util.ShutdownHookManager>SPK> Shutdown hook called
16/10/20 17:13:13 Thread-3 INFO
org.apache.spark.util.ShutdownHookManager>SPK> Deleting directory
/data/home/spark_tmp/spark-5b9f434b-5837-46e6-9625-c4656b86af9e

Thanks.

Re: Can i display message on console when use spark on yarn?

Posted by ayan guha <gu...@gmail.com>.
What do you exactly mean by Yarn Console? We use spark-submit and it
generates exactly same log as you mentioned on driver console,

On Thu, Oct 20, 2016 at 8:21 PM, Jone Zhang <jo...@gmail.com> wrote:

> I submit spark with "spark-submit --master yarn-cluster --deploy-mode
> cluster"
> How can i display message on yarn console.
> I expect it to be like this:
>
> .....
> 16/10/20 17:12:53 main INFO org.apache.spark.deploy.yarn.Client>SPK>
> Application report for application_1453970859007_481440 (state: RUNNING)
> 16/10/20 17:12:58 main INFO org.apache.spark.deploy.yarn.Client>SPK>
> Application report for application_1453970859007_481440 (state: RUNNING)
> 16/10/20 17:13:03 main INFO org.apache.spark.deploy.yarn.Client>SPK>
> Application report for application_1453970859007_481440 (state: RUNNING)
> 16/10/20 17:13:08 main INFO org.apache.spark.deploy.yarn.Client>SPK>
> Application report for application_1453970859007_481440 (state: RUNNING)
> 16/10/20 17:13:13 main INFO org.apache.spark.deploy.yarn.Client>SPK>
> Application report for application_1453970859007_481440 (state: FINISHED)
> 16/10/20 17:13:13 main INFO org.apache.spark.deploy.yarn.Client>SPK>
>          client token: N/A
>          diagnostics: N/A
>          ApplicationMaster host: 10.51.215.100
>          ApplicationMaster RPC port: 0
>          queue: root.default
>          start time: 1476954698645
>          final status: SUCCEEDED
>          tracking URL: http://10.179.20.47:8080/proxy/application_
> 1453970859007_481440/history/application_1453970859007_481440/1
>          user: mqq
> ===Spark Task Result is ===
> ===some message want to display===
> 16/10/20 17:13:13 Thread-3 INFO org.apache.spark.util.ShutdownHookManager>SPK>
> Shutdown hook called
> 16/10/20 17:13:13 Thread-3 INFO org.apache.spark.util.ShutdownHookManager>SPK>
> Deleting directory /data/home/spark_tmp/spark-5b9f434b-5837-46e6-9625-
> c4656b86af9e
>
> Thanks.
>
>
>


-- 
Best Regards,
Ayan Guha