You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@livy.apache.org by "Partridge, Lucas (GE Aviation)" <Lu...@ge.com> on 2017/11/24 12:27:12 UTC
How to access Python script's stdout when running in Yarn cluster
mode?
Hi,
I'm using Livy's GET /batches/{batchId}/log method to fetch the log lines from a Python script I'm running on Spark in Yarn cluster mode. Unfortunately the stdout from my Python script is not included in the log lines returned by GET /batches/{batchId}/log!
Is this by design, or an unfortunate by-product of running in Yarn cluster mode?
If this is intentional does anyone know how I can access the stdout from my Python script via Livy please? Preferably without having to change my REST client (a Java app) to use Yarn's client deployment mode.
Thanks,
Lucas.
How to access Python script's stdout when running in Yarn cluster mode?
Posted by "Partridge, Lucas (GE Aviation)" <Lu...@ge.com>.
Ok thanks for clarifying. After my previous email I did find some tips about using YarnClient or Spark History Server’s API at https://stackoverflow.com/questions/43610723/how-to-retrieve-yarns-logs-programmatically-using-java
I also realised I had misinterpreted the diagram of Yarn’s client deployment mode at https://www.cloudera.com/documentation/enterprise/5-5-x/topics/cdh_ig_running_spark_on_yarn.html#deployment_modes__section_m5f_skd_ss . I now realise the ‘client application’ in that diagram is the Livy server itself rather than my own app which is acting as a client to the Livy server. So I’m actually happier to run in yarn client mode than before because the loading will be on the Livy server rather than on my app. And if the loading should ultimately be moved to the Yarn cluster instead we can always move to yarn cluster mode in future and then retrieve the logs programmatically using YarnClient or whatever.
Thanks, Lucas.
From: Saisai Shao [mailto:sai.sai.shao@gmail.com]
Sent: 27 November 2017 12:12
To: user@livy.incubator.apache.org
Subject: EXT: Re: How to access Python script's stdout when running in Yarn cluster mode?
I think you have plenty of ways to get application log, either via command line or programmatically, even with yarn RM UI. Since there already has several ways to get application log, so currently we don't have a plan to address this.
2017-11-27 20:04 GMT+08:00 Partridge, Lucas (GE Aviation) <Lu...@ge.com>>:
Thanks Jerry.
“Currently there's no Livy api for you to get application log via REST API.”
- That’s a real shame. Are there any plans to address this? Because this really limits the usefulness of Livy when I’m using Yarn cluster mode. It seems I have to make a choice between running my client in a scalable manner (yarn cluster) or being able to see the logs programmatically (yarn client), but not both. My client is multi-threaded; I don’t want it to have to host multiple concurrent Spark driver applications but it looks like I might have no choice about this.
Thanks, Lucas.
From: Saisai Shao [mailto:sai.sai.shao@gmail.com<ma...@gmail.com>]
Sent: 27 November 2017 02:19
To: user@livy.incubator.apache.org<ma...@livy.incubator.apache.org>
Subject: EXT: Re: How to access Python script's stdout when running in Yarn cluster mode?
Since you're running with yarn cluster mode, the output from your python script should be part of your yarn application log. you can get it via yarn command like yarn log -applicationId <applictionId>, or others like Yarn UI. Currently there's no Livy api for you to get application log via REST API.
Thanks
Jerry
2017-11-24 20:27 GMT+08:00 Partridge, Lucas (GE Aviation) <Lu...@ge.com>>:
Hi,
I’m using Livy’s GET /batches/{batchId}/log method to fetch the log lines from a Python script I’m running on Spark in Yarn cluster mode. Unfortunately the stdout from my Python script is not included in the log lines returned by GET /batches/{batchId}/log!
Is this by design, or an unfortunate by-product of running in Yarn cluster mode?
If this is intentional does anyone know how I can access the stdout from my Python script via Livy please? Preferably without having to change my REST client (a Java app) to use Yarn’s client deployment mode.
Thanks,
Lucas.
Re: How to access Python script's stdout when running in Yarn cluster mode?
Posted by Saisai Shao <sa...@gmail.com>.
I think you have plenty of ways to get application log, either via command
line or programmatically, even with yarn RM UI. Since there already has
several ways to get application log, so currently we don't have a plan to
address this.
2017-11-27 20:04 GMT+08:00 Partridge, Lucas (GE Aviation) <
Lucas.Partridge@ge.com>:
> Thanks Jerry.
>
>
>
> “Currently there's no Livy api for you to get application log via REST
> API.”
>
> - That’s a real shame. Are there any plans to address this? Because this
> really limits the usefulness of Livy when I’m using Yarn cluster mode. It
> seems I have to make a choice between running my client in a scalable
> manner (yarn cluster) or being able to see the logs programmatically (yarn
> client), but not both. My client is multi-threaded; I don’t want it to
> have to host multiple concurrent Spark driver applications but it looks
> like I might have no choice about this.
>
>
>
> Thanks, Lucas.
>
>
>
> *From:* Saisai Shao [mailto:sai.sai.shao@gmail.com]
> *Sent:* 27 November 2017 02:19
> *To:* user@livy.incubator.apache.org
> *Subject:* EXT: Re: How to access Python script's stdout when running in
> Yarn cluster mode?
>
>
>
> Since you're running with yarn cluster mode, the output from your python
> script should be part of your yarn application log. you can get it via yarn
> command like yarn log -applicationId <applictionId>, or others like Yarn
> UI. Currently there's no Livy api for you to get application log via REST
> API.
>
>
>
> Thanks
>
> Jerry
>
>
>
> 2017-11-24 20:27 GMT+08:00 Partridge, Lucas (GE Aviation) <
> Lucas.Partridge@ge.com>:
>
> Hi,
>
>
>
> I’m using Livy’s GET /batches/{batchId}/log method to fetch the log lines
> from a Python script I’m running on Spark in Yarn cluster mode.
> Unfortunately the stdout from my Python script is not included in the log
> lines returned by GET /batches/{batchId}/log!
>
>
>
> Is this by design, or an unfortunate by-product of running in Yarn cluster
> mode?
>
>
>
> If this is intentional does anyone know how I can access the stdout from
> my Python script via Livy please? Preferably without having to change my
> REST client (a Java app) to use Yarn’s client deployment mode.
>
>
>
> Thanks,
>
> Lucas.
>
>
>
>
>
How to access Python script's stdout when running in Yarn cluster mode?
Posted by "Partridge, Lucas (GE Aviation)" <Lu...@ge.com>.
Thanks Jerry.
“Currently there's no Livy api for you to get application log via REST API.”
- That’s a real shame. Are there any plans to address this? Because this really limits the usefulness of Livy when I’m using Yarn cluster mode. It seems I have to make a choice between running my client in a scalable manner (yarn cluster) or being able to see the logs programmatically (yarn client), but not both. My client is multi-threaded; I don’t want it to have to host multiple concurrent Spark driver applications but it looks like I might have no choice about this.
Thanks, Lucas.
From: Saisai Shao [mailto:sai.sai.shao@gmail.com]
Sent: 27 November 2017 02:19
To: user@livy.incubator.apache.org
Subject: EXT: Re: How to access Python script's stdout when running in Yarn cluster mode?
Since you're running with yarn cluster mode, the output from your python script should be part of your yarn application log. you can get it via yarn command like yarn log -applicationId <applictionId>, or others like Yarn UI. Currently there's no Livy api for you to get application log via REST API.
Thanks
Jerry
2017-11-24 20:27 GMT+08:00 Partridge, Lucas (GE Aviation) <Lu...@ge.com>>:
Hi,
I’m using Livy’s GET /batches/{batchId}/log method to fetch the log lines from a Python script I’m running on Spark in Yarn cluster mode. Unfortunately the stdout from my Python script is not included in the log lines returned by GET /batches/{batchId}/log!
Is this by design, or an unfortunate by-product of running in Yarn cluster mode?
If this is intentional does anyone know how I can access the stdout from my Python script via Livy please? Preferably without having to change my REST client (a Java app) to use Yarn’s client deployment mode.
Thanks,
Lucas.
Re: How to access Python script's stdout when running in Yarn cluster mode?
Posted by Saisai Shao <sa...@gmail.com>.
Since you're running with yarn cluster mode, the output from your python
script should be part of your yarn application log. you can get it via yarn
command like yarn log -applicationId <applictionId>, or others like Yarn
UI. Currently there's no Livy api for you to get application log via REST
API.
Thanks
Jerry
2017-11-24 20:27 GMT+08:00 Partridge, Lucas (GE Aviation) <
Lucas.Partridge@ge.com>:
> Hi,
>
>
>
> I’m using Livy’s GET /batches/{batchId}/log method to fetch the log lines
> from a Python script I’m running on Spark in Yarn cluster mode.
> Unfortunately the stdout from my Python script is not included in the log
> lines returned by GET /batches/{batchId}/log!
>
>
>
> Is this by design, or an unfortunate by-product of running in Yarn cluster
> mode?
>
>
>
> If this is intentional does anyone know how I can access the stdout from
> my Python script via Livy please? Preferably without having to change my
> REST client (a Java app) to use Yarn’s client deployment mode.
>
>
>
> Thanks,
>
> Lucas.
>
>
>