You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Tao Xia <ta...@udacity.com> on 2018/04/13 00:09:03 UTC
Access logs for a running Flink app in YARN cluster
Any good way to get access container logs from a running Flink app in YARN
cluster in EMR?
You can view the logs through YARN UI. But cannot programmatically access
it and send to other services.
The log aggregator only runs when the application finishes or a minimum
3600 secs copy. Any way we can get the logs more frequently?
Re: Access logs for a running Flink app in YARN cluster
Posted by 杨力 <bi...@gmail.com>.
Maybe you can get them from yarn with rest API.
Tao Xia <ta...@udacity.com> 于 2018年4月13日周五 上午8:09写道：
> Any good way to get access container logs from a running Flink app in YARN
> cluster in EMR?
> You can view the logs through YARN UI. But cannot programmatically access
> it and send to other services.
> The log aggregator only runs when the application finishes or a minimum
> 3600 secs copy. Any way we can get the logs more frequently?
>
Re: Access logs for a running Flink app in YARN cluster
Posted by Tao Xia <ta...@udacity.com>.
Thank you all for the tips. Will give a try.
On Fri, Apr 13, 2018 at 12:13 PM, Gary Yao <ga...@data-artisans.com> wrote:
> Hi,
>
> I see two options:
>
> 1. You can login to the slave machines, which run the NodeManagers, and
> access
> the container logs. The path of the container logs can be configured in
> yarn-site.xml with the key yarn.nodemanager.log-dirs. In my tests with
> EMR, the
> logs are stored at /var/log/hadoop-yarn/containers/ by default. For
> example:
>
> sudo ls /var/log/hadoop-yarn/containers/application_
> 1523645835493_0002/container_1523645835493_0002_01_000002
> taskmanager.err taskmanager.log taskmanager.out
>
> 2. You can download the logs via HTTP from Flink:
>
> http://host:port/jobmanager/log
> http://host:port/taskmanagers/<taskmanager-id>/log
>
> To get a list of taskmanagers:
>
> http://host:port/taskmanagers
>
> Best,
> Gary
>
> On Fri, Apr 13, 2018 at 2:09 AM, Tao Xia <ta...@udacity.com> wrote:
>
>> Any good way to get access container logs from a running Flink app in
>> YARN cluster in EMR?
>> You can view the logs through YARN UI. But cannot programmatically access
>> it and send to other services.
>> The log aggregator only runs when the application finishes or a minimum
>> 3600 secs copy. Any way we can get the logs more frequently?
>>
>
>
Re: Access logs for a running Flink app in YARN cluster
Posted by Gary Yao <ga...@data-artisans.com>.
Hi,
I see two options:
1. You can login to the slave machines, which run the NodeManagers, and
access
the container logs. The path of the container logs can be configured in
yarn-site.xml with the key yarn.nodemanager.log-dirs. In my tests with EMR,
the
logs are stored at /var/log/hadoop-yarn/containers/ by default. For example:
sudo ls
/var/log/hadoop-yarn/containers/application_1523645835493_0002/container_1523645835493_0002_01_000002
taskmanager.err taskmanager.log taskmanager.out
2. You can download the logs via HTTP from Flink:
http://host:port/jobmanager/log
http://host:port/taskmanagers/<taskmanager-id>/log
To get a list of taskmanagers:
http://host:port/taskmanagers
Best,
Gary
On Fri, Apr 13, 2018 at 2:09 AM, Tao Xia <ta...@udacity.com> wrote:
> Any good way to get access container logs from a running Flink app in YARN
> cluster in EMR?
> You can view the logs through YARN UI. But cannot programmatically access
> it and send to other services.
> The log aggregator only runs when the application finishes or a minimum
> 3600 secs copy. Any way we can get the logs more frequently?
>