You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Zain Haider Nemati <za...@retailo.co> on 2022/05/19 14:56:39 UTC
Job Logs - Yarn Application Mode
Hey All,
How can I check logs for my job when it is running in application mode via
yarn
Re: Job Logs - Yarn Application Mode
Posted by Shengkai Fang <fs...@gmail.com>.
If you find the JM in the yarn web ui, I think you can also find the webui
to access the Flink web ui with the JM.
Best,
Shengkai
Re: Job Logs - Yarn Application Mode
Posted by Zain Haider Nemati <za...@retailo.co>.
Hi,
Thanks for your response folks. Is it possible to access logs via flink UI
in yarn application mode? Similar to how we can access them in standalone
mode
On Fri, May 20, 2022 at 11:06 AM Shengkai Fang <fs...@gmail.com> wrote:
> Thanks for Biao's explanation.
>
> Best,
> Shengkai
>
> Biao Geng <bi...@gmail.com> 于2022年5月20日周五 11:16写道：
>
>> Hi there,
>> @Zain, Weihua's suggestion should be able to fulfill the request to check
>> JM logs. If you do want to use YARN cli for running Flink applications, it
>> is possible to check JM's log with the YARN command like:
>> *yarn logs -applicationId application_xxx_yyy -am -1 -logFiles
>> jobmanager.log*
>> For TM log, command would be like:
>> * yarn logs -applicationId <app id> -containerId <container id>
>> -logFiles taskmanager.log*
>> Note, it is not super easy to find the container id of TM. Some
>> workaround would be to check JM's log first and get the container id for TM
>> from that. You can also learn more about the details of above commands from *yarn
>> logs -help*
>>
>> @Shengkai, yes, you are right the actual JM address is managed by YARN.
>> To access the JM launched by YARN, users need to access YARN web ui to find
>> the YARN application by applicationId and then click 'application master
>> url' of that application to be redirected to Flink web ui.
>>
>> Best,
>> Biao Geng
>>
>> Shengkai Fang <fs...@gmail.com> 于2022年5月20日周五 10:59写道：
>>
>>> Hi.
>>>
>>> I am not familiar with the YARN application mode. Because the job
>>> manager is started when submit the jobs. So how can users know the address
>>> of the JM? Do we need to look up the Yarn UI to search the submitted job
>>> with the JobID?
>>>
>>> Best,
>>> Shengkai
>>>
>>> Weihua Hu <hu...@gmail.com> 于2022年5月20日周五 10:23写道：
>>>
>>>> Hi,
>>>> You can get the logs from Flink Web UI if job is running.
>>>> Best,
>>>> Weihua
>>>>
>>>> 2022年5月19日 下午10:56，Zain Haider Nemati <za...@retailo.co> 写道：
>>>>
>>>> Hey All,
>>>> How can I check logs for my job when it is running in application mode
>>>> via yarn
>>>>
>>>>
>>>>
Re: Job Logs - Yarn Application Mode
Posted by Zain Haider Nemati <za...@retailo.co>.
Hi,
Thanks for your response folks. Is it possible to access logs via flink UI
in yarn application mode? Similar to how we can access them in standalone
mode
On Fri, May 20, 2022 at 11:06 AM Shengkai Fang <fs...@gmail.com> wrote:
> Thanks for Biao's explanation.
>
> Best,
> Shengkai
>
> Biao Geng <bi...@gmail.com> 于2022年5月20日周五 11:16写道：
>
>> Hi there,
>> @Zain, Weihua's suggestion should be able to fulfill the request to check
>> JM logs. If you do want to use YARN cli for running Flink applications, it
>> is possible to check JM's log with the YARN command like:
>> *yarn logs -applicationId application_xxx_yyy -am -1 -logFiles
>> jobmanager.log*
>> For TM log, command would be like:
>> * yarn logs -applicationId <app id> -containerId <container id>
>> -logFiles taskmanager.log*
>> Note, it is not super easy to find the container id of TM. Some
>> workaround would be to check JM's log first and get the container id for TM
>> from that. You can also learn more about the details of above commands from *yarn
>> logs -help*
>>
>> @Shengkai, yes, you are right the actual JM address is managed by YARN.
>> To access the JM launched by YARN, users need to access YARN web ui to find
>> the YARN application by applicationId and then click 'application master
>> url' of that application to be redirected to Flink web ui.
>>
>> Best,
>> Biao Geng
>>
>> Shengkai Fang <fs...@gmail.com> 于2022年5月20日周五 10:59写道：
>>
>>> Hi.
>>>
>>> I am not familiar with the YARN application mode. Because the job
>>> manager is started when submit the jobs. So how can users know the address
>>> of the JM? Do we need to look up the Yarn UI to search the submitted job
>>> with the JobID?
>>>
>>> Best,
>>> Shengkai
>>>
>>> Weihua Hu <hu...@gmail.com> 于2022年5月20日周五 10:23写道：
>>>
>>>> Hi,
>>>> You can get the logs from Flink Web UI if job is running.
>>>> Best,
>>>> Weihua
>>>>
>>>> 2022年5月19日 下午10:56，Zain Haider Nemati <za...@retailo.co> 写道：
>>>>
>>>> Hey All,
>>>> How can I check logs for my job when it is running in application mode
>>>> via yarn
>>>>
>>>>
>>>>
Re: Job Logs - Yarn Application Mode
Posted by Shengkai Fang <fs...@gmail.com>.
Thanks for Biao's explanation.
Best,
Shengkai
Biao Geng <bi...@gmail.com> 于2022年5月20日周五 11:16写道：
> Hi there,
> @Zain, Weihua's suggestion should be able to fulfill the request to check
> JM logs. If you do want to use YARN cli for running Flink applications, it
> is possible to check JM's log with the YARN command like:
> *yarn logs -applicationId application_xxx_yyy -am -1 -logFiles
> jobmanager.log*
> For TM log, command would be like:
> * yarn logs -applicationId <app id> -containerId <container id> -logFiles
> taskmanager.log*
> Note, it is not super easy to find the container id of TM. Some workaround
> would be to check JM's log first and get the container id for TM from that.
> You can also learn more about the details of above commands from *yarn
> logs -help*
>
> @Shengkai, yes, you are right the actual JM address is managed by YARN. To
> access the JM launched by YARN, users need to access YARN web ui to find
> the YARN application by applicationId and then click 'application master
> url' of that application to be redirected to Flink web ui.
>
> Best,
> Biao Geng
>
> Shengkai Fang <fs...@gmail.com> 于2022年5月20日周五 10:59写道：
>
>> Hi.
>>
>> I am not familiar with the YARN application mode. Because the job manager
>> is started when submit the jobs. So how can users know the address of the
>> JM? Do we need to look up the Yarn UI to search the submitted job with the
>> JobID?
>>
>> Best,
>> Shengkai
>>
>> Weihua Hu <hu...@gmail.com> 于2022年5月20日周五 10:23写道：
>>
>>> Hi,
>>> You can get the logs from Flink Web UI if job is running.
>>> Best,
>>> Weihua
>>>
>>> 2022年5月19日 下午10:56，Zain Haider Nemati <za...@retailo.co> 写道：
>>>
>>> Hey All,
>>> How can I check logs for my job when it is running in application mode
>>> via yarn
>>>
>>>
>>>
Re: Job Logs - Yarn Application Mode
Posted by Shengkai Fang <fs...@gmail.com>.
Thanks for Biao's explanation.
Best,
Shengkai
Biao Geng <bi...@gmail.com> 于2022年5月20日周五 11:16写道：
> Hi there,
> @Zain, Weihua's suggestion should be able to fulfill the request to check
> JM logs. If you do want to use YARN cli for running Flink applications, it
> is possible to check JM's log with the YARN command like:
> *yarn logs -applicationId application_xxx_yyy -am -1 -logFiles
> jobmanager.log*
> For TM log, command would be like:
> * yarn logs -applicationId <app id> -containerId <container id> -logFiles
> taskmanager.log*
> Note, it is not super easy to find the container id of TM. Some workaround
> would be to check JM's log first and get the container id for TM from that.
> You can also learn more about the details of above commands from *yarn
> logs -help*
>
> @Shengkai, yes, you are right the actual JM address is managed by YARN. To
> access the JM launched by YARN, users need to access YARN web ui to find
> the YARN application by applicationId and then click 'application master
> url' of that application to be redirected to Flink web ui.
>
> Best,
> Biao Geng
>
> Shengkai Fang <fs...@gmail.com> 于2022年5月20日周五 10:59写道：
>
>> Hi.
>>
>> I am not familiar with the YARN application mode. Because the job manager
>> is started when submit the jobs. So how can users know the address of the
>> JM? Do we need to look up the Yarn UI to search the submitted job with the
>> JobID?
>>
>> Best,
>> Shengkai
>>
>> Weihua Hu <hu...@gmail.com> 于2022年5月20日周五 10:23写道：
>>
>>> Hi,
>>> You can get the logs from Flink Web UI if job is running.
>>> Best,
>>> Weihua
>>>
>>> 2022年5月19日 下午10:56，Zain Haider Nemati <za...@retailo.co> 写道：
>>>
>>> Hey All,
>>> How can I check logs for my job when it is running in application mode
>>> via yarn
>>>
>>>
>>>
Re: Job Logs - Yarn Application Mode
Posted by Biao Geng <bi...@gmail.com>.
Hi there,
@Zain, Weihua's suggestion should be able to fulfill the request to check
JM logs. If you do want to use YARN cli for running Flink applications, it
is possible to check JM's log with the YARN command like:
*yarn logs -applicationId application_xxx_yyy -am -1 -logFiles
jobmanager.log*
For TM log, command would be like:
* yarn logs -applicationId <app id> -containerId <container id> -logFiles
taskmanager.log*
Note, it is not super easy to find the container id of TM. Some workaround
would be to check JM's log first and get the container id for TM from that.
You can also learn more about the details of above commands from *yarn logs
-help*
@Shengkai, yes, you are right the actual JM address is managed by YARN. To
access the JM launched by YARN, users need to access YARN web ui to find
the YARN application by applicationId and then click 'application master
url' of that application to be redirected to Flink web ui.
Best,
Biao Geng
Shengkai Fang <fs...@gmail.com> 于2022年5月20日周五 10:59写道：
> Hi.
>
> I am not familiar with the YARN application mode. Because the job manager
> is started when submit the jobs. So how can users know the address of the
> JM? Do we need to look up the Yarn UI to search the submitted job with the
> JobID?
>
> Best,
> Shengkai
>
> Weihua Hu <hu...@gmail.com> 于2022年5月20日周五 10:23写道：
>
>> Hi,
>> You can get the logs from Flink Web UI if job is running.
>> Best,
>> Weihua
>>
>> 2022年5月19日 下午10:56，Zain Haider Nemati <za...@retailo.co> 写道：
>>
>> Hey All,
>> How can I check logs for my job when it is running in application mode
>> via yarn
>>
>>
>>
Re: Job Logs - Yarn Application Mode
Posted by Biao Geng <bi...@gmail.com>.
Hi there,
@Zain, Weihua's suggestion should be able to fulfill the request to check
JM logs. If you do want to use YARN cli for running Flink applications, it
is possible to check JM's log with the YARN command like:
*yarn logs -applicationId application_xxx_yyy -am -1 -logFiles
jobmanager.log*
For TM log, command would be like:
* yarn logs -applicationId <app id> -containerId <container id> -logFiles
taskmanager.log*
Note, it is not super easy to find the container id of TM. Some workaround
would be to check JM's log first and get the container id for TM from that.
You can also learn more about the details of above commands from *yarn logs
-help*
@Shengkai, yes, you are right the actual JM address is managed by YARN. To
access the JM launched by YARN, users need to access YARN web ui to find
the YARN application by applicationId and then click 'application master
url' of that application to be redirected to Flink web ui.
Best,
Biao Geng
Shengkai Fang <fs...@gmail.com> 于2022年5月20日周五 10:59写道：
> Hi.
>
> I am not familiar with the YARN application mode. Because the job manager
> is started when submit the jobs. So how can users know the address of the
> JM? Do we need to look up the Yarn UI to search the submitted job with the
> JobID?
>
> Best,
> Shengkai
>
> Weihua Hu <hu...@gmail.com> 于2022年5月20日周五 10:23写道：
>
>> Hi,
>> You can get the logs from Flink Web UI if job is running.
>> Best,
>> Weihua
>>
>> 2022年5月19日 下午10:56，Zain Haider Nemati <za...@retailo.co> 写道：
>>
>> Hey All,
>> How can I check logs for my job when it is running in application mode
>> via yarn
>>
>>
>>
Re: Job Logs - Yarn Application Mode
Posted by Shengkai Fang <fs...@gmail.com>.
Hi.
I am not familiar with the YARN application mode. Because the job manager
is started when submit the jobs. So how can users know the address of the
JM? Do we need to look up the Yarn UI to search the submitted job with the
JobID?
Best,
Shengkai
Weihua Hu <hu...@gmail.com> 于2022年5月20日周五 10:23写道：
> Hi,
> You can get the logs from Flink Web UI if job is running.
> Best,
> Weihua
>
> 2022年5月19日 下午10:56，Zain Haider Nemati <za...@retailo.co> 写道：
>
> Hey All,
> How can I check logs for my job when it is running in application mode via
> yarn
>
>
>
Re: Job Logs - Yarn Application Mode
Posted by Shengkai Fang <fs...@gmail.com>.
Hi.
I am not familiar with the YARN application mode. Because the job manager
is started when submit the jobs. So how can users know the address of the
JM? Do we need to look up the Yarn UI to search the submitted job with the
JobID?
Best,
Shengkai
Weihua Hu <hu...@gmail.com> 于2022年5月20日周五 10:23写道：
> Hi,
> You can get the logs from Flink Web UI if job is running.
> Best,
> Weihua
>
> 2022年5月19日 下午10:56，Zain Haider Nemati <za...@retailo.co> 写道：
>
> Hey All,
> How can I check logs for my job when it is running in application mode via
> yarn
>
>
>
Re: Job Logs - Yarn Application Mode
Posted by Weihua Hu <hu...@gmail.com>.
Hi,
You can get the logs from Flink Web UI if job is running.
Best,
Weihua
> 2022年5月19日 下午10:56，Zain Haider Nemati <za...@retailo.co> 写道：
>
> Hey All,
> How can I check logs for my job when it is running in application mode via yarn
Re: Job Logs - Yarn Application Mode
Posted by Weihua Hu <hu...@gmail.com>.
Hi,
You can get the logs from Flink Web UI if job is running.
Best,
Weihua
> 2022年5月19日 下午10:56，Zain Haider Nemati <za...@retailo.co> 写道：
>
> Hey All,
> How can I check logs for my job when it is running in application mode via yarn