You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Juho Autio <ju...@rovio.com> on 2018/04/27 09:13:50 UTC
Application logs missing from jobmanager log
The logs logged by my job jar before env.execute can't be found in
jobmanager log. I can't find them anywhere else either.
I can see all the usual logs by Flink components in the jobmanager log,
though. And in taskmanager log I can see both Flink's internal & my
application's logs from the execution time operations. But indeed missing
the logs from before env.execute.
I'm out of ideas how to debug this further. Any suggestions, or is this the
known behaviour?
I'm submitting my flink jobs in yarn cluster mode. Using Flink 1.5-SNAPSHOT.
Re: Application logs missing from jobmanager log
Posted by Fabian Hueske <fh...@gmail.com>.
Great, thank you!
2018-05-11 10:31 GMT+02:00 Juho Autio <ju...@rovio.com>:
> Thanks Fabian, here's the ticket:
> https://issues.apache.org/jira/browse/FLINK-9335
>
> On Wed, May 2, 2018 at 12:53 PM, Fabian Hueske <fh...@gmail.com> wrote:
>
>> Hi Juho,
>>
>> I assume that these logs are generated from a different process, i.e.,
>> the client process and not the JM or TM process.
>> Hence, they end up in a different log file and are not covered by the log
>> collection of the UI.
>> The reason is that this process might also be run on a machine outside of
>> the cluster. So the client log file might not be accessible from the UI
>> process.
>>
>> It seems like good feature request to include the client logs though.
>> Would you mind opening a JIRA issue for this?
>>
>> Thanks, Fabian
>>
>> 2018-04-27 11:27 GMT+02:00 Juho Autio <ju...@rovio.com>:
>>
>>> Ah, found the place! In my case they seem to be going to
>>> /home/hadoop/flink-1.5-SNAPSHOT/log/flink-hadoop-client-ip-10-0-10-29.log
>>> (for example).
>>>
>>> Any reason why these can't be shown in Flink UI, maybe in jobmanager log?
>>>
>>>
>>> On Fri, Apr 27, 2018 at 12:13 PM, Juho Autio <ju...@rovio.com>
>>> wrote:
>>>
>>>> The logs logged by my job jar before env.execute can't be found in
>>>> jobmanager log. I can't find them anywhere else either.
>>>>
>>>> I can see all the usual logs by Flink components in the jobmanager log,
>>>> though. And in taskmanager log I can see both Flink's internal & my
>>>> application's logs from the execution time operations. But indeed missing
>>>> the logs from before env.execute.
>>>>
>>>> I'm out of ideas how to debug this further. Any suggestions, or is this
>>>> the known behaviour?
>>>>
>>>> I'm submitting my flink jobs in yarn cluster mode. Using Flink
>>>> 1.5-SNAPSHOT.
>>>>
>>>>
>>>
>>>
>>
>
Re: Application logs missing from jobmanager log
Posted by Juho Autio <ju...@rovio.com>.
Thanks Fabian, here's the ticket:
https://issues.apache.org/jira/browse/FLINK-9335
On Wed, May 2, 2018 at 12:53 PM, Fabian Hueske <fh...@gmail.com> wrote:
> Hi Juho,
>
> I assume that these logs are generated from a different process, i.e., the
> client process and not the JM or TM process.
> Hence, they end up in a different log file and are not covered by the log
> collection of the UI.
> The reason is that this process might also be run on a machine outside of
> the cluster. So the client log file might not be accessible from the UI
> process.
>
> It seems like good feature request to include the client logs though.
> Would you mind opening a JIRA issue for this?
>
> Thanks, Fabian
>
> 2018-04-27 11:27 GMT+02:00 Juho Autio <ju...@rovio.com>:
>
>> Ah, found the place! In my case they seem to be going to
>> /home/hadoop/flink-1.5-SNAPSHOT/log/flink-hadoop-client-ip-10-0-10-29.log
>> (for example).
>>
>> Any reason why these can't be shown in Flink UI, maybe in jobmanager log?
>>
>>
>> On Fri, Apr 27, 2018 at 12:13 PM, Juho Autio <ju...@rovio.com>
>> wrote:
>>
>>> The logs logged by my job jar before env.execute can't be found in
>>> jobmanager log. I can't find them anywhere else either.
>>>
>>> I can see all the usual logs by Flink components in the jobmanager log,
>>> though. And in taskmanager log I can see both Flink's internal & my
>>> application's logs from the execution time operations. But indeed missing
>>> the logs from before env.execute.
>>>
>>> I'm out of ideas how to debug this further. Any suggestions, or is this
>>> the known behaviour?
>>>
>>> I'm submitting my flink jobs in yarn cluster mode. Using Flink
>>> 1.5-SNAPSHOT.
>>>
>>>
>>
>>
>
Re: Application logs missing from jobmanager log
Posted by Fabian Hueske <fh...@gmail.com>.
Hi Juho,
I assume that these logs are generated from a different process, i.e., the
client process and not the JM or TM process.
Hence, they end up in a different log file and are not covered by the log
collection of the UI.
The reason is that this process might also be run on a machine outside of
the cluster. So the client log file might not be accessible from the UI
process.
It seems like good feature request to include the client logs though.
Would you mind opening a JIRA issue for this?
Thanks, Fabian
2018-04-27 11:27 GMT+02:00 Juho Autio <ju...@rovio.com>:
> Ah, found the place! In my case they seem to be going to
> /home/hadoop/flink-1.5-SNAPSHOT/log/flink-hadoop-client-ip-10-0-10-29.log
> (for example).
>
> Any reason why these can't be shown in Flink UI, maybe in jobmanager log?
>
>
> On Fri, Apr 27, 2018 at 12:13 PM, Juho Autio <ju...@rovio.com> wrote:
>
>> The logs logged by my job jar before env.execute can't be found in
>> jobmanager log. I can't find them anywhere else either.
>>
>> I can see all the usual logs by Flink components in the jobmanager log,
>> though. And in taskmanager log I can see both Flink's internal & my
>> application's logs from the execution time operations. But indeed missing
>> the logs from before env.execute.
>>
>> I'm out of ideas how to debug this further. Any suggestions, or is this
>> the known behaviour?
>>
>> I'm submitting my flink jobs in yarn cluster mode. Using Flink
>> 1.5-SNAPSHOT.
>>
>>
>
>
Re: Application logs missing from jobmanager log
Posted by Juho Autio <ju...@rovio.com>.
Ah, found the place! In my case they seem to be going to
/home/hadoop/flink-1.5-SNAPSHOT/log/flink-hadoop-client-ip-10-0-10-29.log
(for example).
Any reason why these can't be shown in Flink UI, maybe in jobmanager log?
On Fri, Apr 27, 2018 at 12:13 PM, Juho Autio <ju...@rovio.com> wrote:
> The logs logged by my job jar before env.execute can't be found in
> jobmanager log. I can't find them anywhere else either.
>
> I can see all the usual logs by Flink components in the jobmanager log,
> though. And in taskmanager log I can see both Flink's internal & my
> application's logs from the execution time operations. But indeed missing
> the logs from before env.execute.
>
> I'm out of ideas how to debug this further. Any suggestions, or is this
> the known behaviour?
>
> I'm submitting my flink jobs in yarn cluster mode. Using Flink
> 1.5-SNAPSHOT.
>
>