You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by David Kim <da...@braintreepayments.com> on 2016/05/20 17:35:25 UTC
Logging Exceptions
Hello!
Using flink 1.0.2, I noticed that exceptions thrown during a flink program
would show up on the flink dashboard in the 'Exceptions' tab. That's great!
However, I don't think flink currently logs this same exception. I was
hoping there would be an equivalent `log.error` call so that third party
logging frameworks can also act upon such errors.
If this currently the known behavior, would it be troublesome to also make
a `log.error` call around the code that is responsible for sending the
exception to the dashboard?
If there's a misconfiguration on my end, let me know!
Thanks!
David
Re: Logging Exceptions
Posted by David Kim <da...@braintreepayments.com>.
Awesome, thank you!
David
On Wed, May 25, 2016 at 4:54 AM Aljoscha Krettek <al...@apache.org>
wrote:
> Hi David,
> you are right, for some exceptions Flink only forwards to the
> web-dashboard/application client but does not print to the log file. I
> opened a Jira issue to track this: FLINK-3969
> <https://issues.apache.org/jira/browse/FLINK-3969>.
>
> Thanks for reporting!
>
> Aljoscha
>
> On Mon, 23 May 2016 at 22:08 David Kim <da...@braintreepayments.com>
> wrote:
>
>> Hi Max!
>>
>> Unfortunately, that's not the behavior I'm seeing.
>>
>> I verified my log4.properties is configured properly because I do see
>> messages in the /log directory.
>>
>> However, for this stack trace (grabbed from the web dashboard), I do not
>> see it in my log file:
>>
>> java.lang.RuntimeException: Could not forward element to next operator
>> at org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:354)
>> at org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:337)
>> at org.apache.flink.streaming.api.operators.StreamSource$NonTimestampContext.collect(StreamSource.java:158)
>> at org.apache.flink.streaming.api.functions.source.FromElementsFunction.run(FromElementsFunction.java:127)
>> at org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:78)
>> at org.apache.flink.streaming.runtime.tasks.SourceStreamTask.run(SourceStreamTask.java:56)
>> at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:225)
>> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:559)
>> at java.lang.Thread.run(Thread.java:745)
>>
>>
>> Here's my sample program:
>>
>>
>> val env = StreamExecutionEnvironment.getExecutionEnvironment
>> env
>> .fromCollection(List(1))
>> .filter(element => {
>> throw new RuntimeException("Throwing a runtime to test exception logging :)")
>> })
>> env.execute("MyTestJobName")
>>
>>
>> The error is thrown in Task.java[1] and it was hard to track where flink would actually log to the log file.
>>
>>
>> [1] https://github.com/apache/flink/blob/release-1.0.2-rc3/flink-runtime/src/main/java/org/apache/flink/runtime/taskmanager/Task.java#L559
>>
>>
>> Thanks!
>>
>> David
>>
>>
>> On Mon, May 23, 2016 at 12:01 PM Maximilian Michels <mx...@apache.org>
>> wrote:
>>
>>> Hi David,
>>>
>>> I'm afraid Flink logs all exceptions. You'll find the exceptions in the
>>> /log directory.
>>>
>>> Cheers,
>>> Max
>>>
>>> On Mon, May 23, 2016 at 6:18 PM, David Kim <
>>> david.kim@braintreepayments.com> wrote:
>>>
>>>> Hello!
>>>>
>>>> Just wanted to check up on this. :)
>>>>
>>>> I grepped around for `log.error` and it *seems* that currently the
>>>> only events for logging out exceptions are for non-application related
>>>> errors.
>>>>
>>>> Thanks!
>>>> David
>>>>
>>>> On Fri, May 20, 2016 at 12:35 PM David Kim <
>>>> david.kim@braintreepayments.com> wrote:
>>>>
>>>>> Hello!
>>>>>
>>>>> Using flink 1.0.2, I noticed that exceptions thrown during a flink
>>>>> program would show up on the flink dashboard in the 'Exceptions' tab.
>>>>> That's great!
>>>>>
>>>>> However, I don't think flink currently logs this same exception. I was
>>>>> hoping there would be an equivalent `log.error` call so that third party
>>>>> logging frameworks can also act upon such errors.
>>>>>
>>>>> If this currently the known behavior, would it be troublesome to also
>>>>> make a `log.error` call around the code that is responsible for sending the
>>>>> exception to the dashboard?
>>>>>
>>>>> If there's a misconfiguration on my end, let me know!
>>>>>
>>>>> Thanks!
>>>>> David
>>>>>
>>>>
>>>
Re: Logging Exceptions
Posted by Aljoscha Krettek <al...@apache.org>.
Hi David,
you are right, for some exceptions Flink only forwards to the
web-dashboard/application client but does not print to the log file. I
opened a Jira issue to track this: FLINK-3969
<https://issues.apache.org/jira/browse/FLINK-3969>.
Thanks for reporting!
Aljoscha
On Mon, 23 May 2016 at 22:08 David Kim <da...@braintreepayments.com>
wrote:
> Hi Max!
>
> Unfortunately, that's not the behavior I'm seeing.
>
> I verified my log4.properties is configured properly because I do see
> messages in the /log directory.
>
> However, for this stack trace (grabbed from the web dashboard), I do not
> see it in my log file:
>
> java.lang.RuntimeException: Could not forward element to next operator
> at org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:354)
> at org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:337)
> at org.apache.flink.streaming.api.operators.StreamSource$NonTimestampContext.collect(StreamSource.java:158)
> at org.apache.flink.streaming.api.functions.source.FromElementsFunction.run(FromElementsFunction.java:127)
> at org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:78)
> at org.apache.flink.streaming.runtime.tasks.SourceStreamTask.run(SourceStreamTask.java:56)
> at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:225)
> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:559)
> at java.lang.Thread.run(Thread.java:745)
>
>
> Here's my sample program:
>
>
> val env = StreamExecutionEnvironment.getExecutionEnvironment
> env
> .fromCollection(List(1))
> .filter(element => {
> throw new RuntimeException("Throwing a runtime to test exception logging :)")
> })
> env.execute("MyTestJobName")
>
>
> The error is thrown in Task.java[1] and it was hard to track where flink would actually log to the log file.
>
>
> [1] https://github.com/apache/flink/blob/release-1.0.2-rc3/flink-runtime/src/main/java/org/apache/flink/runtime/taskmanager/Task.java#L559
>
>
> Thanks!
>
> David
>
>
> On Mon, May 23, 2016 at 12:01 PM Maximilian Michels <mx...@apache.org>
> wrote:
>
>> Hi David,
>>
>> I'm afraid Flink logs all exceptions. You'll find the exceptions in the
>> /log directory.
>>
>> Cheers,
>> Max
>>
>> On Mon, May 23, 2016 at 6:18 PM, David Kim <
>> david.kim@braintreepayments.com> wrote:
>>
>>> Hello!
>>>
>>> Just wanted to check up on this. :)
>>>
>>> I grepped around for `log.error` and it *seems* that currently the only
>>> events for logging out exceptions are for non-application related errors.
>>>
>>> Thanks!
>>> David
>>>
>>> On Fri, May 20, 2016 at 12:35 PM David Kim <
>>> david.kim@braintreepayments.com> wrote:
>>>
>>>> Hello!
>>>>
>>>> Using flink 1.0.2, I noticed that exceptions thrown during a flink
>>>> program would show up on the flink dashboard in the 'Exceptions' tab.
>>>> That's great!
>>>>
>>>> However, I don't think flink currently logs this same exception. I was
>>>> hoping there would be an equivalent `log.error` call so that third party
>>>> logging frameworks can also act upon such errors.
>>>>
>>>> If this currently the known behavior, would it be troublesome to also
>>>> make a `log.error` call around the code that is responsible for sending the
>>>> exception to the dashboard?
>>>>
>>>> If there's a misconfiguration on my end, let me know!
>>>>
>>>> Thanks!
>>>> David
>>>>
>>>
>>
Re: Logging Exceptions
Posted by David Kim <da...@braintreepayments.com>.
Hi Max!
Unfortunately, that's not the behavior I'm seeing.
I verified my log4.properties is configured properly because I do see
messages in the /log directory.
However, for this stack trace (grabbed from the web dashboard), I do not
see it in my log file:
java.lang.RuntimeException: Could not forward element to next operator
at org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:354)
at org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:337)
at org.apache.flink.streaming.api.operators.StreamSource$NonTimestampContext.collect(StreamSource.java:158)
at org.apache.flink.streaming.api.functions.source.FromElementsFunction.run(FromElementsFunction.java:127)
at org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:78)
at org.apache.flink.streaming.runtime.tasks.SourceStreamTask.run(SourceStreamTask.java:56)
at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:225)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:559)
at java.lang.Thread.run(Thread.java:745)
Here's my sample program:
val env = StreamExecutionEnvironment.getExecutionEnvironment
env
.fromCollection(List(1))
.filter(element => {
throw new RuntimeException("Throwing a runtime to test exception
logging :)")
})
env.execute("MyTestJobName")
The error is thrown in Task.java[1] and it was hard to track where
flink would actually log to the log file.
[1] https://github.com/apache/flink/blob/release-1.0.2-rc3/flink-runtime/src/main/java/org/apache/flink/runtime/taskmanager/Task.java#L559
Thanks!
David
On Mon, May 23, 2016 at 12:01 PM Maximilian Michels <mx...@apache.org> wrote:
> Hi David,
>
> I'm afraid Flink logs all exceptions. You'll find the exceptions in the
> /log directory.
>
> Cheers,
> Max
>
> On Mon, May 23, 2016 at 6:18 PM, David Kim <
> david.kim@braintreepayments.com> wrote:
>
>> Hello!
>>
>> Just wanted to check up on this. :)
>>
>> I grepped around for `log.error` and it *seems* that currently the only
>> events for logging out exceptions are for non-application related errors.
>>
>> Thanks!
>> David
>>
>> On Fri, May 20, 2016 at 12:35 PM David Kim <
>> david.kim@braintreepayments.com> wrote:
>>
>>> Hello!
>>>
>>> Using flink 1.0.2, I noticed that exceptions thrown during a flink
>>> program would show up on the flink dashboard in the 'Exceptions' tab.
>>> That's great!
>>>
>>> However, I don't think flink currently logs this same exception. I was
>>> hoping there would be an equivalent `log.error` call so that third party
>>> logging frameworks can also act upon such errors.
>>>
>>> If this currently the known behavior, would it be troublesome to also
>>> make a `log.error` call around the code that is responsible for sending the
>>> exception to the dashboard?
>>>
>>> If there's a misconfiguration on my end, let me know!
>>>
>>> Thanks!
>>> David
>>>
>>
>
Re: Logging Exceptions
Posted by Maximilian Michels <mx...@apache.org>.
Hi David,
I'm afraid Flink logs all exceptions. You'll find the exceptions in the
/log directory.
Cheers,
Max
On Mon, May 23, 2016 at 6:18 PM, David Kim <da...@braintreepayments.com>
wrote:
> Hello!
>
> Just wanted to check up on this. :)
>
> I grepped around for `log.error` and it *seems* that currently the only
> events for logging out exceptions are for non-application related errors.
>
> Thanks!
> David
>
> On Fri, May 20, 2016 at 12:35 PM David Kim <
> david.kim@braintreepayments.com> wrote:
>
>> Hello!
>>
>> Using flink 1.0.2, I noticed that exceptions thrown during a flink
>> program would show up on the flink dashboard in the 'Exceptions' tab.
>> That's great!
>>
>> However, I don't think flink currently logs this same exception. I was
>> hoping there would be an equivalent `log.error` call so that third party
>> logging frameworks can also act upon such errors.
>>
>> If this currently the known behavior, would it be troublesome to also
>> make a `log.error` call around the code that is responsible for sending the
>> exception to the dashboard?
>>
>> If there's a misconfiguration on my end, let me know!
>>
>> Thanks!
>> David
>>
>
Re: Logging Exceptions
Posted by David Kim <da...@braintreepayments.com>.
Hello!
Just wanted to check up on this. :)
I grepped around for `log.error` and it *seems* that currently the only
events for logging out exceptions are for non-application related errors.
Thanks!
David
On Fri, May 20, 2016 at 12:35 PM David Kim <da...@braintreepayments.com>
wrote:
> Hello!
>
> Using flink 1.0.2, I noticed that exceptions thrown during a flink program
> would show up on the flink dashboard in the 'Exceptions' tab. That's great!
>
> However, I don't think flink currently logs this same exception. I was
> hoping there would be an equivalent `log.error` call so that third party
> logging frameworks can also act upon such errors.
>
> If this currently the known behavior, would it be troublesome to also make
> a `log.error` call around the code that is responsible for sending the
> exception to the dashboard?
>
> If there's a misconfiguration on my end, let me know!
>
> Thanks!
> David
>