Re: [External] Re: Consuming Messages from Kafka

["Conlin, Joshua [USA]" <co...@bah.com>]

"StringLogSink" just looks like:


System.out.println(msg);

LOG.info("Logging message: " + msg);


And LOG is from slf4j.  In the Flink UI that is running on Yarn, I see no counts, nor log statements or stdout under JobManager.  It seems to make no difference if I submit the job through yarn via command line or the Flink UI session already running under yarn.  Where would you recommend I look in the Yarn containers?


Thanks again for your help.


Josh

From: Robert Metzger <rm...@apache.org>>
Reply-To: "user@flink.apache.org<ma...@flink.apache.org>" <us...@flink.apache.org>>
Date: Tuesday, April 26, 2016 at 3:30 PM
To: "user@flink.apache.org<ma...@flink.apache.org>" <us...@flink.apache.org>>
Subject: [External] Re: Consuming Messages from Kafka

Hi,

the web interface is a good idea for checking if everything is working as expected. However in this case I expect the counts for the task be 0 because the source and sink are chained together into one task (upcoming Flink releases will fix this behavior).

I assume the "StringLogSink" is logging all incoming events. How do you do that? Using slf4j ? our by System.out.println?
I'm asking to make sure you're looking at the right place to capture the output. It will be at the YARN containers.

Regards,
Robert


On Tue, Apr 26, 2016 at 8:34 PM, Dominik Choma <do...@gmail.com>> wrote:
Hi,

You can check if any messages are going through dataflow on flink web dashboard
https://flink.apache.org/img/blog/new-dashboard-screenshot.png



Dominik Choma

Wiadomość napisana przez Conlin, Joshua [USA] <co...@bah.com>> w dniu 26 kwi 2016, o godz. 20:16:

re messages being sent to Kafka on that topic, I just never see anything in Flink.  Any help/insight you could provide would be greatly appreciated.  If it makes a difference this is running on YARN.  Also, here's what I see in the logs:

Re: [External] Re: Consuming Messages from Kafka

[Robert Metzger <rm...@apache.org>]

Hi Josh,

The JobManager log won't contain this output.

Check out these slides I did a while ago, they explain how you can retrieve
the logs from the TaskManagers:
http://www.slideshare.net/robertmetzger1/apache-flink-hands-on#14



On Tue, Apr 26, 2016 at 9:41 PM, Conlin, Joshua [USA] <conlin_joshua@bah.com
> wrote:

> “StringLogSink” just looks like:
>
> System.out.println(msg);
>
> LOG.info("Logging message: " + msg);
>
>
> And LOG is from slf4j.  In the Flink UI that is running on Yarn, I see no
> counts, nor log statements or stdout under JobManager.  It seems to make no
> difference if I submit the job through yarn via command line or the Flink
> UI session already running under yarn.  Where would you recommend I look in
> the Yarn containers?
>
>
> Thanks again for your help.
>
>
> Josh
>
> From: Robert Metzger <rm...@apache.org>
> Reply-To: "user@flink.apache.org" <us...@flink.apache.org>
> Date: Tuesday, April 26, 2016 at 3:30 PM
> To: "user@flink.apache.org" <us...@flink.apache.org>
> Subject: [External] Re: Consuming Messages from Kafka
>
> Hi,
>
> the web interface is a good idea for checking if everything is working as
> expected. However in this case I expect the counts for the task be 0
> because the source and sink are chained together into one task (upcoming
> Flink releases will fix this behavior).
>
> I assume the "StringLogSink" is logging all incoming events. How do you do
> that? Using slf4j ? our by System.out.println?
> I'm asking to make sure you're looking at the right place to capture the
> output. It will be at the YARN containers.
>
> Regards,
> Robert
>
>
> On Tue, Apr 26, 2016 at 8:34 PM, Dominik Choma <do...@gmail.com>
> wrote:
>
>> Hi,
>>
>> You can check if any messages are going through dataflow on flink web
>> dashboard
>> https://flink.apache.org/img/blog/new-dashboard-screenshot.png
>>
>>
>>
>> Dominik Choma
>>
>> Wiadomość napisana przez Conlin, Joshua [USA] <co...@bah.com> w
>> dniu 26 kwi 2016, o godz. 20:16:
>>
>> re messages being sent to Kafka on that topic, I just never see anything
>> in Flink.  Any help/insight you could provide would be greatly
>> appreciated.  If it makes a difference this is running on YARN.  Also,
>> here’s what I see in the logs:
>>
>>
>>
>