You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Rex Fenley <Re...@remind101.com> on 2020/10/26 23:03:26 UTC
EMR Logging Woes
Hello,
After lots of testing in local environments we're now trying to get our
cluster running on AWS EMR. We followed much of the documentation from both
AWS and Flink and have gotten to the point of creating a yarn session and
submitting jobs. We successfully get back a Job ID and in the Yarn Timeline
Server UI it says our application is running. However, we are having a hard
time with logging.
2 main issues:
1. Logs for the jobmanager and taskmanager seem to take a long time to show
up or in some cases just seem to never show up in the Yarn / Hadoop UI,
even though we can see them just fine when ssh'ing into the cluster's
nodes. Anything we can do to speed this up?
2. We can't seem to see anything except for WARN and ERROR logs for the
jobmanager and taskmanager, we need at least INFO right now to confirm
things are working as expected. We have been jumping through hoops going
through a multitude of configuration files including
log4j-session.properties and log4j.properties setting level to DEBUG but it
has not helped. Are these the correct configuration files?
Thanks!
--
Rex Fenley | Software Engineer - Mobile and Backend
Remind.com <https://www.remind.com/> | BLOG <http://blog.remind.com/>
| FOLLOW
US <https://twitter.com/remindhq> | LIKE US
<https://www.facebook.com/remindhq>
Re: EMR Logging Woes
Posted by Rex Fenley <Re...@remind101.com>.
Thanks! I'll check these out.
On Tue, Oct 27, 2020 at 12:58 AM Robert Metzger <rm...@apache.org> wrote:
> Hi Rex,
>
> 1. You can also use the Flink UI for retrieving logs. That usually works
> quite fast (unless your logs are huge).
>
> 2. These are the correct configuration files for setting the log level.
> Are you running on a vanilla EMR cluster, or are there modifications? The
> "problem" is that Flink on YARN adds jar files (and other files) provided
> by the environment (YARN) to its classpath. The vanilla EMR configuration
> should be fine to not interfere with Flink's logging. But maybe there are
> some changes in your environment that cause problems?
>
> Since you are SSHing into the machines already: At the top of each Flink
> log file, we are logging the location of the log4j configuration file
> (search for "-Dlog4j.configuration="). Try to open that file to verify
> what's in there.
>
> Hope this helps!
>
> Robert
>
>
> On Tue, Oct 27, 2020 at 12:03 AM Rex Fenley <Re...@remind101.com> wrote:
>
>> Hello,
>>
>> After lots of testing in local environments we're now trying to get our
>> cluster running on AWS EMR. We followed much of the documentation from both
>> AWS and Flink and have gotten to the point of creating a yarn session and
>> submitting jobs. We successfully get back a Job ID and in the Yarn Timeline
>> Server UI it says our application is running. However, we are having a hard
>> time with logging.
>>
>> 2 main issues:
>> 1. Logs for the jobmanager and taskmanager seem to take a long time to
>> show up or in some cases just seem to never show up in the Yarn / Hadoop
>> UI, even though we can see them just fine when ssh'ing into the cluster's
>> nodes. Anything we can do to speed this up?
>>
>> 2. We can't seem to see anything except for WARN and ERROR logs for the
>> jobmanager and taskmanager, we need at least INFO right now to confirm
>> things are working as expected. We have been jumping through hoops going
>> through a multitude of configuration files including
>> log4j-session.properties and log4j.properties setting level to DEBUG but
>> it has not helped. Are these the correct configuration files?
>>
>> Thanks!
>>
>> --
>>
>> Rex Fenley | Software Engineer - Mobile and Backend
>>
>>
>> Remind.com <https://www.remind.com/> | BLOG <http://blog.remind.com/>
>> | FOLLOW US <https://twitter.com/remindhq> | LIKE US
>> <https://www.facebook.com/remindhq>
>>
>
--
Rex Fenley | Software Engineer - Mobile and Backend
Remind.com <https://www.remind.com/> | BLOG <http://blog.remind.com/>
| FOLLOW
US <https://twitter.com/remindhq> | LIKE US
<https://www.facebook.com/remindhq>
Re: EMR Logging Woes
Posted by Robert Metzger <rm...@apache.org>.
Hi Rex,
1. You can also use the Flink UI for retrieving logs. That usually works
quite fast (unless your logs are huge).
2. These are the correct configuration files for setting the log level. Are
you running on a vanilla EMR cluster, or are there modifications? The
"problem" is that Flink on YARN adds jar files (and other files) provided
by the environment (YARN) to its classpath. The vanilla EMR configuration
should be fine to not interfere with Flink's logging. But maybe there are
some changes in your environment that cause problems?
Since you are SSHing into the machines already: At the top of each Flink
log file, we are logging the location of the log4j configuration file
(search for "-Dlog4j.configuration="). Try to open that file to verify
what's in there.
Hope this helps!
Robert
On Tue, Oct 27, 2020 at 12:03 AM Rex Fenley <Re...@remind101.com> wrote:
> Hello,
>
> After lots of testing in local environments we're now trying to get our
> cluster running on AWS EMR. We followed much of the documentation from both
> AWS and Flink and have gotten to the point of creating a yarn session and
> submitting jobs. We successfully get back a Job ID and in the Yarn Timeline
> Server UI it says our application is running. However, we are having a hard
> time with logging.
>
> 2 main issues:
> 1. Logs for the jobmanager and taskmanager seem to take a long time to
> show up or in some cases just seem to never show up in the Yarn / Hadoop
> UI, even though we can see them just fine when ssh'ing into the cluster's
> nodes. Anything we can do to speed this up?
>
> 2. We can't seem to see anything except for WARN and ERROR logs for the
> jobmanager and taskmanager, we need at least INFO right now to confirm
> things are working as expected. We have been jumping through hoops going
> through a multitude of configuration files including
> log4j-session.properties and log4j.properties setting level to DEBUG but
> it has not helped. Are these the correct configuration files?
>
> Thanks!
>
> --
>
> Rex Fenley | Software Engineer - Mobile and Backend
>
>
> Remind.com <https://www.remind.com/> | BLOG <http://blog.remind.com/> |
> FOLLOW US <https://twitter.com/remindhq> | LIKE US
> <https://www.facebook.com/remindhq>
>