You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Filip Łęczycki <fi...@gmail.com> on 2015/12/08 18:56:16 UTC
Using memory logging in Flink
Hi,
I am trying to enable logging of memory usage on flink 0.10.0 by adding:
taskmanager.debug.memory.startLogThread: true
to config.yaml and setting log4j level to DEBUG however in the logs
after running the job I cannot see any info regarding memory usage.My
job lasted 30s so it should catch few intervals. Should I change
something else in the configuration?
Best regards/Pozdrawiam,
Filip Łęczycki
Re: Using memory logging in Flink
Posted by Till Rohrmann <tr...@apache.org>.
How do you start the flink cluster? If it's a standalone cluster and you
don't use a shared directory, then you'll find the log of the taskmanager
on the machine on which the taskmanager runs. If you use YARN then you can
activate log aggregation to retrieve the log easily after the job has
finished.
Cheers,
Till
On Wed, Dec 9, 2015 at 2:30 PM, Filip Łęczycki <fi...@gmail.com>
wrote:
> Hi Stephan, Till
>
> I am looking for the memory usage info in flink_<user>_jobmanager... log.
> As there are no other log files in /log dir I assumed this is the right
> place to look. I will try to setup a new clean instance and check whether
> it will work.
>
> Thank you very much for your assistance!
> Best regards,
> Filip Łęczycki
>
> Pozdrawiam,
> Filip Łęczycki
>
> 2015-12-09 11:13 GMT+01:00 Stephan Ewen <se...@apache.org>:
>
>> Hi Filip!
>>
>> Someone else just used the memory logging with the exact described
>> settings - it worked.
>>
>> There is probably some mixup, you may be looking into the wrong log file,
>> or may setting the a value in a different config...
>>
>> Stephan
>>
>>
>> On Wed, Dec 9, 2015 at 10:58 AM, Till Rohrmann <tr...@apache.org>
>> wrote:
>>
>>> I assume you're looking in the taskmanager log file for the memory usage
>>> logging statements, right?
>>>
>>> Cheers,
>>> Till
>>>
>>> On Wed, Dec 9, 2015 at 12:15 AM, Filip Łęczycki <filipleczycki@gmail.com
>>> > wrote:
>>>
>>>> Hi,
>>>>
>>>> Thank you for your reply!
>>>>
>>>> I have made sure I restarted the TaskManager after changing config, but
>>>> it didn't resolve the issue.The config is loaded as I can see the following
>>>> line in the log:
>>>> 09:12:2015 00:00:21,894 DEBUG
>>>> org.apache.flink.configuration.GlobalConfiguration - Loading
>>>> configuration property: taskmanager.debug.memory.startLogThread, true
>>>>
>>>> I am running a job on local standalone flink instance and my
>>>> log4j.properties look like this:
>>>> log4j.rootLogger=DEBUG, file
>>>>
>>>> # Log all infos in the given file
>>>> log4j.appender.file=org.apache.log4j.FileAppender
>>>> log4j.appender.file.file=${log.file}
>>>> log4j.appender.file.append=false
>>>> log4j.appender.file.layout=org.apache.log4j.PatternLayout
>>>> log4j.appender.file.layout.ConversionPattern=%d{dd:MM:YYY HH:mm:ss,SSS}
>>>> %-5p %-60c %x - %m%n
>>>>
>>>> # suppress the irrelevant (wrong) warnings from the netty channel
>>>> handler
>>>> log4j.logger.org.jboss.netty.channel.DefaultChannelPipeline=ERROR, file
>>>>
>>>> Maybe I can set some property in the job's code so that it would force
>>>> such verbose logging? I need those logs to run some flink performance
>>>> tests but maybe I can somehow extract the benchmark results without them, do
>>>> you know any other way to monitor Flink Job's memory usage and GC time,
>>>> other than looking at web interface?
>>>>
>>>> Best regards,
>>>> Filip Łęczycki
>>>>
>>>> Pozdrawiam,
>>>> Filip Łęczycki
>>>>
>>>> 2015-12-08 20:48 GMT+01:00 Stephan Ewen <se...@apache.org>:
>>>>
>>>>> Hi!
>>>>>
>>>>> That is exactly the right way to do it. Logging has to be at least
>>>>> INFO and the parameter "taskmanager.debug.memory.startLogThread" set to
>>>>> true.
>>>>> The log output should be under
>>>>> "org.apache.flink.runtime.taskmanager.TaskManager".
>>>>>
>>>>> Do you see other outputs for that class in the log?
>>>>>
>>>>> Make sure you restarted the TaskManager processes after you changed
>>>>> the config file.
>>>>>
>>>>> Greetings,
>>>>> Stephan
>>>>>
>>>>>
>>>>>
>>>>> On Tue, Dec 8, 2015 at 6:56 PM, Filip Łęczycki <
>>>>> filipleczycki@gmail.com> wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> I am trying to enable logging of memory usage on flink 0.10.0 by
>>>>>> adding:
>>>>>>
>>>>>> taskmanager.debug.memory.startLogThread: true
>>>>>>
>>>>>> to config.yaml and setting log4j level to DEBUG however in the logs after running the job I cannot see any info regarding memory usage.My job lasted 30s so it should catch few intervals. Should I change something else in the configuration?
>>>>>>
>>>>>>
>>>>>> Best regards/Pozdrawiam,
>>>>>>
>>>>>> Filip Łęczycki
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>
Re: Using memory logging in Flink
Posted by Filip Łęczycki <fi...@gmail.com>.
Hi Stephan, Till
I am looking for the memory usage info in flink_<user>_jobmanager... log.
As there are no other log files in /log dir I assumed this is the right
place to look. I will try to setup a new clean instance and check whether
it will work.
Thank you very much for your assistance!
Best regards,
Filip Łęczycki
Pozdrawiam,
Filip Łęczycki
2015-12-09 11:13 GMT+01:00 Stephan Ewen <se...@apache.org>:
> Hi Filip!
>
> Someone else just used the memory logging with the exact described
> settings - it worked.
>
> There is probably some mixup, you may be looking into the wrong log file,
> or may setting the a value in a different config...
>
> Stephan
>
>
> On Wed, Dec 9, 2015 at 10:58 AM, Till Rohrmann <tr...@apache.org>
> wrote:
>
>> I assume you're looking in the taskmanager log file for the memory usage
>> logging statements, right?
>>
>> Cheers,
>> Till
>>
>> On Wed, Dec 9, 2015 at 12:15 AM, Filip Łęczycki <fi...@gmail.com>
>> wrote:
>>
>>> Hi,
>>>
>>> Thank you for your reply!
>>>
>>> I have made sure I restarted the TaskManager after changing config, but
>>> it didn't resolve the issue.The config is loaded as I can see the following
>>> line in the log:
>>> 09:12:2015 00:00:21,894 DEBUG
>>> org.apache.flink.configuration.GlobalConfiguration - Loading
>>> configuration property: taskmanager.debug.memory.startLogThread, true
>>>
>>> I am running a job on local standalone flink instance and my
>>> log4j.properties look like this:
>>> log4j.rootLogger=DEBUG, file
>>>
>>> # Log all infos in the given file
>>> log4j.appender.file=org.apache.log4j.FileAppender
>>> log4j.appender.file.file=${log.file}
>>> log4j.appender.file.append=false
>>> log4j.appender.file.layout=org.apache.log4j.PatternLayout
>>> log4j.appender.file.layout.ConversionPattern=%d{dd:MM:YYY HH:mm:ss,SSS}
>>> %-5p %-60c %x - %m%n
>>>
>>> # suppress the irrelevant (wrong) warnings from the netty channel handler
>>> log4j.logger.org.jboss.netty.channel.DefaultChannelPipeline=ERROR, file
>>>
>>> Maybe I can set some property in the job's code so that it would force
>>> such verbose logging? I need those logs to run some flink performance
>>> tests but maybe I can somehow extract the benchmark results without them, do
>>> you know any other way to monitor Flink Job's memory usage and GC time,
>>> other than looking at web interface?
>>>
>>> Best regards,
>>> Filip Łęczycki
>>>
>>> Pozdrawiam,
>>> Filip Łęczycki
>>>
>>> 2015-12-08 20:48 GMT+01:00 Stephan Ewen <se...@apache.org>:
>>>
>>>> Hi!
>>>>
>>>> That is exactly the right way to do it. Logging has to be at least INFO
>>>> and the parameter "taskmanager.debug.memory.startLogThread" set to true.
>>>> The log output should be under
>>>> "org.apache.flink.runtime.taskmanager.TaskManager".
>>>>
>>>> Do you see other outputs for that class in the log?
>>>>
>>>> Make sure you restarted the TaskManager processes after you changed the
>>>> config file.
>>>>
>>>> Greetings,
>>>> Stephan
>>>>
>>>>
>>>>
>>>> On Tue, Dec 8, 2015 at 6:56 PM, Filip Łęczycki <filipleczycki@gmail.com
>>>> > wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> I am trying to enable logging of memory usage on flink 0.10.0 by
>>>>> adding:
>>>>>
>>>>> taskmanager.debug.memory.startLogThread: true
>>>>>
>>>>> to config.yaml and setting log4j level to DEBUG however in the logs after running the job I cannot see any info regarding memory usage.My job lasted 30s so it should catch few intervals. Should I change something else in the configuration?
>>>>>
>>>>>
>>>>> Best regards/Pozdrawiam,
>>>>>
>>>>> Filip Łęczycki
>>>>>
>>>>
>>>>
>>>
>>
>
Re: Using memory logging in Flink
Posted by Stephan Ewen <se...@apache.org>.
Hi Filip!
Someone else just used the memory logging with the exact described settings
- it worked.
There is probably some mixup, you may be looking into the wrong log file,
or may setting the a value in a different config...
Stephan
On Wed, Dec 9, 2015 at 10:58 AM, Till Rohrmann <tr...@apache.org> wrote:
> I assume you're looking in the taskmanager log file for the memory usage
> logging statements, right?
>
> Cheers,
> Till
>
> On Wed, Dec 9, 2015 at 12:15 AM, Filip Łęczycki <fi...@gmail.com>
> wrote:
>
>> Hi,
>>
>> Thank you for your reply!
>>
>> I have made sure I restarted the TaskManager after changing config, but
>> it didn't resolve the issue.The config is loaded as I can see the following
>> line in the log:
>> 09:12:2015 00:00:21,894 DEBUG
>> org.apache.flink.configuration.GlobalConfiguration - Loading
>> configuration property: taskmanager.debug.memory.startLogThread, true
>>
>> I am running a job on local standalone flink instance and my
>> log4j.properties look like this:
>> log4j.rootLogger=DEBUG, file
>>
>> # Log all infos in the given file
>> log4j.appender.file=org.apache.log4j.FileAppender
>> log4j.appender.file.file=${log.file}
>> log4j.appender.file.append=false
>> log4j.appender.file.layout=org.apache.log4j.PatternLayout
>> log4j.appender.file.layout.ConversionPattern=%d{dd:MM:YYY HH:mm:ss,SSS}
>> %-5p %-60c %x - %m%n
>>
>> # suppress the irrelevant (wrong) warnings from the netty channel handler
>> log4j.logger.org.jboss.netty.channel.DefaultChannelPipeline=ERROR, file
>>
>> Maybe I can set some property in the job's code so that it would force
>> such verbose logging? I need those logs to run some flink performance
>> tests but maybe I can somehow extract the benchmark results without them, do
>> you know any other way to monitor Flink Job's memory usage and GC time,
>> other than looking at web interface?
>>
>> Best regards,
>> Filip Łęczycki
>>
>> Pozdrawiam,
>> Filip Łęczycki
>>
>> 2015-12-08 20:48 GMT+01:00 Stephan Ewen <se...@apache.org>:
>>
>>> Hi!
>>>
>>> That is exactly the right way to do it. Logging has to be at least INFO
>>> and the parameter "taskmanager.debug.memory.startLogThread" set to true.
>>> The log output should be under
>>> "org.apache.flink.runtime.taskmanager.TaskManager".
>>>
>>> Do you see other outputs for that class in the log?
>>>
>>> Make sure you restarted the TaskManager processes after you changed the
>>> config file.
>>>
>>> Greetings,
>>> Stephan
>>>
>>>
>>>
>>> On Tue, Dec 8, 2015 at 6:56 PM, Filip Łęczycki <fi...@gmail.com>
>>> wrote:
>>>
>>>> Hi,
>>>>
>>>> I am trying to enable logging of memory usage on flink 0.10.0 by adding:
>>>>
>>>> taskmanager.debug.memory.startLogThread: true
>>>>
>>>> to config.yaml and setting log4j level to DEBUG however in the logs after running the job I cannot see any info regarding memory usage.My job lasted 30s so it should catch few intervals. Should I change something else in the configuration?
>>>>
>>>>
>>>> Best regards/Pozdrawiam,
>>>>
>>>> Filip Łęczycki
>>>>
>>>
>>>
>>
>
Re: Using memory logging in Flink
Posted by Till Rohrmann <tr...@apache.org>.
I assume you're looking in the taskmanager log file for the memory usage
logging statements, right?
Cheers,
Till
On Wed, Dec 9, 2015 at 12:15 AM, Filip Łęczycki <fi...@gmail.com>
wrote:
> Hi,
>
> Thank you for your reply!
>
> I have made sure I restarted the TaskManager after changing config, but it
> didn't resolve the issue.The config is loaded as I can see the following
> line in the log:
> 09:12:2015 00:00:21,894 DEBUG
> org.apache.flink.configuration.GlobalConfiguration - Loading
> configuration property: taskmanager.debug.memory.startLogThread, true
>
> I am running a job on local standalone flink instance and my
> log4j.properties look like this:
> log4j.rootLogger=DEBUG, file
>
> # Log all infos in the given file
> log4j.appender.file=org.apache.log4j.FileAppender
> log4j.appender.file.file=${log.file}
> log4j.appender.file.append=false
> log4j.appender.file.layout=org.apache.log4j.PatternLayout
> log4j.appender.file.layout.ConversionPattern=%d{dd:MM:YYY HH:mm:ss,SSS}
> %-5p %-60c %x - %m%n
>
> # suppress the irrelevant (wrong) warnings from the netty channel handler
> log4j.logger.org.jboss.netty.channel.DefaultChannelPipeline=ERROR, file
>
> Maybe I can set some property in the job's code so that it would force
> such verbose logging? I need those logs to run some flink performance
> tests but maybe I can somehow extract the benchmark results without them, do
> you know any other way to monitor Flink Job's memory usage and GC time,
> other than looking at web interface?
>
> Best regards,
> Filip Łęczycki
>
> Pozdrawiam,
> Filip Łęczycki
>
> 2015-12-08 20:48 GMT+01:00 Stephan Ewen <se...@apache.org>:
>
>> Hi!
>>
>> That is exactly the right way to do it. Logging has to be at least INFO
>> and the parameter "taskmanager.debug.memory.startLogThread" set to true.
>> The log output should be under
>> "org.apache.flink.runtime.taskmanager.TaskManager".
>>
>> Do you see other outputs for that class in the log?
>>
>> Make sure you restarted the TaskManager processes after you changed the
>> config file.
>>
>> Greetings,
>> Stephan
>>
>>
>>
>> On Tue, Dec 8, 2015 at 6:56 PM, Filip Łęczycki <fi...@gmail.com>
>> wrote:
>>
>>> Hi,
>>>
>>> I am trying to enable logging of memory usage on flink 0.10.0 by adding:
>>>
>>> taskmanager.debug.memory.startLogThread: true
>>>
>>> to config.yaml and setting log4j level to DEBUG however in the logs after running the job I cannot see any info regarding memory usage.My job lasted 30s so it should catch few intervals. Should I change something else in the configuration?
>>>
>>>
>>> Best regards/Pozdrawiam,
>>>
>>> Filip Łęczycki
>>>
>>
>>
>
Re: Using memory logging in Flink
Posted by Filip Łęczycki <fi...@gmail.com>.
Hi,
Thank you for your reply!
I have made sure I restarted the TaskManager after changing config, but it
didn't resolve the issue.The config is loaded as I can see the following
line in the log:
09:12:2015 00:00:21,894 DEBUG
org.apache.flink.configuration.GlobalConfiguration - Loading
configuration property: taskmanager.debug.memory.startLogThread, true
I am running a job on local standalone flink instance and my
log4j.properties look like this:
log4j.rootLogger=DEBUG, file
# Log all infos in the given file
log4j.appender.file=org.apache.log4j.FileAppender
log4j.appender.file.file=${log.file}
log4j.appender.file.append=false
log4j.appender.file.layout=org.apache.log4j.PatternLayout
log4j.appender.file.layout.ConversionPattern=%d{dd:MM:YYY HH:mm:ss,SSS}
%-5p %-60c %x - %m%n
# suppress the irrelevant (wrong) warnings from the netty channel handler
log4j.logger.org.jboss.netty.channel.DefaultChannelPipeline=ERROR, file
Maybe I can set some property in the job's code so that it would force such
verbose logging? I need those logs to run some flink performance tests but
maybe I can somehow extract the benchmark results without them, do you
know any other way to monitor Flink Job's memory usage and GC time, other
than looking at web interface?
Best regards,
Filip Łęczycki
Pozdrawiam,
Filip Łęczycki
2015-12-08 20:48 GMT+01:00 Stephan Ewen <se...@apache.org>:
> Hi!
>
> That is exactly the right way to do it. Logging has to be at least INFO
> and the parameter "taskmanager.debug.memory.startLogThread" set to true.
> The log output should be under
> "org.apache.flink.runtime.taskmanager.TaskManager".
>
> Do you see other outputs for that class in the log?
>
> Make sure you restarted the TaskManager processes after you changed the
> config file.
>
> Greetings,
> Stephan
>
>
>
> On Tue, Dec 8, 2015 at 6:56 PM, Filip Łęczycki <fi...@gmail.com>
> wrote:
>
>> Hi,
>>
>> I am trying to enable logging of memory usage on flink 0.10.0 by adding:
>>
>> taskmanager.debug.memory.startLogThread: true
>>
>> to config.yaml and setting log4j level to DEBUG however in the logs after running the job I cannot see any info regarding memory usage.My job lasted 30s so it should catch few intervals. Should I change something else in the configuration?
>>
>>
>> Best regards/Pozdrawiam,
>>
>> Filip Łęczycki
>>
>
>
Re: Using memory logging in Flink
Posted by Stephan Ewen <se...@apache.org>.
Hi!
That is exactly the right way to do it. Logging has to be at least INFO and
the parameter "taskmanager.debug.memory.startLogThread" set to true.
The log output should be under
"org.apache.flink.runtime.taskmanager.TaskManager".
Do you see other outputs for that class in the log?
Make sure you restarted the TaskManager processes after you changed the
config file.
Greetings,
Stephan
On Tue, Dec 8, 2015 at 6:56 PM, Filip Łęczycki <fi...@gmail.com>
wrote:
> Hi,
>
> I am trying to enable logging of memory usage on flink 0.10.0 by adding:
>
> taskmanager.debug.memory.startLogThread: true
>
> to config.yaml and setting log4j level to DEBUG however in the logs after running the job I cannot see any info regarding memory usage.My job lasted 30s so it should catch few intervals. Should I change something else in the configuration?
>
>
> Best regards/Pozdrawiam,
>
> Filip Łęczycki
>