You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@mesos.apache.org by Frank Scholten <fr...@frankscholten.nl> on 2016/10/04 13:12:56 UTC
Mapped diagnostics context - Adding internal Mesos IDs as context to
the logs
Hi,
On JIRA I found several issues about making logs less spammy as well
as making them easier to understand for day to day operations.
https://issues.apache.org/jira/browse/MESOS-4432 Condense (redundant)
log messages related to task launch/status/finish
https://issues.apache.org/jira/browse/MESOS-5467 offer DECLINE /
ACCEPT + Recovered resource messages are spammy
https://issues.apache.org/jira/browse/MESOS-4430 Identify and change
logging level for message that don't contain specific
task/framework/slave info
Besides reducing the logs would it be possible to add more context? In
the Java world the technique of 'Mapped diagnostic context' is used
with Logback where each log line contains a few fields with context.
See http://logback.qos.ch/manual/mdc.html
To translate this to Mesos how about adding the internal IDs such as
agent, framework, and task IDs at the beginning of the log, so this
information is separated from the textual, human readble log message.
At the moment this information is tangled which makes it hard to
interpret, especially when there are so many logs for each task.
For example, change this
I1004 12:24:12.118803 3780 status_update_manager.cpp:320] Received
status update TASK_FAILED (UUID: a1d03948-30bf-46b3-9599-cfcfc7cbc27b)
for task weave-demo_database_catalogue-db.6d415c17-8a2d-11e6-90a7-0242458f9469
of framework f1546295-ab46-496a-8cf9-91756fece4ed-0000
to
I1004 12:24:12.118803 3780 status_update_manager.cpp:320
A:agent12318032910 F:f1546295-ab46-496a-8cf9-91756fece4ed-0000
T:xyz-a1d03948-30bf-46b3-9599-cfcfc7cbc27b] Received status update
TASK_FAILED for task 'xyz'
In this case the header contains A:$AGENT_ID, F:$FRAMEWORK_ID,
T:$TASK_ID and as the context and lines with similar context can be
more easily correlated visually.
Is this feasible?
Cheers,
Frank
Re: Mapped diagnostics context - Adding internal Mesos IDs as context
to the logs
Posted by tommy xiao <xi...@gmail.com>.
vote +1, how about add a epic on this concern topic?
2016-10-06 21:33 GMT+08:00 Joris Van Remoortere <jo...@mesosphere.io>:
> It's adding *an* extension to the header of the log. In this case after the
> file name.
> In the comment above the code you can see that it's:
>
> > often used to append the port we're listening on to the logfile
>
> That means we can tag our own IDs in there such as Agent ID, Executor ID,
> Master ID, etc.
>
> —
> *Joris Van Remoortere*
> Mesosphere
>
> On Wed, Oct 5, 2016 at 11:34 AM, Benjamin Hindman <be...@mesosphere.io>
> wrote:
>
> > How does adding the filename extension help with the context problem?
> >
> > On Wed, Oct 5, 2016 at 6:14 AM, Joris Van Remoortere <
> joris@mesosphere.io>
> > wrote:
> >
> >> This is technically feasible:
> >>> //
> >>> // Specify an "extension" added to the filename specified via
> >>> // SetLogDestination. This applies to all severity levels. It's
> >>> // often used to append the port we're listening on to the logfile
> >>> // name. Thread-safe.
> >>> //
> >>> GOOGLE_GLOG_DLL_DECL void SetLogFilenameExtension(
> >>> const char* filename_extension);
> >>
> >>
> >> @benm @vinod @benh what are your thoughts on actually doing it?
> >>
> >> —
> >> *Joris Van Remoortere*
> >> Mesosphere
> >>
> >> On Tue, Oct 4, 2016 at 2:39 PM, Zameer Manji <zm...@apache.org> wrote:
> >>
> >>> I don't know if this is feasible or not, but I would be a strong +1 to
> >>> this.
> >>>
> >>> This would make tracing failures much easier.
> >>>
> >>> On Tue, Oct 4, 2016 at 6:12 AM, Frank Scholten <frank@frankscholten.nl
> >
> >>> wrote:
> >>>
> >>> > Hi,
> >>> >
> >>> > On JIRA I found several issues about making logs less spammy as well
> >>> > as making them easier to understand for day to day operations.
> >>> >
> >>> > https://issues.apache.org/jira/browse/MESOS-4432 Condense
> (redundant)
> >>> > log messages related to task launch/status/finish
> >>> > https://issues.apache.org/jira/browse/MESOS-5467 offer DECLINE /
> >>> > ACCEPT + Recovered resource messages are spammy
> >>> > https://issues.apache.org/jira/browse/MESOS-4430 Identify and change
> >>> > logging level for message that don't contain specific
> >>> > task/framework/slave info
> >>> >
> >>> > Besides reducing the logs would it be possible to add more context?
> In
> >>> > the Java world the technique of 'Mapped diagnostic context' is used
> >>> > with Logback where each log line contains a few fields with context.
> >>> > See http://logback.qos.ch/manual/mdc.html
> >>> >
> >>> > To translate this to Mesos how about adding the internal IDs such as
> >>> > agent, framework, and task IDs at the beginning of the log, so this
> >>> > information is separated from the textual, human readble log message.
> >>> > At the moment this information is tangled which makes it hard to
> >>> > interpret, especially when there are so many logs for each task.
> >>> >
> >>> > For example, change this
> >>> >
> >>> > I1004 12:24:12.118803 3780 status_update_manager.cpp:320] Received
> >>> > status update TASK_FAILED (UUID: a1d03948-30bf-46b3-9599-
> cfcfc7cbc27b)
> >>> > for task weave-demo_database_catalogue-db.6d415c17-8a2d-11e6-90a7-
> >>> > 0242458f9469
> >>> > of framework f1546295-ab46-496a-8cf9-91756fece4ed-0000
> >>> >
> >>> > to
> >>> >
> >>> > I1004 12:24:12.118803 3780 status_update_manager.cpp:320
> >>> > A:agent12318032910 F:f1546295-ab46-496a-8cf9-91756fece4ed-0000
> >>> > T:xyz-a1d03948-30bf-46b3-9599-cfcfc7cbc27b] Received status update
> >>> > TASK_FAILED for task 'xyz'
> >>> >
> >>> > In this case the header contains A:$AGENT_ID, F:$FRAMEWORK_ID,
> >>> > T:$TASK_ID and as the context and lines with similar context can be
> >>> > more easily correlated visually.
> >>> >
> >>> > Is this feasible?
> >>> >
> >>> > Cheers,
> >>> >
> >>> > Frank
> >>> >
> >>> > --
> >>> > Zameer Manji
> >>> >
> >>>
> >>
> >>
> >
> >
> > --
> > Benjamin Hindman
> > Founder of Mesosphere and Co-Creator of Apache Mesos
> > Mesosphere Inc. <http://www.mesosphere.io/>
> >
> > Follow us on Twitter: @mesosphere <http://twitter.com/mesosphere>
> >
> > [image: Watch the Video]
> > <http://smart.mesosphere.io/v2/a/watchvideoimg/57f51dac29a64565e77ecedc-
> zrRrz/> [image:
> > .]See how DC/OS elastically
> > powers the modern application
> > <http://smart.mesosphere.io/v2/a/watchvideotext2/
> 57f51dac29a64565e77ecedc-zrRrz/>
> >
> >
>
--
Deshi Xiao
Twitter: xds2000
E-mail: xiaods(AT)gmail.com
Re: Mapped diagnostics context - Adding internal Mesos IDs as context
to the logs
Posted by Joris Van Remoortere <jo...@mesosphere.io>.
It's adding *an* extension to the header of the log. In this case after the
file name.
In the comment above the code you can see that it's:
> often used to append the port we're listening on to the logfile
That means we can tag our own IDs in there such as Agent ID, Executor ID,
Master ID, etc.
—
*Joris Van Remoortere*
Mesosphere
On Wed, Oct 5, 2016 at 11:34 AM, Benjamin Hindman <be...@mesosphere.io>
wrote:
> How does adding the filename extension help with the context problem?
>
> On Wed, Oct 5, 2016 at 6:14 AM, Joris Van Remoortere <jo...@mesosphere.io>
> wrote:
>
>> This is technically feasible:
>>> //
>>> // Specify an "extension" added to the filename specified via
>>> // SetLogDestination. This applies to all severity levels. It's
>>> // often used to append the port we're listening on to the logfile
>>> // name. Thread-safe.
>>> //
>>> GOOGLE_GLOG_DLL_DECL void SetLogFilenameExtension(
>>> const char* filename_extension);
>>
>>
>> @benm @vinod @benh what are your thoughts on actually doing it?
>>
>> —
>> *Joris Van Remoortere*
>> Mesosphere
>>
>> On Tue, Oct 4, 2016 at 2:39 PM, Zameer Manji <zm...@apache.org> wrote:
>>
>>> I don't know if this is feasible or not, but I would be a strong +1 to
>>> this.
>>>
>>> This would make tracing failures much easier.
>>>
>>> On Tue, Oct 4, 2016 at 6:12 AM, Frank Scholten <fr...@frankscholten.nl>
>>> wrote:
>>>
>>> > Hi,
>>> >
>>> > On JIRA I found several issues about making logs less spammy as well
>>> > as making them easier to understand for day to day operations.
>>> >
>>> > https://issues.apache.org/jira/browse/MESOS-4432 Condense (redundant)
>>> > log messages related to task launch/status/finish
>>> > https://issues.apache.org/jira/browse/MESOS-5467 offer DECLINE /
>>> > ACCEPT + Recovered resource messages are spammy
>>> > https://issues.apache.org/jira/browse/MESOS-4430 Identify and change
>>> > logging level for message that don't contain specific
>>> > task/framework/slave info
>>> >
>>> > Besides reducing the logs would it be possible to add more context? In
>>> > the Java world the technique of 'Mapped diagnostic context' is used
>>> > with Logback where each log line contains a few fields with context.
>>> > See http://logback.qos.ch/manual/mdc.html
>>> >
>>> > To translate this to Mesos how about adding the internal IDs such as
>>> > agent, framework, and task IDs at the beginning of the log, so this
>>> > information is separated from the textual, human readble log message.
>>> > At the moment this information is tangled which makes it hard to
>>> > interpret, especially when there are so many logs for each task.
>>> >
>>> > For example, change this
>>> >
>>> > I1004 12:24:12.118803 3780 status_update_manager.cpp:320] Received
>>> > status update TASK_FAILED (UUID: a1d03948-30bf-46b3-9599-cfcfc7cbc27b)
>>> > for task weave-demo_database_catalogue-db.6d415c17-8a2d-11e6-90a7-
>>> > 0242458f9469
>>> > of framework f1546295-ab46-496a-8cf9-91756fece4ed-0000
>>> >
>>> > to
>>> >
>>> > I1004 12:24:12.118803 3780 status_update_manager.cpp:320
>>> > A:agent12318032910 F:f1546295-ab46-496a-8cf9-91756fece4ed-0000
>>> > T:xyz-a1d03948-30bf-46b3-9599-cfcfc7cbc27b] Received status update
>>> > TASK_FAILED for task 'xyz'
>>> >
>>> > In this case the header contains A:$AGENT_ID, F:$FRAMEWORK_ID,
>>> > T:$TASK_ID and as the context and lines with similar context can be
>>> > more easily correlated visually.
>>> >
>>> > Is this feasible?
>>> >
>>> > Cheers,
>>> >
>>> > Frank
>>> >
>>> > --
>>> > Zameer Manji
>>> >
>>>
>>
>>
>
>
> --
> Benjamin Hindman
> Founder of Mesosphere and Co-Creator of Apache Mesos
> Mesosphere Inc. <http://www.mesosphere.io/>
>
> Follow us on Twitter: @mesosphere <http://twitter.com/mesosphere>
>
> [image: Watch the Video]
> <http://smart.mesosphere.io/v2/a/watchvideoimg/57f51dac29a64565e77ecedc-zrRrz/> [image:
> .]See how DC/OS elastically
> powers the modern application
> <http://smart.mesosphere.io/v2/a/watchvideotext2/57f51dac29a64565e77ecedc-zrRrz/>
>
>
Re: Mapped diagnostics context - Adding internal Mesos IDs as context
to the logs
Posted by Joris Van Remoortere <jo...@mesosphere.io>.
>
> This is technically feasible:
> //
> // Specify an "extension" added to the filename specified via
> // SetLogDestination. This applies to all severity levels. It's
> // often used to append the port we're listening on to the logfile
> // name. Thread-safe.
> //
> GOOGLE_GLOG_DLL_DECL void SetLogFilenameExtension(
> const char* filename_extension);
@benm @vinod @benh what are your thoughts on actually doing it?
—
*Joris Van Remoortere*
Mesosphere
On Tue, Oct 4, 2016 at 2:39 PM, Zameer Manji <zm...@apache.org> wrote:
> I don't know if this is feasible or not, but I would be a strong +1 to
> this.
>
> This would make tracing failures much easier.
>
> On Tue, Oct 4, 2016 at 6:12 AM, Frank Scholten <fr...@frankscholten.nl>
> wrote:
>
> > Hi,
> >
> > On JIRA I found several issues about making logs less spammy as well
> > as making them easier to understand for day to day operations.
> >
> > https://issues.apache.org/jira/browse/MESOS-4432 Condense (redundant)
> > log messages related to task launch/status/finish
> > https://issues.apache.org/jira/browse/MESOS-5467 offer DECLINE /
> > ACCEPT + Recovered resource messages are spammy
> > https://issues.apache.org/jira/browse/MESOS-4430 Identify and change
> > logging level for message that don't contain specific
> > task/framework/slave info
> >
> > Besides reducing the logs would it be possible to add more context? In
> > the Java world the technique of 'Mapped diagnostic context' is used
> > with Logback where each log line contains a few fields with context.
> > See http://logback.qos.ch/manual/mdc.html
> >
> > To translate this to Mesos how about adding the internal IDs such as
> > agent, framework, and task IDs at the beginning of the log, so this
> > information is separated from the textual, human readble log message.
> > At the moment this information is tangled which makes it hard to
> > interpret, especially when there are so many logs for each task.
> >
> > For example, change this
> >
> > I1004 12:24:12.118803 3780 status_update_manager.cpp:320] Received
> > status update TASK_FAILED (UUID: a1d03948-30bf-46b3-9599-cfcfc7cbc27b)
> > for task weave-demo_database_catalogue-db.6d415c17-8a2d-11e6-90a7-
> > 0242458f9469
> > of framework f1546295-ab46-496a-8cf9-91756fece4ed-0000
> >
> > to
> >
> > I1004 12:24:12.118803 3780 status_update_manager.cpp:320
> > A:agent12318032910 F:f1546295-ab46-496a-8cf9-91756fece4ed-0000
> > T:xyz-a1d03948-30bf-46b3-9599-cfcfc7cbc27b] Received status update
> > TASK_FAILED for task 'xyz'
> >
> > In this case the header contains A:$AGENT_ID, F:$FRAMEWORK_ID,
> > T:$TASK_ID and as the context and lines with similar context can be
> > more easily correlated visually.
> >
> > Is this feasible?
> >
> > Cheers,
> >
> > Frank
> >
> > --
> > Zameer Manji
> >
>
Re: Mapped diagnostics context - Adding internal Mesos IDs as context
to the logs
Posted by Zameer Manji <zm...@apache.org>.
I don't know if this is feasible or not, but I would be a strong +1 to this.
This would make tracing failures much easier.
On Tue, Oct 4, 2016 at 6:12 AM, Frank Scholten <fr...@frankscholten.nl>
wrote:
> Hi,
>
> On JIRA I found several issues about making logs less spammy as well
> as making them easier to understand for day to day operations.
>
> https://issues.apache.org/jira/browse/MESOS-4432 Condense (redundant)
> log messages related to task launch/status/finish
> https://issues.apache.org/jira/browse/MESOS-5467 offer DECLINE /
> ACCEPT + Recovered resource messages are spammy
> https://issues.apache.org/jira/browse/MESOS-4430 Identify and change
> logging level for message that don't contain specific
> task/framework/slave info
>
> Besides reducing the logs would it be possible to add more context? In
> the Java world the technique of 'Mapped diagnostic context' is used
> with Logback where each log line contains a few fields with context.
> See http://logback.qos.ch/manual/mdc.html
>
> To translate this to Mesos how about adding the internal IDs such as
> agent, framework, and task IDs at the beginning of the log, so this
> information is separated from the textual, human readble log message.
> At the moment this information is tangled which makes it hard to
> interpret, especially when there are so many logs for each task.
>
> For example, change this
>
> I1004 12:24:12.118803 3780 status_update_manager.cpp:320] Received
> status update TASK_FAILED (UUID: a1d03948-30bf-46b3-9599-cfcfc7cbc27b)
> for task weave-demo_database_catalogue-db.6d415c17-8a2d-11e6-90a7-
> 0242458f9469
> of framework f1546295-ab46-496a-8cf9-91756fece4ed-0000
>
> to
>
> I1004 12:24:12.118803 3780 status_update_manager.cpp:320
> A:agent12318032910 F:f1546295-ab46-496a-8cf9-91756fece4ed-0000
> T:xyz-a1d03948-30bf-46b3-9599-cfcfc7cbc27b] Received status update
> TASK_FAILED for task 'xyz'
>
> In this case the header contains A:$AGENT_ID, F:$FRAMEWORK_ID,
> T:$TASK_ID and as the context and lines with similar context can be
> more easily correlated visually.
>
> Is this feasible?
>
> Cheers,
>
> Frank
>
> --
> Zameer Manji
>