You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@beam.apache.org by Aviem Zur <av...@gmail.com> on 2017/03/22 03:58:08 UTC

[DISCUSSION] Consistent use of loggers

Hi all,

There have been a few reports lately (On JIRA [1] and on Slack) from users
regarding inconsistent loggers used across Beam's modules.

While we use SLF4J, different modules use a different logger behind it
(JUL, log4j, etc)
So when people add a log4j.properties file to their classpath for instance,
they expect this to affect all of their dependencies on Beam modules, but
it doesn’t and they miss out on some logs they thought they would see.

I think we should strive for consistency in which logger is used behind
SLF4J, and try to enforce this in our modules.
I for one think it should be slf4j-log4j. However, if performance of
logging is critical we might want to consider logback.

Note: SLF4J will still be the facade for logging across the project. The
only change would be the logger SLF4J delegates to.

Once we have something like this it would also be useful to add
documentation on logging in Beam to the website.

[1] https://issues.apache.org/jira/browse/BEAM-1757

Re: [DISCUSSION] Consistent use of loggers

Posted by Ted Yu <yu...@gmail.com>.
+1 to what JB said. 

> On Mar 21, 2017, at 9:46 PM, Aviem Zur <av...@gmail.com> wrote:
> 
> +1 to what JB said.
> 
> Will just have to be documented well as if we provide no binding there will
> be no logging out of the box unless the user adds a binding.
> 
> On Wed, Mar 22, 2017 at 6:24 AM Jean-Baptiste Onofré <jb...@nanthrax.net>
> wrote:
> 
>> Hi Aviem,
>> 
>> Good point.
>> 
>> I think, in our dependencies set, we should just depend to slf4j-api and
>> let the
>> user provides the binding he wants (slf4j-log4j12, slf4j-simple, whatever).
>> 
>> We define a binding only with test scope in our modules.
>> 
>> Regards
>> JB
>> 
>>> On 03/22/2017 04:58 AM, Aviem Zur wrote:
>>> Hi all,
>>> 
>>> There have been a few reports lately (On JIRA [1] and on Slack) from
>> users
>>> regarding inconsistent loggers used across Beam's modules.
>>> 
>>> While we use SLF4J, different modules use a different logger behind it
>>> (JUL, log4j, etc)
>>> So when people add a log4j.properties file to their classpath for
>> instance,
>>> they expect this to affect all of their dependencies on Beam modules, but
>>> it doesn’t and they miss out on some logs they thought they would see.
>>> 
>>> I think we should strive for consistency in which logger is used behind
>>> SLF4J, and try to enforce this in our modules.
>>> I for one think it should be slf4j-log4j. However, if performance of
>>> logging is critical we might want to consider logback.
>>> 
>>> Note: SLF4J will still be the facade for logging across the project. The
>>> only change would be the logger SLF4J delegates to.
>>> 
>>> Once we have something like this it would also be useful to add
>>> documentation on logging in Beam to the website.
>>> 
>>> [1] https://issues.apache.org/jira/browse/BEAM-1757
>> 
>> --
>> Jean-Baptiste Onofré
>> jbonofre@apache.org
>> http://blog.nanthrax.net
>> Talend - http://www.talend.com
>> 

Re: [DISCUSSION] Consistent use of loggers

Posted by Dan Halperin <dh...@google.com.INVALID>.
For examples (which I think is auto-propagated to examples archetype), and
I think also manually done for starter archetype:

* Every runner, including DirectRunner, is in a profile: -Pdirect-runner:
https://github.com/apache/beam/blob/master/examples/java/pom.xml#L43
* The slf4j-jdk14 is already already used, but not in a profile:
https://github.com/apache/beam/blob/master/examples/java/pom.xml#L517

I would be supportive of moving the slf4j-jdk14 to only Direct and Dataflow
profiles if that is what you think is best.

And yes, the intent is that only 1 runner profile in these cases is active
at a time.

On Thu, Apr 6, 2017 at 9:21 PM, Aviem Zur <av...@gmail.com> wrote:

> >IMO I don't think the DirectRunner should depend directly on any specific
> logging
> backend (at least, not in the compile or runtime scopes). I think it should
> depend on JUL in the test scope, so that there are logs when executing
> DirectRunner tests.
> >My reasoning: I can see in any binary version of Beam that the SDK, the
> DirectRunner,
> and 1 or more other runners will all be on the classpath.
> >Ideally this should work regardless of whatever other runner is used;
> presumably
> the DirectRunner would "automagically" pick up the logging config of the
> other runner.
> That sounds like a very plausible scenario and this would "protect" the
> runner's binding from an intruding binding from direct runner, since it
> would have no binding.
> However, there is also the scenario that a user runs the examples using
> direct runner,
> this is their first interaction with Beam, and they see no logs whatsoever,
> they would have to add a binding.
> We could solve this by adding a binding in the 'direct-runner' profile in
> examples module and the maven archetypes (And allow only one runner profile
> to be specified at a time, in case their logger binding clashes).
>
> >I like the use of slf4j as it enables lots of publishers of logs, but I
> don't
> want to supply a default/required consumer of logs because that will
> restrict
> use cases in the future...
> I agree, forcing log4j binding might give the user a false sense of: "all
> runners use log4j" while this might not be true for future (and isn't true
> today, for Dataflow runner), but we can't assure that future runners could
> support this.
>
> So it seems we're left with:
> 1) Add documentation around logging in each runner.
> 2) Consider enabling a binding (JUL) for direct runner profile in examples
> module and maven archetypes.
> 3) Allow only one runner profile to be active at a time in examples and
> maven archetypes as their logger binding might clash.
>
> Thoughts?
>
> On Tue, Apr 4, 2017 at 8:51 AM Dan Halperin <dh...@google.com.invalid>
> wrote:
>
> > At this point, I'm a little unclear on what is the proposal. Can you
> > refresh a simplified/aggregated view after this conversation?
> >
> > IMO I don't think the DirectRunner should depend directly on any specific
> > logging backend (at least, not in the compile or runtime scopes). I think
> > it should depend on JUL in the test scope, so that there are logs when
> > executing DirectRunner tests.
> >
> > My reasoning: I can see in any binary version of Beam that the SDK, the
> > DirectRunner, and 1 or more other runners will all be on the classpath.
> > Ideally this should work regardless of whatever other runner is used;
> > presumably the DirectRunner would "automagically" pick up the logging
> > config of the other runner.
> >
> > I like the use of slf4j as it enables lots of publishers of logs, but I
> > don't want to supply a default/required consumer of logs because that
> will
> > restrict use cases in the future...
> >
> > On Mon, Apr 3, 2017 at 8:14 PM, Jean-Baptiste Onofré <jb...@nanthrax.net>
> > wrote:
> >
> > > Fair enough. +1 especially for the documentation.
> > >
> > > Regards
> > > JB
> > >
> > >
> > > On 04/03/2017 08:48 PM, Aviem Zur wrote:
> > >
> > >> Upon further inspection there seems to be an issue we may have
> > overlooked:
> > >> In cluster mode, some of the runners will have dependencies added
> > directly
> > >> to the classpath by the cluster, and since SLF4J can only work with
> one
> > >> binding, the first one in the classpath will be used.
> > >>
> > >> So while what we suggested would work in local mode, the user's chosen
> > >> binding and configuration might be ignored in cluster mode, which is
> > >> detrimental to what we wanted to accomplish.
> > >>
> > >> So I believe what we should do instead is:
> > >>
> > >>    1. Add better documentation regarding logging in each runner, which
> > >>    binding is used, perhaps examples of how to configure logging for
> > that
> > >>    runner.
> > >>    2. Have direct runner use the most common binding among runners
> (this
> > >>    appears to be log4j which is used by Spark runner, Flink runner and
> > >> Apex
> > >>    runner).
> > >>
> > >>
> > >> On Mon, Apr 3, 2017 at 7:02 PM Aljoscha Krettek <al...@apache.org>
> > >> wrote:
> > >>
> > >> Yes, I think we can exclude log4j from the Flink dependencies. It’s
> > >>> somewhat annoying that they are there in the first place.
> > >>>
> > >>> The Flink doc has this to say about the topic:
> > >>> https://ci.apache.org/projects/flink/flink-docs-release-1.3/
> > >>> monitoring/logging.html
> > >>>
> > >>>> On 3. Apr 2017, at 17:56, Aviem Zur <av...@gmail.com> wrote:
> > >>>>
> > >>>> * java.util.logging could be a good choice for the Direct Runner
> > >>>>>
> > >>>> Yes, this will be great for users (Instead of having no logging when
> > >>>>
> > >>> using
> > >>>
> > >>>> direct runner).
> > >>>>
> > >>>> * Logging backend could be runner-specific, particularly if it needs
> > to
> > >>>>> integrate into some other experience
> > >>>>>
> > >>>> Good point, let's take a look at the current state of runners:
> > >>>> Direct runner - will use JUL as suggested.
> > >>>> Dataflow runner - looks like there is already no binding (There is a
> > >>>> binding in tests only).
> > >>>> Spark runner - currently uses slf4j-log4j12. does not require any
> > >>>>
> > >>> specific
> > >>>
> > >>>> logger, we can change this to no binding.
> > >>>> Flink runner - uses slf4j-log4j12 transitively from Flink
> > dependencies.
> > >>>>
> > >>> I'm
> > >>>
> > >>>> assuming this is not a must and we can default to no binding here.
> > >>>> @aljoscha please confirm.
> > >>>> Apex runner - uses slf4j-log4j12 transitively from Apex
> dependencies.
> > >>>> I'm
> > >>>> assuming this is not a must and we can default to no binding here.
> > @thw
> > >>>> please confirm.
> > >>>>
> > >>>> It might be a good idea to use a consistent binding in tests (Since
> > >>>> we'll
> > >>>> use JUL for direct runner, let this be JUL).
> > >>>>
> > >>>> On Wed, Mar 29, 2017 at 7:23 PM Davor Bonaci <da...@apache.org>
> > wrote:
> > >>>>
> > >>>> +1 on consistency across Beam modules on the logging facade
> > >>>> +1 on enforcing consistency
> > >>>> +1 on clearly documenting how to do logging
> > >>>>
> > >>>> Mixed feelings:
> > >>>> * Logging backend could be runner-specific, particularly if it needs
> > to
> > >>>> integrate into some other experience
> > >>>> * java.util.logging could be a good choice for the Direct Runner
> > >>>>
> > >>>> On Tue, Mar 28, 2017 at 6:50 PM, Ahmet Altay
> <altay@google.com.invalid
> > >
> > >>>> wrote:
> > >>>>
> > >>>> On Wed, Mar 22, 2017 at 10:38 AM, Tibor Kiss <tkiss@hortonworks.com
> >
> > >>>>> wrote:
> > >>>>>
> > >>>>> This is a great idea!
> > >>>>>>
> > >>>>>> I believe Python-SDK's logging could also be enhanced (a bit
> > >>>>>>
> > >>>>> differently):
> > >>>>>
> > >>>>>> Currently we are not instantiating the logger, just using the
> class
> > >>>>>>
> > >>>>> what
> > >>>
> > >>>> logging package provides.
> > >>>>>> Shortcoming of this approach is that the user cannot set the log
> > level
> > >>>>>>
> > >>>>> on
> > >>>>
> > >>>>> a per module basis as all log messages
> > >>>>>> end up in the root level.
> > >>>>>>
> > >>>>>>
> > >>>>> +1 to this. Python SDK needs to expands its logging capabilities.
> > Filed
> > >>>>>
> > >>>> [1]
> > >>>>
> > >>>>> for this.
> > >>>>>
> > >>>>> Ahmet
> > >>>>>
> > >>>>> [1] https://issues.apache.org/jira/browse/BEAM-1825
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>>> On 3/22/17, 5:46 AM, "Aviem Zur" <av...@gmail.com> wrote:
> > >>>>>>
> > >>>>>>    +1 to what JB said.
> > >>>>>>
> > >>>>>>    Will just have to be documented well as if we provide no
> binding
> > >>>>>>
> > >>>>> there
> > >>>>>
> > >>>>>> will
> > >>>>>>    be no logging out of the box unless the user adds a binding.
> > >>>>>>
> > >>>>>>    On Wed, Mar 22, 2017 at 6:24 AM Jean-Baptiste Onofré <
> > >>>>>>
> > >>>>> jb@nanthrax.net>
> > >>>>>
> > >>>>>>    wrote:
> > >>>>>>
> > >>>>>> Hi Aviem,
> > >>>>>>>
> > >>>>>>> Good point.
> > >>>>>>>
> > >>>>>>> I think, in our dependencies set, we should just depend to
> > >>>>>>>
> > >>>>>> slf4j-api
> > >>>>>
> > >>>>>> and
> > >>>>>>
> > >>>>>>> let the
> > >>>>>>> user provides the binding he wants (slf4j-log4j12, slf4j-simple,
> > >>>>>>>
> > >>>>>> whatever).
> > >>>>>>
> > >>>>>>>
> > >>>>>>> We define a binding only with test scope in our modules.
> > >>>>>>>
> > >>>>>>> Regards
> > >>>>>>> JB
> > >>>>>>>
> > >>>>>>> On 03/22/2017 04:58 AM, Aviem Zur wrote:
> > >>>>>>>
> > >>>>>>>> Hi all,
> > >>>>>>>>
> > >>>>>>>> There have been a few reports lately (On JIRA [1] and on Slack)
> > >>>>>>>>
> > >>>>>>> from
> > >>>>>>
> > >>>>>>> users
> > >>>>>>>
> > >>>>>>>> regarding inconsistent loggers used across Beam's modules.
> > >>>>>>>>
> > >>>>>>>> While we use SLF4J, different modules use a different logger
> > >>>>>>>>
> > >>>>>>> behind it
> > >>>>>>
> > >>>>>>> (JUL, log4j, etc)
> > >>>>>>>> So when people add a log4j.properties file to their classpath
> > >>>>>>>>
> > >>>>>>> for
> > >>>>
> > >>>>> instance,
> > >>>>>>>
> > >>>>>>>> they expect this to affect all of their dependencies on Beam
> > >>>>>>>>
> > >>>>>>> modules, but
> > >>>>>>
> > >>>>>>> it doesn’t and they miss out on some logs they thought they
> > >>>>>>>>
> > >>>>>>> would
> > >>>>
> > >>>>> see.
> > >>>>>>
> > >>>>>>>
> > >>>>>>>> I think we should strive for consistency in which logger is used
> > >>>>>>>>
> > >>>>>>> behind
> > >>>>>>
> > >>>>>>> SLF4J, and try to enforce this in our modules.
> > >>>>>>>> I for one think it should be slf4j-log4j. However, if
> > >>>>>>>>
> > >>>>>>> performance
> > >>>>
> > >>>>> of
> > >>>>>>
> > >>>>>>> logging is critical we might want to consider logback.
> > >>>>>>>>
> > >>>>>>>> Note: SLF4J will still be the facade for logging across the
> > >>>>>>>>
> > >>>>>>> project. The
> > >>>>>>
> > >>>>>>> only change would be the logger SLF4J delegates to.
> > >>>>>>>>
> > >>>>>>>> Once we have something like this it would also be useful to add
> > >>>>>>>> documentation on logging in Beam to the website.
> > >>>>>>>>
> > >>>>>>>> [1] https://issues.apache.org/jira/browse/BEAM-1757
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>> --
> > >>>>>>> Jean-Baptiste Onofré
> > >>>>>>> jbonofre@apache.org
> > >>>>>>> http://blog.nanthrax.net
> > >>>>>>> Talend - http://www.talend.com
> > >>>>>>>
> > >>>>>>>
> > >>>>>>
> > >>>>>>
> > >>>>>>
> > >>>>>
> > >>>
> > >>>
> > >>
> > > --
> > > Jean-Baptiste Onofré
> > > jbonofre@apache.org
> > > http://blog.nanthrax.net
> > > Talend - http://www.talend.com
> > >
> >
>

Re: [DISCUSSION] Consistent use of loggers

Posted by Aviem Zur <av...@gmail.com>.
>IMO I don't think the DirectRunner should depend directly on any specific logging
backend (at least, not in the compile or runtime scopes). I think it should
depend on JUL in the test scope, so that there are logs when executing
DirectRunner tests.
>My reasoning: I can see in any binary version of Beam that the SDK, the DirectRunner,
and 1 or more other runners will all be on the classpath.
>Ideally this should work regardless of whatever other runner is used; presumably
the DirectRunner would "automagically" pick up the logging config of the
other runner.
That sounds like a very plausible scenario and this would "protect" the
runner's binding from an intruding binding from direct runner, since it
would have no binding.
However, there is also the scenario that a user runs the examples using
direct runner,
this is their first interaction with Beam, and they see no logs whatsoever,
they would have to add a binding.
We could solve this by adding a binding in the 'direct-runner' profile in
examples module and the maven archetypes (And allow only one runner profile
to be specified at a time, in case their logger binding clashes).

>I like the use of slf4j as it enables lots of publishers of logs, but I don't
want to supply a default/required consumer of logs because that will restrict
use cases in the future...
I agree, forcing log4j binding might give the user a false sense of: "all
runners use log4j" while this might not be true for future (and isn't true
today, for Dataflow runner), but we can't assure that future runners could
support this.

So it seems we're left with:
1) Add documentation around logging in each runner.
2) Consider enabling a binding (JUL) for direct runner profile in examples
module and maven archetypes.
3) Allow only one runner profile to be active at a time in examples and
maven archetypes as their logger binding might clash.

Thoughts?

On Tue, Apr 4, 2017 at 8:51 AM Dan Halperin <dh...@google.com.invalid>
wrote:

> At this point, I'm a little unclear on what is the proposal. Can you
> refresh a simplified/aggregated view after this conversation?
>
> IMO I don't think the DirectRunner should depend directly on any specific
> logging backend (at least, not in the compile or runtime scopes). I think
> it should depend on JUL in the test scope, so that there are logs when
> executing DirectRunner tests.
>
> My reasoning: I can see in any binary version of Beam that the SDK, the
> DirectRunner, and 1 or more other runners will all be on the classpath.
> Ideally this should work regardless of whatever other runner is used;
> presumably the DirectRunner would "automagically" pick up the logging
> config of the other runner.
>
> I like the use of slf4j as it enables lots of publishers of logs, but I
> don't want to supply a default/required consumer of logs because that will
> restrict use cases in the future...
>
> On Mon, Apr 3, 2017 at 8:14 PM, Jean-Baptiste Onofré <jb...@nanthrax.net>
> wrote:
>
> > Fair enough. +1 especially for the documentation.
> >
> > Regards
> > JB
> >
> >
> > On 04/03/2017 08:48 PM, Aviem Zur wrote:
> >
> >> Upon further inspection there seems to be an issue we may have
> overlooked:
> >> In cluster mode, some of the runners will have dependencies added
> directly
> >> to the classpath by the cluster, and since SLF4J can only work with one
> >> binding, the first one in the classpath will be used.
> >>
> >> So while what we suggested would work in local mode, the user's chosen
> >> binding and configuration might be ignored in cluster mode, which is
> >> detrimental to what we wanted to accomplish.
> >>
> >> So I believe what we should do instead is:
> >>
> >>    1. Add better documentation regarding logging in each runner, which
> >>    binding is used, perhaps examples of how to configure logging for
> that
> >>    runner.
> >>    2. Have direct runner use the most common binding among runners (this
> >>    appears to be log4j which is used by Spark runner, Flink runner and
> >> Apex
> >>    runner).
> >>
> >>
> >> On Mon, Apr 3, 2017 at 7:02 PM Aljoscha Krettek <al...@apache.org>
> >> wrote:
> >>
> >> Yes, I think we can exclude log4j from the Flink dependencies. It’s
> >>> somewhat annoying that they are there in the first place.
> >>>
> >>> The Flink doc has this to say about the topic:
> >>> https://ci.apache.org/projects/flink/flink-docs-release-1.3/
> >>> monitoring/logging.html
> >>>
> >>>> On 3. Apr 2017, at 17:56, Aviem Zur <av...@gmail.com> wrote:
> >>>>
> >>>> * java.util.logging could be a good choice for the Direct Runner
> >>>>>
> >>>> Yes, this will be great for users (Instead of having no logging when
> >>>>
> >>> using
> >>>
> >>>> direct runner).
> >>>>
> >>>> * Logging backend could be runner-specific, particularly if it needs
> to
> >>>>> integrate into some other experience
> >>>>>
> >>>> Good point, let's take a look at the current state of runners:
> >>>> Direct runner - will use JUL as suggested.
> >>>> Dataflow runner - looks like there is already no binding (There is a
> >>>> binding in tests only).
> >>>> Spark runner - currently uses slf4j-log4j12. does not require any
> >>>>
> >>> specific
> >>>
> >>>> logger, we can change this to no binding.
> >>>> Flink runner - uses slf4j-log4j12 transitively from Flink
> dependencies.
> >>>>
> >>> I'm
> >>>
> >>>> assuming this is not a must and we can default to no binding here.
> >>>> @aljoscha please confirm.
> >>>> Apex runner - uses slf4j-log4j12 transitively from Apex dependencies.
> >>>> I'm
> >>>> assuming this is not a must and we can default to no binding here.
> @thw
> >>>> please confirm.
> >>>>
> >>>> It might be a good idea to use a consistent binding in tests (Since
> >>>> we'll
> >>>> use JUL for direct runner, let this be JUL).
> >>>>
> >>>> On Wed, Mar 29, 2017 at 7:23 PM Davor Bonaci <da...@apache.org>
> wrote:
> >>>>
> >>>> +1 on consistency across Beam modules on the logging facade
> >>>> +1 on enforcing consistency
> >>>> +1 on clearly documenting how to do logging
> >>>>
> >>>> Mixed feelings:
> >>>> * Logging backend could be runner-specific, particularly if it needs
> to
> >>>> integrate into some other experience
> >>>> * java.util.logging could be a good choice for the Direct Runner
> >>>>
> >>>> On Tue, Mar 28, 2017 at 6:50 PM, Ahmet Altay <altay@google.com.invalid
> >
> >>>> wrote:
> >>>>
> >>>> On Wed, Mar 22, 2017 at 10:38 AM, Tibor Kiss <tk...@hortonworks.com>
> >>>>> wrote:
> >>>>>
> >>>>> This is a great idea!
> >>>>>>
> >>>>>> I believe Python-SDK's logging could also be enhanced (a bit
> >>>>>>
> >>>>> differently):
> >>>>>
> >>>>>> Currently we are not instantiating the logger, just using the class
> >>>>>>
> >>>>> what
> >>>
> >>>> logging package provides.
> >>>>>> Shortcoming of this approach is that the user cannot set the log
> level
> >>>>>>
> >>>>> on
> >>>>
> >>>>> a per module basis as all log messages
> >>>>>> end up in the root level.
> >>>>>>
> >>>>>>
> >>>>> +1 to this. Python SDK needs to expands its logging capabilities.
> Filed
> >>>>>
> >>>> [1]
> >>>>
> >>>>> for this.
> >>>>>
> >>>>> Ahmet
> >>>>>
> >>>>> [1] https://issues.apache.org/jira/browse/BEAM-1825
> >>>>>
> >>>>>
> >>>>>
> >>>>>> On 3/22/17, 5:46 AM, "Aviem Zur" <av...@gmail.com> wrote:
> >>>>>>
> >>>>>>    +1 to what JB said.
> >>>>>>
> >>>>>>    Will just have to be documented well as if we provide no binding
> >>>>>>
> >>>>> there
> >>>>>
> >>>>>> will
> >>>>>>    be no logging out of the box unless the user adds a binding.
> >>>>>>
> >>>>>>    On Wed, Mar 22, 2017 at 6:24 AM Jean-Baptiste Onofré <
> >>>>>>
> >>>>> jb@nanthrax.net>
> >>>>>
> >>>>>>    wrote:
> >>>>>>
> >>>>>> Hi Aviem,
> >>>>>>>
> >>>>>>> Good point.
> >>>>>>>
> >>>>>>> I think, in our dependencies set, we should just depend to
> >>>>>>>
> >>>>>> slf4j-api
> >>>>>
> >>>>>> and
> >>>>>>
> >>>>>>> let the
> >>>>>>> user provides the binding he wants (slf4j-log4j12, slf4j-simple,
> >>>>>>>
> >>>>>> whatever).
> >>>>>>
> >>>>>>>
> >>>>>>> We define a binding only with test scope in our modules.
> >>>>>>>
> >>>>>>> Regards
> >>>>>>> JB
> >>>>>>>
> >>>>>>> On 03/22/2017 04:58 AM, Aviem Zur wrote:
> >>>>>>>
> >>>>>>>> Hi all,
> >>>>>>>>
> >>>>>>>> There have been a few reports lately (On JIRA [1] and on Slack)
> >>>>>>>>
> >>>>>>> from
> >>>>>>
> >>>>>>> users
> >>>>>>>
> >>>>>>>> regarding inconsistent loggers used across Beam's modules.
> >>>>>>>>
> >>>>>>>> While we use SLF4J, different modules use a different logger
> >>>>>>>>
> >>>>>>> behind it
> >>>>>>
> >>>>>>> (JUL, log4j, etc)
> >>>>>>>> So when people add a log4j.properties file to their classpath
> >>>>>>>>
> >>>>>>> for
> >>>>
> >>>>> instance,
> >>>>>>>
> >>>>>>>> they expect this to affect all of their dependencies on Beam
> >>>>>>>>
> >>>>>>> modules, but
> >>>>>>
> >>>>>>> it doesn’t and they miss out on some logs they thought they
> >>>>>>>>
> >>>>>>> would
> >>>>
> >>>>> see.
> >>>>>>
> >>>>>>>
> >>>>>>>> I think we should strive for consistency in which logger is used
> >>>>>>>>
> >>>>>>> behind
> >>>>>>
> >>>>>>> SLF4J, and try to enforce this in our modules.
> >>>>>>>> I for one think it should be slf4j-log4j. However, if
> >>>>>>>>
> >>>>>>> performance
> >>>>
> >>>>> of
> >>>>>>
> >>>>>>> logging is critical we might want to consider logback.
> >>>>>>>>
> >>>>>>>> Note: SLF4J will still be the facade for logging across the
> >>>>>>>>
> >>>>>>> project. The
> >>>>>>
> >>>>>>> only change would be the logger SLF4J delegates to.
> >>>>>>>>
> >>>>>>>> Once we have something like this it would also be useful to add
> >>>>>>>> documentation on logging in Beam to the website.
> >>>>>>>>
> >>>>>>>> [1] https://issues.apache.org/jira/browse/BEAM-1757
> >>>>>>>>
> >>>>>>>>
> >>>>>>> --
> >>>>>>> Jean-Baptiste Onofré
> >>>>>>> jbonofre@apache.org
> >>>>>>> http://blog.nanthrax.net
> >>>>>>> Talend - http://www.talend.com
> >>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>
> >>>
> >>>
> >>
> > --
> > Jean-Baptiste Onofré
> > jbonofre@apache.org
> > http://blog.nanthrax.net
> > Talend - http://www.talend.com
> >
>

Re: [DISCUSSION] Consistent use of loggers

Posted by Dan Halperin <dh...@google.com.INVALID>.
At this point, I'm a little unclear on what is the proposal. Can you
refresh a simplified/aggregated view after this conversation?

IMO I don't think the DirectRunner should depend directly on any specific
logging backend (at least, not in the compile or runtime scopes). I think
it should depend on JUL in the test scope, so that there are logs when
executing DirectRunner tests.

My reasoning: I can see in any binary version of Beam that the SDK, the
DirectRunner, and 1 or more other runners will all be on the classpath.
Ideally this should work regardless of whatever other runner is used;
presumably the DirectRunner would "automagically" pick up the logging
config of the other runner.

I like the use of slf4j as it enables lots of publishers of logs, but I
don't want to supply a default/required consumer of logs because that will
restrict use cases in the future...

On Mon, Apr 3, 2017 at 8:14 PM, Jean-Baptiste Onofré <jb...@nanthrax.net>
wrote:

> Fair enough. +1 especially for the documentation.
>
> Regards
> JB
>
>
> On 04/03/2017 08:48 PM, Aviem Zur wrote:
>
>> Upon further inspection there seems to be an issue we may have overlooked:
>> In cluster mode, some of the runners will have dependencies added directly
>> to the classpath by the cluster, and since SLF4J can only work with one
>> binding, the first one in the classpath will be used.
>>
>> So while what we suggested would work in local mode, the user's chosen
>> binding and configuration might be ignored in cluster mode, which is
>> detrimental to what we wanted to accomplish.
>>
>> So I believe what we should do instead is:
>>
>>    1. Add better documentation regarding logging in each runner, which
>>    binding is used, perhaps examples of how to configure logging for that
>>    runner.
>>    2. Have direct runner use the most common binding among runners (this
>>    appears to be log4j which is used by Spark runner, Flink runner and
>> Apex
>>    runner).
>>
>>
>> On Mon, Apr 3, 2017 at 7:02 PM Aljoscha Krettek <al...@apache.org>
>> wrote:
>>
>> Yes, I think we can exclude log4j from the Flink dependencies. It’s
>>> somewhat annoying that they are there in the first place.
>>>
>>> The Flink doc has this to say about the topic:
>>> https://ci.apache.org/projects/flink/flink-docs-release-1.3/
>>> monitoring/logging.html
>>>
>>>> On 3. Apr 2017, at 17:56, Aviem Zur <av...@gmail.com> wrote:
>>>>
>>>> * java.util.logging could be a good choice for the Direct Runner
>>>>>
>>>> Yes, this will be great for users (Instead of having no logging when
>>>>
>>> using
>>>
>>>> direct runner).
>>>>
>>>> * Logging backend could be runner-specific, particularly if it needs to
>>>>> integrate into some other experience
>>>>>
>>>> Good point, let's take a look at the current state of runners:
>>>> Direct runner - will use JUL as suggested.
>>>> Dataflow runner - looks like there is already no binding (There is a
>>>> binding in tests only).
>>>> Spark runner - currently uses slf4j-log4j12. does not require any
>>>>
>>> specific
>>>
>>>> logger, we can change this to no binding.
>>>> Flink runner - uses slf4j-log4j12 transitively from Flink dependencies.
>>>>
>>> I'm
>>>
>>>> assuming this is not a must and we can default to no binding here.
>>>> @aljoscha please confirm.
>>>> Apex runner - uses slf4j-log4j12 transitively from Apex dependencies.
>>>> I'm
>>>> assuming this is not a must and we can default to no binding here. @thw
>>>> please confirm.
>>>>
>>>> It might be a good idea to use a consistent binding in tests (Since
>>>> we'll
>>>> use JUL for direct runner, let this be JUL).
>>>>
>>>> On Wed, Mar 29, 2017 at 7:23 PM Davor Bonaci <da...@apache.org> wrote:
>>>>
>>>> +1 on consistency across Beam modules on the logging facade
>>>> +1 on enforcing consistency
>>>> +1 on clearly documenting how to do logging
>>>>
>>>> Mixed feelings:
>>>> * Logging backend could be runner-specific, particularly if it needs to
>>>> integrate into some other experience
>>>> * java.util.logging could be a good choice for the Direct Runner
>>>>
>>>> On Tue, Mar 28, 2017 at 6:50 PM, Ahmet Altay <al...@google.com.invalid>
>>>> wrote:
>>>>
>>>> On Wed, Mar 22, 2017 at 10:38 AM, Tibor Kiss <tk...@hortonworks.com>
>>>>> wrote:
>>>>>
>>>>> This is a great idea!
>>>>>>
>>>>>> I believe Python-SDK's logging could also be enhanced (a bit
>>>>>>
>>>>> differently):
>>>>>
>>>>>> Currently we are not instantiating the logger, just using the class
>>>>>>
>>>>> what
>>>
>>>> logging package provides.
>>>>>> Shortcoming of this approach is that the user cannot set the log level
>>>>>>
>>>>> on
>>>>
>>>>> a per module basis as all log messages
>>>>>> end up in the root level.
>>>>>>
>>>>>>
>>>>> +1 to this. Python SDK needs to expands its logging capabilities. Filed
>>>>>
>>>> [1]
>>>>
>>>>> for this.
>>>>>
>>>>> Ahmet
>>>>>
>>>>> [1] https://issues.apache.org/jira/browse/BEAM-1825
>>>>>
>>>>>
>>>>>
>>>>>> On 3/22/17, 5:46 AM, "Aviem Zur" <av...@gmail.com> wrote:
>>>>>>
>>>>>>    +1 to what JB said.
>>>>>>
>>>>>>    Will just have to be documented well as if we provide no binding
>>>>>>
>>>>> there
>>>>>
>>>>>> will
>>>>>>    be no logging out of the box unless the user adds a binding.
>>>>>>
>>>>>>    On Wed, Mar 22, 2017 at 6:24 AM Jean-Baptiste Onofré <
>>>>>>
>>>>> jb@nanthrax.net>
>>>>>
>>>>>>    wrote:
>>>>>>
>>>>>> Hi Aviem,
>>>>>>>
>>>>>>> Good point.
>>>>>>>
>>>>>>> I think, in our dependencies set, we should just depend to
>>>>>>>
>>>>>> slf4j-api
>>>>>
>>>>>> and
>>>>>>
>>>>>>> let the
>>>>>>> user provides the binding he wants (slf4j-log4j12, slf4j-simple,
>>>>>>>
>>>>>> whatever).
>>>>>>
>>>>>>>
>>>>>>> We define a binding only with test scope in our modules.
>>>>>>>
>>>>>>> Regards
>>>>>>> JB
>>>>>>>
>>>>>>> On 03/22/2017 04:58 AM, Aviem Zur wrote:
>>>>>>>
>>>>>>>> Hi all,
>>>>>>>>
>>>>>>>> There have been a few reports lately (On JIRA [1] and on Slack)
>>>>>>>>
>>>>>>> from
>>>>>>
>>>>>>> users
>>>>>>>
>>>>>>>> regarding inconsistent loggers used across Beam's modules.
>>>>>>>>
>>>>>>>> While we use SLF4J, different modules use a different logger
>>>>>>>>
>>>>>>> behind it
>>>>>>
>>>>>>> (JUL, log4j, etc)
>>>>>>>> So when people add a log4j.properties file to their classpath
>>>>>>>>
>>>>>>> for
>>>>
>>>>> instance,
>>>>>>>
>>>>>>>> they expect this to affect all of their dependencies on Beam
>>>>>>>>
>>>>>>> modules, but
>>>>>>
>>>>>>> it doesn’t and they miss out on some logs they thought they
>>>>>>>>
>>>>>>> would
>>>>
>>>>> see.
>>>>>>
>>>>>>>
>>>>>>>> I think we should strive for consistency in which logger is used
>>>>>>>>
>>>>>>> behind
>>>>>>
>>>>>>> SLF4J, and try to enforce this in our modules.
>>>>>>>> I for one think it should be slf4j-log4j. However, if
>>>>>>>>
>>>>>>> performance
>>>>
>>>>> of
>>>>>>
>>>>>>> logging is critical we might want to consider logback.
>>>>>>>>
>>>>>>>> Note: SLF4J will still be the facade for logging across the
>>>>>>>>
>>>>>>> project. The
>>>>>>
>>>>>>> only change would be the logger SLF4J delegates to.
>>>>>>>>
>>>>>>>> Once we have something like this it would also be useful to add
>>>>>>>> documentation on logging in Beam to the website.
>>>>>>>>
>>>>>>>> [1] https://issues.apache.org/jira/browse/BEAM-1757
>>>>>>>>
>>>>>>>>
>>>>>>> --
>>>>>>> Jean-Baptiste Onofré
>>>>>>> jbonofre@apache.org
>>>>>>> http://blog.nanthrax.net
>>>>>>> Talend - http://www.talend.com
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>
>>>
>>
> --
> Jean-Baptiste Onofré
> jbonofre@apache.org
> http://blog.nanthrax.net
> Talend - http://www.talend.com
>

Re: [DISCUSSION] Consistent use of loggers

Posted by Jean-Baptiste Onofré <jb...@nanthrax.net>.
Fair enough. +1 especially for the documentation.

Regards
JB

On 04/03/2017 08:48 PM, Aviem Zur wrote:
> Upon further inspection there seems to be an issue we may have overlooked:
> In cluster mode, some of the runners will have dependencies added directly
> to the classpath by the cluster, and since SLF4J can only work with one
> binding, the first one in the classpath will be used.
>
> So while what we suggested would work in local mode, the user's chosen
> binding and configuration might be ignored in cluster mode, which is
> detrimental to what we wanted to accomplish.
>
> So I believe what we should do instead is:
>
>    1. Add better documentation regarding logging in each runner, which
>    binding is used, perhaps examples of how to configure logging for that
>    runner.
>    2. Have direct runner use the most common binding among runners (this
>    appears to be log4j which is used by Spark runner, Flink runner and Apex
>    runner).
>
>
> On Mon, Apr 3, 2017 at 7:02 PM Aljoscha Krettek <al...@apache.org> wrote:
>
>> Yes, I think we can exclude log4j from the Flink dependencies. It\u2019s
>> somewhat annoying that they are there in the first place.
>>
>> The Flink doc has this to say about the topic:
>> https://ci.apache.org/projects/flink/flink-docs-release-1.3/monitoring/logging.html
>>> On 3. Apr 2017, at 17:56, Aviem Zur <av...@gmail.com> wrote:
>>>
>>>> * java.util.logging could be a good choice for the Direct Runner
>>> Yes, this will be great for users (Instead of having no logging when
>> using
>>> direct runner).
>>>
>>>> * Logging backend could be runner-specific, particularly if it needs to
>>>> integrate into some other experience
>>> Good point, let's take a look at the current state of runners:
>>> Direct runner - will use JUL as suggested.
>>> Dataflow runner - looks like there is already no binding (There is a
>>> binding in tests only).
>>> Spark runner - currently uses slf4j-log4j12. does not require any
>> specific
>>> logger, we can change this to no binding.
>>> Flink runner - uses slf4j-log4j12 transitively from Flink dependencies.
>> I'm
>>> assuming this is not a must and we can default to no binding here.
>>> @aljoscha please confirm.
>>> Apex runner - uses slf4j-log4j12 transitively from Apex dependencies. I'm
>>> assuming this is not a must and we can default to no binding here. @thw
>>> please confirm.
>>>
>>> It might be a good idea to use a consistent binding in tests (Since we'll
>>> use JUL for direct runner, let this be JUL).
>>>
>>> On Wed, Mar 29, 2017 at 7:23 PM Davor Bonaci <da...@apache.org> wrote:
>>>
>>> +1 on consistency across Beam modules on the logging facade
>>> +1 on enforcing consistency
>>> +1 on clearly documenting how to do logging
>>>
>>> Mixed feelings:
>>> * Logging backend could be runner-specific, particularly if it needs to
>>> integrate into some other experience
>>> * java.util.logging could be a good choice for the Direct Runner
>>>
>>> On Tue, Mar 28, 2017 at 6:50 PM, Ahmet Altay <al...@google.com.invalid>
>>> wrote:
>>>
>>>> On Wed, Mar 22, 2017 at 10:38 AM, Tibor Kiss <tk...@hortonworks.com>
>>>> wrote:
>>>>
>>>>> This is a great idea!
>>>>>
>>>>> I believe Python-SDK's logging could also be enhanced (a bit
>>>> differently):
>>>>> Currently we are not instantiating the logger, just using the class
>> what
>>>>> logging package provides.
>>>>> Shortcoming of this approach is that the user cannot set the log level
>>> on
>>>>> a per module basis as all log messages
>>>>> end up in the root level.
>>>>>
>>>>
>>>> +1 to this. Python SDK needs to expands its logging capabilities. Filed
>>> [1]
>>>> for this.
>>>>
>>>> Ahmet
>>>>
>>>> [1] https://issues.apache.org/jira/browse/BEAM-1825
>>>>
>>>>
>>>>>
>>>>> On 3/22/17, 5:46 AM, "Aviem Zur" <av...@gmail.com> wrote:
>>>>>
>>>>>    +1 to what JB said.
>>>>>
>>>>>    Will just have to be documented well as if we provide no binding
>>>> there
>>>>> will
>>>>>    be no logging out of the box unless the user adds a binding.
>>>>>
>>>>>    On Wed, Mar 22, 2017 at 6:24 AM Jean-Baptiste Onofr� <
>>>> jb@nanthrax.net>
>>>>>    wrote:
>>>>>
>>>>>> Hi Aviem,
>>>>>>
>>>>>> Good point.
>>>>>>
>>>>>> I think, in our dependencies set, we should just depend to
>>>> slf4j-api
>>>>> and
>>>>>> let the
>>>>>> user provides the binding he wants (slf4j-log4j12, slf4j-simple,
>>>>> whatever).
>>>>>>
>>>>>> We define a binding only with test scope in our modules.
>>>>>>
>>>>>> Regards
>>>>>> JB
>>>>>>
>>>>>> On 03/22/2017 04:58 AM, Aviem Zur wrote:
>>>>>>> Hi all,
>>>>>>>
>>>>>>> There have been a few reports lately (On JIRA [1] and on Slack)
>>>>> from
>>>>>> users
>>>>>>> regarding inconsistent loggers used across Beam's modules.
>>>>>>>
>>>>>>> While we use SLF4J, different modules use a different logger
>>>>> behind it
>>>>>>> (JUL, log4j, etc)
>>>>>>> So when people add a log4j.properties file to their classpath
>>> for
>>>>>> instance,
>>>>>>> they expect this to affect all of their dependencies on Beam
>>>>> modules, but
>>>>>>> it doesn\u2019t and they miss out on some logs they thought they
>>> would
>>>>> see.
>>>>>>>
>>>>>>> I think we should strive for consistency in which logger is used
>>>>> behind
>>>>>>> SLF4J, and try to enforce this in our modules.
>>>>>>> I for one think it should be slf4j-log4j. However, if
>>> performance
>>>>> of
>>>>>>> logging is critical we might want to consider logback.
>>>>>>>
>>>>>>> Note: SLF4J will still be the facade for logging across the
>>>>> project. The
>>>>>>> only change would be the logger SLF4J delegates to.
>>>>>>>
>>>>>>> Once we have something like this it would also be useful to add
>>>>>>> documentation on logging in Beam to the website.
>>>>>>>
>>>>>>> [1] https://issues.apache.org/jira/browse/BEAM-1757
>>>>>>>
>>>>>>
>>>>>> --
>>>>>> Jean-Baptiste Onofr�
>>>>>> jbonofre@apache.org
>>>>>> http://blog.nanthrax.net
>>>>>> Talend - http://www.talend.com
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>
>>
>

-- 
Jean-Baptiste Onofr�
jbonofre@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com

Re: [DISCUSSION] Consistent use of loggers

Posted by Ted Yu <yu...@gmail.com>.
+1

> On Apr 3, 2017, at 11:48 AM, Aviem Zur <av...@gmail.com> wrote:
> 
> Upon further inspection there seems to be an issue we may have overlooked:
> In cluster mode, some of the runners will have dependencies added directly
> to the classpath by the cluster, and since SLF4J can only work with one
> binding, the first one in the classpath will be used.
> 
> So while what we suggested would work in local mode, the user's chosen
> binding and configuration might be ignored in cluster mode, which is
> detrimental to what we wanted to accomplish.
> 
> So I believe what we should do instead is:
> 
>   1. Add better documentation regarding logging in each runner, which
>   binding is used, perhaps examples of how to configure logging for that
>   runner.
>   2. Have direct runner use the most common binding among runners (this
>   appears to be log4j which is used by Spark runner, Flink runner and Apex
>   runner).
> 
> 
>> On Mon, Apr 3, 2017 at 7:02 PM Aljoscha Krettek <al...@apache.org> wrote:
>> 
>> Yes, I think we can exclude log4j from the Flink dependencies. It’s
>> somewhat annoying that they are there in the first place.
>> 
>> The Flink doc has this to say about the topic:
>> https://ci.apache.org/projects/flink/flink-docs-release-1.3/monitoring/logging.html
>>>> On 3. Apr 2017, at 17:56, Aviem Zur <av...@gmail.com> wrote:
>>>> 
>>>> * java.util.logging could be a good choice for the Direct Runner
>>> Yes, this will be great for users (Instead of having no logging when
>> using
>>> direct runner).
>>> 
>>>> * Logging backend could be runner-specific, particularly if it needs to
>>>> integrate into some other experience
>>> Good point, let's take a look at the current state of runners:
>>> Direct runner - will use JUL as suggested.
>>> Dataflow runner - looks like there is already no binding (There is a
>>> binding in tests only).
>>> Spark runner - currently uses slf4j-log4j12. does not require any
>> specific
>>> logger, we can change this to no binding.
>>> Flink runner - uses slf4j-log4j12 transitively from Flink dependencies.
>> I'm
>>> assuming this is not a must and we can default to no binding here.
>>> @aljoscha please confirm.
>>> Apex runner - uses slf4j-log4j12 transitively from Apex dependencies. I'm
>>> assuming this is not a must and we can default to no binding here. @thw
>>> please confirm.
>>> 
>>> It might be a good idea to use a consistent binding in tests (Since we'll
>>> use JUL for direct runner, let this be JUL).
>>> 
>>> On Wed, Mar 29, 2017 at 7:23 PM Davor Bonaci <da...@apache.org> wrote:
>>> 
>>> +1 on consistency across Beam modules on the logging facade
>>> +1 on enforcing consistency
>>> +1 on clearly documenting how to do logging
>>> 
>>> Mixed feelings:
>>> * Logging backend could be runner-specific, particularly if it needs to
>>> integrate into some other experience
>>> * java.util.logging could be a good choice for the Direct Runner
>>> 
>>> On Tue, Mar 28, 2017 at 6:50 PM, Ahmet Altay <al...@google.com.invalid>
>>> wrote:
>>> 
>>>> On Wed, Mar 22, 2017 at 10:38 AM, Tibor Kiss <tk...@hortonworks.com>
>>>> wrote:
>>>> 
>>>>> This is a great idea!
>>>>> 
>>>>> I believe Python-SDK's logging could also be enhanced (a bit
>>>> differently):
>>>>> Currently we are not instantiating the logger, just using the class
>> what
>>>>> logging package provides.
>>>>> Shortcoming of this approach is that the user cannot set the log level
>>> on
>>>>> a per module basis as all log messages
>>>>> end up in the root level.
>>>> 
>>>> +1 to this. Python SDK needs to expands its logging capabilities. Filed
>>> [1]
>>>> for this.
>>>> 
>>>> Ahmet
>>>> 
>>>> [1] https://issues.apache.org/jira/browse/BEAM-1825
>>>> 
>>>> 
>>>>> 
>>>>> On 3/22/17, 5:46 AM, "Aviem Zur" <av...@gmail.com> wrote:
>>>>> 
>>>>>   +1 to what JB said.
>>>>> 
>>>>>   Will just have to be documented well as if we provide no binding
>>>> there
>>>>> will
>>>>>   be no logging out of the box unless the user adds a binding.
>>>>> 
>>>>>   On Wed, Mar 22, 2017 at 6:24 AM Jean-Baptiste Onofré <
>>>> jb@nanthrax.net>
>>>>>   wrote:
>>>>> 
>>>>>> Hi Aviem,
>>>>>> 
>>>>>> Good point.
>>>>>> 
>>>>>> I think, in our dependencies set, we should just depend to
>>>> slf4j-api
>>>>> and
>>>>>> let the
>>>>>> user provides the binding he wants (slf4j-log4j12, slf4j-simple,
>>>>> whatever).
>>>>>> 
>>>>>> We define a binding only with test scope in our modules.
>>>>>> 
>>>>>> Regards
>>>>>> JB
>>>>>> 
>>>>>>> On 03/22/2017 04:58 AM, Aviem Zur wrote:
>>>>>>> Hi all,
>>>>>>> 
>>>>>>> There have been a few reports lately (On JIRA [1] and on Slack)
>>>>> from
>>>>>> users
>>>>>>> regarding inconsistent loggers used across Beam's modules.
>>>>>>> 
>>>>>>> While we use SLF4J, different modules use a different logger
>>>>> behind it
>>>>>>> (JUL, log4j, etc)
>>>>>>> So when people add a log4j.properties file to their classpath
>>> for
>>>>>> instance,
>>>>>>> they expect this to affect all of their dependencies on Beam
>>>>> modules, but
>>>>>>> it doesn’t and they miss out on some logs they thought they
>>> would
>>>>> see.
>>>>>>> 
>>>>>>> I think we should strive for consistency in which logger is used
>>>>> behind
>>>>>>> SLF4J, and try to enforce this in our modules.
>>>>>>> I for one think it should be slf4j-log4j. However, if
>>> performance
>>>>> of
>>>>>>> logging is critical we might want to consider logback.
>>>>>>> 
>>>>>>> Note: SLF4J will still be the facade for logging across the
>>>>> project. The
>>>>>>> only change would be the logger SLF4J delegates to.
>>>>>>> 
>>>>>>> Once we have something like this it would also be useful to add
>>>>>>> documentation on logging in Beam to the website.
>>>>>>> 
>>>>>>> [1] https://issues.apache.org/jira/browse/BEAM-1757
>>>>>> 
>>>>>> --
>>>>>> Jean-Baptiste Onofré
>>>>>> jbonofre@apache.org
>>>>>> http://blog.nanthrax.net
>>>>>> Talend - http://www.talend.com
>> 
>> 

Re: [DISCUSSION] Consistent use of loggers

Posted by Aviem Zur <av...@gmail.com>.
Upon further inspection there seems to be an issue we may have overlooked:
In cluster mode, some of the runners will have dependencies added directly
to the classpath by the cluster, and since SLF4J can only work with one
binding, the first one in the classpath will be used.

So while what we suggested would work in local mode, the user's chosen
binding and configuration might be ignored in cluster mode, which is
detrimental to what we wanted to accomplish.

So I believe what we should do instead is:

   1. Add better documentation regarding logging in each runner, which
   binding is used, perhaps examples of how to configure logging for that
   runner.
   2. Have direct runner use the most common binding among runners (this
   appears to be log4j which is used by Spark runner, Flink runner and Apex
   runner).


On Mon, Apr 3, 2017 at 7:02 PM Aljoscha Krettek <al...@apache.org> wrote:

> Yes, I think we can exclude log4j from the Flink dependencies. It’s
> somewhat annoying that they are there in the first place.
>
> The Flink doc has this to say about the topic:
> https://ci.apache.org/projects/flink/flink-docs-release-1.3/monitoring/logging.html
> > On 3. Apr 2017, at 17:56, Aviem Zur <av...@gmail.com> wrote:
> >
> >> * java.util.logging could be a good choice for the Direct Runner
> > Yes, this will be great for users (Instead of having no logging when
> using
> > direct runner).
> >
> >> * Logging backend could be runner-specific, particularly if it needs to
> >> integrate into some other experience
> > Good point, let's take a look at the current state of runners:
> > Direct runner - will use JUL as suggested.
> > Dataflow runner - looks like there is already no binding (There is a
> > binding in tests only).
> > Spark runner - currently uses slf4j-log4j12. does not require any
> specific
> > logger, we can change this to no binding.
> > Flink runner - uses slf4j-log4j12 transitively from Flink dependencies.
> I'm
> > assuming this is not a must and we can default to no binding here.
> > @aljoscha please confirm.
> > Apex runner - uses slf4j-log4j12 transitively from Apex dependencies. I'm
> > assuming this is not a must and we can default to no binding here. @thw
> > please confirm.
> >
> > It might be a good idea to use a consistent binding in tests (Since we'll
> > use JUL for direct runner, let this be JUL).
> >
> > On Wed, Mar 29, 2017 at 7:23 PM Davor Bonaci <da...@apache.org> wrote:
> >
> > +1 on consistency across Beam modules on the logging facade
> > +1 on enforcing consistency
> > +1 on clearly documenting how to do logging
> >
> > Mixed feelings:
> > * Logging backend could be runner-specific, particularly if it needs to
> > integrate into some other experience
> > * java.util.logging could be a good choice for the Direct Runner
> >
> > On Tue, Mar 28, 2017 at 6:50 PM, Ahmet Altay <al...@google.com.invalid>
> > wrote:
> >
> >> On Wed, Mar 22, 2017 at 10:38 AM, Tibor Kiss <tk...@hortonworks.com>
> >> wrote:
> >>
> >>> This is a great idea!
> >>>
> >>> I believe Python-SDK's logging could also be enhanced (a bit
> >> differently):
> >>> Currently we are not instantiating the logger, just using the class
> what
> >>> logging package provides.
> >>> Shortcoming of this approach is that the user cannot set the log level
> > on
> >>> a per module basis as all log messages
> >>> end up in the root level.
> >>>
> >>
> >> +1 to this. Python SDK needs to expands its logging capabilities. Filed
> > [1]
> >> for this.
> >>
> >> Ahmet
> >>
> >> [1] https://issues.apache.org/jira/browse/BEAM-1825
> >>
> >>
> >>>
> >>> On 3/22/17, 5:46 AM, "Aviem Zur" <av...@gmail.com> wrote:
> >>>
> >>>    +1 to what JB said.
> >>>
> >>>    Will just have to be documented well as if we provide no binding
> >> there
> >>> will
> >>>    be no logging out of the box unless the user adds a binding.
> >>>
> >>>    On Wed, Mar 22, 2017 at 6:24 AM Jean-Baptiste Onofré <
> >> jb@nanthrax.net>
> >>>    wrote:
> >>>
> >>>> Hi Aviem,
> >>>>
> >>>> Good point.
> >>>>
> >>>> I think, in our dependencies set, we should just depend to
> >> slf4j-api
> >>> and
> >>>> let the
> >>>> user provides the binding he wants (slf4j-log4j12, slf4j-simple,
> >>> whatever).
> >>>>
> >>>> We define a binding only with test scope in our modules.
> >>>>
> >>>> Regards
> >>>> JB
> >>>>
> >>>> On 03/22/2017 04:58 AM, Aviem Zur wrote:
> >>>>> Hi all,
> >>>>>
> >>>>> There have been a few reports lately (On JIRA [1] and on Slack)
> >>> from
> >>>> users
> >>>>> regarding inconsistent loggers used across Beam's modules.
> >>>>>
> >>>>> While we use SLF4J, different modules use a different logger
> >>> behind it
> >>>>> (JUL, log4j, etc)
> >>>>> So when people add a log4j.properties file to their classpath
> > for
> >>>> instance,
> >>>>> they expect this to affect all of their dependencies on Beam
> >>> modules, but
> >>>>> it doesn’t and they miss out on some logs they thought they
> > would
> >>> see.
> >>>>>
> >>>>> I think we should strive for consistency in which logger is used
> >>> behind
> >>>>> SLF4J, and try to enforce this in our modules.
> >>>>> I for one think it should be slf4j-log4j. However, if
> > performance
> >>> of
> >>>>> logging is critical we might want to consider logback.
> >>>>>
> >>>>> Note: SLF4J will still be the facade for logging across the
> >>> project. The
> >>>>> only change would be the logger SLF4J delegates to.
> >>>>>
> >>>>> Once we have something like this it would also be useful to add
> >>>>> documentation on logging in Beam to the website.
> >>>>>
> >>>>> [1] https://issues.apache.org/jira/browse/BEAM-1757
> >>>>>
> >>>>
> >>>> --
> >>>> Jean-Baptiste Onofré
> >>>> jbonofre@apache.org
> >>>> http://blog.nanthrax.net
> >>>> Talend - http://www.talend.com
> >>>>
> >>>
> >>>
> >>>
> >>
>
>

Re: [DISCUSSION] Consistent use of loggers

Posted by Aljoscha Krettek <al...@apache.org>.
Yes, I think we can exclude log4j from the Flink dependencies. It’s somewhat annoying that they are there in the first place.

The Flink doc has this to say about the topic: https://ci.apache.org/projects/flink/flink-docs-release-1.3/monitoring/logging.html
> On 3. Apr 2017, at 17:56, Aviem Zur <av...@gmail.com> wrote:
> 
>> * java.util.logging could be a good choice for the Direct Runner
> Yes, this will be great for users (Instead of having no logging when using
> direct runner).
> 
>> * Logging backend could be runner-specific, particularly if it needs to
>> integrate into some other experience
> Good point, let's take a look at the current state of runners:
> Direct runner - will use JUL as suggested.
> Dataflow runner - looks like there is already no binding (There is a
> binding in tests only).
> Spark runner - currently uses slf4j-log4j12. does not require any specific
> logger, we can change this to no binding.
> Flink runner - uses slf4j-log4j12 transitively from Flink dependencies. I'm
> assuming this is not a must and we can default to no binding here.
> @aljoscha please confirm.
> Apex runner - uses slf4j-log4j12 transitively from Apex dependencies. I'm
> assuming this is not a must and we can default to no binding here. @thw
> please confirm.
> 
> It might be a good idea to use a consistent binding in tests (Since we'll
> use JUL for direct runner, let this be JUL).
> 
> On Wed, Mar 29, 2017 at 7:23 PM Davor Bonaci <da...@apache.org> wrote:
> 
> +1 on consistency across Beam modules on the logging facade
> +1 on enforcing consistency
> +1 on clearly documenting how to do logging
> 
> Mixed feelings:
> * Logging backend could be runner-specific, particularly if it needs to
> integrate into some other experience
> * java.util.logging could be a good choice for the Direct Runner
> 
> On Tue, Mar 28, 2017 at 6:50 PM, Ahmet Altay <al...@google.com.invalid>
> wrote:
> 
>> On Wed, Mar 22, 2017 at 10:38 AM, Tibor Kiss <tk...@hortonworks.com>
>> wrote:
>> 
>>> This is a great idea!
>>> 
>>> I believe Python-SDK's logging could also be enhanced (a bit
>> differently):
>>> Currently we are not instantiating the logger, just using the class what
>>> logging package provides.
>>> Shortcoming of this approach is that the user cannot set the log level
> on
>>> a per module basis as all log messages
>>> end up in the root level.
>>> 
>> 
>> +1 to this. Python SDK needs to expands its logging capabilities. Filed
> [1]
>> for this.
>> 
>> Ahmet
>> 
>> [1] https://issues.apache.org/jira/browse/BEAM-1825
>> 
>> 
>>> 
>>> On 3/22/17, 5:46 AM, "Aviem Zur" <av...@gmail.com> wrote:
>>> 
>>>    +1 to what JB said.
>>> 
>>>    Will just have to be documented well as if we provide no binding
>> there
>>> will
>>>    be no logging out of the box unless the user adds a binding.
>>> 
>>>    On Wed, Mar 22, 2017 at 6:24 AM Jean-Baptiste Onofré <
>> jb@nanthrax.net>
>>>    wrote:
>>> 
>>>> Hi Aviem,
>>>> 
>>>> Good point.
>>>> 
>>>> I think, in our dependencies set, we should just depend to
>> slf4j-api
>>> and
>>>> let the
>>>> user provides the binding he wants (slf4j-log4j12, slf4j-simple,
>>> whatever).
>>>> 
>>>> We define a binding only with test scope in our modules.
>>>> 
>>>> Regards
>>>> JB
>>>> 
>>>> On 03/22/2017 04:58 AM, Aviem Zur wrote:
>>>>> Hi all,
>>>>> 
>>>>> There have been a few reports lately (On JIRA [1] and on Slack)
>>> from
>>>> users
>>>>> regarding inconsistent loggers used across Beam's modules.
>>>>> 
>>>>> While we use SLF4J, different modules use a different logger
>>> behind it
>>>>> (JUL, log4j, etc)
>>>>> So when people add a log4j.properties file to their classpath
> for
>>>> instance,
>>>>> they expect this to affect all of their dependencies on Beam
>>> modules, but
>>>>> it doesn’t and they miss out on some logs they thought they
> would
>>> see.
>>>>> 
>>>>> I think we should strive for consistency in which logger is used
>>> behind
>>>>> SLF4J, and try to enforce this in our modules.
>>>>> I for one think it should be slf4j-log4j. However, if
> performance
>>> of
>>>>> logging is critical we might want to consider logback.
>>>>> 
>>>>> Note: SLF4J will still be the facade for logging across the
>>> project. The
>>>>> only change would be the logger SLF4J delegates to.
>>>>> 
>>>>> Once we have something like this it would also be useful to add
>>>>> documentation on logging in Beam to the website.
>>>>> 
>>>>> [1] https://issues.apache.org/jira/browse/BEAM-1757
>>>>> 
>>>> 
>>>> --
>>>> Jean-Baptiste Onofré
>>>> jbonofre@apache.org
>>>> http://blog.nanthrax.net
>>>> Talend - http://www.talend.com
>>>> 
>>> 
>>> 
>>> 
>> 


Re: [DISCUSSION] Consistent use of loggers

Posted by Aviem Zur <av...@gmail.com>.
>* java.util.logging could be a good choice for the Direct Runner
Yes, this will be great for users (Instead of having no logging when using
direct runner).

>* Logging backend could be runner-specific, particularly if it needs to
>integrate into some other experience
Good point, let's take a look at the current state of runners:
Direct runner - will use JUL as suggested.
Dataflow runner - looks like there is already no binding (There is a
binding in tests only).
Spark runner - currently uses slf4j-log4j12. does not require any specific
logger, we can change this to no binding.
Flink runner - uses slf4j-log4j12 transitively from Flink dependencies. I'm
assuming this is not a must and we can default to no binding here.
@aljoscha please confirm.
Apex runner - uses slf4j-log4j12 transitively from Apex dependencies. I'm
assuming this is not a must and we can default to no binding here. @thw
please confirm.

It might be a good idea to use a consistent binding in tests (Since we'll
use JUL for direct runner, let this be JUL).

On Wed, Mar 29, 2017 at 7:23 PM Davor Bonaci <da...@apache.org> wrote:

+1 on consistency across Beam modules on the logging facade
+1 on enforcing consistency
+1 on clearly documenting how to do logging

Mixed feelings:
* Logging backend could be runner-specific, particularly if it needs to
integrate into some other experience
* java.util.logging could be a good choice for the Direct Runner

On Tue, Mar 28, 2017 at 6:50 PM, Ahmet Altay <al...@google.com.invalid>
wrote:

> On Wed, Mar 22, 2017 at 10:38 AM, Tibor Kiss <tk...@hortonworks.com>
> wrote:
>
> > This is a great idea!
> >
> > I believe Python-SDK's logging could also be enhanced (a bit
> differently):
> > Currently we are not instantiating the logger, just using the class what
> > logging package provides.
> > Shortcoming of this approach is that the user cannot set the log level
on
> > a per module basis as all log messages
> > end up in the root level.
> >
>
> +1 to this. Python SDK needs to expands its logging capabilities. Filed
[1]
> for this.
>
> Ahmet
>
> [1] https://issues.apache.org/jira/browse/BEAM-1825
>
>
> >
> > On 3/22/17, 5:46 AM, "Aviem Zur" <av...@gmail.com> wrote:
> >
> >     +1 to what JB said.
> >
> >     Will just have to be documented well as if we provide no binding
> there
> > will
> >     be no logging out of the box unless the user adds a binding.
> >
> >     On Wed, Mar 22, 2017 at 6:24 AM Jean-Baptiste Onofré <
> jb@nanthrax.net>
> >     wrote:
> >
> >     > Hi Aviem,
> >     >
> >     > Good point.
> >     >
> >     > I think, in our dependencies set, we should just depend to
> slf4j-api
> > and
> >     > let the
> >     > user provides the binding he wants (slf4j-log4j12, slf4j-simple,
> > whatever).
> >     >
> >     > We define a binding only with test scope in our modules.
> >     >
> >     > Regards
> >     > JB
> >     >
> >     > On 03/22/2017 04:58 AM, Aviem Zur wrote:
> >     > > Hi all,
> >     > >
> >     > > There have been a few reports lately (On JIRA [1] and on Slack)
> > from
> >     > users
> >     > > regarding inconsistent loggers used across Beam's modules.
> >     > >
> >     > > While we use SLF4J, different modules use a different logger
> > behind it
> >     > > (JUL, log4j, etc)
> >     > > So when people add a log4j.properties file to their classpath
for
> >     > instance,
> >     > > they expect this to affect all of their dependencies on Beam
> > modules, but
> >     > > it doesn’t and they miss out on some logs they thought they
would
> > see.
> >     > >
> >     > > I think we should strive for consistency in which logger is used
> > behind
> >     > > SLF4J, and try to enforce this in our modules.
> >     > > I for one think it should be slf4j-log4j. However, if
performance
> > of
> >     > > logging is critical we might want to consider logback.
> >     > >
> >     > > Note: SLF4J will still be the facade for logging across the
> > project. The
> >     > > only change would be the logger SLF4J delegates to.
> >     > >
> >     > > Once we have something like this it would also be useful to add
> >     > > documentation on logging in Beam to the website.
> >     > >
> >     > > [1] https://issues.apache.org/jira/browse/BEAM-1757
> >     > >
> >     >
> >     > --
> >     > Jean-Baptiste Onofré
> >     > jbonofre@apache.org
> >     > http://blog.nanthrax.net
> >     > Talend - http://www.talend.com
> >     >
> >
> >
> >
>

Re: [DISCUSSION] Consistent use of loggers

Posted by Davor Bonaci <da...@apache.org>.
+1 on consistency across Beam modules on the logging facade
+1 on enforcing consistency
+1 on clearly documenting how to do logging

Mixed feelings:
* Logging backend could be runner-specific, particularly if it needs to
integrate into some other experience
* java.util.logging could be a good choice for the Direct Runner

On Tue, Mar 28, 2017 at 6:50 PM, Ahmet Altay <al...@google.com.invalid>
wrote:

> On Wed, Mar 22, 2017 at 10:38 AM, Tibor Kiss <tk...@hortonworks.com>
> wrote:
>
> > This is a great idea!
> >
> > I believe Python-SDK's logging could also be enhanced (a bit
> differently):
> > Currently we are not instantiating the logger, just using the class what
> > logging package provides.
> > Shortcoming of this approach is that the user cannot set the log level on
> > a per module basis as all log messages
> > end up in the root level.
> >
>
> +1 to this. Python SDK needs to expands its logging capabilities. Filed [1]
> for this.
>
> Ahmet
>
> [1] https://issues.apache.org/jira/browse/BEAM-1825
>
>
> >
> > On 3/22/17, 5:46 AM, "Aviem Zur" <av...@gmail.com> wrote:
> >
> >     +1 to what JB said.
> >
> >     Will just have to be documented well as if we provide no binding
> there
> > will
> >     be no logging out of the box unless the user adds a binding.
> >
> >     On Wed, Mar 22, 2017 at 6:24 AM Jean-Baptiste Onofré <
> jb@nanthrax.net>
> >     wrote:
> >
> >     > Hi Aviem,
> >     >
> >     > Good point.
> >     >
> >     > I think, in our dependencies set, we should just depend to
> slf4j-api
> > and
> >     > let the
> >     > user provides the binding he wants (slf4j-log4j12, slf4j-simple,
> > whatever).
> >     >
> >     > We define a binding only with test scope in our modules.
> >     >
> >     > Regards
> >     > JB
> >     >
> >     > On 03/22/2017 04:58 AM, Aviem Zur wrote:
> >     > > Hi all,
> >     > >
> >     > > There have been a few reports lately (On JIRA [1] and on Slack)
> > from
> >     > users
> >     > > regarding inconsistent loggers used across Beam's modules.
> >     > >
> >     > > While we use SLF4J, different modules use a different logger
> > behind it
> >     > > (JUL, log4j, etc)
> >     > > So when people add a log4j.properties file to their classpath for
> >     > instance,
> >     > > they expect this to affect all of their dependencies on Beam
> > modules, but
> >     > > it doesn’t and they miss out on some logs they thought they would
> > see.
> >     > >
> >     > > I think we should strive for consistency in which logger is used
> > behind
> >     > > SLF4J, and try to enforce this in our modules.
> >     > > I for one think it should be slf4j-log4j. However, if performance
> > of
> >     > > logging is critical we might want to consider logback.
> >     > >
> >     > > Note: SLF4J will still be the facade for logging across the
> > project. The
> >     > > only change would be the logger SLF4J delegates to.
> >     > >
> >     > > Once we have something like this it would also be useful to add
> >     > > documentation on logging in Beam to the website.
> >     > >
> >     > > [1] https://issues.apache.org/jira/browse/BEAM-1757
> >     > >
> >     >
> >     > --
> >     > Jean-Baptiste Onofré
> >     > jbonofre@apache.org
> >     > http://blog.nanthrax.net
> >     > Talend - http://www.talend.com
> >     >
> >
> >
> >
>

Re: [DISCUSSION] Consistent use of loggers

Posted by Ahmet Altay <al...@google.com.INVALID>.
On Wed, Mar 22, 2017 at 10:38 AM, Tibor Kiss <tk...@hortonworks.com> wrote:

> This is a great idea!
>
> I believe Python-SDK's logging could also be enhanced (a bit differently):
> Currently we are not instantiating the logger, just using the class what
> logging package provides.
> Shortcoming of this approach is that the user cannot set the log level on
> a per module basis as all log messages
> end up in the root level.
>

+1 to this. Python SDK needs to expands its logging capabilities. Filed [1]
for this.

Ahmet

[1] https://issues.apache.org/jira/browse/BEAM-1825


>
> On 3/22/17, 5:46 AM, "Aviem Zur" <av...@gmail.com> wrote:
>
>     +1 to what JB said.
>
>     Will just have to be documented well as if we provide no binding there
> will
>     be no logging out of the box unless the user adds a binding.
>
>     On Wed, Mar 22, 2017 at 6:24 AM Jean-Baptiste Onofré <jb...@nanthrax.net>
>     wrote:
>
>     > Hi Aviem,
>     >
>     > Good point.
>     >
>     > I think, in our dependencies set, we should just depend to slf4j-api
> and
>     > let the
>     > user provides the binding he wants (slf4j-log4j12, slf4j-simple,
> whatever).
>     >
>     > We define a binding only with test scope in our modules.
>     >
>     > Regards
>     > JB
>     >
>     > On 03/22/2017 04:58 AM, Aviem Zur wrote:
>     > > Hi all,
>     > >
>     > > There have been a few reports lately (On JIRA [1] and on Slack)
> from
>     > users
>     > > regarding inconsistent loggers used across Beam's modules.
>     > >
>     > > While we use SLF4J, different modules use a different logger
> behind it
>     > > (JUL, log4j, etc)
>     > > So when people add a log4j.properties file to their classpath for
>     > instance,
>     > > they expect this to affect all of their dependencies on Beam
> modules, but
>     > > it doesn’t and they miss out on some logs they thought they would
> see.
>     > >
>     > > I think we should strive for consistency in which logger is used
> behind
>     > > SLF4J, and try to enforce this in our modules.
>     > > I for one think it should be slf4j-log4j. However, if performance
> of
>     > > logging is critical we might want to consider logback.
>     > >
>     > > Note: SLF4J will still be the facade for logging across the
> project. The
>     > > only change would be the logger SLF4J delegates to.
>     > >
>     > > Once we have something like this it would also be useful to add
>     > > documentation on logging in Beam to the website.
>     > >
>     > > [1] https://issues.apache.org/jira/browse/BEAM-1757
>     > >
>     >
>     > --
>     > Jean-Baptiste Onofré
>     > jbonofre@apache.org
>     > http://blog.nanthrax.net
>     > Talend - http://www.talend.com
>     >
>
>
>

Re: [DISCUSSION] Consistent use of loggers

Posted by Tibor Kiss <tk...@hortonworks.com>.
This is a great idea!

I believe Python-SDK's logging could also be enhanced (a bit differently):
Currently we are not instantiating the logger, just using the class what logging package provides.
Shortcoming of this approach is that the user cannot set the log level on a per module basis as all log messages
end up in the root level.

On 3/22/17, 5:46 AM, "Aviem Zur" <av...@gmail.com> wrote:

    +1 to what JB said.
    
    Will just have to be documented well as if we provide no binding there will
    be no logging out of the box unless the user adds a binding.
    
    On Wed, Mar 22, 2017 at 6:24 AM Jean-Baptiste Onofré <jb...@nanthrax.net>
    wrote:
    
    > Hi Aviem,
    >
    > Good point.
    >
    > I think, in our dependencies set, we should just depend to slf4j-api and
    > let the
    > user provides the binding he wants (slf4j-log4j12, slf4j-simple, whatever).
    >
    > We define a binding only with test scope in our modules.
    >
    > Regards
    > JB
    >
    > On 03/22/2017 04:58 AM, Aviem Zur wrote:
    > > Hi all,
    > >
    > > There have been a few reports lately (On JIRA [1] and on Slack) from
    > users
    > > regarding inconsistent loggers used across Beam's modules.
    > >
    > > While we use SLF4J, different modules use a different logger behind it
    > > (JUL, log4j, etc)
    > > So when people add a log4j.properties file to their classpath for
    > instance,
    > > they expect this to affect all of their dependencies on Beam modules, but
    > > it doesn’t and they miss out on some logs they thought they would see.
    > >
    > > I think we should strive for consistency in which logger is used behind
    > > SLF4J, and try to enforce this in our modules.
    > > I for one think it should be slf4j-log4j. However, if performance of
    > > logging is critical we might want to consider logback.
    > >
    > > Note: SLF4J will still be the facade for logging across the project. The
    > > only change would be the logger SLF4J delegates to.
    > >
    > > Once we have something like this it would also be useful to add
    > > documentation on logging in Beam to the website.
    > >
    > > [1] https://issues.apache.org/jira/browse/BEAM-1757
    > >
    >
    > --
    > Jean-Baptiste Onofré
    > jbonofre@apache.org
    > http://blog.nanthrax.net
    > Talend - http://www.talend.com
    >
    


Re: [DISCUSSION] Consistent use of loggers

Posted by Aviem Zur <av...@gmail.com>.
+1 to what JB said.

Will just have to be documented well as if we provide no binding there will
be no logging out of the box unless the user adds a binding.

On Wed, Mar 22, 2017 at 6:24 AM Jean-Baptiste Onofré <jb...@nanthrax.net>
wrote:

> Hi Aviem,
>
> Good point.
>
> I think, in our dependencies set, we should just depend to slf4j-api and
> let the
> user provides the binding he wants (slf4j-log4j12, slf4j-simple, whatever).
>
> We define a binding only with test scope in our modules.
>
> Regards
> JB
>
> On 03/22/2017 04:58 AM, Aviem Zur wrote:
> > Hi all,
> >
> > There have been a few reports lately (On JIRA [1] and on Slack) from
> users
> > regarding inconsistent loggers used across Beam's modules.
> >
> > While we use SLF4J, different modules use a different logger behind it
> > (JUL, log4j, etc)
> > So when people add a log4j.properties file to their classpath for
> instance,
> > they expect this to affect all of their dependencies on Beam modules, but
> > it doesn’t and they miss out on some logs they thought they would see.
> >
> > I think we should strive for consistency in which logger is used behind
> > SLF4J, and try to enforce this in our modules.
> > I for one think it should be slf4j-log4j. However, if performance of
> > logging is critical we might want to consider logback.
> >
> > Note: SLF4J will still be the facade for logging across the project. The
> > only change would be the logger SLF4J delegates to.
> >
> > Once we have something like this it would also be useful to add
> > documentation on logging in Beam to the website.
> >
> > [1] https://issues.apache.org/jira/browse/BEAM-1757
> >
>
> --
> Jean-Baptiste Onofré
> jbonofre@apache.org
> http://blog.nanthrax.net
> Talend - http://www.talend.com
>

Re: [DISCUSSION] Consistent use of loggers

Posted by Jean-Baptiste Onofré <jb...@nanthrax.net>.
Hi Aviem,

Good point.

I think, in our dependencies set, we should just depend to slf4j-api and let the 
user provides the binding he wants (slf4j-log4j12, slf4j-simple, whatever).

We define a binding only with test scope in our modules.

Regards
JB

On 03/22/2017 04:58 AM, Aviem Zur wrote:
> Hi all,
>
> There have been a few reports lately (On JIRA [1] and on Slack) from users
> regarding inconsistent loggers used across Beam's modules.
>
> While we use SLF4J, different modules use a different logger behind it
> (JUL, log4j, etc)
> So when people add a log4j.properties file to their classpath for instance,
> they expect this to affect all of their dependencies on Beam modules, but
> it doesn\u2019t and they miss out on some logs they thought they would see.
>
> I think we should strive for consistency in which logger is used behind
> SLF4J, and try to enforce this in our modules.
> I for one think it should be slf4j-log4j. However, if performance of
> logging is critical we might want to consider logback.
>
> Note: SLF4J will still be the facade for logging across the project. The
> only change would be the logger SLF4J delegates to.
>
> Once we have something like this it would also be useful to add
> documentation on logging in Beam to the website.
>
> [1] https://issues.apache.org/jira/browse/BEAM-1757
>

-- 
Jean-Baptiste Onofr�
jbonofre@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com