You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@beam.apache.org by Pablo Estrada <pa...@google.com> on 2019/12/13 00:33:57 UTC
Root logger configuration
Hello all,
It has been pointed out to me by Chad, and also by others, that my logging
changes have caused logs to start getting lost.
It seems that by never logging on the root logger, initialization for a
root handler is skipped; and that's what causes the failures.
I will work on a fix for this. I am thinking of providing a very simple
apache_beam.utils.get_logger function that does something like this:
def get_logger(name):
logging.basicConfig()
return logging.getLogger(name)
And specific paths that need special handling of the logs should override
this config by adding their own handlers (e.g. sdk_worker, fn_api_runner,
etc).
I hope I can have a fix for this by tomorrow.
Best
-P.
Re: Root logger configuration
Posted by Pablo Estrada <pa...@google.com>.
A fix, calling basicconfig in pipeline and pipelineoptions:
https://github.com/apache/beam/pull/10396
On Tue, Dec 17, 2019 at 3:17 PM Robert Bradshaw <ro...@google.com> wrote:
> The generally expected behavior is that if you don't do anything,
> logging goes to stderr. Logging to non-root loggers breaks this.
> (Arguably it's a bug in the Python logging libraries to have this
> inconsistency, but so be it...)
>
> On the other hand, if you do set something up, that is respected.
>
> I think setting up default logging (emulating what is done by
> logging.info(), etc. which is calling basicConfig to get a stderr
> logger iff no root loggers are present) make sense. We could do this
> in Pipeline.__init__, but should also probably do it when parsing
> options. These two entry points should be sufficient.
>
> We may need to update non-Pipeline-creating unit-tests as well in some
> cases, but I think generally the testing framework sets up its
> logging.
>
> On Tue, Dec 17, 2019 at 3:09 PM Pablo Estrada <pa...@google.com> wrote:
> >
> > The Python basicConfig[1] sets up a StreamHandler[2], which by default
> publishes to stderr. This configuration is applied by default whenever
> someone logs anything on the root logger (e.g. logging.info('abc')). With
> the module-based logging changes, the basic config is never called on the
> pipeline construction side, so logging depends 100% on the user, instead of
> having a 'reasonable default'.
> >
> > The default configuration gets setup if the user ever logs on the root
> logger (logging.info, ...). If they don't then, nothing is set up.
> >
> > I'd say that it's best to be consistent with our previous behavior,
> which was to have a default configuration by virtue of logging on the root
> logger. By calling basicConfig in Pipeline.__init__, we allow the user to
> set up a configuration before creating the Pipeline object, or we set up a
> default at that point.
> >
> > Thoughts?
> >
> > [1] https://docs.python.org/3/library/logging.html#logging.basicConfig
> > [2]
> https://docs.python.org/3/library/logging.handlers.html#logging.StreamHandler
> >
> > On Tue, Dec 17, 2019 at 2:42 PM Luke Cwik <lc...@google.com> wrote:
> >>
> >> In Beam Java, the expectation has always been that pipeline authors are
> responsible for setting up logging correctly during pipeline construction
> time and that the Beam SDK is responsible for setting up logging at
> pipeline execution time.
> >>
> >> Is this something we can solve by documenting and telling users to
> configure logging at pipeline construction time or is the expected python
> user experience that logging always works out of the box?
> >>
> >> On Tue, Dec 17, 2019 at 2:15 PM Pablo Estrada <pa...@google.com>
> wrote:
> >>>
> >>> It should not affect debuggability at pipeline runtime, as the
> sdk_worker already does the appropriate setup of handlers, but it may
> affect debugging of e.g. ptransform expansion, and pipeline construction
> issues.
> >>> Best
> >>> -P.
> >>>
> >>> On Tue, Dec 17, 2019 at 1:59 PM Udi Meiri <eh...@google.com> wrote:
> >>>>
> >>>> Pablo, does the issue affect debuggability of pipelines?
> >>>>
> >>>> On Mon, Dec 16, 2019 at 6:23 PM Chad Dombrova <ch...@gmail.com>
> wrote:
> >>>>>
> >>>>>
> >>>>>
> >>>>> On Mon, Dec 16, 2019 at 5:59 PM Pablo Estrada <pa...@google.com>
> wrote:
> >>>>>>
> >>>>>> +chadrik@gmail.com is this consistent with behavior that you
> observed?
> >>>>>
> >>>>>
> >>>>> I honestly can't recall, sorry. I just remember that while I was
> testing I updated sdk version and some logging stopped. I *think* I was
> missing the state/message stream, which would be on the client side after
> pipeline construction.
> >>>>>
> >>>>> -chad
> >>>>>
>
Re: Root logger configuration
Posted by Robert Bradshaw <ro...@google.com>.
The generally expected behavior is that if you don't do anything,
logging goes to stderr. Logging to non-root loggers breaks this.
(Arguably it's a bug in the Python logging libraries to have this
inconsistency, but so be it...)
On the other hand, if you do set something up, that is respected.
I think setting up default logging (emulating what is done by
logging.info(), etc. which is calling basicConfig to get a stderr
logger iff no root loggers are present) make sense. We could do this
in Pipeline.__init__, but should also probably do it when parsing
options. These two entry points should be sufficient.
We may need to update non-Pipeline-creating unit-tests as well in some
cases, but I think generally the testing framework sets up its
logging.
On Tue, Dec 17, 2019 at 3:09 PM Pablo Estrada <pa...@google.com> wrote:
>
> The Python basicConfig[1] sets up a StreamHandler[2], which by default publishes to stderr. This configuration is applied by default whenever someone logs anything on the root logger (e.g. logging.info('abc')). With the module-based logging changes, the basic config is never called on the pipeline construction side, so logging depends 100% on the user, instead of having a 'reasonable default'.
>
> The default configuration gets setup if the user ever logs on the root logger (logging.info, ...). If they don't then, nothing is set up.
>
> I'd say that it's best to be consistent with our previous behavior, which was to have a default configuration by virtue of logging on the root logger. By calling basicConfig in Pipeline.__init__, we allow the user to set up a configuration before creating the Pipeline object, or we set up a default at that point.
>
> Thoughts?
>
> [1] https://docs.python.org/3/library/logging.html#logging.basicConfig
> [2] https://docs.python.org/3/library/logging.handlers.html#logging.StreamHandler
>
> On Tue, Dec 17, 2019 at 2:42 PM Luke Cwik <lc...@google.com> wrote:
>>
>> In Beam Java, the expectation has always been that pipeline authors are responsible for setting up logging correctly during pipeline construction time and that the Beam SDK is responsible for setting up logging at pipeline execution time.
>>
>> Is this something we can solve by documenting and telling users to configure logging at pipeline construction time or is the expected python user experience that logging always works out of the box?
>>
>> On Tue, Dec 17, 2019 at 2:15 PM Pablo Estrada <pa...@google.com> wrote:
>>>
>>> It should not affect debuggability at pipeline runtime, as the sdk_worker already does the appropriate setup of handlers, but it may affect debugging of e.g. ptransform expansion, and pipeline construction issues.
>>> Best
>>> -P.
>>>
>>> On Tue, Dec 17, 2019 at 1:59 PM Udi Meiri <eh...@google.com> wrote:
>>>>
>>>> Pablo, does the issue affect debuggability of pipelines?
>>>>
>>>> On Mon, Dec 16, 2019 at 6:23 PM Chad Dombrova <ch...@gmail.com> wrote:
>>>>>
>>>>>
>>>>>
>>>>> On Mon, Dec 16, 2019 at 5:59 PM Pablo Estrada <pa...@google.com> wrote:
>>>>>>
>>>>>> +chadrik@gmail.com is this consistent with behavior that you observed?
>>>>>
>>>>>
>>>>> I honestly can't recall, sorry. I just remember that while I was testing I updated sdk version and some logging stopped. I *think* I was missing the state/message stream, which would be on the client side after pipeline construction.
>>>>>
>>>>> -chad
>>>>>
Re: Root logger configuration
Posted by Pablo Estrada <pa...@google.com>.
The Python basicConfig[1] sets up a StreamHandler[2], which by default
publishes to stderr. This configuration is applied by default whenever
someone logs anything on the root logger (e.g. logging.info('abc')). With
the module-based logging changes, the basic config is never called on the
pipeline construction side, so logging depends 100% on the user, instead of
having a 'reasonable default'.
The default configuration gets setup if the user ever logs on the root
logger (logging.info, ...). If they don't then, nothing is set up.
I'd say that it's best to be consistent with our previous behavior, which
was to have a default configuration by virtue of logging on the root
logger. By calling basicConfig in Pipeline.__init__, we allow the user to
set up a configuration before creating the Pipeline object, or we set up a
default at that point.
Thoughts?
[1] https://docs.python.org/3/library/logging.html#logging.basicConfig
[2]
https://docs.python.org/3/library/logging.handlers.html#logging.StreamHandler
On Tue, Dec 17, 2019 at 2:42 PM Luke Cwik <lc...@google.com> wrote:
> In Beam Java, the expectation has always been that pipeline authors are
> responsible for setting up logging correctly during pipeline construction
> time and that the Beam SDK is responsible for setting up logging at
> pipeline execution time.
>
> Is this something we can solve by documenting and telling users to
> configure logging at pipeline construction time or is the expected python
> user experience that logging always works out of the box?
>
> On Tue, Dec 17, 2019 at 2:15 PM Pablo Estrada <pa...@google.com> wrote:
>
>> It should not affect debuggability at pipeline runtime, as the sdk_worker
>> already does the appropriate setup of handlers, but it may affect debugging
>> of e.g. ptransform expansion, and pipeline construction issues.
>> Best
>> -P.
>>
>> On Tue, Dec 17, 2019 at 1:59 PM Udi Meiri <eh...@google.com> wrote:
>>
>>> Pablo, does the issue affect debuggability of pipelines?
>>>
>>> On Mon, Dec 16, 2019 at 6:23 PM Chad Dombrova <ch...@gmail.com> wrote:
>>>
>>>>
>>>>
>>>> On Mon, Dec 16, 2019 at 5:59 PM Pablo Estrada <pa...@google.com>
>>>> wrote:
>>>>
>>>>> +chadrik@gmail.com <ch...@gmail.com> is this consistent with
>>>>> behavior that you observed?
>>>>>
>>>>
>>>> I honestly can't recall, sorry. I just remember that while I was
>>>> testing I updated sdk version and some logging stopped. I *think* I was
>>>> missing the state/message stream, which would be on the client side after
>>>> pipeline construction.
>>>>
>>>> -chad
>>>>
>>>>
Re: Root logger configuration
Posted by Luke Cwik <lc...@google.com>.
In Beam Java, the expectation has always been that pipeline authors are
responsible for setting up logging correctly during pipeline construction
time and that the Beam SDK is responsible for setting up logging at
pipeline execution time.
Is this something we can solve by documenting and telling users to
configure logging at pipeline construction time or is the expected python
user experience that logging always works out of the box?
On Tue, Dec 17, 2019 at 2:15 PM Pablo Estrada <pa...@google.com> wrote:
> It should not affect debuggability at pipeline runtime, as the sdk_worker
> already does the appropriate setup of handlers, but it may affect debugging
> of e.g. ptransform expansion, and pipeline construction issues.
> Best
> -P.
>
> On Tue, Dec 17, 2019 at 1:59 PM Udi Meiri <eh...@google.com> wrote:
>
>> Pablo, does the issue affect debuggability of pipelines?
>>
>> On Mon, Dec 16, 2019 at 6:23 PM Chad Dombrova <ch...@gmail.com> wrote:
>>
>>>
>>>
>>> On Mon, Dec 16, 2019 at 5:59 PM Pablo Estrada <pa...@google.com>
>>> wrote:
>>>
>>>> +chadrik@gmail.com <ch...@gmail.com> is this consistent with
>>>> behavior that you observed?
>>>>
>>>
>>> I honestly can't recall, sorry. I just remember that while I was
>>> testing I updated sdk version and some logging stopped. I *think* I was
>>> missing the state/message stream, which would be on the client side after
>>> pipeline construction.
>>>
>>> -chad
>>>
>>>
Re: Root logger configuration
Posted by Pablo Estrada <pa...@google.com>.
It should not affect debuggability at pipeline runtime, as the sdk_worker
already does the appropriate setup of handlers, but it may affect debugging
of e.g. ptransform expansion, and pipeline construction issues.
Best
-P.
On Tue, Dec 17, 2019 at 1:59 PM Udi Meiri <eh...@google.com> wrote:
> Pablo, does the issue affect debuggability of pipelines?
>
> On Mon, Dec 16, 2019 at 6:23 PM Chad Dombrova <ch...@gmail.com> wrote:
>
>>
>>
>> On Mon, Dec 16, 2019 at 5:59 PM Pablo Estrada <pa...@google.com> wrote:
>>
>>> +chadrik@gmail.com <ch...@gmail.com> is this consistent with behavior
>>> that you observed?
>>>
>>
>> I honestly can't recall, sorry. I just remember that while I was testing
>> I updated sdk version and some logging stopped. I *think* I was missing
>> the state/message stream, which would be on the client side after pipeline
>> construction.
>>
>> -chad
>>
>>
Re: Root logger configuration
Posted by Udi Meiri <eh...@google.com>.
Pablo, does the issue affect debuggability of pipelines?
On Mon, Dec 16, 2019 at 6:23 PM Chad Dombrova <ch...@gmail.com> wrote:
>
>
> On Mon, Dec 16, 2019 at 5:59 PM Pablo Estrada <pa...@google.com> wrote:
>
>> +chadrik@gmail.com <ch...@gmail.com> is this consistent with behavior
>> that you observed?
>>
>
> I honestly can't recall, sorry. I just remember that while I was testing
> I updated sdk version and some logging stopped. I *think* I was missing
> the state/message stream, which would be on the client side after pipeline
> construction.
>
> -chad
>
>
Re: Root logger configuration
Posted by Chad Dombrova <ch...@gmail.com>.
On Mon, Dec 16, 2019 at 5:59 PM Pablo Estrada <pa...@google.com> wrote:
> +chadrik@gmail.com <ch...@gmail.com> is this consistent with behavior
> that you observed?
>
I honestly can't recall, sorry. I just remember that while I was testing I
updated sdk version and some logging stopped. I *think* I was missing the
state/message stream, which would be on the client side after pipeline
construction.
-chad
Re: Root logger configuration
Posted by Pablo Estrada <pa...@google.com>.
+chadrik@gmail.com <ch...@gmail.com> is this consistent with behavior
that you observed?
+Udi Meiri <eh...@google.com> sorry about the short notice on this. I know
you are working on RC1. But do you think this should be a blocker for
2.18.0? I have https://issues.apache.org/jira/browse/BEAM-8976 to track
that.
On Mon, Dec 16, 2019 at 5:49 PM Pablo Estrada <pa...@google.com> wrote:
> Hi all,
> So there are two places where we log:
> 1. At pipeline construction.
> 2. At worker startup, to execute user code in workers.
>
> The workers have always set up their logging handlers properly, but
> pipeline construction setup is up to the user.
>
> If the user doesn't set it up, we could have a default configuration - and
> I propose doing it when we create the Pipeline object. See my proposed fix:
> https://github.com/apache/beam/pull/10396 - @Robert Bradshaw
> <ro...@google.com> wdyt?
>
> If this keeps holding out, then I have a quickfix to make all l9ogging
> happen in the root logger for now:
> https://github.com/apache/beam/pull/10394
>
> Best
> -P.
>
> On Fri, Dec 13, 2019 at 7:45 PM Ahmet Altay <al...@google.com> wrote:
>
>>
>>
>> On Fri, Dec 13, 2019 at 1:08 PM Robert Bradshaw <ro...@google.com>
>> wrote:
>>
>>> The default behavior of Python logging is
>>>
>>> (1) Logging handlers may get added (usually in main or very close to it).
>>> (2) Logging goes to those handlers, or a default handler if none were
>>> added.
>>>
>>> With this proposal, we would have
>>>
>>> (0) A default handler gets added.
>>> (1) Logging handlers may get added (usually in main or very close to
>>> it) in addition to the default.
>>> (2) Logging goes to these handlers and the default.
>>>
>>> On Fri, Dec 13, 2019 at 12:46 PM Pablo Estrada <pa...@google.com>
>>> wrote:
>>> >
>>> > I looked at the documentation for basicConfig:
>>> https://docs.python.org/3/library/logging.html#logging.basicConfig
>>> >
>>> > Specifically, the following line:
>>> >
>>> > > This function does nothing if the root logger already has handlers
>>> configured, unless the keyword argument force is set to True.
>>> >
>>> > That would mean that anyone can override the handling later on - which
>>> the workers do?
>>> > Best
>>> > -P.
>>> >
>>> > On Fri, Dec 13, 2019 at 10:55 AM Robert Bradshaw <ro...@google.com>
>>> wrote:
>>> >>
>>> >> Thanks for looking into this.
>>> >>
>>> >> I'm not sure unconditionally calling logging.basicConfig() on module
>>> >> import is the correct solution--this prevents modules that wish to set
>>> >> up handlers in place of the default handler from being able to do so.
>>> >> (This is why logging.basicConfig is lazily done at the first logging
>>> >> statement for the root logger, rather than earlier.)
>>> >>
>>> >> On Thu, Dec 12, 2019 at 4:34 PM Pablo Estrada <pa...@google.com>
>>> wrote:
>>> >> >
>>> >> > Hello all,
>>> >> > It has been pointed out to me by Chad, and also by others, that my
>>> logging changes have caused logs to start getting lost.
>>>
>>
>> Does this include losing user logs? Is there a JIRA, and should we fix it
>> in Beam 2.18 branch?
>>
>>
>>> >> >
>>> >> > It seems that by never logging on the root logger, initialization
>>> for a root handler is skipped; and that's what causes the failures.
>>> >> >
>>> >> > I will work on a fix for this. I am thinking of providing a very
>>> simple apache_beam.utils.get_logger function that does something like this:
>>> >> >
>>> >> > def get_logger(name):
>>> >> > logging.basicConfig()
>>> >> > return logging.getLogger(name)
>>> >> >
>>> >> > And specific paths that need special handling of the logs should
>>> override this config by adding their own handlers (e.g. sdk_worker,
>>> fn_api_runner, etc).
>>> >> >
>>> >> > I hope I can have a fix for this by tomorrow.
>>> >> > Best
>>> >> > -P.
>>>
>>
Re: Root logger configuration
Posted by Pablo Estrada <pa...@google.com>.
Hi all,
So there are two places where we log:
1. At pipeline construction.
2. At worker startup, to execute user code in workers.
The workers have always set up their logging handlers properly, but
pipeline construction setup is up to the user.
If the user doesn't set it up, we could have a default configuration - and
I propose doing it when we create the Pipeline object. See my proposed fix:
https://github.com/apache/beam/pull/10396 - @Robert Bradshaw
<ro...@google.com> wdyt?
If this keeps holding out, then I have a quickfix to make all l9ogging
happen in the root logger for now: https://github.com/apache/beam/pull/10394
Best
-P.
On Fri, Dec 13, 2019 at 7:45 PM Ahmet Altay <al...@google.com> wrote:
>
>
> On Fri, Dec 13, 2019 at 1:08 PM Robert Bradshaw <ro...@google.com>
> wrote:
>
>> The default behavior of Python logging is
>>
>> (1) Logging handlers may get added (usually in main or very close to it).
>> (2) Logging goes to those handlers, or a default handler if none were
>> added.
>>
>> With this proposal, we would have
>>
>> (0) A default handler gets added.
>> (1) Logging handlers may get added (usually in main or very close to
>> it) in addition to the default.
>> (2) Logging goes to these handlers and the default.
>>
>> On Fri, Dec 13, 2019 at 12:46 PM Pablo Estrada <pa...@google.com>
>> wrote:
>> >
>> > I looked at the documentation for basicConfig:
>> https://docs.python.org/3/library/logging.html#logging.basicConfig
>> >
>> > Specifically, the following line:
>> >
>> > > This function does nothing if the root logger already has handlers
>> configured, unless the keyword argument force is set to True.
>> >
>> > That would mean that anyone can override the handling later on - which
>> the workers do?
>> > Best
>> > -P.
>> >
>> > On Fri, Dec 13, 2019 at 10:55 AM Robert Bradshaw <ro...@google.com>
>> wrote:
>> >>
>> >> Thanks for looking into this.
>> >>
>> >> I'm not sure unconditionally calling logging.basicConfig() on module
>> >> import is the correct solution--this prevents modules that wish to set
>> >> up handlers in place of the default handler from being able to do so.
>> >> (This is why logging.basicConfig is lazily done at the first logging
>> >> statement for the root logger, rather than earlier.)
>> >>
>> >> On Thu, Dec 12, 2019 at 4:34 PM Pablo Estrada <pa...@google.com>
>> wrote:
>> >> >
>> >> > Hello all,
>> >> > It has been pointed out to me by Chad, and also by others, that my
>> logging changes have caused logs to start getting lost.
>>
>
> Does this include losing user logs? Is there a JIRA, and should we fix it
> in Beam 2.18 branch?
>
>
>> >> >
>> >> > It seems that by never logging on the root logger, initialization
>> for a root handler is skipped; and that's what causes the failures.
>> >> >
>> >> > I will work on a fix for this. I am thinking of providing a very
>> simple apache_beam.utils.get_logger function that does something like this:
>> >> >
>> >> > def get_logger(name):
>> >> > logging.basicConfig()
>> >> > return logging.getLogger(name)
>> >> >
>> >> > And specific paths that need special handling of the logs should
>> override this config by adding their own handlers (e.g. sdk_worker,
>> fn_api_runner, etc).
>> >> >
>> >> > I hope I can have a fix for this by tomorrow.
>> >> > Best
>> >> > -P.
>>
>
Re: Root logger configuration
Posted by Ahmet Altay <al...@google.com>.
On Fri, Dec 13, 2019 at 1:08 PM Robert Bradshaw <ro...@google.com> wrote:
> The default behavior of Python logging is
>
> (1) Logging handlers may get added (usually in main or very close to it).
> (2) Logging goes to those handlers, or a default handler if none were
> added.
>
> With this proposal, we would have
>
> (0) A default handler gets added.
> (1) Logging handlers may get added (usually in main or very close to
> it) in addition to the default.
> (2) Logging goes to these handlers and the default.
>
> On Fri, Dec 13, 2019 at 12:46 PM Pablo Estrada <pa...@google.com> wrote:
> >
> > I looked at the documentation for basicConfig:
> https://docs.python.org/3/library/logging.html#logging.basicConfig
> >
> > Specifically, the following line:
> >
> > > This function does nothing if the root logger already has handlers
> configured, unless the keyword argument force is set to True.
> >
> > That would mean that anyone can override the handling later on - which
> the workers do?
> > Best
> > -P.
> >
> > On Fri, Dec 13, 2019 at 10:55 AM Robert Bradshaw <ro...@google.com>
> wrote:
> >>
> >> Thanks for looking into this.
> >>
> >> I'm not sure unconditionally calling logging.basicConfig() on module
> >> import is the correct solution--this prevents modules that wish to set
> >> up handlers in place of the default handler from being able to do so.
> >> (This is why logging.basicConfig is lazily done at the first logging
> >> statement for the root logger, rather than earlier.)
> >>
> >> On Thu, Dec 12, 2019 at 4:34 PM Pablo Estrada <pa...@google.com>
> wrote:
> >> >
> >> > Hello all,
> >> > It has been pointed out to me by Chad, and also by others, that my
> logging changes have caused logs to start getting lost.
>
Does this include losing user logs? Is there a JIRA, and should we fix it
in Beam 2.18 branch?
> >> >
> >> > It seems that by never logging on the root logger, initialization for
> a root handler is skipped; and that's what causes the failures.
> >> >
> >> > I will work on a fix for this. I am thinking of providing a very
> simple apache_beam.utils.get_logger function that does something like this:
> >> >
> >> > def get_logger(name):
> >> > logging.basicConfig()
> >> > return logging.getLogger(name)
> >> >
> >> > And specific paths that need special handling of the logs should
> override this config by adding their own handlers (e.g. sdk_worker,
> fn_api_runner, etc).
> >> >
> >> > I hope I can have a fix for this by tomorrow.
> >> > Best
> >> > -P.
>
Re: Root logger configuration
Posted by Robert Bradshaw <ro...@google.com>.
The default behavior of Python logging is
(1) Logging handlers may get added (usually in main or very close to it).
(2) Logging goes to those handlers, or a default handler if none were added.
With this proposal, we would have
(0) A default handler gets added.
(1) Logging handlers may get added (usually in main or very close to
it) in addition to the default.
(2) Logging goes to these handlers and the default.
On Fri, Dec 13, 2019 at 12:46 PM Pablo Estrada <pa...@google.com> wrote:
>
> I looked at the documentation for basicConfig: https://docs.python.org/3/library/logging.html#logging.basicConfig
>
> Specifically, the following line:
>
> > This function does nothing if the root logger already has handlers configured, unless the keyword argument force is set to True.
>
> That would mean that anyone can override the handling later on - which the workers do?
> Best
> -P.
>
> On Fri, Dec 13, 2019 at 10:55 AM Robert Bradshaw <ro...@google.com> wrote:
>>
>> Thanks for looking into this.
>>
>> I'm not sure unconditionally calling logging.basicConfig() on module
>> import is the correct solution--this prevents modules that wish to set
>> up handlers in place of the default handler from being able to do so.
>> (This is why logging.basicConfig is lazily done at the first logging
>> statement for the root logger, rather than earlier.)
>>
>> On Thu, Dec 12, 2019 at 4:34 PM Pablo Estrada <pa...@google.com> wrote:
>> >
>> > Hello all,
>> > It has been pointed out to me by Chad, and also by others, that my logging changes have caused logs to start getting lost.
>> >
>> > It seems that by never logging on the root logger, initialization for a root handler is skipped; and that's what causes the failures.
>> >
>> > I will work on a fix for this. I am thinking of providing a very simple apache_beam.utils.get_logger function that does something like this:
>> >
>> > def get_logger(name):
>> > logging.basicConfig()
>> > return logging.getLogger(name)
>> >
>> > And specific paths that need special handling of the logs should override this config by adding their own handlers (e.g. sdk_worker, fn_api_runner, etc).
>> >
>> > I hope I can have a fix for this by tomorrow.
>> > Best
>> > -P.
Re: Root logger configuration
Posted by Pablo Estrada <pa...@google.com>.
I looked at the documentation for basicConfig:
https://docs.python.org/3/library/logging.html#logging.basicConfig
Specifically, the following line:
> This function does nothing if the root logger already has handlers
configured, unless the keyword argument force is set to True.
That would mean that anyone can override the handling later on - which the
workers do?
Best
-P.
On Fri, Dec 13, 2019 at 10:55 AM Robert Bradshaw <ro...@google.com>
wrote:
> Thanks for looking into this.
>
> I'm not sure unconditionally calling logging.basicConfig() on module
> import is the correct solution--this prevents modules that wish to set
> up handlers in place of the default handler from being able to do so.
> (This is why logging.basicConfig is lazily done at the first logging
> statement for the root logger, rather than earlier.)
>
> On Thu, Dec 12, 2019 at 4:34 PM Pablo Estrada <pa...@google.com> wrote:
> >
> > Hello all,
> > It has been pointed out to me by Chad, and also by others, that my
> logging changes have caused logs to start getting lost.
> >
> > It seems that by never logging on the root logger, initialization for a
> root handler is skipped; and that's what causes the failures.
> >
> > I will work on a fix for this. I am thinking of providing a very simple
> apache_beam.utils.get_logger function that does something like this:
> >
> > def get_logger(name):
> > logging.basicConfig()
> > return logging.getLogger(name)
> >
> > And specific paths that need special handling of the logs should
> override this config by adding their own handlers (e.g. sdk_worker,
> fn_api_runner, etc).
> >
> > I hope I can have a fix for this by tomorrow.
> > Best
> > -P.
>
Re: Root logger configuration
Posted by Robert Bradshaw <ro...@google.com>.
Thanks for looking into this.
I'm not sure unconditionally calling logging.basicConfig() on module
import is the correct solution--this prevents modules that wish to set
up handlers in place of the default handler from being able to do so.
(This is why logging.basicConfig is lazily done at the first logging
statement for the root logger, rather than earlier.)
On Thu, Dec 12, 2019 at 4:34 PM Pablo Estrada <pa...@google.com> wrote:
>
> Hello all,
> It has been pointed out to me by Chad, and also by others, that my logging changes have caused logs to start getting lost.
>
> It seems that by never logging on the root logger, initialization for a root handler is skipped; and that's what causes the failures.
>
> I will work on a fix for this. I am thinking of providing a very simple apache_beam.utils.get_logger function that does something like this:
>
> def get_logger(name):
> logging.basicConfig()
> return logging.getLogger(name)
>
> And specific paths that need special handling of the logs should override this config by adding their own handlers (e.g. sdk_worker, fn_api_runner, etc).
>
> I hope I can have a fix for this by tomorrow.
> Best
> -P.