Custom log appenders with Dataflow runner

[peay <pe...@protonmail.com>]

Hello,

I've been trying to use a custom appender with slf4j and either log4j, logback or `java.util.logging` as a backend (I am not picky), to report log entries above a certain level to a third party service.

I can set a custom appender in those backends, and that works fine when submitting the jobs. However, no matter what I do, at runtime in the Dataflow pipeline workers, all my slf4j loggers get replaced by `java.util.logging.Logger` loggers, with a single handler set to `com.google.cloud.dataflow.worker.runners.worker.logging.DataflowWorkerLoggingHandler`.

I have not found a good workaround. I can manually add my `java.util.logging` handler for the loggers I create, but this is a bit hacky and I don't get all the loggers related to the pipeline worker itself. For instance, if an exception and `UserCodeException` are thrown, the error is logged by the pipeline worker and my custom appender/handler won't apply.

This is more about some undocumented internals of the Dataflow pipeline worker, so this may be slightly out of scope for this mailer, but Google Support couldn't help and I am curious to see if anyone has some experience with that.

Thanks!

Re: Custom log appenders with Dataflow runner

[Devon Meunier <de...@shopify.com>]

peay,

Let me know if you get this working, I've run into the same issue.

On Wed, May 17, 2017 at 5:09 PM Lukasz Cwik <lc...@google.com> wrote:

> Have you tried installing a logger onto the root JUL logger once the
> pipeline starts executing in the worker (so inside one of your DoFn(s)
> setup methods)?
>
> ROOT_LOGGER_NAME = ""
>
> LogManager.getLogManager().getLogger(ROOT_LOGGER_NAME).addHandler(myCustomHandler);
>
> Also, the logging integration seems to be heavily tied to Google Cloud
> Logging integration so even if you get this working, its effectively
> unsupported and likely to break in the future. Your best bet is once Apache
> Beam uses docker containers and Google Cloud Dataflow supports user
> containers. Then you'll have a lot of control over the environment.
>
> On Wed, May 17, 2017 at 1:35 PM, peay <pe...@protonmail.com> wrote:
>
>> Hello,
>>
>> I've been trying to use a custom appender with slf4j and either log4j,
>> logback or `java.util.logging` as a backend (I am not picky), to report log
>> entries above a certain level to a third party service.
>>
>> I can set a custom appender in those backends, and that works fine when
>> submitting the jobs. However, no matter what I do, at runtime in the
>> Dataflow pipeline workers, all my slf4j loggers get replaced by
>> `java.util.logging.Logger` loggers, with a single handler set to
>> `com.google.cloud.dataflow.worker.runners.worker.logging.DataflowWorkerLoggingHandler`.
>>
>> I have not found a good workaround. I can manually add my
>> `java.util.logging` handler for the loggers I create, but this is a bit
>> hacky and I don't get all the loggers related to the pipeline worker
>> itself. For instance, if an exception and `UserCodeException` are thrown,
>> the error is logged by the pipeline worker and my custom appender/handler
>> won't apply.
>>
>> This is more about some undocumented internals of the Dataflow pipeline
>> worker, so this may be slightly out of scope for this mailer, but Google
>> Support couldn't help and I am curious to see if anyone has some experience
>> with that.
>>
>> Thanks!
>>
>>
>

Re: Custom log appenders with Dataflow runner

[Lukasz Cwik <lc...@google.com>]

Have you tried installing a logger onto the root JUL logger once the
pipeline starts executing in the worker (so inside one of your DoFn(s)
setup methods)?

ROOT_LOGGER_NAME = ""
LogManager.getLogManager().getLogger(ROOT_LOGGER_NAME).addHandler(myCustomHandler);

Also, the logging integration seems to be heavily tied to Google Cloud
Logging integration so even if you get this working, its effectively
unsupported and likely to break in the future. Your best bet is once Apache
Beam uses docker containers and Google Cloud Dataflow supports user
containers. Then you'll have a lot of control over the environment.

On Wed, May 17, 2017 at 1:35 PM, peay <pe...@protonmail.com> wrote:

> Hello,
>
> I've been trying to use a custom appender with slf4j and either log4j,
> logback or `java.util.logging` as a backend (I am not picky), to report log
> entries above a certain level to a third party service.
>
> I can set a custom appender in those backends, and that works fine when
> submitting the jobs. However, no matter what I do, at runtime in the
> Dataflow pipeline workers, all my slf4j loggers get replaced by
> `java.util.logging.Logger` loggers, with a single handler set to
> `com.google.cloud.dataflow.worker.runners.worker.logging.
> DataflowWorkerLoggingHandler`.
>
> I have not found a good workaround. I can manually add my
> `java.util.logging` handler for the loggers I create, but this is a bit
> hacky and I don't get all the loggers related to the pipeline worker
> itself. For instance, if an exception and `UserCodeException` are thrown,
> the error is logged by the pipeline worker and my custom appender/handler
> won't apply.
>
> This is more about some undocumented internals of the Dataflow pipeline
> worker, so this may be slightly out of scope for this mailer, but Google
> Support couldn't help and I am curious to see if anyone has some experience
> with that.
>
> Thanks!
>
>