Why does Spark need to set log levels

[Daan Debie <de...@gmail.com>]

Hi all!

I would love to use Spark with a somewhat more modern logging framework
than Log4j 1.2. I have Logback in mind, mostly because it integrates well
with central logging solutions such as the ELK stack. I've read up a bit on
getting Spark 2.0 (that's what I'm using currently) to work with anything
else than Log4j 1.2, and it seems nigh impossible.

If I understood the problem correctly from the various JIRA issues I read,
it seems Spark needs to be able to set the log-level programmatically,
which slf4j doesn't support, and as such, Spark integrates with Log4j 1.2
on a deep level.

My question: why would Spark want to set log levels programmatically? Why
not leave it to the user of Spark to provide a logging configuration that
suits his/her needs? That way, the offending code that integrates with
Log4j directly, could be removed, and Spark can start relying only on the
slf4j API, as any good library should.

I'm curious about the motivations of the Spark dev team on this!

Daan

Re: Why does Spark need to set log levels

[Steve Loughran <st...@hortonworks.com>]

> On 9 Oct 2017, at 16:49, Daan Debie <de...@gmail.com> wrote:
> 
> Hi all!
> 
> I would love to use Spark with a somewhat more modern logging framework than Log4j 1.2. I have Logback in mind, mostly because it integrates well with central logging solutions such as the ELK stack. I've read up a bit on getting Spark 2.0 (that's what I'm using currently) to work with anything else than Log4j 1.2, and it seems nigh impossible.
> 
> If I understood the problem correctly from the various JIRA issues I read, it seems Spark needs to be able to set the log-level programmatically, which slf4j doesn't support, and as such, Spark integrates with Log4j 1.2 on a deep level.
> 
> My question: why would Spark want to set log levels programmatically? Why not leave it to the user of Spark to provide a logging configuration that suits his/her needs? That way, the offending code that integrates with Log4j directly, could be removed, and Spark can start relying only on the slf4j API, as any good library should.
> 
> I'm curious about the motivations of the Spark dev team on this!
> 
> Daan

AFAIK log level setting is primarily done in test code & not production: spark has gone to a lot of effort to be logger independent; if you look at org.apache.spark.internal.Logging it's explicitly probing for Log4J as the logger back end and only doing some config stuff (falling back to a base log4j.properties) file there isn't a default one.

I'd expect the dependencies to follow, with commons-logging being the general bridge API for anything yet to fully embrace SLF4J. Though some things will inevitably use java.util.logging & so you'll get stuff appearing on stderr no matter how hard you try, usually something important like Kerberos errors....


---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org