You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by "Mendelson, Assaf" <As...@rsa.com> on 2017/02/05 14:53:56 UTC
using an alternative slf4j implementation
Hi,
Spark seems to explicitly use log4j.
This means that if I use an alternative backend for my application (e.g. ch.qos.logback) I have a conflict.
Sure I can exclude logback but that means my application cannot use our internal tools.
Is there a way to use logback as a backend logging while using spark?
Assaf.
Re: using an alternative slf4j implementation
Posted by Steve Loughran <st...@hortonworks.com>.
> On 6 Feb 2017, at 11:06, Mendelson, Assaf <As...@rsa.com> wrote:
>
> Found some questions (without answers) and I found some jira (https://issues.apache.org/jira/browse/SPARK-4147 and https://issues.apache.org/jira/browse/SPARK-14703), however they do not solve the issue.
> Nominally, a library should not explicitly set a binding, however spark, does so (I imagine this is so spark-submit can package everything and have a logger). If a dependency does this, the nominal solution would be to exclude the binding (and maybe add a relevant bridge), however, since spark adds the relevant jars in spark-submit this I don't see how to do this.
>
> Is there any way of forcing logback as the binding?
>
What happens if you just take the log4j stuff out of SPARK_HOME/lib and drop in the binding you want?
> -----Original Message-----
> From: Jacek Laskowski [mailto:jacek@japila.pl]
> Sent: Monday, February 06, 2017 10:46 AM
> To: Mendelson, Assaf
> Cc: user
> Subject: Re: using an alternative slf4j implementation
>
> Hi,
>
> Sounds like a quite involved development for me. I can't help here.
> I'd suggest going through the dev and user mailing lists for the past year and JIRA issues regarding the issue as I vaguely remember some discussions about logging in Spark (that would merit to do the migration to logback eventually).
>
> Pozdrawiam,
> Jacek Laskowski
> ----
> https://medium.com/@jaceklaskowski/
> Mastering Apache Spark 2.0 https://bit.ly/mastering-apache-spark
> Follow me at https://twitter.com/jaceklaskowski
>
>
> On Mon, Feb 6, 2017 at 9:06 AM, Mendelson, Assaf <As...@rsa.com> wrote:
>> Shading doesn’t help (we already shaded everything).
>>
>> According to https://www.slf4j.org/codes.html#multiple_bindings only
>> one binding can be used. The problem is that once we link to spark
>> jars then we automatically inherit spark’s binding (for log4j).
>>
>> I would like to find a way to either send spark’s logs to log4j and my
>> logs to logback or send everything to logback.
>>
>> Assaf.
>>
>>
>>
>> From: Jacek Laskowski [mailto:jacek@japila.pl]
>> Sent: Monday, February 06, 2017 12:47 AM
>> To: Mendelson, Assaf
>> Cc: user
>> Subject: Re: using an alternative slf4j implementation
>>
>>
>>
>> Hi,
>>
>>
>>
>> Shading conflicting dependencies?
>>
>>
>>
>> Jacek
>>
>>
>>
>> On 5 Feb 2017 3:56 p.m., "Mendelson, Assaf" <As...@rsa.com> wrote:
>>
>> Hi,
>>
>> Spark seems to explicitly use log4j.
>>
>> This means that if I use an alternative backend for my application (e.g.
>> ch.qos.logback) I have a conflict.
>>
>> Sure I can exclude logback but that means my application cannot use
>> our internal tools.
>>
>>
>>
>> Is there a way to use logback as a backend logging while using spark?
>>
>> Assaf.
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: user-unsubscribe@spark.apache.org
>
---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org
RE: using an alternative slf4j implementation
Posted by "Mendelson, Assaf" <As...@rsa.com>.
Found some questions (without answers) and I found some jira (https://issues.apache.org/jira/browse/SPARK-4147 and https://issues.apache.org/jira/browse/SPARK-14703), however they do not solve the issue.
Nominally, a library should not explicitly set a binding, however spark, does so (I imagine this is so spark-submit can package everything and have a logger). If a dependency does this, the nominal solution would be to exclude the binding (and maybe add a relevant bridge), however, since spark adds the relevant jars in spark-submit this I don't see how to do this.
Is there any way of forcing logback as the binding?
-----Original Message-----
From: Jacek Laskowski [mailto:jacek@japila.pl]
Sent: Monday, February 06, 2017 10:46 AM
To: Mendelson, Assaf
Cc: user
Subject: Re: using an alternative slf4j implementation
Hi,
Sounds like a quite involved development for me. I can't help here.
I'd suggest going through the dev and user mailing lists for the past year and JIRA issues regarding the issue as I vaguely remember some discussions about logging in Spark (that would merit to do the migration to logback eventually).
Pozdrawiam,
Jacek Laskowski
----
https://medium.com/@jaceklaskowski/
Mastering Apache Spark 2.0 https://bit.ly/mastering-apache-spark
Follow me at https://twitter.com/jaceklaskowski
On Mon, Feb 6, 2017 at 9:06 AM, Mendelson, Assaf <As...@rsa.com> wrote:
> Shading doesn’t help (we already shaded everything).
>
> According to https://www.slf4j.org/codes.html#multiple_bindings only
> one binding can be used. The problem is that once we link to spark
> jars then we automatically inherit spark’s binding (for log4j).
>
> I would like to find a way to either send spark’s logs to log4j and my
> logs to logback or send everything to logback.
>
> Assaf.
>
>
>
> From: Jacek Laskowski [mailto:jacek@japila.pl]
> Sent: Monday, February 06, 2017 12:47 AM
> To: Mendelson, Assaf
> Cc: user
> Subject: Re: using an alternative slf4j implementation
>
>
>
> Hi,
>
>
>
> Shading conflicting dependencies?
>
>
>
> Jacek
>
>
>
> On 5 Feb 2017 3:56 p.m., "Mendelson, Assaf" <As...@rsa.com> wrote:
>
> Hi,
>
> Spark seems to explicitly use log4j.
>
> This means that if I use an alternative backend for my application (e.g.
> ch.qos.logback) I have a conflict.
>
> Sure I can exclude logback but that means my application cannot use
> our internal tools.
>
>
>
> Is there a way to use logback as a backend logging while using spark?
>
> Assaf.
---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org
Re: using an alternative slf4j implementation
Posted by Jacek Laskowski <ja...@japila.pl>.
Hi,
Sounds like a quite involved development for me. I can't help here.
I'd suggest going through the dev and user mailing lists for the past
year and JIRA issues regarding the issue as I vaguely remember some
discussions about logging in Spark (that would merit to do the
migration to logback eventually).
Pozdrawiam,
Jacek Laskowski
----
https://medium.com/@jaceklaskowski/
Mastering Apache Spark 2.0 https://bit.ly/mastering-apache-spark
Follow me at https://twitter.com/jaceklaskowski
On Mon, Feb 6, 2017 at 9:06 AM, Mendelson, Assaf
<As...@rsa.com> wrote:
> Shading doesn’t help (we already shaded everything).
>
> According to https://www.slf4j.org/codes.html#multiple_bindings only one
> binding can be used. The problem is that once we link to spark jars then we
> automatically inherit spark’s binding (for log4j).
>
> I would like to find a way to either send spark’s logs to log4j and my logs
> to logback or send everything to logback.
>
> Assaf.
>
>
>
> From: Jacek Laskowski [mailto:jacek@japila.pl]
> Sent: Monday, February 06, 2017 12:47 AM
> To: Mendelson, Assaf
> Cc: user
> Subject: Re: using an alternative slf4j implementation
>
>
>
> Hi,
>
>
>
> Shading conflicting dependencies?
>
>
>
> Jacek
>
>
>
> On 5 Feb 2017 3:56 p.m., "Mendelson, Assaf" <As...@rsa.com> wrote:
>
> Hi,
>
> Spark seems to explicitly use log4j.
>
> This means that if I use an alternative backend for my application (e.g.
> ch.qos.logback) I have a conflict.
>
> Sure I can exclude logback but that means my application cannot use our
> internal tools.
>
>
>
> Is there a way to use logback as a backend logging while using spark?
>
> Assaf.
---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org
RE: using an alternative slf4j implementation
Posted by "Mendelson, Assaf" <As...@rsa.com>.
Shading doesn’t help (we already shaded everything).
According to https://www.slf4j.org/codes.html#multiple_bindings only one binding can be used. The problem is that once we link to spark jars then we automatically inherit spark’s binding (for log4j).
I would like to find a way to either send spark’s logs to log4j and my logs to logback or send everything to logback.
Assaf.
From: Jacek Laskowski [mailto:jacek@japila.pl]
Sent: Monday, February 06, 2017 12:47 AM
To: Mendelson, Assaf
Cc: user
Subject: Re: using an alternative slf4j implementation
Hi,
Shading conflicting dependencies?
Jacek
On 5 Feb 2017 3:56 p.m., "Mendelson, Assaf" <As...@rsa.com>> wrote:
Hi,
Spark seems to explicitly use log4j.
This means that if I use an alternative backend for my application (e.g. ch.qos.logback) I have a conflict.
Sure I can exclude logback but that means my application cannot use our internal tools.
Is there a way to use logback as a backend logging while using spark?
Assaf.
Re: using an alternative slf4j implementation
Posted by Jacek Laskowski <ja...@japila.pl>.
Hi,
Shading conflicting dependencies?
Jacek
On 5 Feb 2017 3:56 p.m., "Mendelson, Assaf" <As...@rsa.com> wrote:
> Hi,
>
> Spark seems to explicitly use log4j.
>
> This means that if I use an alternative backend for my application (e.g.
> ch.qos.logback) I have a conflict.
>
> Sure I can exclude logback but that means my application cannot use our
> internal tools.
>
>
>
> Is there a way to use logback as a backend logging while using spark?
>
> Assaf.
>