You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Prithish <pr...@gmail.com> on 2017/02/26 16:31:55 UTC
Custom log4j.properties on AWS EMR
Hoping someone can answer this.
I am unable to override and use a Custom log4j.properties on Amazon EMR. I
am running Spark on EMR (Yarn) and have tried all the below combinations in
the Spark-Submit to try and use the custom log4j.
In Client mode
--driver-java-options
"-Dlog4j.configuration=hdfs://host:port/user/hadoop/log4j.properties"
In Cluster mode
--conf
"spark.driver.extraJavaOptions=-Dlog4j.configuration=hdfs://host:port/user/hadoop/log4j.properties"
I have also tried picking from local filesystem using file://// instead of
hdfs. None of this seem to work. However, I can get this working when
running on my local Yarn setup.
Any ideas?
I have also posted on Stackoverflow (link below)
http://stackoverflow.com/questions/42452622/custom-log4j-properties-on-aws-emr
Re: Custom log4j.properties on AWS EMR
Posted by Prithish <pr...@gmail.com>.
Thanks for your response Jonathan. Yes, this works. I also added another
way of achieving this to the Stackoverflow post. Thanks for the help.
On Tue, Feb 28, 2017 at 11:58 PM, Jonathan Kelly <jo...@gmail.com>
wrote:
> Prithish,
>
> I saw you posted this on SO, so I responded there just now. See
> http://stackoverflow.com/questions/42452622/custom-
> log4j-properties-on-aws-emr/42516161#42516161
>
> In short, an hdfs:// path can't be used to configure log4j because log4j
> knows nothing about hdfs. Instead, since you are using EMR, you should use
> the Configuration API when creating your cluster to configure the
> spark-log4j configuration classification. See http://docs.aws.amazon.
> com/emr/latest/ReleaseGuide/emr-configure-apps.html for more info.
>
> ~ Jonathan
>
> On Sun, Feb 26, 2017 at 8:17 PM Prithish <pr...@gmail.com> wrote:
>
>> Steve, I tried that, but didn't work. Any other ideas?
>>
>> On Mon, Feb 27, 2017 at 1:42 AM, Steve Loughran <st...@hortonworks.com>
>> wrote:
>>
>> try giving a resource of a file in the JAR, e.g add a file
>> "log4j-debugging.properties into the jar, and give a config option of
>> -Dlog4j.configuration=/log4j-debugging.properties (maybe also try
>> without the "/")
>>
>>
>> On 26 Feb 2017, at 16:31, Prithish <pr...@gmail.com> wrote:
>>
>> Hoping someone can answer this.
>>
>> I am unable to override and use a Custom log4j.properties on Amazon EMR.
>> I am running Spark on EMR (Yarn) and have tried all the below combinations
>> in the Spark-Submit to try and use the custom log4j.
>>
>> In Client mode
>> --driver-java-options "-Dlog4j.configuration=hdfs://
>> host:port/user/hadoop/log4j.properties"
>>
>> In Cluster mode
>> --conf "spark.driver.extraJavaOptions=-Dlog4j.configuration=hdfs://host:
>> port/user/hadoop/log4j.properties"
>>
>> I have also tried picking from local filesystem using file://// instead
>> of hdfs. None of this seem to work. However, I can get this working when
>> running on my local Yarn setup.
>>
>> Any ideas?
>>
>> I have also posted on Stackoverflow (link below)
>> http://stackoverflow.com/questions/42452622/custom-
>> log4j-properties-on-aws-emr
>>
>>
>>
>>
Re: Custom log4j.properties on AWS EMR
Posted by Jonathan Kelly <jo...@gmail.com>.
Prithish,
I saw you posted this on SO, so I responded there just now. See
http://stackoverflow.com/questions/42452622/custom-log4j-properties-on-aws-emr/42516161#42516161
In short, an hdfs:// path can't be used to configure log4j because log4j
knows nothing about hdfs. Instead, since you are using EMR, you should use
the Configuration API when creating your cluster to configure the
spark-log4j configuration classification. See
http://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-configure-apps.html for
more info.
~ Jonathan
On Sun, Feb 26, 2017 at 8:17 PM Prithish <pr...@gmail.com> wrote:
> Steve, I tried that, but didn't work. Any other ideas?
>
> On Mon, Feb 27, 2017 at 1:42 AM, Steve Loughran <st...@hortonworks.com>
> wrote:
>
> try giving a resource of a file in the JAR, e.g add a file
> "log4j-debugging.properties into the jar, and give a config option of
> -Dlog4j.configuration=/log4j-debugging.properties (maybe also try without
> the "/")
>
>
> On 26 Feb 2017, at 16:31, Prithish <pr...@gmail.com> wrote:
>
> Hoping someone can answer this.
>
> I am unable to override and use a Custom log4j.properties on Amazon EMR. I
> am running Spark on EMR (Yarn) and have tried all the below combinations in
> the Spark-Submit to try and use the custom log4j.
>
> In Client mode
> --driver-java-options "-Dlog4j.configuration=
> hdfs://host:port/user/hadoop/log4j.properties"
>
> In Cluster mode
> --conf "spark.driver.extraJavaOptions=-Dlog4j.configuration=
> hdfs://host:port/user/hadoop/log4j.properties"
>
> I have also tried picking from local filesystem using file://// instead
> of hdfs. None of this seem to work. However, I can get this working when
> running on my local Yarn setup.
>
> Any ideas?
>
> I have also posted on Stackoverflow (link below)
>
> http://stackoverflow.com/questions/42452622/custom-log4j-properties-on-aws-emr
>
>
>
>
Re: Custom log4j.properties on AWS EMR
Posted by Prithish <pr...@gmail.com>.
Steve, I tried that, but didn't work. Any other ideas?
On Mon, Feb 27, 2017 at 1:42 AM, Steve Loughran <st...@hortonworks.com>
wrote:
> try giving a resource of a file in the JAR, e.g add a file
> "log4j-debugging.properties into the jar, and give a config option of
> -Dlog4j.configuration=/log4j-debugging.properties (maybe also try
> without the "/")
>
>
> On 26 Feb 2017, at 16:31, Prithish <pr...@gmail.com> wrote:
>
> Hoping someone can answer this.
>
> I am unable to override and use a Custom log4j.properties on Amazon EMR. I
> am running Spark on EMR (Yarn) and have tried all the below combinations in
> the Spark-Submit to try and use the custom log4j.
>
> In Client mode
> --driver-java-options "-Dlog4j.configuration=hdfs://
> host:port/user/hadoop/log4j.properties"
>
> In Cluster mode
> --conf "spark.driver.extraJavaOptions=-Dlog4j.configuration=hdfs://host:
> port/user/hadoop/log4j.properties"
>
> I have also tried picking from local filesystem using file://// instead
> of hdfs. None of this seem to work. However, I can get this working when
> running on my local Yarn setup.
>
> Any ideas?
>
> I have also posted on Stackoverflow (link below)
> http://stackoverflow.com/questions/42452622/custom-
> log4j-properties-on-aws-emr
>
>
>
Re: Custom log4j.properties on AWS EMR
Posted by Steve Loughran <st...@hortonworks.com>.
try giving a resource of a file in the JAR, e.g add a file "log4j-debugging.properties into the jar, and give a config option of -Dlog4j.configuration=/log4j-debugging.properties (maybe also try without the "/")
On 26 Feb 2017, at 16:31, Prithish <pr...@gmail.com>> wrote:
Hoping someone can answer this.
I am unable to override and use a Custom log4j.properties on Amazon EMR. I am running Spark on EMR (Yarn) and have tried all the below combinations in the Spark-Submit to try and use the custom log4j.
In Client mode
--driver-java-options "-Dlog4j.configuration=hdfs://host:port/user/hadoop/log4j.properties"
In Cluster mode
--conf "spark.driver.extraJavaOptions=-Dlog4j.configuration=hdfs://host:port/user/hadoop/log4j.properties"
I have also tried picking from local filesystem using file://// instead of hdfs. None of this seem to work. However, I can get this working when running on my local Yarn setup.
Any ideas?
I have also posted on Stackoverflow (link below)
http://stackoverflow.com/questions/42452622/custom-log4j-properties-on-aws-emr