You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Tobias Pfeiffer <tg...@preferred.jp> on 2014/11/20 09:20:03 UTC
spark-submit and logging
Hi,
I am using spark-submit to submit my application jar to a YARN cluster. I
want to deliver a single jar file to my users, so I would like to avoid to
tell them "also, please put that log4j.xml file somewhere and add that path
to the spark-submit command".
I thought it would be sufficient that my application jar file contains a
log4j.xml file, but that does not seem to be the case. If I don't add a
log4j.xml file to the classpath before launching spark-submit, the one
bundled with spark will be used -- which has a negative influence on my
program execution. Is there any way I can tell spark-submit to use the
log4j configuration bundled in my jar file?
Thanks
Tobias
Re: spark-submit and logging
Posted by Marcelo Vanzin <va...@cloudera.com>.
Hi Tobias,
With the current Yarn code, packaging the configuration in your app's
jar and adding the "-Dlog4j.configuration=log4jConf.xml" argument to
the extraJavaOptions configs should work.
That's not the recommended way for get it to work, though, since this
behavior may change in the future. The recommended way is to do what
Sean mentions - upload the configuration as a separate file. If you
upload a "log4j.properties" file that's all you need, but if you
really want to use an XML config file you'll also need to pass the
extra options to the driver and executors.
On Thu, Nov 20, 2014 at 12:20 AM, Tobias Pfeiffer <tg...@preferred.jp> wrote:
> Hi,
>
> I am using spark-submit to submit my application jar to a YARN cluster. I
> want to deliver a single jar file to my users, so I would like to avoid to
> tell them "also, please put that log4j.xml file somewhere and add that path
> to the spark-submit command".
>
> I thought it would be sufficient that my application jar file contains a
> log4j.xml file, but that does not seem to be the case. If I don't add a
> log4j.xml file to the classpath before launching spark-submit, the one
> bundled with spark will be used -- which has a negative influence on my
> program execution. Is there any way I can tell spark-submit to use the
> log4j configuration bundled in my jar file?
>
> Thanks
> Tobias
--
Marcelo
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: spark-submit and logging
Posted by Marcelo Vanzin <va...@cloudera.com>.
Check the "--files" argument in the output "spark-submit -h".
On Thu, Nov 20, 2014 at 7:51 AM, Matt Narrell <ma...@gmail.com> wrote:
> How do I configure the files to be uploaded to YARN containers. So far, I’ve only seen "--conf spark.yarn.jar=hdfs://….” which allows me to specify the HDFS location of the Spark JAR, but I’m not sure how to prescribe other files for uploading (e.g., spark-env.sh)
>
> mn
>
>> On Nov 20, 2014, at 4:08 AM, Sean Owen <so...@cloudera.com> wrote:
>>
>> I think the standard practice is to include your log config file among
>> the files uploaded to YARN containers, and then set
>> -Dlog4j.configuration=yourfile.xml in
>> spark.{executor.driver}.extraJavaOptions ?
>>
>> http://spark.apache.org/docs/latest/running-on-yarn.html
>>
>> On Thu, Nov 20, 2014 at 9:20 AM, Tobias Pfeiffer <tg...@preferred.jp> wrote:
>>> Hi,
>>>
>>> I am using spark-submit to submit my application jar to a YARN cluster. I
>>> want to deliver a single jar file to my users, so I would like to avoid to
>>> tell them "also, please put that log4j.xml file somewhere and add that path
>>> to the spark-submit command".
>>>
>>> I thought it would be sufficient that my application jar file contains a
>>> log4j.xml file, but that does not seem to be the case. If I don't add a
>>> log4j.xml file to the classpath before launching spark-submit, the one
>>> bundled with spark will be used -- which has a negative influence on my
>>> program execution. Is there any way I can tell spark-submit to use the
>>> log4j configuration bundled in my jar file?
>>>
>>> Thanks
>>> Tobias
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>> For additional commands, e-mail: user-help@spark.apache.org
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
--
Marcelo
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: spark-submit and logging
Posted by Matt Narrell <ma...@gmail.com>.
How do I configure the files to be uploaded to YARN containers. So far, I’ve only seen "--conf spark.yarn.jar=hdfs://….” which allows me to specify the HDFS location of the Spark JAR, but I’m not sure how to prescribe other files for uploading (e.g., spark-env.sh)
mn
> On Nov 20, 2014, at 4:08 AM, Sean Owen <so...@cloudera.com> wrote:
>
> I think the standard practice is to include your log config file among
> the files uploaded to YARN containers, and then set
> -Dlog4j.configuration=yourfile.xml in
> spark.{executor.driver}.extraJavaOptions ?
>
> http://spark.apache.org/docs/latest/running-on-yarn.html
>
> On Thu, Nov 20, 2014 at 9:20 AM, Tobias Pfeiffer <tg...@preferred.jp> wrote:
>> Hi,
>>
>> I am using spark-submit to submit my application jar to a YARN cluster. I
>> want to deliver a single jar file to my users, so I would like to avoid to
>> tell them "also, please put that log4j.xml file somewhere and add that path
>> to the spark-submit command".
>>
>> I thought it would be sufficient that my application jar file contains a
>> log4j.xml file, but that does not seem to be the case. If I don't add a
>> log4j.xml file to the classpath before launching spark-submit, the one
>> bundled with spark will be used -- which has a negative influence on my
>> program execution. Is there any way I can tell spark-submit to use the
>> log4j configuration bundled in my jar file?
>>
>> Thanks
>> Tobias
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: spark-submit and logging
Posted by Sean Owen <so...@cloudera.com>.
I think the standard practice is to include your log config file among
the files uploaded to YARN containers, and then set
-Dlog4j.configuration=yourfile.xml in
spark.{executor.driver}.extraJavaOptions ?
http://spark.apache.org/docs/latest/running-on-yarn.html
On Thu, Nov 20, 2014 at 9:20 AM, Tobias Pfeiffer <tg...@preferred.jp> wrote:
> Hi,
>
> I am using spark-submit to submit my application jar to a YARN cluster. I
> want to deliver a single jar file to my users, so I would like to avoid to
> tell them "also, please put that log4j.xml file somewhere and add that path
> to the spark-submit command".
>
> I thought it would be sufficient that my application jar file contains a
> log4j.xml file, but that does not seem to be the case. If I don't add a
> log4j.xml file to the classpath before launching spark-submit, the one
> bundled with spark will be used -- which has a negative influence on my
> program execution. Is there any way I can tell spark-submit to use the
> log4j configuration bundled in my jar file?
>
> Thanks
> Tobias
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org