You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by HARSH TAKKAR <ta...@gmail.com> on 2016/02/22 07:44:30 UTC
[Please Help] Log redirection on EMR
Hi
In am using an EMR cluster for running my spark jobs, but after the job
finishes logs disappear,
I have added a log4j.properties in my jar, but all the logs still redirects
to EMR resource manager which vanishes after jobs completes, is there a way
i could redirect the logs to a location in file syatem, I am working on
price points and its very critical for me to maintain logs.
Just to add i get following error when my application starts.
java.io.FileNotFoundException: /etc/spark/conf/log4j.properties (No
such file or directory)
at java.io.FileInputStream.open(Native Method)
at java.io.FileInputStream.<init>(FileInputStream.java:146)
at java.io.FileInputStream.<init>(FileInputStream.java:101)
at sun.net.www.protocol.file.FileURLConnection.connect(FileURLConnection.java:90)
at sun.net.www.protocol.file.FileURLConnection.getInputStream(FileURLConnection.java:188)
at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:557)
at org.apache.log4j.helpers.OptionConverter.selectAndConfigure(OptionConverter.java:526)
at org.apache.log4j.LogManager.<clinit>(LogManager.java:127)
at org.apache.spark.Logging$class.initializeLogging(Logging.scala:122)
at org.apache.spark.Logging$class.initializeIfNecessary(Logging.scala:107)
at org.apache.spark.Logging$class.log(Logging.scala:51)
at org.apache.spark.deploy.yarn.ApplicationMaster$.log(ApplicationMaster.scala:607)
at org.apache.spark.deploy.yarn.ApplicationMaster$.main(ApplicationMaster.scala:621)
at org.apache.spark.deploy.yarn.ApplicationMaster.main(ApplicationMaster.scala)
Re: [Please Help] Log redirection on EMR
Posted by HARSH TAKKAR <ta...@gmail.com>.
Hi Sabarish
Thanks for your help, i was able to get the logs from archive, is there a
way i can adjust archival policy, say i want to persist the logs of last 2
jobs on resource manage and archive others on the file system.
On Mon, Feb 22, 2016 at 12:25 PM Sabarish Sasidharan <
sabarish.sasidharan@manthan.com> wrote:
> Your logs are getting archived in your logs bucket in S3.
>
>
> http://docs.aws.amazon.com/ElasticMapReduce/latest/DeveloperGuide/emr-plan-debugging.html
>
> Regards
> Sab
>
> On Mon, Feb 22, 2016 at 12:14 PM, HARSH TAKKAR <ta...@gmail.com>
> wrote:
>
>> Hi
>>
>> In am using an EMR cluster for running my spark jobs, but after the job
>> finishes logs disappear,
>>
>> I have added a log4j.properties in my jar, but all the logs still
>> redirects to EMR resource manager which vanishes after jobs completes, is
>> there a way i could redirect the logs to a location in file syatem, I am
>> working on price points and its very critical for me to maintain logs.
>>
>> Just to add i get following error when my application starts.
>>
>> java.io.FileNotFoundException: /etc/spark/conf/log4j.properties (No such file or directory)
>> at java.io.FileInputStream.open(Native Method)
>> at java.io.FileInputStream.<init>(FileInputStream.java:146)
>> at java.io.FileInputStream.<init>(FileInputStream.java:101)
>> at sun.net.www.protocol.file.FileURLConnection.connect(FileURLConnection.java:90)
>> at sun.net.www.protocol.file.FileURLConnection.getInputStream(FileURLConnection.java:188)
>> at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:557)
>> at org.apache.log4j.helpers.OptionConverter.selectAndConfigure(OptionConverter.java:526)
>> at org.apache.log4j.LogManager.<clinit>(LogManager.java:127)
>> at org.apache.spark.Logging$class.initializeLogging(Logging.scala:122)
>> at org.apache.spark.Logging$class.initializeIfNecessary(Logging.scala:107)
>> at org.apache.spark.Logging$class.log(Logging.scala:51)
>> at org.apache.spark.deploy.yarn.ApplicationMaster$.log(ApplicationMaster.scala:607)
>> at org.apache.spark.deploy.yarn.ApplicationMaster$.main(ApplicationMaster.scala:621)
>> at org.apache.spark.deploy.yarn.ApplicationMaster.main(ApplicationMaster.scala)
>>
>>
>>
>
>
> --
>
> Architect - Big Data
> Ph: +91 99805 99458
>
> Manthan Systems | *Company of the year - Analytics (2014 Frost and
> Sullivan India ICT)*
> +++
>
Re: [Please Help] Log redirection on EMR
Posted by Sabarish Sasidharan <sa...@manthan.com>.
Your logs are getting archived in your logs bucket in S3.
http://docs.aws.amazon.com/ElasticMapReduce/latest/DeveloperGuide/emr-plan-debugging.html
Regards
Sab
On Mon, Feb 22, 2016 at 12:14 PM, HARSH TAKKAR <ta...@gmail.com>
wrote:
> Hi
>
> In am using an EMR cluster for running my spark jobs, but after the job
> finishes logs disappear,
>
> I have added a log4j.properties in my jar, but all the logs still
> redirects to EMR resource manager which vanishes after jobs completes, is
> there a way i could redirect the logs to a location in file syatem, I am
> working on price points and its very critical for me to maintain logs.
>
> Just to add i get following error when my application starts.
>
> java.io.FileNotFoundException: /etc/spark/conf/log4j.properties (No such file or directory)
> at java.io.FileInputStream.open(Native Method)
> at java.io.FileInputStream.<init>(FileInputStream.java:146)
> at java.io.FileInputStream.<init>(FileInputStream.java:101)
> at sun.net.www.protocol.file.FileURLConnection.connect(FileURLConnection.java:90)
> at sun.net.www.protocol.file.FileURLConnection.getInputStream(FileURLConnection.java:188)
> at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:557)
> at org.apache.log4j.helpers.OptionConverter.selectAndConfigure(OptionConverter.java:526)
> at org.apache.log4j.LogManager.<clinit>(LogManager.java:127)
> at org.apache.spark.Logging$class.initializeLogging(Logging.scala:122)
> at org.apache.spark.Logging$class.initializeIfNecessary(Logging.scala:107)
> at org.apache.spark.Logging$class.log(Logging.scala:51)
> at org.apache.spark.deploy.yarn.ApplicationMaster$.log(ApplicationMaster.scala:607)
> at org.apache.spark.deploy.yarn.ApplicationMaster$.main(ApplicationMaster.scala:621)
> at org.apache.spark.deploy.yarn.ApplicationMaster.main(ApplicationMaster.scala)
>
>
>
--
Architect - Big Data
Ph: +91 99805 99458
Manthan Systems | *Company of the year - Analytics (2014 Frost and Sullivan
India ICT)*
+++