You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by durga <du...@gmail.com> on 2014/12/31 01:21:22 UTC
JetS3T settings spark
I am not sure , the way I can pass jets3t.properties file for spark-submit.
--file option seems not working.
can some one please help me. My production spark jobs get hung up when
reading s3 file sporadically.
Thanks,
-D
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/JetS3T-settings-spark-tp20916.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: JetS3T settings spark
Posted by durga katakam <du...@gmail.com>.
Thanks Matei.
-D
On Tue, Dec 30, 2014 at 4:49 PM, Matei Zaharia <ma...@gmail.com>
wrote:
> This file needs to be on your CLASSPATH actually, not just in a directory.
> The best way to pass it in is probably to package it into your application
> JAR. You can put it in src/main/resources in a Maven or SBT project, and
> check that it makes it into the JAR using jar tf yourfile.jar.
>
> Matei
>
> > On Dec 30, 2014, at 4:21 PM, durga <du...@gmail.com> wrote:
> >
> > I am not sure , the way I can pass jets3t.properties file for
> spark-submit.
> > --file option seems not working.
> > can some one please help me. My production spark jobs get hung up when
> > reading s3 file sporadically.
> >
> > Thanks,
> > -D
> >
> >
> >
> > --
> > View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/JetS3T-settings-spark-tp20916.html
> > Sent from the Apache Spark User List mailing list archive at Nabble.com.
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> > For additional commands, e-mail: user-help@spark.apache.org
> >
>
>
Re: JetS3T settings spark
Posted by Matei Zaharia <ma...@gmail.com>.
This file needs to be on your CLASSPATH actually, not just in a directory. The best way to pass it in is probably to package it into your application JAR. You can put it in src/main/resources in a Maven or SBT project, and check that it makes it into the JAR using jar tf yourfile.jar.
Matei
> On Dec 30, 2014, at 4:21 PM, durga <du...@gmail.com> wrote:
>
> I am not sure , the way I can pass jets3t.properties file for spark-submit.
> --file option seems not working.
> can some one please help me. My production spark jobs get hung up when
> reading s3 file sporadically.
>
> Thanks,
> -D
>
>
>
> --
> View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/JetS3T-settings-spark-tp20916.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org