You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by ar7 <as...@gmail.com> on 2016/06/15 10:50:34 UTC
Adding h5 files in a zip to use with PySpark
I am using PySpark 1.6.1 for my spark application. I have additional modules
which I am loading using the argument --py-files. I also have a h5 file
which I need to access from one of the modules for initializing the
ApolloNet.
Is there any way I could access those files from the modules if I put them
in the same archive? I tried this approach but it was throwing an error
because the files are not there in every worker. I can think of one solution
which is copying the file to each of the workers but I want to know if there
are better ways to do it?
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Adding-h5-files-in-a-zip-to-use-with-PySpark-tp27173.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: Adding h5 files in a zip to use with PySpark
Posted by Ashwin Raaghav <as...@gmail.com>.
Thanks! That worked! :)
And to read the files, I used pyspark.SparkFiles module.
On Thu, Jun 16, 2016 at 7:12 AM, Sun Rui <su...@163.com> wrote:
> have you tried
> --files <your h5 file>?
> > On Jun 15, 2016, at 18:50, ar7 <as...@gmail.com> wrote:
> >
> > I am using PySpark 1.6.1 for my spark application. I have additional
> modules
> > which I am loading using the argument --py-files. I also have a h5 file
> > which I need to access from one of the modules for initializing the
> > ApolloNet.
> >
> > Is there any way I could access those files from the modules if I put
> them
> > in the same archive? I tried this approach but it was throwing an error
> > because the files are not there in every worker. I can think of one
> solution
> > which is copying the file to each of the workers but I want to know if
> there
> > are better ways to do it?
> >
> >
> >
> > --
> > View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/Adding-h5-files-in-a-zip-to-use-with-PySpark-tp27173.html
> > Sent from the Apache Spark User List mailing list archive at Nabble.com.
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> > For additional commands, e-mail: user-help@spark.apache.org
> >
>
>
>
--
Regards,
Ashwin Raaghav
Re: Adding h5 files in a zip to use with PySpark
Posted by Sun Rui <su...@163.com>.
have you tried
--files <your h5 file>?
> On Jun 15, 2016, at 18:50, ar7 <as...@gmail.com> wrote:
>
> I am using PySpark 1.6.1 for my spark application. I have additional modules
> which I am loading using the argument --py-files. I also have a h5 file
> which I need to access from one of the modules for initializing the
> ApolloNet.
>
> Is there any way I could access those files from the modules if I put them
> in the same archive? I tried this approach but it was throwing an error
> because the files are not there in every worker. I can think of one solution
> which is copying the file to each of the workers but I want to know if there
> are better ways to do it?
>
>
>
> --
> View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Adding-h5-files-in-a-zip-to-use-with-PySpark-tp27173.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org