You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by pouryas <po...@adbrain.com> on 2014/09/30 16:43:52 UTC
Re: S3 - Extra $_folder$ files for every directory node
I would like to know a way for not adding those $_folder$ files to S3 as
well. I can go ahead and delete them but it would be nice if Spark handles
this for you.
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/S3-Extra-folder-files-for-every-directory-node-tp15078p15402.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: S3 - Extra $_folder$ files for every directory node
Posted by Nicholas Chammas <ni...@gmail.com>.
Those files are created by the Hadoop API that Spark leverages. Spark does
not directly control that.
You may be able to check with the Hadoop project on whether they are
looking at changing this behavior. I believe it was introduced because S3
at one point required it, though it doesn't anymore.
On Tue, Sep 30, 2014 at 10:43 AM, pouryas <po...@adbrain.com> wrote:
> I would like to know a way for not adding those $_folder$ files to S3 as
> well. I can go ahead and delete them but it would be nice if Spark handles
> this for you.
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/S3-Extra-folder-files-for-every-directory-node-tp15078p15402.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>