You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@spark.apache.org by Grega Kešpret <gr...@celtra.com> on 2014/02/22 21:11:41 UTC

Get RDD partition location

Is it possible to get location (e.g. file name) of RDD partition?

Let's say I do

val logs = sc.textFile("s3n://some/path/*/*")

This automatically gets me all the files in all directories of path
"some/path/*". When processing those files, is it possible to get the
information about the underlying partition's location ?

If not, there's always possibility of constructing several RDDs with
sc.textFile and keeping track of the paths myself.


Grega

Re: Get RDD partition location

Posted by Tathagata Das <ta...@gmail.com>.

I dont think there is a clean way to do that. Its best to create separate
RDDs yourself.

TD


On Sat, Feb 22, 2014 at 12:11 PM, Grega Kešpret <gr...@celtra.com> wrote:

> Is it possible to get location (e.g. file name) of RDD partition?
>
> Let's say I do
>
> val logs = sc.textFile("s3n://some/path/*/*")
>
> This automatically gets me all the files in all directories of path
> "some/path/*". When processing those files, is it possible to get the
> information about the underlying partition's location ?
>
> If not, there's always possibility of constructing several RDDs with
> sc.textFile and keeping track of the paths myself.
>
>
> Grega
>
>