Adding RDD function to segment an RDD (like substring)

["Ganelin, Ilya" <Il...@capitalone.com>]

Hi all – a utility that I’ve found useful several times now when working with RDDs is to be able to reason about segments of the RDD.

For example, if I have two large RDDs and I want to combine them in a way that would be intractable in terms of memory or disk storage (e.g. A cartesian) but a piece-wise approach is tractable, there are not many methods that enable this.

Existing relevant methods are :
takeSample – Which allows me to take a number of random values from an RDD but does not let me process the entire RDD
randomSplit – Which segments the RDD into an array of smaller RDDs – with this, however, I’ve seen numerous memory issues during execution on larger datasets or when splitting into many RDDs
forEach or collect – Which require the RDD to fit into memory – which is not possible for larger datasets

What I am proposing is the addition of a simple method to operate on an RDD which would operate equivalently to a substring operation in String:

For an RDD[T]:

def sample(startIdx : Int, endIdx : Int) : RDD[T] = {
val zipped = this.zipWithIndex()
val sampled = filter(idx => (idx >= startIdx) && (idx < endIdx))
sampled.map(_._1)
}

I would appreciate any feedback – thank you!


________________________________________________________

The information contained in this e-mail is confidential and/or proprietary to Capital One and/or its affiliates. The information transmitted herewith is intended only for use by the individual or entity to which it is addressed.  If the reader of this message is not the intended recipient, you are hereby notified that any review, retransmission, dissemination, distribution, copying or other use of, or taking of any action in reliance upon this information is strictly prohibited. If you have received this communication in error, please contact the sender and delete the material from your computer.

Re: Adding RDD function to segment an RDD (like substring)

[Mark Hamstra <ma...@clearstorydata.com>]

`zipWithIndex` is both compute intensive and breaks Spark's
"transformations are lazy" model, so it is probably not appropriate to add
this to the public RDD API.  If `zipWithIndex` weren't already what I
consider to be broken, I'd be much friendlier to building something more on
top of it, but I really don't like `zipWithIndex` or this `sample` idea of
yours as they stand.  Aside from those concerns, the name `sample` doesn't
really work since it is too easy to assume that it means some kind of
random sampling.

None of that is to say that you can't make effective use of `zipWithIndex`
and your `sample` in your own code; but I'm not a fan of extending the
Spark public API in this way.

On Tue, Dec 9, 2014 at 12:38 PM, Ganelin, Ilya <Il...@capitalone.com>
wrote:

> Hi all – a utility that I’ve found useful several times now when working
> with RDDs is to be able to reason about segments of the RDD.
>
> For example, if I have two large RDDs and I want to combine them in a way
> that would be intractable in terms of memory or disk storage (e.g. A
> cartesian) but a piece-wise approach is tractable, there are not many
> methods that enable this.
>
> Existing relevant methods are :
> takeSample – Which allows me to take a number of random values from an RDD
> but does not let me process the entire RDD
> randomSplit – Which segments the RDD into an array of smaller RDDs – with
> this, however, I’ve seen numerous memory issues during execution on larger
> datasets or when splitting into many RDDs
> forEach or collect – Which require the RDD to fit into memory – which is
> not possible for larger datasets
>
> What I am proposing is the addition of a simple method to operate on an
> RDD which would operate equivalently to a substring operation in String:
>
> For an RDD[T]:
>
> def sample(startIdx : Int, endIdx : Int) : RDD[T] = {
> val zipped = this.zipWithIndex()
> val sampled = filter(idx => (idx >= startIdx) && (idx < endIdx))
> sampled.map(_._1)
> }
>
> I would appreciate any feedback – thank you!
>
>
> ________________________________________________________
>
> The information contained in this e-mail is confidential and/or
> proprietary to Capital One and/or its affiliates. The information
> transmitted herewith is intended only for use by the individual or entity
> to which it is addressed.  If the reader of this message is not the
> intended recipient, you are hereby notified that any review,
> retransmission, dissemination, distribution, copying or other use of, or
> taking of any action in reliance upon this information is strictly
> prohibited. If you have received this communication in error, please
> contact the sender and delete the material from your computer.
>