RDD operation examples with data?

[Chris Curtin <cu...@gmail.com>]

Hi,

I'm learning Spark and I am confused about when to use the many different
operations on RDDs. Does anyone have any examples which show example inputs
and resulting outputs for the various RDD operations and if the operation
takes an Function a simple example of the code?

For example, something like this for flatMap

One row -> "the quick brown fox"

Passed to:

JavaRDD<String> words = lines.flatMap(new FlatMapFunction<String, String>() {
      @Override
      public Iterable<String> call(String s) {
        return Arrays.asList(SPACE.split(s));
      }
    });

When completed: words would contain
"the"
"quick"
"brown"
"fox"

(Yes this one is pretty obvious but some of the others aren't).

If such examples don't exist, is there a shared wiki or someplace we
could start building one?

Thanks,

Chris

Re: RDD operation examples with data?

[Jacob Eisinger <ei...@gmail.com>]

I would check out the source examples on Spark's Github:
https://github.com/apache/spark/tree/master/examples/src/main/scala/org/apache/spark/examples

And, Zhen He put together a great web page with summaries and examples of
each function:
http://apache-spark-user-list.1001560.n3.nabble.com/A-new-resource-for-getting-examples-of-Spark-RDD-API-calls-td5529.html

Hope this helps!

Jacob


On Thu, Jul 31, 2014 at 3:00 PM, Chris Curtin <cu...@gmail.com>
wrote:

> Hi,
>
> I'm learning Spark and I am confused about when to use the many different
> operations on RDDs. Does anyone have any examples which show example inputs
> and resulting outputs for the various RDD operations and if the operation
> takes an Function a simple example of the code?
>
> For example, something like this for flatMap
>
> One row -> "the quick brown fox"
>
> Passed to:
>
> JavaRDD<String> words = lines.flatMap(new FlatMapFunction<String, String>() {
>       @Override
>       public Iterable<String> call(String s) {
>         return Arrays.asList(SPACE.split(s));
>       }
>     });
>
> When completed: words would contain
> "the"
> "quick"
> "brown"
> "fox"
>
> (Yes this one is pretty obvious but some of the others aren't).
>
> If such examples don't exist, is there a shared wiki or someplace we could start building one?
>
> Thanks,
>
> Chris
>
>