You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@beam.apache.org by Jesse Anderson <je...@smokinghand.com> on 2016/06/07 16:10:34 UTC

ParDo Docs Question

I have a question about the ParDo JavaDocs. The JavaDocs say:
The ParDo processing style is similar to what happens inside the "Mapper"
or "Reducer" class of a MapReduce-style algorithm.

I think a ParDo's DoFn is only what a Mapper class would do. A DoFn doesn't
seem to run after a shuffle sort like a reducer does. Is my understanding
correct?

Thanks,

Jesse

Re: ParDo Docs Question

Posted by Frances Perry <fj...@google.com>.
Yup, exactly. A MapReduce is equivalent to a ParDo-GroupByKey-ParDo.

On Tue, Jun 7, 2016 at 9:55 AM, Jesse Anderson <je...@smokinghand.com>
wrote:

> Pawel,
>
> I think I understand it better. By running a ParDo after a GroupByKey, the
> DoFn runs as what a MapReduce-style would do in the reducer.
>
> Thanks,
>
> Jesse
>
> On Tue, Jun 7, 2016 at 12:17 PM Pawel Szczur <pa...@gmail.com>
> wrote:
>
>> There's no shuffle sort (at least no documented), but they may be shuffle
>> e.g. when one use GroupByKey. In such case, the result is:
>> PCollection<KV<?,Iterable<?>>
>> if you apply DoFn, it's similar to running reducer.
>>
>> 2016-06-07 18:10 GMT+02:00 Jesse Anderson <je...@smokinghand.com>:
>>
>>> I have a question about the ParDo JavaDocs. The JavaDocs say:
>>> The ParDo processing style is similar to what happens inside the
>>> "Mapper" or "Reducer" class of a MapReduce-style algorithm.
>>>
>>> I think a ParDo's DoFn is only what a Mapper class would do. A DoFn
>>> doesn't seem to run after a shuffle sort like a reducer does. Is my
>>> understanding correct?
>>>
>>> Thanks,
>>>
>>> Jesse
>>>
>>
>>

Re: ParDo Docs Question

Posted by Jesse Anderson <je...@smokinghand.com>.
Pawel,

I think I understand it better. By running a ParDo after a GroupByKey, the
DoFn runs as what a MapReduce-style would do in the reducer.

Thanks,

Jesse

On Tue, Jun 7, 2016 at 12:17 PM Pawel Szczur <pa...@gmail.com> wrote:

> There's no shuffle sort (at least no documented), but they may be shuffle
> e.g. when one use GroupByKey. In such case, the result is:
> PCollection<KV<?,Iterable<?>>
> if you apply DoFn, it's similar to running reducer.
>
> 2016-06-07 18:10 GMT+02:00 Jesse Anderson <je...@smokinghand.com>:
>
>> I have a question about the ParDo JavaDocs. The JavaDocs say:
>> The ParDo processing style is similar to what happens inside the "Mapper"
>> or "Reducer" class of a MapReduce-style algorithm.
>>
>> I think a ParDo's DoFn is only what a Mapper class would do. A DoFn
>> doesn't seem to run after a shuffle sort like a reducer does. Is my
>> understanding correct?
>>
>> Thanks,
>>
>> Jesse
>>
>
>

Re: ParDo Docs Question

Posted by Pawel Szczur <pa...@gmail.com>.
There's no shuffle sort (at least no documented), but they may be shuffle
e.g. when one use GroupByKey. In such case, the result is:
PCollection<KV<?,Iterable<?>>
if you apply DoFn, it's similar to running reducer.

2016-06-07 18:10 GMT+02:00 Jesse Anderson <je...@smokinghand.com>:

> I have a question about the ParDo JavaDocs. The JavaDocs say:
> The ParDo processing style is similar to what happens inside the "Mapper"
> or "Reducer" class of a MapReduce-style algorithm.
>
> I think a ParDo's DoFn is only what a Mapper class would do. A DoFn
> doesn't seem to run after a shuffle sort like a reducer does. Is my
> understanding correct?
>
> Thanks,
>
> Jesse
>