You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Saiph Kappa <sa...@gmail.com> on 2014/09/29 19:15:15 UTC
Simple Question: Spark Streaming Applications
Hi,
Do all spark streaming applications use the map operation? or the majority
of them?
Thanks.
Re: Simple Question: Spark Streaming Applications
Posted by Tathagata Das <ta...@gmail.com>.
More importantly, why are you asking this question? :)
Also let me generalize the answer by saying that most applications that do
some useful computations use map-like operations. And by map-like
operations I mean simple operations like map, filter, flatMap,
mapPartitions. The only category of applications that dont use any
transformation are ones where the data is received from a source and dumped
to some output sink as is, without any transformation.
On Tue, Sep 30, 2014 at 6:39 PM, Tobias Pfeiffer <tg...@preferred.jp> wrote:
> Hi,
>
> On Wed, Oct 1, 2014 at 12:20 AM, Saiph Kappa <sa...@gmail.com>
> wrote:
>
>> But most applications use transformations, and map in particular, correct?
>>
>
> Yes, I would claim that most applications that do some useful computation
> use map().
>
> Tobias
>
>
Re: Simple Question: Spark Streaming Applications
Posted by Tobias Pfeiffer <tg...@preferred.jp>.
Hi,
On Wed, Oct 1, 2014 at 12:20 AM, Saiph Kappa <sa...@gmail.com> wrote:
> But most applications use transformations, and map in particular, correct?
>
Yes, I would claim that most applications that do some useful computation
use map().
Tobias
Re: Simple Question: Spark Streaming Applications
Posted by Saiph Kappa <sa...@gmail.com>.
But most applications use transformations, and map in particular, correct?
On Mon, Sep 29, 2014 at 6:21 PM, Liquan Pei <li...@gmail.com> wrote:
> Hi Saiph,
>
> Map is used for transformation on your input RDD. If you don't need
> transformation of your input, you don't need to use map.
>
> Thanks,
> Liquan
>
> On Mon, Sep 29, 2014 at 10:15 AM, Saiph Kappa <sa...@gmail.com>
> wrote:
>
>> Hi,
>>
>> Do all spark streaming applications use the map operation? or the
>> majority of them?
>>
>> Thanks.
>>
>
>
>
> --
> Liquan Pei
> Department of Physics
> University of Massachusetts Amherst
>
Re: Simple Question: Spark Streaming Applications
Posted by Liquan Pei <li...@gmail.com>.
Hi Saiph,
Map is used for transformation on your input RDD. If you don't need
transformation of your input, you don't need to use map.
Thanks,
Liquan
On Mon, Sep 29, 2014 at 10:15 AM, Saiph Kappa <sa...@gmail.com> wrote:
> Hi,
>
> Do all spark streaming applications use the map operation? or the majority
> of them?
>
> Thanks.
>
--
Liquan Pei
Department of Physics
University of Massachusetts Amherst