can I think of JavaDStream<> foreachRDD() as being 'for each mini batch' ?

[Andy Davidson <An...@SantaCruzIntegration.com>]

Hi 

I am new to Spark Streaming. Can I think of  JavaDStream<> foreachRDD() as
being 'for each mini batch¹? The java doc does not say much about this
function.

Here is the background. I am writing a little test program to figure out how
to use streams. At some point I wanted to calculate an aggregate. In my
first attempt my driver did a couple of transformations and tries to get the
count(). I wrote this code like I would a spark core app and added some
logging. My log statements where only executed once, how ever
JavaDStream<>print() appears to run on each mini batch.

When I put my logging and aggregation code inside foreachRDD() things work
as expected my aggegrate and logging appear to be executed on each
minibactch

I am running on a 4 machine cluster.  I create a message with the value of
each mini batch aggregate and use logging and System.out. Given the message
shows up on my console is it safe to assume that this output code is
executing in my driver? The ip shows it is running on my Master.

I thought maybe the message is showing up here because I do not have enough
data in the steam to force load onto the workers?


Any in sites would be greatly appreciated.

Andy

Re: can I think of JavaDStream<> foreachRDD() as being 'for each mini batch' ?

[Andy Davidson <An...@SantaCruzIntegration.com>]

Hi Sean

Many many thanks. This really clears a lot up for me

Andy

From:  Sean Owen <so...@cloudera.com>
Date:  Wednesday, October 1, 2014 at 11:27 AM
To:  Andrew Davidson <An...@SantaCruzIntegration.com>
Cc:  "user@spark.apache.org" <us...@spark.apache.org>
Subject:  Re: can I think of JavaDStream<> foreachRDD() as being 'for each
mini batch' ?

> Yes, foreachRDD will do your something for each RDD, which is what you
> get for each mini-batch of input.
> 
> The operations you express on a DStream (or JavaDStream) are all,
> really, "for each RDD", including print(). Logging is a little harder
> to reason about since the logging will happen on a potentially remote
> receiver. I am not sure if this explains your observed behavior; it
> depends on what you were logging.
> 
> On Wed, Oct 1, 2014 at 6:51 PM, Andy Davidson
> <An...@santacruzintegration.com> wrote:
>>  Hi
>> 
>>  I am new to Spark Streaming. Can I think of  JavaDStream<> foreachRDD() as
>>  being 'for each mini batch¹? The java doc does not say much about this
>>  function.
>> 
>>  Here is the background. I am writing a little test program to figure out how
>>  to use streams. At some point I wanted to calculate an aggregate. In my
>>  first attempt my driver did a couple of transformations and tries to get the
>>  count(). I wrote this code like I would a spark core app and added some
>>  logging. My log statements where only executed once, how ever
>>  JavaDStream<>print() appears to run on each mini batch.
>> 
>>  When I put my logging and aggregation code inside foreachRDD() things work
>>  as expected my aggegrate and logging appear to be executed on each
>>  minibactch
>> 
>>  I am running on a 4 machine cluster.  I create a message with the value of
>>  each mini batch aggregate and use logging and System.out. Given the message
>>  shows up on my console is it safe to assume that this output code is
>>  executing in my driver? The ip shows it is running on my Master.
>> 
>>  I thought maybe the message is showing up here because I do not have enough
>>  data in the steam to force load onto the workers?
>> 
>> 
>>  Any in sites would be greatly appreciated.
>> 
>>  Andy
> 

Re: can I think of JavaDStream<> foreachRDD() as being 'for each mini batch' ?

[Sean Owen <so...@cloudera.com>]

Yes, foreachRDD will do your something for each RDD, which is what you
get for each mini-batch of input.

The operations you express on a DStream (or JavaDStream) are all,
really, "for each RDD", including print(). Logging is a little harder
to reason about since the logging will happen on a potentially remote
receiver. I am not sure if this explains your observed behavior; it
depends on what you were logging.

On Wed, Oct 1, 2014 at 6:51 PM, Andy Davidson
<An...@santacruzintegration.com> wrote:
> Hi
>
> I am new to Spark Streaming. Can I think of  JavaDStream<> foreachRDD() as
> being 'for each mini batch’? The java doc does not say much about this
> function.
>
> Here is the background. I am writing a little test program to figure out how
> to use streams. At some point I wanted to calculate an aggregate. In my
> first attempt my driver did a couple of transformations and tries to get the
> count(). I wrote this code like I would a spark core app and added some
> logging. My log statements where only executed once, how ever
> JavaDStream<>print() appears to run on each mini batch.
>
> When I put my logging and aggregation code inside foreachRDD() things work
> as expected my aggegrate and logging appear to be executed on each
> minibactch
>
> I am running on a 4 machine cluster.  I create a message with the value of
> each mini batch aggregate and use logging and System.out. Given the message
> shows up on my console is it safe to assume that this output code is
> executing in my driver? The ip shows it is running on my Master.
>
> I thought maybe the message is showing up here because I do not have enough
> data in the steam to force load onto the workers?
>
>
> Any in sites would be greatly appreciated.
>
> Andy

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org