You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Sean Owen <so...@cloudera.com> on 2015/02/23 10:03:46 UTC
Re: Why must the dstream.foreachRDD(...) parameter be serializable?
I reached a similar conclusion about checkpointing . It requires your
entire computation to be serializable, even all of the 'local' bits.
Which makes sense. In my case I do not use checkpointing and it is
fine to restart the driver in the case of failure and not try to
recover its state.
What I haven't investigated is whether you can enable checkpointing
for the state in updateStateByKey separately from this mechanism,
which is exactly your question. What happens if you set a checkpoint
dir, but do *not* use StreamingContext.getOrCreate, but *do* call
DStream.checkpoint?
On Mon, Feb 23, 2015 at 8:47 AM, Tobias Pfeiffer <tg...@preferred.jp> wrote:
> Hi,
>
> On Wed, Jan 28, 2015 at 7:01 PM, Sean Owen <so...@cloudera.com> wrote:
>>
>> PS see https://issues.apache.org/jira/browse/SPARK-4196 for an example.
>
>
> OK, I am only realizing now the huge effect this has on my project. Even the
> simplest
> stream.foreachRDD( ... logger.info(...) ... )
> pieces need to be rewritten when checkpointing is enabled, this is not so
> good. However, it should be possible, I think.
>
> But, I think this makes it impossible to use SparkSQL together with Spark
> Streaming, as every
> stream.foreachRDD( ... sqlc.sql("SELECT ...") ...)
> will fail with
> java.io.NotSerializableException:
> scala.util.parsing.combinator.Parsers$$anon$3
> etc.
>
> Actually I don't want checkpointing/failure recovery, I only want to use
> updateStateByKey. Is there maybe any workaround to get updateStateByKey
> working while keeping my foreachRDD functions as they are?
>
> Thanks
> Tobias
>
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: Why must the dstream.foreachRDD(...) parameter be serializable?
Posted by Tobias Pfeiffer <tg...@preferred.jp>.
Hi,
On Tue, Feb 24, 2015 at 4:34 AM, Tathagata Das <ta...@gmail.com>
wrote:
> There are different kinds of checkpointing going on. updateStateByKey
> requires RDD checkpointing which can be enabled only by called
> sparkContext.setCheckpointDirectory. But that does not enable Spark
> Streaming driver checkpoints, which is necessary for recovering from driver
> failures. That is enabled only by streamingContext.checkpoint(...) which
> internally calls sparkContext.setCheckpointDirectory and also enables other
> stuff.
>
I see, thank you very much! I'm happy to see I will not have to rewrite the
entire application :-)
Tobias
Re: Why must the dstream.foreachRDD(...) parameter be serializable?
Posted by Tathagata Das <ta...@gmail.com>.
There are different kinds of checkpointing going on. updateStateByKey
requires RDD checkpointing which can be enabled only by called
sparkContext.setCheckpointDirectory. But that does not enable Spark
Streaming driver checkpoints, which is necessary for recovering from driver
failures. That is enabled only by streamingContext.checkpoint(...) which
internally calls sparkContext.setCheckpointDirectory and also enables other
stuff.
TD
On Mon, Feb 23, 2015 at 1:28 AM, Tobias Pfeiffer <tg...@preferred.jp> wrote:
> Sean,
>
> thanks for your message!
>
> On Mon, Feb 23, 2015 at 6:03 PM, Sean Owen <so...@cloudera.com> wrote:
>>
>> What I haven't investigated is whether you can enable checkpointing
>> for the state in updateStateByKey separately from this mechanism,
>> which is exactly your question. What happens if you set a checkpoint
>> dir, but do *not* use StreamingContext.getOrCreate, but *do* call
>> DStream.checkpoint?
>>
>
> I didn't even use StreamingContext.getOrCreate(), just calling
> streamingContext.checkpoint(...) blew everything up. Well, "blew up" in the
> sense that actor.OneForOneStrategy will print the stack trace of
> the java.io.NotSerializableException every couple of seconds and
> "something" is not going right with execution (I think).
>
> BUT, indeed, just calling sparkContext.setCheckpointDir seems to be
> sufficient for updateStateByKey! Looking at what
> streamingContext.checkpoint() does, I don't get why ;-) and I am not sure
> that this is a robust solution, but in fact that seems to work!
>
> Thanks a lot,
> Tobias
>
>
Re: Why must the dstream.foreachRDD(...) parameter be serializable?
Posted by Tobias Pfeiffer <tg...@preferred.jp>.
Sean,
thanks for your message!
On Mon, Feb 23, 2015 at 6:03 PM, Sean Owen <so...@cloudera.com> wrote:
>
> What I haven't investigated is whether you can enable checkpointing
> for the state in updateStateByKey separately from this mechanism,
> which is exactly your question. What happens if you set a checkpoint
> dir, but do *not* use StreamingContext.getOrCreate, but *do* call
> DStream.checkpoint?
>
I didn't even use StreamingContext.getOrCreate(), just calling
streamingContext.checkpoint(...) blew everything up. Well, "blew up" in the
sense that actor.OneForOneStrategy will print the stack trace of
the java.io.NotSerializableException every couple of seconds and
"something" is not going right with execution (I think).
BUT, indeed, just calling sparkContext.setCheckpointDir seems to be
sufficient for updateStateByKey! Looking at what
streamingContext.checkpoint() does, I don't get why ;-) and I am not sure
that this is a robust solution, but in fact that seems to work!
Thanks a lot,
Tobias