Using mapWithState without a checkpoint

[Daniel Haviv <da...@veracity-group.com>]

Hi,
Is it possible to use mapWithState without checkpointing at all ?
I'd rather have the whole application fail, restart and reload an
initialState RDD than pay for checkpointing every 10 batches.

Thank you,
Daniel

Re: Using mapWithState without a checkpoint

["Shixiong(Ryan) Zhu" <sh...@databricks.com>]

Even if you don't need the checkpointing data for recovery, "mapWithState"
still needs to use "checkpoint" to cut off the RDD lineage.

On Mon, Jan 23, 2017 at 12:30 AM, shyla deshpande <de...@gmail.com>
wrote:

> Hello spark users,
>
> I do have the same question as Daniel.
>
> I would like to save the state in Cassandra and on failure recover using
> the initialState. If some one has already tried this, please share your
> experience and sample code.
>
> Thanks.
>
> On Thu, Nov 17, 2016 at 9:45 AM, Daniel Haviv <
> daniel.haviv@veracity-group.com> wrote:
>
>> Hi,
>> Is it possible to use mapWithState without checkpointing at all ?
>> I'd rather have the whole application fail, restart and reload an
>> initialState RDD than pay for checkpointing every 10 batches.
>>
>> Thank you,
>> Daniel
>>
>
>

Re: Using mapWithState without a checkpoint

[shyla deshpande <de...@gmail.com>]

Hello spark users,

I do have the same question as Daniel.

I would like to save the state in Cassandra and on failure recover using
the initialState. If some one has already tried this, please share your
experience and sample code.

Thanks.

On Thu, Nov 17, 2016 at 9:45 AM, Daniel Haviv <
daniel.haviv@veracity-group.com> wrote:

> Hi,
> Is it possible to use mapWithState without checkpointing at all ?
> I'd rather have the whole application fail, restart and reload an
> initialState RDD than pay for checkpointing every 10 batches.
>
> Thank you,
> Daniel
>