You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by V Dineshkumar <de...@gmail.com> on 2015/02/24 06:41:03 UTC
Write ahead Logs and checkpoint
Hi,
My spark streaming application is pulling data from Kafka.To prevent data
loss I have implemented WAL and enable checkpointing.On killing my
application and restarting it I am able to prevent data loss now but
however I am getting duplicate messages.
Is it because the application got killed before it was able checkpoint the
current processing state??
If yes how to tackle the duplicate messages?
Thanks,
Dinesh
Re: Write ahead Logs and checkpoint
Posted by Tathagata Das <td...@databricks.com>.
Exactly, that is the reason.
To avoid that, in Spark 1.3 to-be-released, we have added a new Kafka API
(called direct stream) which does not use Zookeeper at all to keep track of
progress, and maintains offset within Spark Streaming. That can guarantee
all records being received exactly-once. Its experimental for now, but we
will make it stable. Please try it out.
TD
On Mon, Feb 23, 2015 at 9:41 PM, V Dineshkumar <developer.dinesh89@gmail.com
> wrote:
> Hi,
>
> My spark streaming application is pulling data from Kafka.To prevent data
> loss I have implemented WAL and enable checkpointing.On killing my
> application and restarting it I am able to prevent data loss now but
> however I am getting duplicate messages.
>
> Is it because the application got killed before it was able checkpoint the
> current processing state??
> If yes how to tackle the duplicate messages?
>
> Thanks,
> Dinesh
>