You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by ponkin <al...@ya.ru> on 2015/11/24 10:46:16 UTC

[streaming] KafkaUtils.createDirectStream - how to start streming from checkpoints?

HI,

When I create stream with KafkaUtils.createDirectStream I can explicitly define the position "largest" or "smallest" - where to read topic from.
What if I have previous checkpoints( in HDFS for example) with offsets, and I want to start reading from the last checkpoint?
In source code of KafkaUtils.createDirectStream I see the following

 val reset = kafkaParams.get("auto.offset.reset").map(_.toLowerCase)
 
     (for {
       topicPartitions <- kc.getPartitions(topics).right
       leaderOffsets <- (if (reset == Some("smallest")) {
         kc.getEarliestLeaderOffsets(topicPartitions)
       } else {
         kc.getLatestLeaderOffsets(topicPartitions)
       }).right
...

So it turns out that, I have no options to start reading from checkpoints(and offsets)?
Am I right?
How can I force Spark to start reading from saved offesets(in checkpoints)? Is it possible at all or I need to store offsets in external datastore?

Alexey Ponkin




--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/streaming-KafkaUtils-createDirectStream-how-to-start-streming-from-checkpoints-tp25461.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

Re: [streaming] KafkaUtils.createDirectStream - how to start streming from checkpoints?

Posted by Понькин Алексей <al...@ya.ru>.
Great, thank you.
Sorry for being so inattentive) Need to read docs carefully.

-- 
Яндекс.Почта — надёжная почта
http://mail.yandex.ru/neo2/collect/?exp=1&t=1


24.11.2015, 15:15, "Deng Ching-Mallete" <oc...@apache.org>:
> Hi,
>
> If you wish to read from checkpoints, you need to use StreamingContext.getOrCreate(checkpointDir, functionToCreateContext) to create the streaming context that you pass in to KafkaUtils.createDirectStream(...). You may refer to http://spark.apache.org/docs/latest/streaming-programming-guide.html#checkpointing for an example.
>
> HTH,
> Deng
>
> On Tue, Nov 24, 2015 at 5:46 PM, ponkin <al...@ya.ru> wrote:
>> HI,
>>
>> When I create stream with KafkaUtils.createDirectStream I can explicitly define the position "largest" or "smallest" - where to read topic from.
>> What if I have previous checkpoints( in HDFS for example) with offsets, and I want to start reading from the last checkpoint?
>> In source code of KafkaUtils.createDirectStream I see the following
>>
>>  val reset = kafkaParams.get("auto.offset.reset").map(_.toLowerCase)
>>
>>      (for {
>>        topicPartitions <- kc.getPartitions(topics).right
>>        leaderOffsets <- (if (reset == Some("smallest")) {
>>          kc.getEarliestLeaderOffsets(topicPartitions)
>>        } else {
>>          kc.getLatestLeaderOffsets(topicPartitions)
>>        }).right
>> ...
>>
>> So it turns out that, I have no options to start reading from checkpoints(and offsets)?
>> Am I right?
>> How can I force Spark to start reading from saved offesets(in checkpoints)? Is it possible at all or I need to store offsets in external datastore?
>>
>> Alexey Ponkin
>>
>> ----------------------------------------
>> View this message in context: [streaming] KafkaUtils.createDirectStream - how to start streming from checkpoints?
>> Sent from the Apache Spark User List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org


Re: [streaming] KafkaUtils.createDirectStream - how to start streming from checkpoints?

Posted by Deng Ching-Mallete <oc...@apache.org>.
Hi,

If you wish to read from checkpoints, you need to use
StreamingContext.getOrCreate(checkpointDir, functionToCreateContext) to
create the streaming context that you pass in to
KafkaUtils.createDirectStream(...). You may refer to
http://spark.apache.org/docs/latest/streaming-programming-guide.html#checkpointing
for an example.

HTH,
Deng

On Tue, Nov 24, 2015 at 5:46 PM, ponkin <al...@ya.ru> wrote:

> HI,
>
> When I create stream with KafkaUtils.createDirectStream I can explicitly
> define the position "largest" or "smallest" - where to read topic from.
> What if I have previous checkpoints( in HDFS for example) with offsets,
> and I want to start reading from the last checkpoint?
> In source code of KafkaUtils.createDirectStream I see the following
>
>  val reset = kafkaParams.get("auto.offset.reset").map(_.toLowerCase)
>
>      (for {
>        topicPartitions <- kc.getPartitions(topics).right
>        leaderOffsets <- (if (reset == Some("smallest")) {
>          kc.getEarliestLeaderOffsets(topicPartitions)
>        } else {
>          kc.getLatestLeaderOffsets(topicPartitions)
>        }).right
> ...
>
> So it turns out that, I have no options to start reading from
> checkpoints(and offsets)?
> Am I right?
> How can I force Spark to start reading from saved offesets(in
> checkpoints)? Is it possible at all or I need to store offsets in external
> datastore?
>
> Alexey Ponkin
>
> ------------------------------
> View this message in context: [streaming] KafkaUtils.createDirectStream -
> how to start streming from checkpoints?
> <http://apache-spark-user-list.1001560.n3.nabble.com/streaming-KafkaUtils-createDirectStream-how-to-start-streming-from-checkpoints-tp25461.html>
> Sent from the Apache Spark User List mailing list archive
> <http://apache-spark-user-list.1001560.n3.nabble.com/> at Nabble.com.
>

Re: [streaming] KafkaUtils.createDirectStream - how to start streming from checkpoints?

Posted by Cody Koeninger <co...@koeninger.org>.
Just to be clear, spark checkpoints have nothing to do with zookeeper,
they're stored in the filesystem you specify.

On Sun, Dec 6, 2015 at 1:25 AM, manasdebashiskar <po...@gmail.com>
wrote:

> When you enable check pointing your offsets get written in zookeeper. If
> you
> program dies or shutdowns and later restarted kafkadirectstream api knows
> where to start by looking at those offsets from zookeeper.
>
> This is as easy as it gets.
> However if you are planning to re-use the same checkpoint folder among
> different spark version that is currently not supported.
> In that case you might want to go for writing the offset and topic in your
> favorite database. Assuming that DB is high available you can later retried
> the previously worked offset and start from there.
>
> Take a look at the blog post of cody.(the guy who wrote kafkadirectstream)
> https://github.com/koeninger/kafka-exactly-once/blob/master/blogpost.md
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/streaming-KafkaUtils-createDirectStream-how-to-start-streming-from-checkpoints-tp25461p25597.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>

Re: [streaming] KafkaUtils.createDirectStream - how to start streming from checkpoints?

Posted by manasdebashiskar <po...@gmail.com>.
When you enable check pointing your offsets get written in zookeeper. If you
program dies or shutdowns and later restarted kafkadirectstream api knows
where to start by looking at those offsets from zookeeper.

This is as easy as it gets.
However if you are planning to re-use the same checkpoint folder among
different spark version that is currently not supported.
In that case you might want to go for writing the offset and topic in your
favorite database. Assuming that DB is high available you can later retried
the previously worked offset and start from there.

Take a look at the blog post of cody.(the guy who wrote kafkadirectstream)
https://github.com/koeninger/kafka-exactly-once/blob/master/blogpost.md



--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/streaming-KafkaUtils-createDirectStream-how-to-start-streming-from-checkpoints-tp25461p25597.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org