Spark streaming Kafka receiver WriteAheadLog question

[Renyi Xiong <re...@gmail.com>]

Hi,

Is it possible for Kafka receiver generated WriteAheadLogBackedBlockRDD to
hold corresponded Kafka offset range so that during recovery the RDD can
refer back to Kafka queue instead of paying the cost of write ahead log?

I guess there must be a reason here. Could anyone please help me understand?

Thanks,
Renyi.

Re: Spark streaming Kafka receiver WriteAheadLog question

[Cody Koeninger <co...@koeninger.org>]

If you want to refer back to Kafka based on offset ranges, why not use
createDirectStream?

On Fri, Apr 22, 2016 at 11:49 PM, Renyi Xiong <re...@gmail.com> wrote:
> Hi,
>
> Is it possible for Kafka receiver generated WriteAheadLogBackedBlockRDD to
> hold corresponded Kafka offset range so that during recovery the RDD can
> refer back to Kafka queue instead of paying the cost of write ahead log?
>
> I guess there must be a reason here. Could anyone please help me understand?
>
> Thanks,
> Renyi.

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org