How to replay stream between 2 offsets?

[ranjitreddy <ra...@yahoo.com>]

I'm using Spark(2.1.1)  streaming as Consumer for Kakfa messages.

I've a data pipeline where I continuously (hourly) save offsets to  HBase.
So that I can replay stream from certain offset in case of errors in data
transformations.

I'm able to start the stream at a certain offset, but don't know of a way to
stop the stream after I reach the end offset. I'm using the following API:

        Map<TopicPartition, Long> replayOffset = new HashMap<>();
        replayOffsets.put(new TopicPartition(topicPrefix, 0), offsets[0]);

        List<TopicPartition> topicPartitionList = new ArrayList<>();
        topicPartitionList.add(new TopicPartition(topicPrefix, 0));

        JavaInputDStream<ConsumerRecord&lt;String, String>> messages =
KafkaUtils.createDirectStream(
            jssc,
            LocationStrategies.PreferConsistent(),
            ConsumerStrategies.<String, String>Assign(topicPartitionList,
kafkaParams, replayOffset)
        );

I would like to know if there is an API or a solution for this use case.

Thanks,

-Ranjit





--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/How-to-replay-stream-between-2-offsets-tp28685.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org