The last batch of data is never qualified for processing by the punctuate method. - Kafka Streams

[Harish Subramaniam <ha...@autogeneral.com.au>]

Hi Team

I am seeing inconsistent behaviour in the punctuate method , let me explain
the issue

Use Case :

a) I am getting data from a topic X .
b) Processor receives the data as on when the data is published by the
KafkaProducer (each an every data is received on real time , no problem
with that) . Data is sent with a delay of 1 sec.

c) Punctuate method is supposed to be called every 2 seconds while the data
publishing takes place until the last data is published in the topic .
However , the last message published in the topic is not processed by the
punctuate method.

Comments received for processing : 12345 : Line Number : 9Line Text : 9 - Log
in the process method

Comments processed 2017-08-22T12:44:52.697 - Log in the punctuate method

Comments received for processing : 4567 : Line Number : 9Line Text : 9 - - Log
in the process method

Comments received for processing : 12345 : Line Number : 10Line Text : 10 -
- Log in the process method
Comments received for processing : 4567 : Line Number : 10Line Text : 10 --
- Log in the process method

------ No Logs for the last batch to be processed by punctuate method
---------

I am using the below Kafka - Streams dependency :

//Kafka
compile("org.apache.kafka:kafka-streams:0.11.0.0")

Regards,
Harish

-- 


This email is sent by Auto & General Insurance Company Ltd, Auto & General Services Pty Ltd, Auto & General Holdings Pty Ltd or a related body corporate (Auto & General) and is for the intended addressee.
The views expressed in this email and attachments (email) reflect the views of the stated author but may not reflect views of Auto & General. This email is confidential and subject to copyright. 
It may be privileged. If you are not the intended addressee, confidentiality and privilege have not been waived and any use, interference with, or disclosure of this email is unauthorised. 
If you are not the intended addressee please immediately notify the sender and then delete the email. Auto & General does not warrant that this email is error or virus free.

Re: The last batch of data is never qualified for processing by the punctuate method. - Kafka Streams

["Matthias J. Sax" <ma...@confluent.io>]

The call to punctuate depend on progress on "stream time" (internally
tracked progress of time, depending whatever is returned by
TimestampExtractor).

Thus, if you don't receive any more date, "stream time" does not make
progress anymore and thus, punctuate is not called anymore.

Does this explain your observation?


-Matthias


On 8/21/17 8:25 PM, Harish Subramaniam wrote:
> Hi Team
> 
> I am seeing inconsistent behaviour in the punctuate method , let me explain
> the issue
> 
> Use Case :
> 
> a) I am getting data from a topic X .
> b) Processor receives the data as on when the data is published by the
> KafkaProducer (each an every data is received on real time , no problem
> with that) . Data is sent with a delay of 1 sec.
> 
> c) Punctuate method is supposed to be called every 2 seconds while the data
> publishing takes place until the last data is published in the topic .
> However , the last message published in the topic is not processed by the
> punctuate method.
> 
> Comments received for processing : 12345 : Line Number : 9Line Text : 9 - Log
> in the process method
> 
> Comments processed 2017-08-22T12:44:52.697 - Log in the punctuate method
> 
> Comments received for processing : 4567 : Line Number : 9Line Text : 9 - - Log
> in the process method
> 
> Comments received for processing : 12345 : Line Number : 10Line Text : 10 -
> - Log in the process method
> Comments received for processing : 4567 : Line Number : 10Line Text : 10 --
> - Log in the process method
> 
> ------ No Logs for the last batch to be processed by punctuate method
> ---------
> 
> I am using the below Kafka - Streams dependency :
> 
> //Kafka
> compile("org.apache.kafka:kafka-streams:0.11.0.0")
> 
> Regards,
> Harish
>