You are viewing a plain text version of this content. The canonical link for it is here.

Posted to users@kafka.apache.org by Eli Jordan <el...@gmail.com> on 2017/08/24 23:57:04 UTC

Kafka Streams: Pseudo Wallclock Punctuate

Hi

I have a kafka streams application, that uses a couple of stateful (i.e. using key value stores) custom transformer implementations.

I'm aware that the scheduling of the punctuate method is based on stream time. However, for my use case, this is not what I want, I need wallclock scheduling.

My question is, what is the recommended way to achieve this, until KIP138 provides first class support for wallclock punctuate?

At the moment, I schedule a task to run using ScheduledExecutorService. However, I'm uncertain whether it is safe to access and modify a KeyValueStore from a non kafka streams thread. Is this a safe approach, or will there be problems?

Thanks
Eli

Re: Kafka Streams: Pseudo Wallclock Punctuate

Posted by Eli Jordan <el...@gmail.com>.

Thanks Matthias

Yeah I also had this idea, and started implementing it. Hopefully it works out.

Thanks
Eli

> On 26 Aug 2017, at 04:25, Matthias J. Sax <ma...@confluent.io> wrote:
> 
> Eli,
> 
> One think you could do, is to send "tick tuples" through your topology
> and use WallclockTimestampExtractor. It's not a nice solution, but I
> don't have any better idea atm.
> 
> 
> -Matthias
> 
>> On 8/24/17 9:37 PM, Eli Jordan wrote:
>> Update on this. Modifying the state store on another thread actually doesn’t work. I receive the error
>> 
>> java.lang.IllegalStateException: This should not happen as offset() should only be called while a record is processed
>>    at org.apache.kafka.streams.processor.internals.AbstractProcessorContext.offset(AbstractProcessorContext.java:138)
>>    at org.apache.kafka.streams.state.internals.CachingKeyValueStore.put(CachingKeyValueStore.java:203)
>>    at org.apache.kafka.streams.state.internals.CachingKeyValueStore.put(CachingKeyValueStore.java:198)
>> 
>> So is there a way to do this?
>> 
>>> On 25 Aug 2017, at 9:57 am, Eli Jordan <el...@gmail.com> wrote:
>>> 
>>> Hi
>>> 
>>> I have a kafka streams application, that uses a couple of stateful (i.e. using key value stores) custom transformer implementations.
>>> 
>>> I'm aware that the scheduling of the punctuate method is based on stream time. However, for my use case, this is not what I want, I need wallclock scheduling.
>>> 
>>> My question is, what is the recommended way to achieve this, until KIP138 provides first class support for wallclock punctuate?
>>> 
>>> At the moment, I schedule a task to run using ScheduledExecutorService. However, I'm uncertain whether it is safe to access and modify a KeyValueStore from a non kafka streams thread. Is this a safe approach, or will there be problems?
>>> 
>>> Thanks
>>> Eli
>> 
>

Re: Kafka Streams: Pseudo Wallclock Punctuate

Posted by "Matthias J. Sax" <ma...@confluent.io>.

Eli,

One think you could do, is to send "tick tuples" through your topology
and use WallclockTimestampExtractor. It's not a nice solution, but I
don't have any better idea atm.


-Matthias

On 8/24/17 9:37 PM, Eli Jordan wrote:
> Update on this. Modifying the state store on another thread actually doesn’t work. I receive the error
> 
> java.lang.IllegalStateException: This should not happen as offset() should only be called while a record is processed
> 	at org.apache.kafka.streams.processor.internals.AbstractProcessorContext.offset(AbstractProcessorContext.java:138)
> 	at org.apache.kafka.streams.state.internals.CachingKeyValueStore.put(CachingKeyValueStore.java:203)
> 	at org.apache.kafka.streams.state.internals.CachingKeyValueStore.put(CachingKeyValueStore.java:198)
> 
> So is there a way to do this?
> 
>> On 25 Aug 2017, at 9:57 am, Eli Jordan <el...@gmail.com> wrote:
>>
>> Hi
>>
>> I have a kafka streams application, that uses a couple of stateful (i.e. using key value stores) custom transformer implementations.
>>
>> I'm aware that the scheduling of the punctuate method is based on stream time. However, for my use case, this is not what I want, I need wallclock scheduling.
>>
>> My question is, what is the recommended way to achieve this, until KIP138 provides first class support for wallclock punctuate?
>>
>> At the moment, I schedule a task to run using ScheduledExecutorService. However, I'm uncertain whether it is safe to access and modify a KeyValueStore from a non kafka streams thread. Is this a safe approach, or will there be problems?
>>
>> Thanks
>> Eli
>

Re: Kafka Streams: Pseudo Wallclock Punctuate

Posted by Eli Jordan <el...@gmail.com>.

Update on this. Modifying the state store on another thread actually doesn’t work. I receive the error

java.lang.IllegalStateException: This should not happen as offset() should only be called while a record is processed
	at org.apache.kafka.streams.processor.internals.AbstractProcessorContext.offset(AbstractProcessorContext.java:138)
	at org.apache.kafka.streams.state.internals.CachingKeyValueStore.put(CachingKeyValueStore.java:203)
	at org.apache.kafka.streams.state.internals.CachingKeyValueStore.put(CachingKeyValueStore.java:198)

So is there a way to do this?

> On 25 Aug 2017, at 9:57 am, Eli Jordan <el...@gmail.com> wrote:
> 
> Hi
> 
> I have a kafka streams application, that uses a couple of stateful (i.e. using key value stores) custom transformer implementations.
> 
> I'm aware that the scheduling of the punctuate method is based on stream time. However, for my use case, this is not what I want, I need wallclock scheduling.
> 
> My question is, what is the recommended way to achieve this, until KIP138 provides first class support for wallclock punctuate?
> 
> At the moment, I schedule a task to run using ScheduledExecutorService. However, I'm uncertain whether it is safe to access and modify a KeyValueStore from a non kafka streams thread. Is this a safe approach, or will there be problems?
> 
> Thanks
> Eli