Re: kafka streams application design on state stores

["Matthias J. Sax" <ma...@confluent.io>]

I am not sure if I understand your scenario. What do you exactly mean by
"back ground job" and by "sweeps the records in the store using range
calls" ?

As you noted, only the `Processor` that have access to the store should
write/delete anything from it. You should not try to modify the store
from "outside" (and this would only be possible if you break up the
public APIs anyway...)

Maybe it would help to understand in more detail what you try to achieve
to give better advice?


-Matthias


On 9/26/19 9:45 AM, ChienHsing Wu wrote:
> Hi,
> 
> I am using Kafka Stream's RocksDB state stores in an application and the is a back ground job that sweeps the records in the store using range calls. During processing, we need to ensure that a record is not included in the next range call when that background jobs runs next time. I understand that exposed state store interface to the application is read only and the main reason is to ensure the only way to change records inside RocksDB is from the stream topology. However, there is no way to guarantee stream processing of the removal of the record occurs before that background job runs again.
> 
> Only way I can think of is to remove that record in RocksDB directly.
> 
> I would like to hear any suggestions about
> 
> 
>   1.  If it's OK to remove records in RocksDB out side of the stream topology
>   2.  Any other suggestions to address my goal
> 
> Thanks in advance for your valuable inputs.
> 
> ChienHsing Wu
>