kafka streams application design on state stores

[ChienHsing Wu <ch...@opentext.com>]

Hi,

I am using Kafka Stream's RocksDB state stores in an application and the is a back ground job that sweeps the records in the store using range calls. During processing, we need to ensure that a record is not included in the next range call when that background jobs runs next time. I understand that exposed state store interface to the application is read only and the main reason is to ensure the only way to change records inside RocksDB is from the stream topology. However, there is no way to guarantee stream processing of the removal of the record occurs before that background job runs again.

Only way I can think of is to remove that record in RocksDB directly.

I would like to hear any suggestions about


  1.  If it's OK to remove records in RocksDB out side of the stream topology
  2.  Any other suggestions to address my goal

Thanks in advance for your valuable inputs.

ChienHsing Wu