[jira] [Updated] (KAFKA-14976) Left/outer stream-stream joins create KV stores that aren't customizable

["A. Sophie Blee-Goldman (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/KAFKA-14976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

A. Sophie Blee-Goldman updated KAFKA-14976:
-------------------------------------------
    Labels: needs-kip  (was: )

> Left/outer stream-stream joins create KV stores that aren't customizable
> ------------------------------------------------------------------------
>
>                 Key: KAFKA-14976
>                 URL: https://issues.apache.org/jira/browse/KAFKA-14976
>             Project: Kafka
>          Issue Type: Bug
>          Components: streams
>            Reporter: A. Sophie Blee-Goldman
>            Priority: Major
>              Labels: needs-kip
>
> It appears that we only give the illusion of full customizability when it comes to the state stores of a windowed join. This arose due to an [optimization|https://github.com/apache/kafka/pull/11252] for the performance of the spurious results fix, and means that these joins now come with one additional, and possibly unexpected, state store:
>  
> {code:java}
> final StoreBuilder<KeyValueStore<TimestampedKeyAndJoinSide<K>, LeftOrRightValue<V1, V2>>> builder =
>             new ListValueStoreBuilder<>(
>          |--[   persistent ? 
> this-->  |         Stores.persistentKeyValueStore(storeName) : 
>          |--[      Stores.inMemoryKeyValueStore(storeName),
>                 timestampedKeyAndJoinSideSerde,
>                 leftOrRightValueSerde,
>                 Time.SYSTEM
>             ); {code}
>  
> where persistent is defined above that as
> {code:java}
> final boolean persistent = streamJoinedInternal.thisStoreSupplier() == null || streamJoinedInternal.thisStoreSupplier().get().persistent(); {code}
>  
> This means regardless of whether a custom state store implementation was passed in to the join, we will still insert one of our RocksDB or InMemory state stores. Which might be very surprising since the API makes it seem like the underlying stores are fully configurable.
> I'm adding a warning line for this in PR [#13682|https://github.com/apache/kafka/pull/13682/files#diff-9ce43046fdef1233ab762e728abd1d3d44d7c270b28dcf6b63aa31a93a30af07R334-R336] but we should really make this hidden state store fully configurable like the window stores currently are (which will require a KIP)
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)