[jira] [Commented] (KAFKA-15215) The default.dsl.store config is not compatible with custom state stores

["Guozhang Wang (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KAFKA-15215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17746102#comment-17746102 ] 

Guozhang Wang commented on KAFKA-15215:
---------------------------------------

[~ableegoldman] Thanks for bringing this up! I agree that it would be very beneficial to customized stores with a configurable store type, what we need to figure out though is how to ask users to provide the customized store impls. Some quick options on top of my head:

1. Ask users to "register" a set of customized stores, one for each store type (i.e. kv, time-ranged, versioned, etc) with a naming convention like a prefix so that if the dsl.store configured as "XYZ", the runtime would try to find such impl classes named as "XYZ.keyvalue.." in class loading.

2. Provide a "StoreImplSelector" API to extend (or replace) the config-based "default.dsl.store". Its default implementation could still follow the config value, to return the corresponding in-mem / rocksDB impl classes given the store type; users can override this API to select new impl classes of their own (again, potentially still rely on the config with new config values; and if the corresponding config name is not expected, then error out).

Personally I feel just among these two, the second option seems a clear winner --- of course there might be other smarter options as we brainstorm more --- since it a) would not enforce users to provide one impl classes for each possible store type, b) also as a quality bar to restrict us trying to add more store types casually in the future as it will impact all the API instantiations.

> The default.dsl.store config is not compatible with custom state stores
> -----------------------------------------------------------------------
>
>                 Key: KAFKA-15215
>                 URL: https://issues.apache.org/jira/browse/KAFKA-15215
>             Project: Kafka
>          Issue Type: New Feature
>          Components: streams
>            Reporter: A. Sophie Blee-Goldman
>            Assignee: Almog Gavra
>            Priority: Major
>              Labels: needs-kip
>
> Sort of a bug, sort of a new/missing feature. When we added the long-awaited default.dsl.store config, it was decided to scope the initial KIP to just the two out-of-the-box state stores types offered by Streams, rocksdb and in-memory. The reason being that this would address a large number of the relevant use cases, and could always be followed up with another KIP for custom state stores if/when the demand arose.
> Of course, since rocksdb is the default anyways, the only beneficiaries of this KIP right now are the people who specifically want only in-memory stores – yet custom state stores users are probably by far the ones with the greatest need for an easier way to configure the store type across an entire application. And unfortunately, because the config currently relies on enum definitions for the known OOTB store types, there's not really any way to extend this feature as it is to work with custom implementations.
> I think this is a great feature, which is why I hope to see it extended to the broader user base. Most likely we'll want to introduce a new config for this, though whether it replaces the old default.dsl.store config or complements it will have to be decided during the KIP discussion



--
This message was sent by Atlassian Jira
(v8.20.10#820010)