[jira] [Commented] (KAFKA-3522) Consider adding version information into rocksDB storage format

["Ewen Cheslack-Postava (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KAFKA-3522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15823413#comment-15823413 ] 

Ewen Cheslack-Postava commented on KAFKA-3522:
----------------------------------------------

[~imandhan] There's a KIP proposal for exactly once semantics, so now might be a good time to restart this discussion. Although I'm not clear how they interact since this seems tied to local state that can easily be rebuilt, perhaps [~guozhang] can expand on his thoughts. Regardless, I dropped priority here because the proposed solution is easily added incrementally by first introducing the extra on-disk info (in any upcoming version) and then adding support for a newer RocksDB format in a later version and making the upgrade path required clear to users via release notes (and possibly configs, depending on what support for different versions looked like in RocksDB).

> Consider adding version information into rocksDB storage format
> ---------------------------------------------------------------
>
>                 Key: KAFKA-3522
>                 URL: https://issues.apache.org/jira/browse/KAFKA-3522
>             Project: Kafka
>          Issue Type: Bug
>          Components: streams
>            Reporter: Guozhang Wang
>            Assignee: Ishita Mandhan
>              Labels: architecture
>             Fix For: 0.10.2.0
>
>
> Kafka Streams does not introduce any modifications to the data format in the underlying Kafka protocol, but it does use RocksDB for persistent state storage, and currently its data format is fixed and hard-coded. We want to consider the evolution path in the future we we change the data format, and hence having some version info stored along with the storage file / directory would be useful.
> And this information could be even out of the storage file; for example, we can just use a small "version indicator" file in the rocksdb directory for this purposes. Thoughts? [~enothereska] [~jkreps]



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)