[jira] [Commented] (SAMZA-938) KV-store API changes should be backward compatible to 0.10

["Jake Maes (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SAMZA-938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15250599#comment-15250599 ] 

Jake Maes commented on SAMZA-938:
---------------------------------

Taking a step back, it seems what SAMZA-813 was really trying to accomplish is to optimize performance for using RocksDB as a queue. 

Unfortunately, the patch exposed new methods in all the interfaces and store implementations that are only actually functional in the RocksDB implementation (and really only needed for a store that has the same tombstoning/compaction semantics as RocksDB). I personally don't think this is the way to go, even for a major release where backward incompatible changes are permissible. Even if we split the new methods into a separate interface, all the extension classes (LoggedStore, CachedStore) would need to have a bunch of "instanceof" checks to avoid calling the method on instances that only implement the old interfaces. 

Interestingly, there were some suggestions for the queue use case posted last year:
https://github.com/facebook/rocksdb/wiki/Implement-Queue-Service-Using-RocksDB

>From those suggestions, it seems we could implement a subclass of RocksDBKeyValueStore which employs those optimizations without changing the KV store interfaces. Namely, it could persist the start/end SCNs of the queue and the iterator would only iterate over that range. We also might want to enforce that only the most recently accessed key is deleted (to enforce queue semantics). This way, users can interact with the store just like any other KVStore, primarily using the existing put(), delete(), and iterate() methods. I think this is much cleaner. 

I think it will be a fair amount of work to implement and test this queue implementation though, so we could either bite the bullet now or back out the SAMZA-813 patch and revisit the suggested solution after the release. 

@yipan, thoughts? Do you agree on the suggestion for a queue-oriented store? Do you think we should prioritize it now, or save it for after the release?

> KV-store API changes should be backward compatible to 0.10
> ----------------------------------------------------------
>
>                 Key: SAMZA-938
>                 URL: https://issues.apache.org/jira/browse/SAMZA-938
>             Project: Samza
>          Issue Type: Bug
>    Affects Versions: 0.10.1
>            Reporter: Yi Pan (Data Infrastructure)
>            Assignee: Jake Maes
>             Fix For: 0.10.1
>
>
> SAMZA-813 introduced new methods in KeyValueStore and KeyValueIterator interface classes, which in JDK7, breaks the backward compatibility for customized implementation of KeyValueStore and KeyValueIterator.
> As 0.10.1 is a bug-fix release, we should make sure that the backward compatibility is retained. Hence, it would be better to have the extended interface classes to implement the seekable functions, instead of changing the existing classes.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)