Re: Distributed key valute store using bookkeeper

[Flavio Junqueira <fp...@apache.org>]

Hi Enrico,

Your suggestion of an embeddable kv-store backed by BooKeeper sounds nice. I don't think anyone has made open a system that does it and is production quality. Is the API of your kv-store just get, put, delete, or are you thinking of more sophisticated operations, like conditional writes, read-modify-writes, etc?

Thanks,
-Flavio

> On 26 Jan 2016, at 07:09, Enrico Olivelli <eo...@gmail.com> wrote:
> 
> Thank you. I was looking something like HBase but more simple and embeddable in a JVM. 
> My use case is the following:
> I have a service actually implemented as a group of JVM which in turn use a shared traditional Sql database like Mysql.
>  I do not want an external storage engine but I would like that data can be managed by the service jvms themselves which can  be added to the cluster dynamically.
> One advantage in having an embedded storage is that a single machine installation can be setup by only installing the JVM program. Actually we use H2 for this scenario but it cannot be used in production. Even for this single instance case we need at least a Mysql db which is really a burden for installers and for maintenance. 
> The second advantage is to have a storage that can be expanded very simply by adding service machines to the cluster.
> My typical cluster size is no more that 10 machines due to the nature of the service.
> 
> We are already using bookkeeper as core low level service for our business and I hoped that something was already available.
> 
> Thanks
> -- Enrico
> 
> 
> Il giorno Lun 25 Gen 2016 21:20 Sijie Guo <sijie@apache.org <ma...@apache.org>> ha scritto:
> Twitter Manhattan is using distributedlog (built upon bookkeeper) for strong consistent operations. 
> 
> https://blog.twitter.com/2014/manhattan-our-real-time-multi-tenant-distributed-database-for-twitter-scale <https://blog.twitter.com/2014/manhattan-our-real-time-multi-tenant-distributed-database-for-twitter-scale>
> https://blog.twitter.com/2015/building-distributedlog-twitter-s-high-performance-replicated-log-service <https://blog.twitter.com/2015/building-distributedlog-twitter-s-high-performance-replicated-log-service>
> 
> I think Yahoo has similar usage. 
> 
> Other than that, I am not aware about any open sourced production-quality key/value store that built upon bookkeeper.
> 
> - Sijie
> 
> 
> 
> On Mon, Jan 25, 2016 at 11:54 AM, Enrico Olivelli <eolivelli@gmail.com <ma...@gmail.com>> wrote:
> Hi, do you know any  production quality key value store built upon bookkeeper?
> 
> -- Enrico
> 

Re: Distributed key valute store using bookkeeper

[Enrico Olivelli <eo...@gmail.com>]

Actually we need at least CAS operations ("set foo='a' if foo='b'") and
something like "secondary" indexes.
Another important fact is that in my case write operations will be a lot
more than read operations.
We need to scale up to thousands of tables.
We do not need "joins".
We need to have transactions which span multiple tables but not potentially
all tables. We work in a multi-tenant environment and so tables can be
grouped by 'tenant', every read/write access is relative only to the tables
of a single tenant.
We would like physical data to reside "near" the JVMs which use data more
times.

As we are using a lot Bookkeeper we are evaluating to start a new (open
source) project which will implement the basic operations we need.

Thanks
- Enrico

2016-02-14 11:32 GMT+01:00 Flavio Junqueira <fp...@apache.org>:

> Hi Enrico,
>
> Your suggestion of an embeddable kv-store backed by BooKeeper sounds nice.
> I don't think anyone has made open a system that does it and is production
> quality. Is the API of your kv-store just get, put, delete, or are you
> thinking of more sophisticated operations, like conditional writes,
> read-modify-writes, etc?
>
> Thanks,
> -Flavio
>
> On 26 Jan 2016, at 07:09, Enrico Olivelli <eo...@gmail.com> wrote:
>
> Thank you. I was looking something like HBase but more simple and
> embeddable in a JVM.
> My use case is the following:
> I have a service actually implemented as a group of JVM which in turn use
> a shared traditional Sql database like Mysql.
>  I do not want an external storage engine but I would like that data can
> be managed by the service jvms themselves which can  be added to the
> cluster dynamically.
> One advantage in having an embedded storage is that a single machine
> installation can be setup by only installing the JVM program. Actually we
> use H2 for this scenario but it cannot be used in production. Even for this
> single instance case we need at least a Mysql db which is really a burden
> for installers and for maintenance.
> The second advantage is to have a storage that can be expanded very simply
> by adding service machines to the cluster.
> My typical cluster size is no more that 10 machines due to the nature of
> the service.
>
> We are already using bookkeeper as core low level service for our business
> and I hoped that something was already available.
>
> Thanks
> -- Enrico
>
>
> Il giorno Lun 25 Gen 2016 21:20 Sijie Guo <si...@apache.org> ha scritto:
>
>> Twitter Manhattan is using distributedlog (built upon bookkeeper) for
>> strong consistent operations.
>>
>>
>> https://blog.twitter.com/2014/manhattan-our-real-time-multi-tenant-distributed-database-for-twitter-scale
>>
>> https://blog.twitter.com/2015/building-distributedlog-twitter-s-high-performance-replicated-log-service
>>
>> I think Yahoo has similar usage.
>>
>> Other than that, I am not aware about any open sourced production-quality
>> key/value store that built upon bookkeeper.
>>
>> - Sijie
>>
>>
>>
>> On Mon, Jan 25, 2016 at 11:54 AM, Enrico Olivelli <eo...@gmail.com>
>> wrote:
>>
>>> Hi, do you know any  production quality key value store built upon
>>> bookkeeper?
>>>
>>> -- Enrico
>>>
>>
>>
>