You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@cassandra.apache.org by Dikang Gu <di...@gmail.com> on 2017/11/03 18:22:48 UTC
Pluggable storage engine discussion
Hi,
We are having discussions about the pluggable storage engine plan on the
jira: https://issues.apache.org/jira/browse/CASSANDRA-13475.
We are trying to figure out a plan for the pluggable storage engine effort.
Right now, the discussion is mainly happening between couple C* committers,
like Blake and me. But I want to increase the visibility, and I'm very
welcome more developers to be involved in the discussion. It will help us
on moving forward on this effort.
Also, I have a quip as a (very high level) design doc for this project.
https://quip.com/bhw5ABUCi3co
Thanks
Dikang.
Re: Pluggable storage engine discussion
Posted by Dikang Gu <di...@gmail.com>.
Hi Stefan,
Yeah, my team has been working for 6+ months on the development and testing
of the RocksDB based storage engine, and we are in the process of rolling
it out to our production deployment inside Instagram! We see huge gains in
reducing the latency and footprint of our Cassandra clusters. I will
publish those when I have more metrics after the rollout.
I do hope someone in community can try it out as well, in your shadow
cluster of course. We have pushed the code here:
https://github.com/Instagram/cassandra/commits/rocks_3.0 .
On Fri, Nov 3, 2017 at 1:48 PM, Stefan Podkowinski <sp...@apache.org> wrote:
> Hi Dikang
>
> Have you been able to continue evaluating RocksDB? I'm afraid we might
> be a bit too much ahead in the discussion by already talking about a
> pluggable architecture, while we haven't fully evaluated yet if we can
> and want to support an alternative RocksDB engine implementation at all.
> Because if we don't, we also don't need a pluggable architecture at this
> point, do we? There's little to be gained from a major refactoring, just
> to find out that alternative engines we thought of didn't turn out to be
> a good fit for production for whatever reasons.
>
> On the other hand, if RocksDB is (by whatever standards) a better
> storage implementation, why not completely switch, instead of just
> making it an option? But if it's not, is a major refactoring still worth
> it?
>
>
> On 03.11.17 19:22, Dikang Gu wrote:
> > Hi,
> >
> > We are having discussions about the pluggable storage engine plan on the
> > jira: https://issues.apache.org/jira/browse/CASSANDRA-13475.
> >
> > We are trying to figure out a plan for the pluggable storage engine
> effort.
> > Right now, the discussion is mainly happening between couple C*
> committers,
> > like Blake and me. But I want to increase the visibility, and I'm very
> > welcome more developers to be involved in the discussion. It will help us
> > on moving forward on this effort.
> >
> > Also, I have a quip as a (very high level) design doc for this project.
> > https://quip.com/bhw5ABUCi3co
> >
> > Thanks
> > Dikang.
> >
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@cassandra.apache.org
> For additional commands, e-mail: dev-help@cassandra.apache.org
>
>
--
Dikang
Re: Pluggable storage engine discussion
Posted by Michael Burman <mi...@redhat.com>.
Hi,
There's a ticket also for columnar storage option, which I guess is
something that many might want. Not least because in many cases it could
reduce the storage footprint by a large margin (and enable more
sophisticated compression options), even if we discount the possible
query advantages. For range queries it might be seriously faster - as
Evan Chan reported with FiloDB and I have personally done in
Hawkular-Metrics also (storing "columnar blocks" in a single cell lead
to ~95% reduction in query times for any non-trivial query even when it
had to be deserialized at the client side).
CASSANDRA-7447
While anything can obviously be built on top of any storage engine,
that's not necessarily the most effective way.
- Micke
On 11/03/2017 10:48 PM, Stefan Podkowinski wrote:
> Hi Dikang
>
> Have you been able to continue evaluating RocksDB? I'm afraid we might
> be a bit too much ahead in the discussion by already talking about a
> pluggable architecture, while we haven't fully evaluated yet if we can
> and want to support an alternative RocksDB engine implementation at all.
> Because if we don't, we also don't need a pluggable architecture at this
> point, do we? There's little to be gained from a major refactoring, just
> to find out that alternative engines we thought of didn't turn out to be
> a good fit for production for whatever reasons.
>
> On the other hand, if RocksDB is (by whatever standards) a better
> storage implementation, why not completely switch, instead of just
> making it an option? But if it's not, is a major refactoring still worth it?
>
>
> On 03.11.17 19:22, Dikang Gu wrote:
>> Hi,
>>
>> We are having discussions about the pluggable storage engine plan on the
>> jira: https://issues.apache.org/jira/browse/CASSANDRA-13475.
>>
>> We are trying to figure out a plan for the pluggable storage engine effort.
>> Right now, the discussion is mainly happening between couple C* committers,
>> like Blake and me. But I want to increase the visibility, and I'm very
>> welcome more developers to be involved in the discussion. It will help us
>> on moving forward on this effort.
>>
>> Also, I have a quip as a (very high level) design doc for this project.
>> https://quip.com/bhw5ABUCi3co
>>
>> Thanks
>> Dikang.
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@cassandra.apache.org
> For additional commands, e-mail: dev-help@cassandra.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@cassandra.apache.org
For additional commands, e-mail: dev-help@cassandra.apache.org
Re: Pluggable storage engine discussion
Posted by Stefan Podkowinski <sp...@apache.org>.
Hi Dikang
Have you been able to continue evaluating RocksDB? I'm afraid we might
be a bit too much ahead in the discussion by already talking about a
pluggable architecture, while we haven't fully evaluated yet if we can
and want to support an alternative RocksDB engine implementation at all.
Because if we don't, we also don't need a pluggable architecture at this
point, do we? There's little to be gained from a major refactoring, just
to find out that alternative engines we thought of didn't turn out to be
a good fit for production for whatever reasons.
On the other hand, if RocksDB is (by whatever standards) a better
storage implementation, why not completely switch, instead of just
making it an option? But if it's not, is a major refactoring still worth it?
On 03.11.17 19:22, Dikang Gu wrote:
> Hi,
>
> We are having discussions about the pluggable storage engine plan on the
> jira: https://issues.apache.org/jira/browse/CASSANDRA-13475.
>
> We are trying to figure out a plan for the pluggable storage engine effort.
> Right now, the discussion is mainly happening between couple C* committers,
> like Blake and me. But I want to increase the visibility, and I'm very
> welcome more developers to be involved in the discussion. It will help us
> on moving forward on this effort.
>
> Also, I have a quip as a (very high level) design doc for this project.
> https://quip.com/bhw5ABUCi3co
>
> Thanks
> Dikang.
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@cassandra.apache.org
For additional commands, e-mail: dev-help@cassandra.apache.org