AW: Gora Cassandra module design rationale

["Zirpins. Christian" <c....@seeburger.de>]

Hi Lewis,

I've finished the implementation of a Gora feature for Cassandra composite primary keys as discussed before.

The extension allows to define primary keys that are represented by avro classes. A mapping specifies how fields of the key class are mapped to the components of composite partition keys and composite column names. This gives users more control with respect to the distribution of data into Cassandra database structures. It is now possible to store data in wide rows with custom indexes that allow for fast range scans on a single node. Also there is no more need for an order-preserving partitioner that is likely to compromise data distribution in the Cassandra cluster.

In essence, composite primary keys with identical partition parts will be written in the same Cassandra row (which is essentially a partition). Within the same row entities are stored in lexical order by their cluster key components. Avro field names are appended as the last component of the composite column name. The current implementation does not substitute super columns. Thus, complex avro fields are still mapped to super columns. Super column families use the same composite primary keys as simple column families. As Gora always fully loads nested complex types, the use of super column families is not really a problem. Yet, super columns could be substituted by another level of column name components below the field qualifiers in future work. It would also be possible to rethink the decomposition of complex nested types beyond the first level.

The implementation uses the concept of Gora partitionQueries in order to decompose row scanning queries into a sets of queries that each operate on a single row. However, such a decomposition is not always possible and real range scans are limited to wide rows (partitions).

The implementation is fully backward compatible. Simple key classes can still be used and row scans are still possible with an order-preserving partitioner. The current junit tests are all passed. Furthermore, I have added an example and some unit tests to demonstrate the use of composite primary keys for time series data.

As mentioned earlier, we are happy to share this extension. I've created a jira issue for it (GORA-267) and will provide the implementation on GitHub (https://github.com/zirpins/gora/tree/GORA-267).

Regards,
Christian



-----Ursprüngliche Nachricht-----
Von: Lewis John Mcgibbney [mailto:lewis.mcgibbney@gmail.com]
Gesendet: Montag, 22. Juli 2013 00:05
An: <de...@gora.apache.org>
Betreff: Re: Gora Cassandra module design rationale

Hi Christian,

On Sat, Jul 20, 2013 at 12:54 PM, <de...@gora.apache.org> wrote:

>
> thanks a lot for sharing your thoughts! I have worked a little on the
> topic during the last days and would like to share the results with you.
> Pls accept my apologies for the long text.
>

No probs

>
> You will have noticed that the current design of the Gora cassandra
> module is not really in conformance with the above points.


No joke ;)


> Still I think Gora did fantastic work in providing basic concepts and
> infrastructure for a big data abstraction layer. Moreover, it should
> be generally possible to extend the design in a way that satisfies our needs.
>

I agree with you very mcuh here. Although you point out some fundamental points, which for some may be blockers for using Gora for your use case, there are other low hanging fruit (in the form of tweaks and improvements)... which I assume you are working on right now. It's just a case of delving down into the module.

[snip]


> The idea to exploit these concepts in Gora builds on adding an
> additional key mapping part to the gora-cassandra-mapping. An avro
> class might define a primary key element. The key is represented by
> another avro class. The mapping defines which fields of the avro key
> class hold the data for which parts of the C* compound primary key.
> Additionally, the field names of the avro value class are being
> appended to the composite column name. Nested complex types that have
> formerly been mapped to sub columns are handled accordingly by
> appending their name and subKeys as column names. As C* composite
> column names can be defined dynamically, also an arbitrary level of nesting complex types would be possible.
>

I think this is the first serious suggestion we've had for a rethink on the logic behind the gora-cassandra storage model. I like it, and I would really like to see something (code) to map/tie the above to. It is not trivial but you seem to have a comprehensive (better than me ;)) vision of how it should and will work.


>
> An open question for me is, if this concept is also viable considering
> other data stores and if it would make sense to add something along
> this lines to the Gora core. At least for Cassandra it seems to do the trick.
>

One important aspect of core that we have successfully maintained to-date, is that the object2datstore mappings are backend specific. Consequently (and although admittedly there have been many improvements/changes in Gora over the last few revisions) we really aim to be backwards compatible until we reach 1.0. We are seeing (in the pipeline) a complete overhaul/upgrade of the HBase API within possibly 0.4. If you wish to propose changes to core then please feel free to do so... maybe a well defined and comprehensively document Jira issue is in order here?


>
> As a POC I am implementing this on top of the current Gora HEAD (just
> the cassandra module). Thanks to the good Gora infrastructure, I've
> finished a first version of the data creation/update part and can give you an example.


On the side, what were the nice parts of the core API which you liked right now? What are the bad?


> The corresponding gora-cassandra-mapping is shown below. The result of
> this is that all meter reading objects of a single year will go into
> one partition (row) and can be retrieved by date ranges. Additionally
> the CF holds aggregations for months and weeks. Additionally, the
> mapping specifies data replication for the keyspace.
>
> There is no doubt, this certainly makes sense for you use case. It is
> very
very powerful indeed.

N.B. I reviewed and acknowledged the avro, xml and Java code you posted, thanks for being so verbose. I'm sure others have thoughts. I for one would like to come back to it once (as below) I can see some code... if at all possible.

> ...
>

I'm trying to finish the Gora C* store this month. I'm happy to share the
> patch with you if you are interested.
>
> It would be really great to have your insight and motivation as part
> of
Gora. Your code would be very welcome here and thanks for dropping in on the dev lists.
Thank you
Lewis



...




SEEBURGER AG            Vorstand/Seeburger Executive Board:
Sitz der Gesellschaft/Registered Office:                Bernd Seeburger, Axel Haas, Michael Kleeberg
Edisonstr. 1
D-75015 Bretten         Vorsitzender des Aufsichtsrats/Chairperson of the Seeburger Supervisory Board:
Tel.: 07252 / 96 - 0            Dr. Franz Scherer
Fax: 07252 / 96 - 2222
Internet: http://www.seeburger.de               Registergericht/Commercial Register:
e-mail: info@seeburger.de               HRB 240708 Mannheim


Dieses E-Mail ist nur für den Empfänger bestimmt, an den es gerichtet ist und kann vertrauliches bzw. unter das Berufsgeheimnis fallendes Material enthalten. Jegliche darin enthaltene Ansicht oder Meinungsäußerung ist die des Autors und stellt nicht notwendigerweise die Ansicht oder Meinung der SEEBURGER AG dar. Sind Sie nicht der Empfänger, so haben Sie diese E-Mail irrtümlich erhalten und jegliche Verwendung, Veröffentlichung, Weiterleitung, Abschrift oder jeglicher Druck dieser E-Mail ist strengstens untersagt. Weder die SEEBURGER AG noch der Absender ( Zirpins. Christian ) übernehmen die Haftung für Viren; es obliegt Ihrer Verantwortung, die E-Mail und deren Anhänge auf Viren zu prüfen.


The present email addresses only the addressee which it targets and may contain confidential material that may be protected by the professional secret. The opinions reflected herein are not necessarily the one of the SEEBURGER AG. If you are not the addressee, you have accidentally got this email and are not enabled to use, publish, forward, copy or print it in any way. Neither SEEBURGER AG , nor the sender (Zirpins. Christian) are liable for viruses, being your own responsibility to check this email and its attachments for this purpose.