You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@cassandra.apache.org by Eric Evans <ee...@rackspace.com> on 2010/10/28 22:40:07 UTC
NoSQL, YesCQL?
I don't know about everyone else, but I've been pretty dissatisfied with
our API situation for some time.
I used to be of the opinion that The Problem was Thrift and The Answer
was Avro. I do still prefer Avro over Thrift as dependencies go, but
I've come to the conclusion that the real problem has more to do with
the object-oriented RPC pattern that both share in common.
I used to think this was a necessary burden in order to expose the full
power of our query API in a client agnostic manner, and that it was the
job of third-party libraries to wrap this layer to create stable
idiomatic interfaces.
And, I used to think a vibrant ecosystem of these idiomatic libraries
would emerge, but it hasn't. Truth is we've seen quite a bit of
proliferation, and very little critical mass.
This shouldn't be construed as criticism levied against library
maintainers, on the contrary, I think they've done an amazing job all
things considered. But in my opinion, our RPC interfaces push way too
many implementation details client-side, making them brittle and
unstable. This results in unnecessarily complex libraries that are
forced to move in lock-step with Cassandra releases, that are never
backward compatible, and which are consistently broken during our
development cycle. I think we can provide client maintainers and users
with a better experience than this.
One solution to this is to implement a server-side query language, with
simple language drivers that manage all of the common functionality in a
consistent way (statement preparation, connection pooling, etc).
Library maintainers would then build their idiomatic interfaces on top
of these, (and I imagine, remove a metric crap-ton of code in the
process).
To this end I've been playing with exactly that. I have enough to do
simple reads and writes, and I have stubbed out drivers for Java and
Python. I'm seeking community feedback to gauge interest, and to
satisfy the much needed desire to bike-shed. :)
http://github.com/eevans/cassandra/tree/CQL
You need to be sure you're checking out the "CQL" branch.
This is implemented in the Avro interface so you'll need to pass the -a
flag when launching the daemon.
It is far from complete, and is surely quite buggy at this stage, take
it as a preview or demonstration; it's meant to be a starting point for
discussion.
There is a bit of documentation on the syntax, you can see that here:
http://github.com/eevans/cassandra/blob/CQL/doc/cql/CQL.textile
There are some system tests in test/system/test_cql.py that should give
you the basic idea, but assuming you have the Avro Python module
installed you should be able to just...
$ cd drivers/py
$ python
>>> import cql
>>> conn = cql.Connection('Keyspace1', 'localhost', 9160)
>>> conn.execute('UPDATE Standard1 WITH ROW("k", COL("c", "hello!"));')
>>> print conn.execute('SELECT FROM Standard1 WHERE KEY = "k"')
[{u'columns': [{u'timestamp': 1288297508761L, u'name': 'c', u'value':
'hello!', u'ttl': None}], u'key': 'k'}]
>>>
I've been chewing on all of this for a while so I'm brimming with ideas,
but I'll let the dust settle on this email before going any further. :)
Thoughts?
P.S. Dear Cliff Moon, I apologize. You were right.
--
Eric Evans
eevans@rackspace.com
Re: NoSQL, YesCQL?
Posted by Ted Zlatanov <tz...@lifelogs.com>.
On Fri, 29 Oct 2010 10:07:43 -0500 (CDT) "Stu Hood" <st...@rackspace.com> wrote:
SH> Most reasonable languages these days have a way to define what looks
SH> like a DSL: giving people a text DSL which is subject to injection
SH> attacks and can't be type checked without support from a client
SH> driver anyway is brain dead.
I don't think SQL-like query languages are DSLs in the classic sense.
Injection attacks are a red herring: they are a client issue, not a
library or a server problem. Type checking is a valid complaint and I
think it's balanced out by the flexibility of a text protocol.
SH> Regarding performance: assuming optimized RPC libraries (which we do
SH> not yet have in Avro, and which Thrift is getting better at),
SH> serializing to a string and back will never be as performant as
SH> using a pre-parsed representation of the statement on both
SH> sides. "Oh but we can add prepared statements!" Poppycock.
Consider that Cassandra's statements will never be as complicated as a
regular RDBMS, so parsing them efficiently is not so hard. The
parameters can be attached to the query, not necessarily inlined. A
native JDBC level 4 driver could be a very efficient answer to this
problem, too.
SH> The stated problem is that backwards compatibility is hard to
SH> provide: if that is the core complaint, then changing to a text
SH> based serialization format with a sexy name in order to add
SH> backwards compatibility is a severe overreaction to the
SH> problem. Instead, I would propose evolving the API in a manner that
SH> simplifies it.
I think it would be great to allow multiple APIs in Cassandra (when I
proposed it in the past, it was not allowed, and AFAIK still isn't
beyond Avro and Thrift). Then this wouldn't be an yes-or-no choice and
the Thrift API would still be available to those who need it.
Ted
Re: NoSQL, YesCQL?
Posted by Stu Hood <st...@rackspace.com>.
Let me preface this by saying: the fact that it looks like SQL doesn't bother me. But: I think this would be a terrible thing to _focus_ on. Something for contrib? Sure.
Most reasonable languages these days have a way to define what looks like a DSL: giving people a text DSL which is subject to injection attacks and can't be type checked without support from a client driver anyway is brain dead. There is a reason that 99% of clients that interact with a SQL server do so programmatically via a library: because that is an easier way to interact than via string concatenation. At least that way the library is performing the concatenation for you.
Regarding performance: assuming optimized RPC libraries (which we do not yet have in Avro, and which Thrift is getting better at), serializing to a string and back will never be as performant as using a pre-parsed representation of the statement on both sides. "Oh but we can add prepared statements!" Poppycock.
----
> Truth is we've seen quite a bit of proliferation, and very little critical mass.
> forced to move in lock-step with Cassandra releases, that are never backward compatible
Expecting stable libraries or API pre-1.0 is not reasonable: neither the devs nor the users expect stability, and yet we are still able to guarantee compatibility within major versions.
The stated problem is that backwards compatibility is hard to provide: if that is the core complaint, then changing to a text based serialization format with a sexy name in order to add backwards compatibility is a severe overreaction to the problem. Instead, I would propose evolving the API in a manner that simplifies it.
Thanks,
Stu
-----Original Message-----
From: "Eric Evans" <ee...@rackspace.com>
Sent: Thursday, October 28, 2010 4:56pm
To: dev@cassandra.apache.org
Subject: Re: NoSQL, YesCQL?
On Thu, 2010-10-28 at 14:46 -0700, Chip Salzenberg wrote:
> Short answer: "YES Please, but we will still want a side channel for
> minimum overhead."
Ok. Though I'm not sure I agree with the "minimum overhead" argument.
I've only done preliminary tests so far, but this seems on par (if not a
little faster) than Thrift is, and no effort to optimize has been made
(using a socket transport instead of HTTP for example would make a big
difference).
> Long answer: Query languages only work reliably when you have data
> binding assistance (insert "Bobby Tables" xkcd here). However, they
> do
> have the wonderful property of evolving aggressively without requiring
> upgrades of the driver plumbing. This is, of course, emphatically
> *not*
> true of anything like the current Thrift and Avro interfaces. So
> that's
> why I say "Yes." On the other hand, a very simple interface for very
> simple queries has a lot of value, too; see, for example,
> http://yoshinorimatsunobu.blogspot.com/2010/10/using-mysql-as-nosql-story-for.html
> So that's why I think we will still want to bypass the full language
> for minimum latency in some circumstances.
I'm also not sure I follow here either. Do you mean simple for the
application developer? CQL-using code should be considerably simpler
than the equivalent Thrift/Avro.
Thanks for the feedback!
--
Eric Evans
eevans@rackspace.com
Re: NoSQL, YesCQL?
Posted by Chip Salzenberg <ch...@pobox.com>.
On 10/28/2010 2:56 PM, Eric Evans wrote:
> On Thu, 2010-10-28 at 14:46 -0700, Chip Salzenberg wrote:
>> Short answer: "YES Please, but we will still want a side channel for
>> minimum overhead."
> Ok. Though I'm not sure I agree with the "minimum overhead" argument.
>
> I've only done preliminary tests so far, but this seems on par (if not a
> little faster) than Thrift is, and no effort to optimize has been made
> (using a socket transport instead of HTTP for example would make a big
> difference).
I didn't mean to imply that Thrift is fast enough. (Hehe, that would be
funny.) Rather, I meant to suggest that any parsed language will
inevitably be slower than a direct API for simple queries, so we should
also provide such a simple API. If the question is, can the language
completely and effectively replace Thrift for Cassandra, the answer is
Yes. AFAICT, anyway.
>> Long answer: Query languages only work reliably when you have data
>> binding assistance (insert "Bobby Tables" xkcd here). However, they
>> do
>> have the wonderful property of evolving aggressively without requiring
>> upgrades of the driver plumbing. This is, of course, emphatically
>> *not*
>> true of anything like the current Thrift and Avro interfaces. So
>> that's
>> why I say "Yes." On the other hand, a very simple interface for very
>> simple queries has a lot of value, too; see, for example,
>> http://yoshinorimatsunobu.blogspot.com/2010/10/using-mysql-as-nosql-story-for.html
>> So that's why I think we will still want to bypass the full language
>> for minimum latency in some circumstances.
> I'm also not sure I follow here either. Do you mean simple for the
> application developer? CQL-using code should be considerably simpler
> than the equivalent Thrift/Avro.
I'm not concerned about application-coding complexity problems with a
language, only about performance in the lowest of low-latency use cases.
> Thanks for the feedback!
>
Re: NoSQL, YesCQL?
Posted by Eric Evans <ee...@rackspace.com>.
On Thu, 2010-10-28 at 14:46 -0700, Chip Salzenberg wrote:
> Short answer: "YES Please, but we will still want a side channel for
> minimum overhead."
Ok. Though I'm not sure I agree with the "minimum overhead" argument.
I've only done preliminary tests so far, but this seems on par (if not a
little faster) than Thrift is, and no effort to optimize has been made
(using a socket transport instead of HTTP for example would make a big
difference).
> Long answer: Query languages only work reliably when you have data
> binding assistance (insert "Bobby Tables" xkcd here). However, they
> do
> have the wonderful property of evolving aggressively without requiring
> upgrades of the driver plumbing. This is, of course, emphatically
> *not*
> true of anything like the current Thrift and Avro interfaces. So
> that's
> why I say "Yes." On the other hand, a very simple interface for very
> simple queries has a lot of value, too; see, for example,
> http://yoshinorimatsunobu.blogspot.com/2010/10/using-mysql-as-nosql-story-for.html
> So that's why I think we will still want to bypass the full language
> for minimum latency in some circumstances.
I'm also not sure I follow here either. Do you mean simple for the
application developer? CQL-using code should be considerably simpler
than the equivalent Thrift/Avro.
Thanks for the feedback!
--
Eric Evans
eevans@rackspace.com
Re: NoSQL, YesCQL?
Posted by Ted Zlatanov <tz...@lifelogs.com>.
On Fri, 29 Oct 2010 09:29:43 -0500 Gary Dusbabek <gd...@gmail.com> wrote:
GD> 2010/10/29 Ted Zlatanov <tz...@lifelogs.com>:
>> On Thu, 28 Oct 2010 14:46:15 -0700 Chip Salzenberg <ch...@pobox.com> wrote:
>>
CS> Short answer: "YES Please, but we will still want a side channel for
CS> minimum overhead."
>>
>> 100% agreed on both counts. But IIRC the fastest side channel is to
>> become a Cassandra node. Is that an option?
GD> Yes. We call it the fat client. It's a non-storage
GD> gossip-participating cassandra node that speaks directly in terms of
GD> RowMutations, etc.
Sorry, I meant to ask Chip "is that an option for you, as opposed to a
general side channel usable from any language?"
Ted
Re: NoSQL, YesCQL?
Posted by Gary Dusbabek <gd...@gmail.com>.
2010/10/29 Ted Zlatanov <tz...@lifelogs.com>:
> On Thu, 28 Oct 2010 14:46:15 -0700 Chip Salzenberg <ch...@pobox.com> wrote:
>
> CS> Short answer: "YES Please, but we will still want a side channel for
> CS> minimum overhead."
>
> 100% agreed on both counts. But IIRC the fastest side channel is to
> become a Cassandra node. Is that an option?
Yes. We call it the fat client. It's a non-storage
gossip-participating cassandra node that speaks directly in terms of
RowMutations, etc.
Gary
>
> Ted
>
>
Re: CQL status
Posted by Eric Evans <ee...@racklabs.com>.
On Wed, 2011-01-12 at 05:21 +0100, Tristan Tarrant wrote:
> what we need now is a jdbc driver...
https://issues.apache.org/jira/browse/CASSANDRA-1710
> On Jan 12, 2011 3:00 AM, "Courtney Robinson" <sa...@live.co.uk> wrote:
> > ?On Tue, Jan 11, 2011 at 7:27 PM, Courtney Robinson <sa...@live.co.uk>
> > wrote:
> >> ?Having been pointed to
> >> https://issues.apache.org/jira/browse/CASSANDRA-1704
> >> I wanted to know what the state of that was.
> >> Eric Evans said changes had been pushed to SVN, can I check out the
> latest
> >> head and play with it?
> >
> >>Yes.
> >
> > Checking out now to have a look.
> >
> >
> >> Looking at the comments, if its not too late here's my 2cents.
> >
> >>These were both addressed in the 1704 design, by the time we were done
> > with it. :)
> >
> > Looking through the rest of the posts on Jira now and saw :-)
> >
> >> In the proposal there was ASC and DESC on the end of the query template,
> >> assuming here that this implies being able to sort records.
> >
> >>There is ASC and DESC but not ORDER BY, so it is only taking advantage
> > of the "natural" sort order.
> >
> > Makes sense
> >
> >> Being able to use an SQL like syntax should be a matter of convenience
> and
> >> only where it will not force incompatibilities.
> >
> >>Correct.
--
Eric Evans
eevans@racklabs.com
Re: CQL status
Posted by Tristan Tarrant <tr...@gmail.com>.
what we need now is a jdbc driver...
On Jan 12, 2011 3:00 AM, "Courtney Robinson" <sa...@live.co.uk> wrote:
> ?On Tue, Jan 11, 2011 at 7:27 PM, Courtney Robinson <sa...@live.co.uk>
> wrote:
>> ?Having been pointed to
>> https://issues.apache.org/jira/browse/CASSANDRA-1704
>> I wanted to know what the state of that was.
>> Eric Evans said changes had been pushed to SVN, can I check out the
latest
>> head and play with it?
>
>>Yes.
>
> Checking out now to have a look.
>
>
>> Looking at the comments, if its not too late here's my 2cents.
>
>>These were both addressed in the 1704 design, by the time we were done
> with it. :)
>
> Looking through the rest of the posts on Jira now and saw :-)
>
>> In the proposal there was ASC and DESC on the end of the query template,
>> assuming here that this implies being able to sort records.
>
>>There is ASC and DESC but not ORDER BY, so it is only taking advantage
> of the "natural" sort order.
>
> Makes sense
>
>> Being able to use an SQL like syntax should be a matter of convenience
and
>> only where it will not force incompatibilities.
>
>>Correct.
>
> :)
> --
> Jonathan Ellis
> Project Chair, Apache Cassandra
> co-founder of Riptano, the source for professional Cassandra support
> http://riptano.com
>
Re: CQL status
Posted by Courtney Robinson <sa...@live.co.uk>.
?On Tue, Jan 11, 2011 at 7:27 PM, Courtney Robinson <sa...@live.co.uk>
wrote:
> ?Having been pointed to
> https://issues.apache.org/jira/browse/CASSANDRA-1704
> I wanted to know what the state of that was.
> Eric Evans said changes had been pushed to SVN, can I check out the latest
> head and play with it?
>Yes.
Checking out now to have a look.
> Looking at the comments, if its not too late here's my 2cents.
>These were both addressed in the 1704 design, by the time we were done
with it. :)
Looking through the rest of the posts on Jira now and saw :-)
> In the proposal there was ASC and DESC on the end of the query template,
> assuming here that this implies being able to sort records.
>There is ASC and DESC but not ORDER BY, so it is only taking advantage
of the "natural" sort order.
Makes sense
> Being able to use an SQL like syntax should be a matter of convenience and
> only where it will not force incompatibilities.
>Correct.
:)
--
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of Riptano, the source for professional Cassandra support
http://riptano.com
Re: CQL status
Posted by Jonathan Ellis <jb...@gmail.com>.
On Tue, Jan 11, 2011 at 7:27 PM, Courtney Robinson <sa...@live.co.uk> wrote:
> ?Having been pointed to https://issues.apache.org/jira/browse/CASSANDRA-1704
> I wanted to know what the state of that was.
> Eric Evans said changes had been pushed to SVN, can I check out the latest
> head and play with it?
Yes.
> Looking at the comments, if its not too late here's my 2cents.
These were both addressed in the 1704 design, by the time we were done
with it. :)
> In the proposal there was ASC and DESC on the end of the query template, assuming here that this implies being able to sort records.
There is ASC and DESC but not ORDER BY, so it is only taking advantage
of the "natural" sort order.
> Being able to use an SQL like syntax should be a matter of convenience and only where it will not force incompatibilities.
Correct.
--
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of Riptano, the source for professional Cassandra support
http://riptano.com
CQL status
Posted by Courtney Robinson <sa...@live.co.uk>.
?Having been pointed to https://issues.apache.org/jira/browse/CASSANDRA-1704
I wanted to know what the state of that was.
Eric Evans said changes had been pushed to SVN, can I check out the latest
head and play with it?
Looking at the comments, if its not too late here's my 2cents.
For what it's worth I think CQL will be useful.
SELECT COL1,COL2 FROM CF WHERE ROWKEY=KEY1
In this semantics we can specify the columns we we're interested in.
Secondly allowing the predicate to specify a row key to start from...
SELECT COL1 TO COL20 FROM CF WHERE ROWKEY=KEY1 TO KEY1000
COL1 TO COL20 obviously specifies the range of columns we'd like back and
key1 to key1000 specifies our rows.
While I think CQL will be nice to have I agree with some of the points
brought up.
In the proposal there was ASC and DESC on the end of the query template,
assuming here that this implies being able to sort records.
I don't believe this would be beneficial since the already established ethos
is to have data sorted prior to querying for the obvious reason.
Being able to use an SQL like syntax should be a matter of convenience and
only where it will not force incompatibilities.
Re: NoSQL, YesCQL?
Posted by Jonathan Ellis <jb...@gmail.com>.
2010/10/29 Ted Zlatanov <tz...@lifelogs.com>:
> I think the sane, reasonable, simple path is to make the query language
> as similar to SQL as possible (which EricQL seems to aim for). Just
> making the queries pure text would be terrific, in any case. Then a
> JDBC driver or a Perl DBD driver (and their parallels in Ruby, Python,
> etc.) would be so much easier to write and Cassandra clients wouldn't
> have to be so damn complicated.
I think this is the biggest win for a "dialect of SQL" support; ORM
implementors are used to having to special case some things on a
per-db basis (e.g. TOP vs LIMIT; CONNECT BY vs WITH RECURSIVE, etc)
but if the basic SELECT and UPDATE work then we are 90% of the way
there.
--
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of Riptano, the source for professional Cassandra support
http://riptano.com
Re: NoSQL, YesCQL?
Posted by Chip Salzenberg <ch...@pobox.com>.
On 10/29/2010 7:13 AM, Ted Zlatanov wrote:
> On Thu, 28 Oct 2010 14:46:15 -0700 Chip Salzenberg <ch...@pobox.com> wrote:
>
> CS> Short answer: "YES Please, but we will still want a side channel for
> CS> minimum overhead."
>
> 100% agreed on both counts. But IIRC the fastest side channel is to
> become a Cassandra node. Is that an option?
If it were written in C or Perl, maybe. In Java, no.
Re: NoSQL, YesCQL?
Posted by Ted Zlatanov <tz...@lifelogs.com>.
On Thu, 28 Oct 2010 14:46:15 -0700 Chip Salzenberg <ch...@pobox.com> wrote:
CS> Short answer: "YES Please, but we will still want a side channel for
CS> minimum overhead."
100% agreed on both counts. But IIRC the fastest side channel is to
become a Cassandra node. Is that an option?
CS> Long answer: Query languages only work reliably when you have data
CS> binding assistance (insert "Bobby Tables" xkcd here). However, they do
CS> have the wonderful property of evolving aggressively without requiring
CS> upgrades of the driver plumbing. This is, of course, emphatically *not*
CS> true of anything like the current Thrift and Avro interfaces. So that's
CS> why I say "Yes." On the other hand, a very simple interface for very
CS> simple queries has a lot of value, too; see, for example,
CS> http://yoshinorimatsunobu.blogspot.com/2010/10/using-mysql-as-nosql-story-for.html
CS> So that's why I think we will still want to bypass the full language for
CS> minimum latency in some circumstances.
I think the sane, reasonable, simple path is to make the query language
as similar to SQL as possible (which EricQL seems to aim for). Just
making the queries pure text would be terrific, in any case. Then a
JDBC driver or a Perl DBD driver (and their parallels in Ruby, Python,
etc.) would be so much easier to write and Cassandra clients wouldn't
have to be so damn complicated. So I'd rather see specialized tools for
minimum latency and overhead, especially for inserts and dumps (like
MySQL provides mysqlinsert and mysqldump).
Ted
Re: NoSQL, YesCQL?
Posted by Chip Salzenberg <ch...@pobox.com>.
Short answer: "YES Please, but we will still want a side channel for
minimum overhead."
Long answer: Query languages only work reliably when you have data
binding assistance (insert "Bobby Tables" xkcd here). However, they do
have the wonderful property of evolving aggressively without requiring
upgrades of the driver plumbing. This is, of course, emphatically *not*
true of anything like the current Thrift and Avro interfaces. So that's
why I say "Yes." On the other hand, a very simple interface for very
simple queries has a lot of value, too; see, for example,
http://yoshinorimatsunobu.blogspot.com/2010/10/using-mysql-as-nosql-story-for.html
So that's why I think we will still want to bypass the full language for
minimum latency in some circumstances.
On 10/28/2010 1:40 PM, Eric Evans wrote:
> I don't know about everyone else, but I've been pretty dissatisfied with
> our API situation for some time.
>
> I used to be of the opinion that The Problem was Thrift and The Answer
> was Avro. I do still prefer Avro over Thrift as dependencies go, but
> I've come to the conclusion that the real problem has more to do with
> the object-oriented RPC pattern that both share in common.
>
> I used to think this was a necessary burden in order to expose the full
> power of our query API in a client agnostic manner, and that it was the
> job of third-party libraries to wrap this layer to create stable
> idiomatic interfaces.
>
> And, I used to think a vibrant ecosystem of these idiomatic libraries
> would emerge, but it hasn't. Truth is we've seen quite a bit of
> proliferation, and very little critical mass.
>
> This shouldn't be construed as criticism levied against library
> maintainers, on the contrary, I think they've done an amazing job all
> things considered. But in my opinion, our RPC interfaces push way too
> many implementation details client-side, making them brittle and
> unstable. This results in unnecessarily complex libraries that are
> forced to move in lock-step with Cassandra releases, that are never
> backward compatible, and which are consistently broken during our
> development cycle. I think we can provide client maintainers and users
> with a better experience than this.
>
> One solution to this is to implement a server-side query language, with
> simple language drivers that manage all of the common functionality in a
> consistent way (statement preparation, connection pooling, etc).
> Library maintainers would then build their idiomatic interfaces on top
> of these, (and I imagine, remove a metric crap-ton of code in the
> process).
>
> To this end I've been playing with exactly that. I have enough to do
> simple reads and writes, and I have stubbed out drivers for Java and
> Python. I'm seeking community feedback to gauge interest, and to
> satisfy the much needed desire to bike-shed. :)
>
> http://github.com/eevans/cassandra/tree/CQL
>
> You need to be sure you're checking out the "CQL" branch.
>
> This is implemented in the Avro interface so you'll need to pass the -a
> flag when launching the daemon.
>
> It is far from complete, and is surely quite buggy at this stage, take
> it as a preview or demonstration; it's meant to be a starting point for
> discussion.
>
> There is a bit of documentation on the syntax, you can see that here:
> http://github.com/eevans/cassandra/blob/CQL/doc/cql/CQL.textile
>
> There are some system tests in test/system/test_cql.py that should give
> you the basic idea, but assuming you have the Avro Python module
> installed you should be able to just...
>
> $ cd drivers/py
> $ python
>>>> import cql
>>>> conn = cql.Connection('Keyspace1', 'localhost', 9160)
>>>> conn.execute('UPDATE Standard1 WITH ROW("k", COL("c", "hello!"));')
>>>> print conn.execute('SELECT FROM Standard1 WHERE KEY = "k"')
> [{u'columns': [{u'timestamp': 1288297508761L, u'name': 'c', u'value':
> 'hello!', u'ttl': None}], u'key': 'k'}]
>
> I've been chewing on all of this for a while so I'm brimming with ideas,
> but I'll let the dust settle on this email before going any further. :)
>
> Thoughts?
>
>
>
> P.S. Dear Cliff Moon, I apologize. You were right.
>
Implementing [was: NoSQL, YesCQL?]
Posted by Eric Evans <ee...@rackspace.com>.
On Thu, 2010-10-28 at 15:40 -0500, Eric Evans wrote:
[ ... ]
> One solution to this is to implement a server-side query language, with
> simple language drivers that manage all of the common functionality in a
> consistent way (statement preparation, connection pooling, etc).
> Library maintainers would then build their idiomatic interfaces on top
> of these, (and I imagine, remove a metric crap-ton of code in the
> process).
[ ... ]
> I've been chewing on all of this for a while so I'm brimming with ideas,
> but I'll let the dust settle on this email before going any further. :)
Since it looks like most people are receptive to the general idea, maybe
I'll push my luck a little further by talking about implementation.
What I've been bouncing around in my head is this:
1. Implement it in the Avro interface.
2. As new statements are added, mark the corresponding (avro) RPC
methods as deprecated (or remove immediately).
3. When the query language has reached 100% feature coverage, the
language specification transitions to 1.0, and <insert sane process TBD
for change management>.
4. (Optionally )code our own transport and result serialization, and
remove Avro as a dependency.
5. Profit.
The rationale behind #1 and #2 is that the Avro interface is still
considered experimental, nothing that goes on there directly impacts the
supported API, and it has little to no existing user-base (that we know
of). The existing parity with the Thrift interface means that the query
language can be built out incrementally since holes in functionality can
be plugged with the RPC methods. Ultimately I favor the complete
elimination of the RPC methods because, well, as I've already stated I'm
not a fan and so see no reason to maintain *two* forms for the duration.
#3 is just about managing expectations. I don't favor anything overly
bureaucratic; I do favor heavily engaging library maintainers.
As for #4, this falls into the nice to have category, but at this stage
in the game, we'd only be using a tiny fraction of what these frameworks
provide. Creating our own wire protocol would allow us to shed a
dependency and optimize for our specific use.
Thoughts?
--
Eric Evans
eevans@rackspace.com
Re: NoSQL, YesCQL?
Posted by Eric Evans <ee...@rackspace.com>.
On Thu, 2010-10-28 at 18:47 -0700, J. Andrew Rogers wrote:
> > http://github.com/eevans/cassandra/tree/CQL
> >
> > You need to be sure you're checking out the "CQL" branch.
>
> Meta-comment: You probably should not call it CQL. That name is
> already used in multiple standards for similar purposes. These aren't
> dead standards either, for some types of non-SQL database applications
> you are required to implement them. I once wrote a query language
> bridge between a couple CQL database standards, and the current mess
> already confuses the hell out of people. It would not surprise me if
> someone is working on a CQL implementation (pick your standard) that
> uses Cassandra as a backend since it would be a good fit for some use
> cases.
Huh, I wasn't aware. The name is wrong on many levels, I guess I have
to add one more to the list. :)
It's just so damn indulgent though. Think about it:
* You can pronounce it C-quell in conversations with your friends. When
the inevitable confusion occurs, you can say, "What? No, I mean
*Cassandra* Query Language.". Bonus points for a side-ways look of
indignation.
* If someone asks if it's a NoSQL database you can answer, "No, it's not
a NoCQL, it's a NoSQL." Think of all the possibilities using double
negatives.
* In one fell swoop we make all other NoSQL databases, and all SQL
databases, NoCQL.
* Considering my history in naming stuff, it's a total setup for any
number of jokes at my expense.
> To highlight the insanity of the "CQL" namespace, some government
> applications require implementation of two CQL standards where the
> "CQL" acronym expands to the *exact same name* and fill almost
> identical roles. This creates several levels of hell for developers.
>
> A less non-unique name would probably be beneficial. :-)
Yeah, maybe.
--
Eric Evans
eevans@rackspace.com
Re: NoSQL, YesCQL?
Posted by Krishna Sankar <ks...@gmail.com>.
What about BDQL (Big data QL) or just NOQL?
Cheers
<k/>
On 10/28/10 Thu Oct 28, 10, "J. Andrew Rogers" <ja...@gmail.com>
wrote:
> On Thu, Oct 28, 2010 at 1:40 PM, Eric Evans <ee...@rackspace.com> wrote:
>>
>> One solution to this is to implement a server-side query language, with
>> simple language drivers that manage all of the common functionality in a
>> consistent way (statement preparation, connection pooling, etc).
>> Library maintainers would then build their idiomatic interfaces on top
>> of these, (and I imagine, remove a metric crap-ton of code in the
>> process).
>>
>> To this end I've been playing with exactly that. I have enough to do
>> simple reads and writes, and I have stubbed out drivers for Java and
>> Python. I'm seeking community feedback to gauge interest, and to
>> satisfy the much needed desire to bike-shed. :)
>>
>> http://github.com/eevans/cassandra/tree/CQL
>>
>> You need to be sure you're checking out the "CQL" branch.
>
>
> Meta-comment: You probably should not call it CQL. That name is
> already used in multiple standards for similar purposes. These aren't
> dead standards either, for some types of non-SQL database applications
> you are required to implement them. I once wrote a query language
> bridge between a couple CQL database standards, and the current mess
> already confuses the hell out of people. It would not surprise me if
> someone is working on a CQL implementation (pick your standard) that
> uses Cassandra as a backend since it would be a good fit for some use
> cases.
>
> To highlight the insanity of the "CQL" namespace, some government
> applications require implementation of two CQL standards where the
> "CQL" acronym expands to the *exact same name* and fill almost
> identical roles. This creates several levels of hell for developers.
>
> A less non-unique name would probably be beneficial. :-)
Re: NoSQL, YesCQL?
Posted by "J. Andrew Rogers" <ja...@gmail.com>.
On Thu, Oct 28, 2010 at 1:40 PM, Eric Evans <ee...@rackspace.com> wrote:
>
> One solution to this is to implement a server-side query language, with
> simple language drivers that manage all of the common functionality in a
> consistent way (statement preparation, connection pooling, etc).
> Library maintainers would then build their idiomatic interfaces on top
> of these, (and I imagine, remove a metric crap-ton of code in the
> process).
>
> To this end I've been playing with exactly that. I have enough to do
> simple reads and writes, and I have stubbed out drivers for Java and
> Python. I'm seeking community feedback to gauge interest, and to
> satisfy the much needed desire to bike-shed. :)
>
> http://github.com/eevans/cassandra/tree/CQL
>
> You need to be sure you're checking out the "CQL" branch.
Meta-comment: You probably should not call it CQL. That name is
already used in multiple standards for similar purposes. These aren't
dead standards either, for some types of non-SQL database applications
you are required to implement them. I once wrote a query language
bridge between a couple CQL database standards, and the current mess
already confuses the hell out of people. It would not surprise me if
someone is working on a CQL implementation (pick your standard) that
uses Cassandra as a backend since it would be a good fit for some use
cases.
To highlight the insanity of the "CQL" namespace, some government
applications require implementation of two CQL standards where the
"CQL" acronym expands to the *exact same name* and fill almost
identical roles. This creates several levels of hell for developers.
A less non-unique name would probably be beneficial. :-)
--
J. Andrew Rogers
Re: NoSQL, YesCQL?
Posted by Vladimir Vivien <vl...@gmail.com>.
How about something simpler/more approachable such as a query api that hides
away the drudgery of thrift/avro rpc api?
On Fri, Oct 29, 2010 at 12:56 AM, Vincent Maher <vi...@vincentmaher.com>wrote:
> Hi all
>
> I think this is a great idea. If you take out Thrift, which is where
> the bulk of the n00b installation pain lies, and add a query language
> and a PHP extension then I think Cassandra will go mainstream.
>
> I still need to write a proper blog post for the team but just so you
> all know, we're happily using Cassie for all sorts of stuff on
> Motribe at the moment. The work you are doing is changing lives, bit
> by bit, and many more people appreciate it than you realise.
>
> Kind regards,
> _________________________________
>
> Vincent Maher
> Founder, Motribe
> http://motribe.com && http://motribe.mobi
>
> "Accidents don't happen to people who take accidents as a personal
> insult" - Don Corleone
>
> Blogs: http://www.vincentmaher.com
> Facebook: http://www.facebook.com/vincentmaher
> Twitter: http://www.twitter.com/vincent_maher
>
> Mobile: +27 82 998 5412
> Google Talk: flobby1@gmail.com
> Skype Name: vincentmaher
>
>
>
>
> On Thu, Oct 28, 2010 at 10:40 PM, Eric Evans <ee...@rackspace.com> wrote:
> >
> > I don't know about everyone else, but I've been pretty dissatisfied with
> > our API situation for some time.
> >
> > I used to be of the opinion that The Problem was Thrift and The Answer
> > was Avro. I do still prefer Avro over Thrift as dependencies go, but
> > I've come to the conclusion that the real problem has more to do with
> > the object-oriented RPC pattern that both share in common.
> >
> > I used to think this was a necessary burden in order to expose the full
> > power of our query API in a client agnostic manner, and that it was the
> > job of third-party libraries to wrap this layer to create stable
> > idiomatic interfaces.
> >
> > And, I used to think a vibrant ecosystem of these idiomatic libraries
> > would emerge, but it hasn't. Truth is we've seen quite a bit of
> > proliferation, and very little critical mass.
> >
> > This shouldn't be construed as criticism levied against library
> > maintainers, on the contrary, I think they've done an amazing job all
> > things considered. But in my opinion, our RPC interfaces push way too
> > many implementation details client-side, making them brittle and
> > unstable. This results in unnecessarily complex libraries that are
> > forced to move in lock-step with Cassandra releases, that are never
> > backward compatible, and which are consistently broken during our
> > development cycle. I think we can provide client maintainers and users
> > with a better experience than this.
> >
> > One solution to this is to implement a server-side query language, with
> > simple language drivers that manage all of the common functionality in a
> > consistent way (statement preparation, connection pooling, etc).
> > Library maintainers would then build their idiomatic interfaces on top
> > of these, (and I imagine, remove a metric crap-ton of code in the
> > process).
> >
> > To this end I've been playing with exactly that. I have enough to do
> > simple reads and writes, and I have stubbed out drivers for Java and
> > Python. I'm seeking community feedback to gauge interest, and to
> > satisfy the much needed desire to bike-shed. :)
> >
> > http://github.com/eevans/cassandra/tree/CQL
> >
> > You need to be sure you're checking out the "CQL" branch.
> >
> > This is implemented in the Avro interface so you'll need to pass the -a
> > flag when launching the daemon.
> >
> > It is far from complete, and is surely quite buggy at this stage, take
> > it as a preview or demonstration; it's meant to be a starting point for
> > discussion.
> >
> > There is a bit of documentation on the syntax, you can see that here:
> > http://github.com/eevans/cassandra/blob/CQL/doc/cql/CQL.textile
> >
> > There are some system tests in test/system/test_cql.py that should give
> > you the basic idea, but assuming you have the Avro Python module
> > installed you should be able to just...
> >
> > $ cd drivers/py
> > $ python
> >>>> import cql
> >>>> conn = cql.Connection('Keyspace1', 'localhost', 9160)
> >>>> conn.execute('UPDATE Standard1 WITH ROW("k", COL("c", "hello!"));')
> >>>> print conn.execute('SELECT FROM Standard1 WHERE KEY = "k"')
> > [{u'columns': [{u'timestamp': 1288297508761L, u'name': 'c', u'value':
> > 'hello!', u'ttl': None}], u'key': 'k'}]
> >>>>
> >
> >
> > I've been chewing on all of this for a while so I'm brimming with ideas,
> > but I'll let the dust settle on this email before going any further. :)
> >
> > Thoughts?
> >
> >
> >
> > P.S. Dear Cliff Moon, I apologize. You were right.
> >
> > --
> > Eric Evans
> > eevans@rackspace.com
> >
> >
> >
> >
> >
>
--
Vladimir Vivien
Re: NoSQL, YesCQL?
Posted by Vincent Maher <vi...@vincentmaher.com>.
Hi all
I think this is a great idea. If you take out Thrift, which is where
the bulk of the n00b installation pain lies, and add a query language
and a PHP extension then I think Cassandra will go mainstream.
I still need to write a proper blog post for the team but just so you
all know, we're happily using Cassie for all sorts of stuff on
Motribe at the moment. The work you are doing is changing lives, bit
by bit, and many more people appreciate it than you realise.
Kind regards,
_________________________________
Vincent Maher
Founder, Motribe
http://motribe.com && http://motribe.mobi
"Accidents don't happen to people who take accidents as a personal
insult" - Don Corleone
Blogs: http://www.vincentmaher.com
Facebook: http://www.facebook.com/vincentmaher
Twitter: http://www.twitter.com/vincent_maher
Mobile: +27 82 998 5412
Google Talk: flobby1@gmail.com
Skype Name: vincentmaher
On Thu, Oct 28, 2010 at 10:40 PM, Eric Evans <ee...@rackspace.com> wrote:
>
> I don't know about everyone else, but I've been pretty dissatisfied with
> our API situation for some time.
>
> I used to be of the opinion that The Problem was Thrift and The Answer
> was Avro. I do still prefer Avro over Thrift as dependencies go, but
> I've come to the conclusion that the real problem has more to do with
> the object-oriented RPC pattern that both share in common.
>
> I used to think this was a necessary burden in order to expose the full
> power of our query API in a client agnostic manner, and that it was the
> job of third-party libraries to wrap this layer to create stable
> idiomatic interfaces.
>
> And, I used to think a vibrant ecosystem of these idiomatic libraries
> would emerge, but it hasn't. Truth is we've seen quite a bit of
> proliferation, and very little critical mass.
>
> This shouldn't be construed as criticism levied against library
> maintainers, on the contrary, I think they've done an amazing job all
> things considered. But in my opinion, our RPC interfaces push way too
> many implementation details client-side, making them brittle and
> unstable. This results in unnecessarily complex libraries that are
> forced to move in lock-step with Cassandra releases, that are never
> backward compatible, and which are consistently broken during our
> development cycle. I think we can provide client maintainers and users
> with a better experience than this.
>
> One solution to this is to implement a server-side query language, with
> simple language drivers that manage all of the common functionality in a
> consistent way (statement preparation, connection pooling, etc).
> Library maintainers would then build their idiomatic interfaces on top
> of these, (and I imagine, remove a metric crap-ton of code in the
> process).
>
> To this end I've been playing with exactly that. I have enough to do
> simple reads and writes, and I have stubbed out drivers for Java and
> Python. I'm seeking community feedback to gauge interest, and to
> satisfy the much needed desire to bike-shed. :)
>
> http://github.com/eevans/cassandra/tree/CQL
>
> You need to be sure you're checking out the "CQL" branch.
>
> This is implemented in the Avro interface so you'll need to pass the -a
> flag when launching the daemon.
>
> It is far from complete, and is surely quite buggy at this stage, take
> it as a preview or demonstration; it's meant to be a starting point for
> discussion.
>
> There is a bit of documentation on the syntax, you can see that here:
> http://github.com/eevans/cassandra/blob/CQL/doc/cql/CQL.textile
>
> There are some system tests in test/system/test_cql.py that should give
> you the basic idea, but assuming you have the Avro Python module
> installed you should be able to just...
>
> $ cd drivers/py
> $ python
>>>> import cql
>>>> conn = cql.Connection('Keyspace1', 'localhost', 9160)
>>>> conn.execute('UPDATE Standard1 WITH ROW("k", COL("c", "hello!"));')
>>>> print conn.execute('SELECT FROM Standard1 WHERE KEY = "k"')
> [{u'columns': [{u'timestamp': 1288297508761L, u'name': 'c', u'value':
> 'hello!', u'ttl': None}], u'key': 'k'}]
>>>>
>
>
> I've been chewing on all of this for a while so I'm brimming with ideas,
> but I'll let the dust settle on this email before going any further. :)
>
> Thoughts?
>
>
>
> P.S. Dear Cliff Moon, I apologize. You were right.
>
> --
> Eric Evans
> eevans@rackspace.com
>
>
>
>
>