You are viewing a plain text version of this content. The canonical link for it is here.
Posted to client-dev@cassandra.apache.org by Theo Hultberg <th...@iconara.net> on 2014/02/22 21:29:21 UTC
How should clients handle the user defined types in 2.1?
Hi,
Is there any documentation on how CQL clients should handle the new user
defined types coming in 2.1? There's nothing in the protocol specification
on how to handle custom types as far as I can understand.
For example, I tried creating the "address" type from the description of
CASSANDRA-5590, and this is how its metadata looks (the metadata for a
query contains a column with a custom type and this is the description of
it):
org.apache.cassandra.db.marshal.UserType(user_defined_types,61646472657373,737472656574:org.apache.cassandra.db.marshal.UTF8Type,63697479:org.apache.cassandra.db.marshal.UTF8Type,7a69705f636f6465:org.apache.cassandra.db.marshal.Int32Type,70686f6e6573:org.apache.cassandra.db.marshal.SetType(org.apache.cassandra.db.marshal.UTF8Type))
Is the client supposed to parse that description, and in that case how? I
could probably figure it out but it would be great if someone could point
me to the right docs.
yours,
Theo (author of cql-rb, the Ruby driver)
Re: How should clients handle the user defined types in 2.1?
Posted by Edward Capriolo <ed...@gmail.com>.
Deeply nester structures in data stores are difficult. Hive allows List
Struct Map to be infinitely nested (to a point). Most Query languages don't
have ways to say:
navigate to x
go to map y inside x
find all the values inside y that are = 10
I think they have a name for this: :)
http://en.wikipedia.org/wiki/Object-relational_impedance_mismatch
So it is pretty cool that cassandra has support for complex types, but how
to access them/ slice and dice them is another question. I run into ths
problem with hive, we can log VERY complex objects but if someone has to
make a UDF to extract meaningful data they are not super useful.
In the past have adopted to store json and then user jsonpath queries to
extract information. (client side or server side)
https://github.com/zznate/intravert-ug/wiki/Server-Side-JsonPath-example
Personally I would rather see udfs/language features like lateral view that
work on the collections that already exist before this feature.
On Wed, Feb 26, 2014 at 2:09 AM, Theo Hultberg <th...@iconara.net> wrote:
> thanks mikhail.
>
> it bothers me a bit that there is the possibility of arbitrarily deeply
> nested types (it's great for usability, but a headache for me as a driver
> implementer). it feels like just throwing a couple of regexes at this is
> the wrong solution, the string has too much structure and hierarchy for
> that and I think that not doing it with a proper parser will cause trouble
> in the future. and then trying to use the parsed type structure to properly
> encode something that the user wants to store will be very complex. kind of
> like having to implement a whole orm. it's going to be a challenge.
>
> I see that you took the parser route in the python driver so I can at least
> get some inspiration from there. what do you do when the user passes an
> object in a custom type field and that object doesn't look like the type at
> all?
>
> T#
>
>
> On Tue, Feb 25, 2014 at 8:16 PM, Mikhail Stepura <
> mikhail.stepura@outlook.com> wrote:
>
> > I just realized that your driver returns fields' names in the type itself
> > (which unfortunately is not the case with python driver) so you don't
> need
> > step #3.
> >
> > -M
> >
> >
> >
> > On 2/25/14, 10:42, Mikhail Stepura wrote:
> >
> >> The driver shall parse that.
> >> I'm not sure if there is a doc for that, but here is what I did for
> CQLSH
> >>
> >> 1. UserType is a CompositeType, where 1s subtype is a KS name, and 2nd
> >> is hex-encoded name of the type
> >> 2. Remainder of subtypes are types of Usertype's columns. So you can
> >> easily decode *values* for fields
> >> 3. Information about field *names* is stored in system.schema_usertypes
> >> table
> >> 4. The driver has to combine pieces 1-3 and create a new class/type for
> >> a user. It was easy in Python, I guess it should be easy in Ruby as well
> >>
> >> -M
> >>
> >>
> >> On 2/22/14, 12:29, Theo Hultberg wrote:
> >>
> >>> Hi,
> >>>
> >>> Is there any documentation on how CQL clients should handle the new
> user
> >>> defined types coming in 2.1? There's nothing in the protocol
> >>> specification
> >>> on how to handle custom types as far as I can understand.
> >>>
> >>> For example, I tried creating the "address" type from the description
> of
> >>> CASSANDRA-5590, and this is how its metadata looks (the metadata for a
> >>> query contains a column with a custom type and this is the description
> of
> >>> it):
> >>>
> >>> org.apache.cassandra.db.marshal.UserType(user_defined_
> >>> types,61646472657373,737472656574:org.apache.
> >>> cassandra.db.marshal.UTF8Type,63697479:org.apache.cassandra.
> >>> db.marshal.UTF8Type,7a69705f636f6465:org.apache.cassandra.db.marshal.
> >>> Int32Type,70686f6e6573:org.apache.cassandra.db.marshal.
> >>> SetType(org.apache.cassandra.db.marshal.UTF8Type))
> >>>
> >>>
> >>> Is the client supposed to parse that description, and in that case
> how? I
> >>> could probably figure it out but it would be great if someone could
> point
> >>> me to the right docs.
> >>>
> >>> yours,
> >>> Theo (author of cql-rb, the Ruby driver)
> >>>
> >>>
> >>
> >>
> >>
> >
> >
>
Re: How should clients handle the user defined types in 2.1?
Posted by Edward Capriolo <ed...@gmail.com>.
I really with I could thrift slice a non compact table and decode it with
user space tools :)
On Thu, Mar 6, 2014 at 9:22 AM, Theo Hultberg <th...@iconara.net> wrote:
> what I meant by step backwards is that the user defined types feature has
> been without adding it to the CQL protocol. it's not a protocol feature,
> and hasn't even got a spec.
>
> T#
>
>
> On Thu, Mar 6, 2014 at 2:58 PM, Edward Capriolo <edlinuxguru@gmail.com
> >wrote:
>
> > its not a step backwards though the emphasis was to remove burden from
> the
> > client developers. The value of a cassandra column could be 2GB but you
> > were limited by a thrift frame. Now with the native protocol you can now
> > throw huge objects in a column users can find new uses to use/break the
> > database. Before you had to break that up across columns.
> >
> >
> > On Thu, Mar 6, 2014 at 1:33 AM, Theo Hultberg <th...@iconara.net> wrote:
> >
> > > I agree, it feels like a step backwards for the CQL protocol.
> > >
> > > T#
> > >
> > >
> > > On Wed, Mar 5, 2014 at 8:10 PM, Edward Capriolo <edlinuxguru@gmail.com
> > > >wrote:
> > >
> > > > Somewhat of an aside. But wasn't issues with client serialization
> > issues
> > > > one of the reasons to get away from thrift? It seems like asking
> client
> > > > language to decode complex objects recreates the problem only with 1
> > > degree
> > > > more complexity.
> > > >
> > > >
> > > > On Wed, Mar 5, 2014 at 1:55 PM, Theo Hultberg <th...@iconara.net>
> > wrote:
> > > >
> > > > > thanks! yeah, I meant user defined types, but thanks for the
> > > description
> > > > of
> > > > > general custom types too, it's good to know.
> > > > >
> > > > > T#
> > > > >
> > > > >
> > > > > On Tue, Mar 4, 2014 at 9:28 PM, Tyler Hobbs <ty...@datastax.com>
> > > wrote:
> > > > >
> > > > > > On Sat, Mar 1, 2014 at 5:01 AM, Theo Hultberg <th...@iconara.net>
> > > > wrote:
> > > > > >
> > > > > > > Mikhail, thanks, but I meant the reverse of that. Say the user
> > > > creates
> > > > > a
> > > > > > > prepared statement where one of the columns is a custom type,
> how
> > > do
> > > > > you
> > > > > > > serialize the arguments to the prepared statement? Do you
> accept
> > > > > anything
> > > > > > > and let C* complain, or do you make a best effort to shoehorn
> the
> > > > > object
> > > > > > > the user passed into something that looks like the custom type?
> > > > > > >
> > > > > >
> > > > > > Just to be clear, by "custom type", you still mean a user-defined
> > > type,
> > > > > > correct?
> > > > > >
> > > > > > At least in the python driver, it's treated the same as any other
> > > > > > (parametrized) type. For each Cassandra type (UTF8Type,
> Int32Type,
> > > > etc),
> > > > > > the driver will accept values of one or more types. If any of
> the
> > > > > subtypes
> > > > > > don't match this, the driver will raise an exception.
> > > > > >
> > > > > > If you're actually talking about custom types and not
> user-defined
> > > > types,
> > > > > > I'll explain what the python driver does. If the typestring
> (e.g.
> > > > > > org.apache.cassandra.db.marshal.MyType) isn't recognized, the
> > driver
> > > > will
> > > > > > expect a binary string that it can pass directly to Cassandra for
> > > > values
> > > > > of
> > > > > > that type. If the user wants to add driver-level support for it
> > (to
> > > > > enable
> > > > > > converting a python object to a binary string and vice-versa),
> they
> > > can
> > > > > > subclass cassandra.cqltypes.CassandraType and define a
> serialize()
> > > and
> > > > > > deserialize() method. The only condition is that the python
> > > classname
> > > > > must
> > > > > > match the typestring from cassandra, so for
> > > > > > org.apache.cassandra.db.marshal.MyType, the user will create a
> > > > > > MyType(CassandraType) class.
> > > > > >
> > > > > > --
> > > > > > Tyler Hobbs
> > > > > > DataStax <http://datastax.com/>
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: How should clients handle the user defined types in 2.1?
Posted by Theo Hultberg <th...@iconara.net>.
what I meant by step backwards is that the user defined types feature has
been without adding it to the CQL protocol. it's not a protocol feature,
and hasn't even got a spec.
T#
On Thu, Mar 6, 2014 at 2:58 PM, Edward Capriolo <ed...@gmail.com>wrote:
> its not a step backwards though the emphasis was to remove burden from the
> client developers. The value of a cassandra column could be 2GB but you
> were limited by a thrift frame. Now with the native protocol you can now
> throw huge objects in a column users can find new uses to use/break the
> database. Before you had to break that up across columns.
>
>
> On Thu, Mar 6, 2014 at 1:33 AM, Theo Hultberg <th...@iconara.net> wrote:
>
> > I agree, it feels like a step backwards for the CQL protocol.
> >
> > T#
> >
> >
> > On Wed, Mar 5, 2014 at 8:10 PM, Edward Capriolo <edlinuxguru@gmail.com
> > >wrote:
> >
> > > Somewhat of an aside. But wasn't issues with client serialization
> issues
> > > one of the reasons to get away from thrift? It seems like asking client
> > > language to decode complex objects recreates the problem only with 1
> > degree
> > > more complexity.
> > >
> > >
> > > On Wed, Mar 5, 2014 at 1:55 PM, Theo Hultberg <th...@iconara.net>
> wrote:
> > >
> > > > thanks! yeah, I meant user defined types, but thanks for the
> > description
> > > of
> > > > general custom types too, it's good to know.
> > > >
> > > > T#
> > > >
> > > >
> > > > On Tue, Mar 4, 2014 at 9:28 PM, Tyler Hobbs <ty...@datastax.com>
> > wrote:
> > > >
> > > > > On Sat, Mar 1, 2014 at 5:01 AM, Theo Hultberg <th...@iconara.net>
> > > wrote:
> > > > >
> > > > > > Mikhail, thanks, but I meant the reverse of that. Say the user
> > > creates
> > > > a
> > > > > > prepared statement where one of the columns is a custom type, how
> > do
> > > > you
> > > > > > serialize the arguments to the prepared statement? Do you accept
> > > > anything
> > > > > > and let C* complain, or do you make a best effort to shoehorn the
> > > > object
> > > > > > the user passed into something that looks like the custom type?
> > > > > >
> > > > >
> > > > > Just to be clear, by "custom type", you still mean a user-defined
> > type,
> > > > > correct?
> > > > >
> > > > > At least in the python driver, it's treated the same as any other
> > > > > (parametrized) type. For each Cassandra type (UTF8Type, Int32Type,
> > > etc),
> > > > > the driver will accept values of one or more types. If any of the
> > > > subtypes
> > > > > don't match this, the driver will raise an exception.
> > > > >
> > > > > If you're actually talking about custom types and not user-defined
> > > types,
> > > > > I'll explain what the python driver does. If the typestring (e.g.
> > > > > org.apache.cassandra.db.marshal.MyType) isn't recognized, the
> driver
> > > will
> > > > > expect a binary string that it can pass directly to Cassandra for
> > > values
> > > > of
> > > > > that type. If the user wants to add driver-level support for it
> (to
> > > > enable
> > > > > converting a python object to a binary string and vice-versa), they
> > can
> > > > > subclass cassandra.cqltypes.CassandraType and define a serialize()
> > and
> > > > > deserialize() method. The only condition is that the python
> > classname
> > > > must
> > > > > match the typestring from cassandra, so for
> > > > > org.apache.cassandra.db.marshal.MyType, the user will create a
> > > > > MyType(CassandraType) class.
> > > > >
> > > > > --
> > > > > Tyler Hobbs
> > > > > DataStax <http://datastax.com/>
> > > > >
> > > >
> > >
> >
>
Re: How should clients handle the user defined types in 2.1?
Posted by Edward Capriolo <ed...@gmail.com>.
its not a step backwards though the emphasis was to remove burden from the
client developers. The value of a cassandra column could be 2GB but you
were limited by a thrift frame. Now with the native protocol you can now
throw huge objects in a column users can find new uses to use/break the
database. Before you had to break that up across columns.
On Thu, Mar 6, 2014 at 1:33 AM, Theo Hultberg <th...@iconara.net> wrote:
> I agree, it feels like a step backwards for the CQL protocol.
>
> T#
>
>
> On Wed, Mar 5, 2014 at 8:10 PM, Edward Capriolo <edlinuxguru@gmail.com
> >wrote:
>
> > Somewhat of an aside. But wasn't issues with client serialization issues
> > one of the reasons to get away from thrift? It seems like asking client
> > language to decode complex objects recreates the problem only with 1
> degree
> > more complexity.
> >
> >
> > On Wed, Mar 5, 2014 at 1:55 PM, Theo Hultberg <th...@iconara.net> wrote:
> >
> > > thanks! yeah, I meant user defined types, but thanks for the
> description
> > of
> > > general custom types too, it's good to know.
> > >
> > > T#
> > >
> > >
> > > On Tue, Mar 4, 2014 at 9:28 PM, Tyler Hobbs <ty...@datastax.com>
> wrote:
> > >
> > > > On Sat, Mar 1, 2014 at 5:01 AM, Theo Hultberg <th...@iconara.net>
> > wrote:
> > > >
> > > > > Mikhail, thanks, but I meant the reverse of that. Say the user
> > creates
> > > a
> > > > > prepared statement where one of the columns is a custom type, how
> do
> > > you
> > > > > serialize the arguments to the prepared statement? Do you accept
> > > anything
> > > > > and let C* complain, or do you make a best effort to shoehorn the
> > > object
> > > > > the user passed into something that looks like the custom type?
> > > > >
> > > >
> > > > Just to be clear, by "custom type", you still mean a user-defined
> type,
> > > > correct?
> > > >
> > > > At least in the python driver, it's treated the same as any other
> > > > (parametrized) type. For each Cassandra type (UTF8Type, Int32Type,
> > etc),
> > > > the driver will accept values of one or more types. If any of the
> > > subtypes
> > > > don't match this, the driver will raise an exception.
> > > >
> > > > If you're actually talking about custom types and not user-defined
> > types,
> > > > I'll explain what the python driver does. If the typestring (e.g.
> > > > org.apache.cassandra.db.marshal.MyType) isn't recognized, the driver
> > will
> > > > expect a binary string that it can pass directly to Cassandra for
> > values
> > > of
> > > > that type. If the user wants to add driver-level support for it (to
> > > enable
> > > > converting a python object to a binary string and vice-versa), they
> can
> > > > subclass cassandra.cqltypes.CassandraType and define a serialize()
> and
> > > > deserialize() method. The only condition is that the python
> classname
> > > must
> > > > match the typestring from cassandra, so for
> > > > org.apache.cassandra.db.marshal.MyType, the user will create a
> > > > MyType(CassandraType) class.
> > > >
> > > > --
> > > > Tyler Hobbs
> > > > DataStax <http://datastax.com/>
> > > >
> > >
> >
>
Re: How should clients handle the user defined types in 2.1?
Posted by Theo Hultberg <th...@iconara.net>.
I agree, it feels like a step backwards for the CQL protocol.
T#
On Wed, Mar 5, 2014 at 8:10 PM, Edward Capriolo <ed...@gmail.com>wrote:
> Somewhat of an aside. But wasn't issues with client serialization issues
> one of the reasons to get away from thrift? It seems like asking client
> language to decode complex objects recreates the problem only with 1 degree
> more complexity.
>
>
> On Wed, Mar 5, 2014 at 1:55 PM, Theo Hultberg <th...@iconara.net> wrote:
>
> > thanks! yeah, I meant user defined types, but thanks for the description
> of
> > general custom types too, it's good to know.
> >
> > T#
> >
> >
> > On Tue, Mar 4, 2014 at 9:28 PM, Tyler Hobbs <ty...@datastax.com> wrote:
> >
> > > On Sat, Mar 1, 2014 at 5:01 AM, Theo Hultberg <th...@iconara.net>
> wrote:
> > >
> > > > Mikhail, thanks, but I meant the reverse of that. Say the user
> creates
> > a
> > > > prepared statement where one of the columns is a custom type, how do
> > you
> > > > serialize the arguments to the prepared statement? Do you accept
> > anything
> > > > and let C* complain, or do you make a best effort to shoehorn the
> > object
> > > > the user passed into something that looks like the custom type?
> > > >
> > >
> > > Just to be clear, by "custom type", you still mean a user-defined type,
> > > correct?
> > >
> > > At least in the python driver, it's treated the same as any other
> > > (parametrized) type. For each Cassandra type (UTF8Type, Int32Type,
> etc),
> > > the driver will accept values of one or more types. If any of the
> > subtypes
> > > don't match this, the driver will raise an exception.
> > >
> > > If you're actually talking about custom types and not user-defined
> types,
> > > I'll explain what the python driver does. If the typestring (e.g.
> > > org.apache.cassandra.db.marshal.MyType) isn't recognized, the driver
> will
> > > expect a binary string that it can pass directly to Cassandra for
> values
> > of
> > > that type. If the user wants to add driver-level support for it (to
> > enable
> > > converting a python object to a binary string and vice-versa), they can
> > > subclass cassandra.cqltypes.CassandraType and define a serialize() and
> > > deserialize() method. The only condition is that the python classname
> > must
> > > match the typestring from cassandra, so for
> > > org.apache.cassandra.db.marshal.MyType, the user will create a
> > > MyType(CassandraType) class.
> > >
> > > --
> > > Tyler Hobbs
> > > DataStax <http://datastax.com/>
> > >
> >
>
Re: How should clients handle the user defined types in 2.1?
Posted by Edward Capriolo <ed...@gmail.com>.
Somewhat of an aside. But wasn't issues with client serialization issues
one of the reasons to get away from thrift? It seems like asking client
language to decode complex objects recreates the problem only with 1 degree
more complexity.
On Wed, Mar 5, 2014 at 1:55 PM, Theo Hultberg <th...@iconara.net> wrote:
> thanks! yeah, I meant user defined types, but thanks for the description of
> general custom types too, it's good to know.
>
> T#
>
>
> On Tue, Mar 4, 2014 at 9:28 PM, Tyler Hobbs <ty...@datastax.com> wrote:
>
> > On Sat, Mar 1, 2014 at 5:01 AM, Theo Hultberg <th...@iconara.net> wrote:
> >
> > > Mikhail, thanks, but I meant the reverse of that. Say the user creates
> a
> > > prepared statement where one of the columns is a custom type, how do
> you
> > > serialize the arguments to the prepared statement? Do you accept
> anything
> > > and let C* complain, or do you make a best effort to shoehorn the
> object
> > > the user passed into something that looks like the custom type?
> > >
> >
> > Just to be clear, by "custom type", you still mean a user-defined type,
> > correct?
> >
> > At least in the python driver, it's treated the same as any other
> > (parametrized) type. For each Cassandra type (UTF8Type, Int32Type, etc),
> > the driver will accept values of one or more types. If any of the
> subtypes
> > don't match this, the driver will raise an exception.
> >
> > If you're actually talking about custom types and not user-defined types,
> > I'll explain what the python driver does. If the typestring (e.g.
> > org.apache.cassandra.db.marshal.MyType) isn't recognized, the driver will
> > expect a binary string that it can pass directly to Cassandra for values
> of
> > that type. If the user wants to add driver-level support for it (to
> enable
> > converting a python object to a binary string and vice-versa), they can
> > subclass cassandra.cqltypes.CassandraType and define a serialize() and
> > deserialize() method. The only condition is that the python classname
> must
> > match the typestring from cassandra, so for
> > org.apache.cassandra.db.marshal.MyType, the user will create a
> > MyType(CassandraType) class.
> >
> > --
> > Tyler Hobbs
> > DataStax <http://datastax.com/>
> >
>
Re: How should clients handle the user defined types in 2.1?
Posted by Theo Hultberg <th...@iconara.net>.
thanks! yeah, I meant user defined types, but thanks for the description of
general custom types too, it's good to know.
T#
On Tue, Mar 4, 2014 at 9:28 PM, Tyler Hobbs <ty...@datastax.com> wrote:
> On Sat, Mar 1, 2014 at 5:01 AM, Theo Hultberg <th...@iconara.net> wrote:
>
> > Mikhail, thanks, but I meant the reverse of that. Say the user creates a
> > prepared statement where one of the columns is a custom type, how do you
> > serialize the arguments to the prepared statement? Do you accept anything
> > and let C* complain, or do you make a best effort to shoehorn the object
> > the user passed into something that looks like the custom type?
> >
>
> Just to be clear, by "custom type", you still mean a user-defined type,
> correct?
>
> At least in the python driver, it's treated the same as any other
> (parametrized) type. For each Cassandra type (UTF8Type, Int32Type, etc),
> the driver will accept values of one or more types. If any of the subtypes
> don't match this, the driver will raise an exception.
>
> If you're actually talking about custom types and not user-defined types,
> I'll explain what the python driver does. If the typestring (e.g.
> org.apache.cassandra.db.marshal.MyType) isn't recognized, the driver will
> expect a binary string that it can pass directly to Cassandra for values of
> that type. If the user wants to add driver-level support for it (to enable
> converting a python object to a binary string and vice-versa), they can
> subclass cassandra.cqltypes.CassandraType and define a serialize() and
> deserialize() method. The only condition is that the python classname must
> match the typestring from cassandra, so for
> org.apache.cassandra.db.marshal.MyType, the user will create a
> MyType(CassandraType) class.
>
> --
> Tyler Hobbs
> DataStax <http://datastax.com/>
>
Re: How should clients handle the user defined types in 2.1?
Posted by Tyler Hobbs <ty...@datastax.com>.
On Sat, Mar 1, 2014 at 5:01 AM, Theo Hultberg <th...@iconara.net> wrote:
> Mikhail, thanks, but I meant the reverse of that. Say the user creates a
> prepared statement where one of the columns is a custom type, how do you
> serialize the arguments to the prepared statement? Do you accept anything
> and let C* complain, or do you make a best effort to shoehorn the object
> the user passed into something that looks like the custom type?
>
Just to be clear, by "custom type", you still mean a user-defined type,
correct?
At least in the python driver, it's treated the same as any other
(parametrized) type. For each Cassandra type (UTF8Type, Int32Type, etc),
the driver will accept values of one or more types. If any of the subtypes
don't match this, the driver will raise an exception.
If you're actually talking about custom types and not user-defined types,
I'll explain what the python driver does. If the typestring (e.g.
org.apache.cassandra.db.marshal.MyType) isn't recognized, the driver will
expect a binary string that it can pass directly to Cassandra for values of
that type. If the user wants to add driver-level support for it (to enable
converting a python object to a binary string and vice-versa), they can
subclass cassandra.cqltypes.CassandraType and define a serialize() and
deserialize() method. The only condition is that the python classname must
match the typestring from cassandra, so for
org.apache.cassandra.db.marshal.MyType, the user will create a
MyType(CassandraType) class.
--
Tyler Hobbs
DataStax <http://datastax.com/>
Re: How should clients handle the user defined types in 2.1?
Posted by Theo Hultberg <th...@iconara.net>.
Mikhail, thanks, but I meant the reverse of that. Say the user creates a
prepared statement where one of the columns is a custom type, how do you
serialize the arguments to the prepared statement? Do you accept anything
and let C* complain, or do you make a best effort to shoehorn the object
the user passed into something that looks like the custom type?
T#
On Sat, Mar 1, 2014 at 1:12 AM, Mikhail Stepura <mikhail.stepura@outlook.com
> wrote:
> Hi Theo,
>
> Take a look at https://github.com/datastax/python-driver/blob/master/
> cassandra/cqltypes.py#L111
>
>
>
> On 2/25/14, 23:09, Theo Hultberg wrote:
>
>> thanks mikhail.
>>
>> it bothers me a bit that there is the possibility of arbitrarily deeply
>> nested types (it's great for usability, but a headache for me as a driver
>> implementer). it feels like just throwing a couple of regexes at this is
>> the wrong solution, the string has too much structure and hierarchy for
>> that and I think that not doing it with a proper parser will cause trouble
>> in the future. and then trying to use the parsed type structure to
>> properly
>> encode something that the user wants to store will be very complex. kind
>> of
>> like having to implement a whole orm. it's going to be a challenge.
>>
>> I see that you took the parser route in the python driver so I can at
>> least
>> get some inspiration from there. what do you do when the user passes an
>> object in a custom type field and that object doesn't look like the type
>> at
>> all?
>>
>> T#
>>
>>
>> On Tue, Feb 25, 2014 at 8:16 PM, Mikhail Stepura <
>> mikhail.stepura@outlook.com> wrote:
>>
>> I just realized that your driver returns fields' names in the type itself
>>> (which unfortunately is not the case with python driver) so you don't
>>> need
>>> step #3.
>>>
>>> -M
>>>
>>>
>>>
>>> On 2/25/14, 10:42, Mikhail Stepura wrote:
>>>
>>> The driver shall parse that.
>>>> I'm not sure if there is a doc for that, but here is what I did for
>>>> CQLSH
>>>>
>>>> 1. UserType is a CompositeType, where 1s subtype is a KS name, and 2nd
>>>> is hex-encoded name of the type
>>>> 2. Remainder of subtypes are types of Usertype's columns. So you can
>>>> easily decode *values* for fields
>>>> 3. Information about field *names* is stored in system.schema_usertypes
>>>> table
>>>> 4. The driver has to combine pieces 1-3 and create a new class/type for
>>>> a user. It was easy in Python, I guess it should be easy in Ruby as well
>>>>
>>>> -M
>>>>
>>>>
>>>> On 2/22/14, 12:29, Theo Hultberg wrote:
>>>>
>>>> Hi,
>>>>>
>>>>> Is there any documentation on how CQL clients should handle the new
>>>>> user
>>>>> defined types coming in 2.1? There's nothing in the protocol
>>>>> specification
>>>>> on how to handle custom types as far as I can understand.
>>>>>
>>>>> For example, I tried creating the "address" type from the description
>>>>> of
>>>>> CASSANDRA-5590, and this is how its metadata looks (the metadata for a
>>>>> query contains a column with a custom type and this is the description
>>>>> of
>>>>> it):
>>>>>
>>>>> org.apache.cassandra.db.marshal.UserType(user_defined_
>>>>> types,61646472657373,737472656574:org.apache.
>>>>> cassandra.db.marshal.UTF8Type,63697479:org.apache.cassandra.
>>>>> db.marshal.UTF8Type,7a69705f636f6465:org.apache.cassandra.db.marshal.
>>>>> Int32Type,70686f6e6573:org.apache.cassandra.db.marshal.
>>>>> SetType(org.apache.cassandra.db.marshal.UTF8Type))
>>>>>
>>>>>
>>>>> Is the client supposed to parse that description, and in that case
>>>>> how? I
>>>>> could probably figure it out but it would be great if someone could
>>>>> point
>>>>> me to the right docs.
>>>>>
>>>>> yours,
>>>>> Theo (author of cql-rb, the Ruby driver)
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>
>
Re: How should clients handle the user defined types in 2.1?
Posted by Mikhail Stepura <mi...@outlook.com>.
Hi Theo,
Take a look at
https://github.com/datastax/python-driver/blob/master/cassandra/cqltypes.py#L111
On 2/25/14, 23:09, Theo Hultberg wrote:
> thanks mikhail.
>
> it bothers me a bit that there is the possibility of arbitrarily deeply
> nested types (it's great for usability, but a headache for me as a driver
> implementer). it feels like just throwing a couple of regexes at this is
> the wrong solution, the string has too much structure and hierarchy for
> that and I think that not doing it with a proper parser will cause trouble
> in the future. and then trying to use the parsed type structure to properly
> encode something that the user wants to store will be very complex. kind of
> like having to implement a whole orm. it's going to be a challenge.
>
> I see that you took the parser route in the python driver so I can at least
> get some inspiration from there. what do you do when the user passes an
> object in a custom type field and that object doesn't look like the type at
> all?
>
> T#
>
>
> On Tue, Feb 25, 2014 at 8:16 PM, Mikhail Stepura <
> mikhail.stepura@outlook.com> wrote:
>
>> I just realized that your driver returns fields' names in the type itself
>> (which unfortunately is not the case with python driver) so you don't need
>> step #3.
>>
>> -M
>>
>>
>>
>> On 2/25/14, 10:42, Mikhail Stepura wrote:
>>
>>> The driver shall parse that.
>>> I'm not sure if there is a doc for that, but here is what I did for CQLSH
>>>
>>> 1. UserType is a CompositeType, where 1s subtype is a KS name, and 2nd
>>> is hex-encoded name of the type
>>> 2. Remainder of subtypes are types of Usertype's columns. So you can
>>> easily decode *values* for fields
>>> 3. Information about field *names* is stored in system.schema_usertypes
>>> table
>>> 4. The driver has to combine pieces 1-3 and create a new class/type for
>>> a user. It was easy in Python, I guess it should be easy in Ruby as well
>>>
>>> -M
>>>
>>>
>>> On 2/22/14, 12:29, Theo Hultberg wrote:
>>>
>>>> Hi,
>>>>
>>>> Is there any documentation on how CQL clients should handle the new user
>>>> defined types coming in 2.1? There's nothing in the protocol
>>>> specification
>>>> on how to handle custom types as far as I can understand.
>>>>
>>>> For example, I tried creating the "address" type from the description of
>>>> CASSANDRA-5590, and this is how its metadata looks (the metadata for a
>>>> query contains a column with a custom type and this is the description of
>>>> it):
>>>>
>>>> org.apache.cassandra.db.marshal.UserType(user_defined_
>>>> types,61646472657373,737472656574:org.apache.
>>>> cassandra.db.marshal.UTF8Type,63697479:org.apache.cassandra.
>>>> db.marshal.UTF8Type,7a69705f636f6465:org.apache.cassandra.db.marshal.
>>>> Int32Type,70686f6e6573:org.apache.cassandra.db.marshal.
>>>> SetType(org.apache.cassandra.db.marshal.UTF8Type))
>>>>
>>>>
>>>> Is the client supposed to parse that description, and in that case how? I
>>>> could probably figure it out but it would be great if someone could point
>>>> me to the right docs.
>>>>
>>>> yours,
>>>> Theo (author of cql-rb, the Ruby driver)
>>>>
>>>>
>>>
>>>
>>>
>>
>>
>
Re: How should clients handle the user defined types in 2.1?
Posted by Theo Hultberg <th...@iconara.net>.
thanks mikhail.
it bothers me a bit that there is the possibility of arbitrarily deeply
nested types (it's great for usability, but a headache for me as a driver
implementer). it feels like just throwing a couple of regexes at this is
the wrong solution, the string has too much structure and hierarchy for
that and I think that not doing it with a proper parser will cause trouble
in the future. and then trying to use the parsed type structure to properly
encode something that the user wants to store will be very complex. kind of
like having to implement a whole orm. it's going to be a challenge.
I see that you took the parser route in the python driver so I can at least
get some inspiration from there. what do you do when the user passes an
object in a custom type field and that object doesn't look like the type at
all?
T#
On Tue, Feb 25, 2014 at 8:16 PM, Mikhail Stepura <
mikhail.stepura@outlook.com> wrote:
> I just realized that your driver returns fields' names in the type itself
> (which unfortunately is not the case with python driver) so you don't need
> step #3.
>
> -M
>
>
>
> On 2/25/14, 10:42, Mikhail Stepura wrote:
>
>> The driver shall parse that.
>> I'm not sure if there is a doc for that, but here is what I did for CQLSH
>>
>> 1. UserType is a CompositeType, where 1s subtype is a KS name, and 2nd
>> is hex-encoded name of the type
>> 2. Remainder of subtypes are types of Usertype's columns. So you can
>> easily decode *values* for fields
>> 3. Information about field *names* is stored in system.schema_usertypes
>> table
>> 4. The driver has to combine pieces 1-3 and create a new class/type for
>> a user. It was easy in Python, I guess it should be easy in Ruby as well
>>
>> -M
>>
>>
>> On 2/22/14, 12:29, Theo Hultberg wrote:
>>
>>> Hi,
>>>
>>> Is there any documentation on how CQL clients should handle the new user
>>> defined types coming in 2.1? There's nothing in the protocol
>>> specification
>>> on how to handle custom types as far as I can understand.
>>>
>>> For example, I tried creating the "address" type from the description of
>>> CASSANDRA-5590, and this is how its metadata looks (the metadata for a
>>> query contains a column with a custom type and this is the description of
>>> it):
>>>
>>> org.apache.cassandra.db.marshal.UserType(user_defined_
>>> types,61646472657373,737472656574:org.apache.
>>> cassandra.db.marshal.UTF8Type,63697479:org.apache.cassandra.
>>> db.marshal.UTF8Type,7a69705f636f6465:org.apache.cassandra.db.marshal.
>>> Int32Type,70686f6e6573:org.apache.cassandra.db.marshal.
>>> SetType(org.apache.cassandra.db.marshal.UTF8Type))
>>>
>>>
>>> Is the client supposed to parse that description, and in that case how? I
>>> could probably figure it out but it would be great if someone could point
>>> me to the right docs.
>>>
>>> yours,
>>> Theo (author of cql-rb, the Ruby driver)
>>>
>>>
>>
>>
>>
>
>
Re: How should clients handle the user defined types in 2.1?
Posted by Mikhail Stepura <mi...@outlook.com>.
I just realized that your driver returns fields' names in the type
itself (which unfortunately is not the case with python driver) so you
don't need step #3.
-M
On 2/25/14, 10:42, Mikhail Stepura wrote:
> The driver shall parse that.
> I'm not sure if there is a doc for that, but here is what I did for CQLSH
>
> 1. UserType is a CompositeType, where 1s subtype is a KS name, and 2nd
> is hex-encoded name of the type
> 2. Remainder of subtypes are types of Usertype's columns. So you can
> easily decode *values* for fields
> 3. Information about field *names* is stored in system.schema_usertypes
> table
> 4. The driver has to combine pieces 1-3 and create a new class/type for
> a user. It was easy in Python, I guess it should be easy in Ruby as well
>
> -M
>
>
> On 2/22/14, 12:29, Theo Hultberg wrote:
>> Hi,
>>
>> Is there any documentation on how CQL clients should handle the new user
>> defined types coming in 2.1? There's nothing in the protocol
>> specification
>> on how to handle custom types as far as I can understand.
>>
>> For example, I tried creating the "address" type from the description of
>> CASSANDRA-5590, and this is how its metadata looks (the metadata for a
>> query contains a column with a custom type and this is the description of
>> it):
>>
>> org.apache.cassandra.db.marshal.UserType(user_defined_types,61646472657373,737472656574:org.apache.cassandra.db.marshal.UTF8Type,63697479:org.apache.cassandra.db.marshal.UTF8Type,7a69705f636f6465:org.apache.cassandra.db.marshal.Int32Type,70686f6e6573:org.apache.cassandra.db.marshal.SetType(org.apache.cassandra.db.marshal.UTF8Type))
>>
>>
>> Is the client supposed to parse that description, and in that case how? I
>> could probably figure it out but it would be great if someone could point
>> me to the right docs.
>>
>> yours,
>> Theo (author of cql-rb, the Ruby driver)
>>
>
>
>
Re: How should clients handle the user defined types in 2.1?
Posted by Mikhail Stepura <mi...@outlook.com>.
The driver shall parse that.
I'm not sure if there is a doc for that, but here is what I did for CQLSH
1. UserType is a CompositeType, where 1s subtype is a KS name, and 2nd
is hex-encoded name of the type
2. Remainder of subtypes are types of Usertype's columns. So you can
easily decode *values* for fields
3. Information about field *names* is stored in system.schema_usertypes
table
4. The driver has to combine pieces 1-3 and create a new class/type for
a user. It was easy in Python, I guess it should be easy in Ruby as well
-M
On 2/22/14, 12:29, Theo Hultberg wrote:
> Hi,
>
> Is there any documentation on how CQL clients should handle the new user
> defined types coming in 2.1? There's nothing in the protocol specification
> on how to handle custom types as far as I can understand.
>
> For example, I tried creating the "address" type from the description of
> CASSANDRA-5590, and this is how its metadata looks (the metadata for a
> query contains a column with a custom type and this is the description of
> it):
>
> org.apache.cassandra.db.marshal.UserType(user_defined_types,61646472657373,737472656574:org.apache.cassandra.db.marshal.UTF8Type,63697479:org.apache.cassandra.db.marshal.UTF8Type,7a69705f636f6465:org.apache.cassandra.db.marshal.Int32Type,70686f6e6573:org.apache.cassandra.db.marshal.SetType(org.apache.cassandra.db.marshal.UTF8Type))
>
> Is the client supposed to parse that description, and in that case how? I
> could probably figure it out but it would be great if someone could point
> me to the right docs.
>
> yours,
> Theo (author of cql-rb, the Ruby driver)
>