You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Christian Decker <de...@gmail.com> on 2010/09/24 17:56:17 UTC

LongType from user input

Hi all,

I'm having quite a dilemma with the CompareWith attribute. The Problem is
that I have numeric IDs that I'd like to use as row keys, only that I also
have to offer a possibility to let users input them from std input. Since I
cannot ask my users to input an 8byte sequence representing the ID they'd
like, I was about to turn to UTF8, when I remembered that they are compared
lexicographically, so that 100 actually comes before 2, which kills key
slices. Also I cannot just code a converter in since this is supposed to be
a used with multiple columnfamilies, so just converting an integer read into
8bytes isn't going to work either.
Any tricks for this one?

Regards,
Chris

Re: LongType from user input

Posted by Christian Decker <de...@gmail.com>.
BTW is there an official Roadmap which states when the 0.7 release is to be
expected?

On Thu, Sep 30, 2010 at 11:38 AM, Christian Decker <
decker.christian@gmail.com> wrote:

> I just read through the tickets on Jira, and it appears that indices are
> implemented in the 0.7 source tree, but I cannot find any pointer on how to
> use them. I'll be trying to create a custom CassandraStorage that loads data
> through the indices, anyone else interested?
>
> Regards,
> Chris
>
>
> On Thu, Sep 30, 2010 at 10:56 AM, Aaron Morton <aa...@thelastpickle.com>wrote:
>
>> AFAIK indexes are still in dev. The only example is in the system_tests.py
>> in the source tree.
>>
>> Aaron
>>
>>
>> On 30 Sep 2010, at 20:10, Christian Decker <de...@gmail.com>
>> wrote:
>>
>> Apparently I have blanked the 0.7 completely out of my memory. I was
>> trying to implement application layer indices and ignored the fact that
>> Cassandra 0.7 is implementing them by default. I found ticket CASSANDRA-749
>> about the indices and am reading through the code right now, but is there a
>> higher level overview and a tutorial on how to get things started with these
>> indices (and maybe some inner workings)? This might actually solve all of my
>> problems I'm having right now :-)
>>
>> Regards,
>> Chris
>>
>>
>> On Mon, Sep 27, 2010 at 3:45 AM, Aaron Morton < <aa...@thelastpickle.com>
>> aaron@thelastpickle.com> wrote:
>>
>>> The only thing I can think of is that values need to be in the correct
>>> byte format when used in indexes in 0.7. Take a look at the types.py module
>>> in the pycassa client  <http://github.com/pycassa/pycassa>
>>> http://github.com/pycassa/pycassa for an example of which values need to
>>> be byte packed.
>>>
>>> How is your pig function working against cassandra? Is it using the
>>> ColumnFamilyRecordReader? . The code in the internal RowIterator for that
>>> class has an example calling the cluster to get to the comparators.
>>>
>>> Aaron
>>>
>>>
>>> On 27 Sep, 2010,at 03:11 AM, Christian Decker <<d...@gmail.com>
>>> decker.christian@gmail.com> wrote:
>>>
>>> Hi Aaron,
>>>
>>> what changes can I expect in the 0.7 release regarding Comparison and
>>> Parameters? My problem is mainly that I want to take Strings from stdin (or
>>> Pig Scripts for that matter) and convert them in such a way that they are
>>> interpreted correctly and converted to the corresponding byte representation
>>> to use them in column names and keys.
>>>
>>> Regards,
>>> Chris
>>>
>>> On Sun, Sep 26, 2010 at 5:20 AM, Aaron Morton <<a...@thelastpickle.com>
>>> aaron@thelastpickle.com> wrote:
>>>
>>>> Things a changing in v0.7, the row keys are byte arrays.
>>>>
>>>> Not sure I understand your other concerns.
>>>>
>>>> Aaron
>>>>
>>>>
>>>> On 25 Sep 2010, at 08:10, Christian Decker <<d...@gmail.com>
>>>> decker.christian@gmail.com> wrote:
>>>>
>>>>
>>>> Thanks for your quick answer, I think I'll use an affix to sort of cast
>>>> the keys, ranges and others from their textual representation (from Pig) to
>>>> the desired byte representation, since I just noticed that the keys for the
>>>> rows themselfs are always UTF8 interpreted, and since I want to make
>>>> key-range as well as slice queries, I'll be better off this way I think.
>>>> I'll just add a 'L' for Long and 'U' for UUID (of any kind).
>>>>  Or is there a better way that I just can't see from my beginners angle?
>>>> :-)thing
>>>>
>>>> Regards,
>>>> Chris
>>>>
>>>>
>>>> On Fri, Sep 24, 2010 at 6:35 PM, Tyler Hobbs < <ty...@riptano.com>
>>>> tyler@riptano.com> wrote:
>>>>
>>>>> Yes, you can use describe_keyspace() and then look through the
>>>>> results.  It's a little ugly in 0.6, but it works
>>>>>
>>>>> - Tyler
>>>>>
>>>>>
>>>>>
>>>>> On Fri, Sep 24, 2010 at 11:25 AM, Christian Decker <<d...@gmail.com>
>>>>> decker.christian@gmail.com> wrote:
>>>>>
>>>>>> Well I'm writing a loading function for Pig, and as it happens I want
>>>>>> to be able to load slices from cassandra which are specified in the pig
>>>>>> script (thus the input from stdin) but the ColumnFamily from which to read
>>>>>> the data is another parameter and some of the CFs have UTF8, UUID, TimeUUID
>>>>>> or Long types for their keys and columns, so simply converting everything I
>>>>>> get to an 8byte long would break compatibility with the others.
>>>>>> Now thinking about it I attacked the whole problem in a weird way,
>>>>>> since UUID types won't work either.
>>>>>> So let me change my question slightly, is there a way in 0.6 to detect
>>>>>> the compareWith type on a running cluster? That way I could convert it to
>>>>>> the right type :D
>>>>>>
>>>>>> Regards,
>>>>>> Chris
>>>>>>
>>>>>>
>>>>>> On Fri, Sep 24, 2010 at 6:09 PM, Tyler Hobbs < <ty...@riptano.com>
>>>>>> tyler@riptano.com> wrote:
>>>>>>
>>>>>>> I'm not sure I understand why using this with multiple column
>>>>>>> families prevents you from converting it.  Could you clarify this?
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Fri, Sep 24, 2010 at 10:56 AM, Christian Decker <<d...@gmail.com>
>>>>>>> decker.christian@gmail.com> wrote:
>>>>>>>
>>>>>>>> Hi all,
>>>>>>>>
>>>>>>>> I'm having quite a dilemma with the CompareWith attribute. The
>>>>>>>> Problem is that I have numeric IDs that I'd like to use as row keys, only
>>>>>>>> that I also have to offer a possibility to let users input them from std
>>>>>>>> input. Since I cannot ask my users to input an 8byte sequence representing
>>>>>>>> the ID they'd like, I was about to turn to UTF8, when I remembered that they
>>>>>>>> are compared lexicographically, so that 100 actually comes before 2, which
>>>>>>>> kills key slices. Also I cannot just code a converter in since this is
>>>>>>>> supposed to be a used with multiple columnfamilies, so just converting an
>>>>>>>> integer read into 8bytes isn't going to work either.
>>>>>>>> Any tricks for this one?
>>>>>>>>
>>>>>>>> Regards,
>>>>>>>> Chris
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>

Re: LongType from user input

Posted by Aaron Morton <aa...@thelastpickle.com>.
It's specified as part of the schema functions such as system_add_keyspace So it will depend on which language/client you are using to add the keyspace. 

Take a look at /test/system/test_thrift_server.py in test_column_validators() line 1237  for an example of using the ThriftAPI in Python (you would normally use a higher level api) and take a look at interface/cassandra.thrift for the raw interface definition. The sample Keyspace1 that is included in the default cassandra.yaml also includes a CF with an indexed column (Indexed1).

Specifically, a KsDef has a list of CFDef's. THe CfDef has an optional list of ColumnDef's , indexed columns are defined in the ColumnDef. 

AFAIK KEYS is the only index_type at the moment, and EQUAL is the only operator supported. 

Aaron


On 01 Oct, 2010,at 08:39 AM, Christian Decker <de...@gmail.com> wrote:

Sorry for my continued questioning, by now I must have bored everyone about this topic. I have one last question: how do you create such an index? I found only ColumnFamilyStoreText.testIndexCreate(), but it appears that this uses internal API, and thus cannot be invoked from a client side or through JMX, right? From there on I should be able to find my way around the system :-)

On Thu, Sep 30, 2010 at 4:56 PM, Stu Hood <st...@rackspace.com> wrote:
Take a look at the get_indexed_slices method in the 0.7.0-beta Thrift interface.


-----Original Message-----
From: "Christian Decker" <de...@gmail.com>
Sent: Thursday, September 30, 2010 4:38am
To: user@cassandra.apache.org
Subject: Re: LongType from user input

I just read through the tickets on Jira, and it appears that indices are
implemented in the 0.7 source tree, but I cannot find any pointer on how to
use them. I'll be trying to create a custom CassandraStorage that loads data
through the indices, anyone else interested?

Regards,
Chris

On Thu, Sep 30, 2010 at 10:56 AM, Aaron Morton <aa...@thelastpickle.com>wrote:

> AFAIK indexes are still in dev. The only example is in the system_tests.py
> in the source tree.
>
> Aaron
>
>
> On 30 Sep 2010, at 20:10, Christian Decker <de...@gmail.com>
> wrote:
>
> Apparently I have blanked the 0.7 completely out of my memory. I was trying
> to implement application layer indices and ignored the fact that Cassandra
> 0.7 is implementing them by default. I found ticket CASSANDRA-749 about the
> indices and am reading through the code right now, but is there a higher
> level overview and a tutorial on how to get things started with these
> indices (and maybe some inner workings)? This might actually solve all of my
> problems I'm having right now :-)
>
> Regards,
> Chris
>
>
> On Mon, Sep 27, 2010 at 3:45 AM, Aaron Morton < <aa...@thelastpickle.com>
> aaron@thelastpickle.com> wrote:
>
>> The only thing I can think of is that values need to be in the correct
>> byte format when used in indexes in 0.7. Take a look at the types.py module
>> in the pycassa client  <http://github.com/pycassa/pycassa>
>> http://github.com/pycassa/pycassa for an example of which values need to
>> be byte packed.
>>
>> How is your pig function working against cassandra? Is it using the
>> ColumnFamilyRecordReader? . The code in the internal RowIterator for that
>> class has an example calling the cluster to get to the comparators.
>>
>> Aaron
>>
>>
>> On 27 Sep, 2010,at 03:11 AM, Christian Decker <<d...@gmail.com>
>> decker.christian@gmail.com> wrote:
>>
>> Hi Aaron,
>>
>> what changes can I expect in the 0.7 release regarding Comparison and
>> Parameters? My problem is mainly that I want to take Strings from stdin (or
>> Pig Scripts for that matter) and convert them in such a way that they are
>> interpreted correctly and converted to the corresponding byte representation
>> to use them in column names and keys.
>>
>> Regards,
>> Chris
>>
>> On Sun, Sep 26, 2010 at 5:20 AM, Aaron Morton < <aa...@thelastpickle.com>
>> aaron@thelastpickle.com> wrote:
>>
>>> Things a changing in v0.7, the row keys are byte arrays.
>>>
>>> Not sure I understand your other concerns.
>>>
>>> Aaron
>>>
>>>
>>> On 25 Sep 2010, at 08:10, Christian Decker <<d...@gmail.com>
>>> decker.christian@gmail.com> wrote:
>>>
>>>
>>> Thanks for your quick answer, I think I'll use an affix to sort of cast
>>> the keys, ranges and others from their textual representation (from Pig) to
>>> the desired byte representation, since I just noticed that the keys for the
>>> rows themselfs are always UTF8 interpreted, and since I want to make
>>> key-range as well as slice queries, I'll be better off this way I think.
>>> I'll just add a 'L' for Long and 'U' for UUID (of any kind).
>>>  Or is there a better way that I just can't see from my beginners angle?
>>> :-)thing
>>>
>>> Regards,
>>> Chris
>>>
>>>
>>> On Fri, Sep 24, 2010 at 6:35 PM, Tyler Hobbs < <ty...@riptano.com>
>>> tyler@riptano.com> wrote:
>>>
>>>> Yes, you can use describe_keyspace() and then look through the results
>>>> It's a little ugly in 0.6, but it works
>>>>
>>>> - Tyler
>>>>
>>>>
>>>>
>>>> On Fri, Sep 24, 2010 at 11:25 AM, Christian Decker <<d...@gmail.com>
>>>> decker.christian@gmail.com> wrote:
>>>>
>>>>> Well I'm writing a loading function for Pig, and as it happens I want
>>>>> to be able to load slices from cassandra which are specified in the pig
>>>>> script (thus the input from stdin) but the ColumnFamily from which to read
>>>>> the data is another parameter and some of the CFs have UTF8, UUID, TimeUUID
>>>>> or Long types for their keys and columns, so simply converting everything I
>>>>> get to an 8byte long would break compatibility with the others.
>>>>> Now thinking about it I attacked the whole problem in a weird way,
>>>>> since UUID types won't work either.
>>>>> So let me change my question slightly, is there a way in 0.6 to detect
>>>>> the compareWith type on a running cluster? That way I could convert it to
>>>>> the right type :D
>>>>>
>>>>> Regards,
>>>>> Chris
>>>>>
>>>>>
>>>>> On Fri, Sep 24, 2010 at 6:09 PM, Tyler Hobbs < <ty...@riptano.com>
>>>>> tyler@riptano.com> wrote:
>>>>>
>>>>>> I'm not sure I understand why using this with multiple column families
>>>>>> prevents you from converting it.  Could you clarify this?
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Fri, Sep 24, 2010 at 10:56 AM, Christian Decker <<d...@gmail.com>

>>>>>> decker.christian@gmail.com> wrote:
>>>>>>
>>>>>>> Hi all,
>>>>>>>
>>>>>>> I'm having quite a dilemma with the CompareWith attribute. The
>>>>>>> Problem is that I have numeric IDs that I'd like to use as row keys, only
>>>>>>> that I also have to offer a possibility to let users input them from std
>>>>>>> input. Since I cannot ask my users to input an 8byte sequence representing
>>>>>>> the ID they'd like, I was about to turn to UTF8, when I remembered that they
>>>>>>> are compared lexicographically, so that 100 actually comes before 2, which
>>>>>>> kills key slices. Also I cannot just code a converter in since this is
>>>>>>> supposed to be a used with multiple columnfamilies, so just converting an
>>>>>>> integer read into 8bytes isn't going to work either.
>>>>>>> Any tricks for this one?
>>>>>>>
>>>>>>> Regards,
>>>>>>> Chris
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>




Re: LongType from user input

Posted by Christian Decker <de...@gmail.com>.
Sorry for my continued questioning, by now I must have bored everyone about
this topic. I have one last question: how do you create such an index? I
found only ColumnFamilyStoreText.testIndexCreate(), but it appears that this
uses internal API, and thus cannot be invoked from a client side or through
JMX, right? From there on I should be able to find my way around the system
:-)

On Thu, Sep 30, 2010 at 4:56 PM, Stu Hood <st...@rackspace.com> wrote:

> Take a look at the get_indexed_slices method in the 0.7.0-beta Thrift
> interface.
>
> -----Original Message-----
> From: "Christian Decker" <de...@gmail.com>
> Sent: Thursday, September 30, 2010 4:38am
> To: user@cassandra.apache.org
> Subject: Re: LongType from user input
>
> I just read through the tickets on Jira, and it appears that indices are
> implemented in the 0.7 source tree, but I cannot find any pointer on how to
> use them. I'll be trying to create a custom CassandraStorage that loads
> data
> through the indices, anyone else interested?
>
> Regards,
> Chris
>
> On Thu, Sep 30, 2010 at 10:56 AM, Aaron Morton <aaron@thelastpickle.com
> >wrote:
>
> > AFAIK indexes are still in dev. The only example is in the
> system_tests.py
> > in the source tree.
> >
> > Aaron
> >
> >
> > On 30 Sep 2010, at 20:10, Christian Decker <de...@gmail.com>
> > wrote:
> >
> > Apparently I have blanked the 0.7 completely out of my memory. I was
> trying
> > to implement application layer indices and ignored the fact that
> Cassandra
> > 0.7 is implementing them by default. I found ticket CASSANDRA-749 about
> the
> > indices and am reading through the code right now, but is there a higher
> > level overview and a tutorial on how to get things started with these
> > indices (and maybe some inner workings)? This might actually solve all of
> my
> > problems I'm having right now :-)
> >
> > Regards,
> > Chris
> >
> >
> > On Mon, Sep 27, 2010 at 3:45 AM, Aaron Morton < <aaron@thelastpickle.com
> >
> > aaron@thelastpickle.com> wrote:
> >
> >> The only thing I can think of is that values need to be in the correct
> >> byte format when used in indexes in 0.7. Take a look at the types.py
> module
> >> in the pycassa client  <http://github.com/pycassa/pycassa>
> >> http://github.com/pycassa/pycassa for an example of which values need
> to
> >> be byte packed.
> >>
> >> How is your pig function working against cassandra? Is it using the
> >> ColumnFamilyRecordReader? . The code in the internal RowIterator for
> that
> >> class has an example calling the cluster to get to the comparators.
> >>
> >> Aaron
> >>
> >>
> >> On 27 Sep, 2010,at 03:11 AM, Christian Decker <<
> decker.christian@gmail.com>
> >> decker.christian@gmail.com> wrote:
> >>
> >> Hi Aaron,
> >>
> >> what changes can I expect in the 0.7 release regarding Comparison and
> >> Parameters? My problem is mainly that I want to take Strings from stdin
> (or
> >> Pig Scripts for that matter) and convert them in such a way that they
> are
> >> interpreted correctly and converted to the corresponding byte
> representation
> >> to use them in column names and keys.
> >>
> >> Regards,
> >> Chris
> >>
> >> On Sun, Sep 26, 2010 at 5:20 AM, Aaron Morton < <
> aaron@thelastpickle.com>
> >> aaron@thelastpickle.com> wrote:
> >>
> >>> Things a changing in v0.7, the row keys are byte arrays.
> >>>
> >>> Not sure I understand your other concerns.
> >>>
> >>> Aaron
> >>>
> >>>
> >>> On 25 Sep 2010, at 08:10, Christian Decker <<
> decker.christian@gmail.com>
> >>> decker.christian@gmail.com> wrote:
> >>>
> >>>
> >>> Thanks for your quick answer, I think I'll use an affix to sort of cast
> >>> the keys, ranges and others from their textual representation (from
> Pig) to
> >>> the desired byte representation, since I just noticed that the keys for
> the
> >>> rows themselfs are always UTF8 interpreted, and since I want to make
> >>> key-range as well as slice queries, I'll be better off this way I
> think.
> >>> I'll just add a 'L' for Long and 'U' for UUID (of any kind).
> >>>  Or is there a better way that I just can't see from my beginners
> angle?
> >>> :-)thing
> >>>
> >>> Regards,
> >>> Chris
> >>>
> >>>
> >>> On Fri, Sep 24, 2010 at 6:35 PM, Tyler Hobbs < <ty...@riptano.com><
> tyler@riptano.com>
> >>> tyler@riptano.com> wrote:
> >>>
> >>>> Yes, you can use describe_keyspace() and then look through the
> results.
> >>>> It's a little ugly in 0.6, but it works
> >>>>
> >>>> - Tyler
> >>>>
> >>>>
> >>>>
> >>>> On Fri, Sep 24, 2010 at 11:25 AM, Christian Decker <<
> decker.christian@gmail.com><de...@gmail.com>
> >>>> decker.christian@gmail.com> wrote:
> >>>>
> >>>>> Well I'm writing a loading function for Pig, and as it happens I want
> >>>>> to be able to load slices from cassandra which are specified in the
> pig
> >>>>> script (thus the input from stdin) but the ColumnFamily from which to
> read
> >>>>> the data is another parameter and some of the CFs have UTF8, UUID,
> TimeUUID
> >>>>> or Long types for their keys and columns, so simply converting
> everything I
> >>>>> get to an 8byte long would break compatibility with the others.
> >>>>> Now thinking about it I attacked the whole problem in a weird way,
> >>>>> since UUID types won't work either.
> >>>>> So let me change my question slightly, is there a way in 0.6 to
> detect
> >>>>> the compareWith type on a running cluster? That way I could convert
> it to
> >>>>> the right type :D
> >>>>>
> >>>>> Regards,
> >>>>> Chris
> >>>>>
> >>>>>
> >>>>> On Fri, Sep 24, 2010 at 6:09 PM, Tyler Hobbs < <ty...@riptano.com><
> tyler@riptano.com>
> >>>>> tyler@riptano.com> wrote:
> >>>>>
> >>>>>> I'm not sure I understand why using this with multiple column
> families
> >>>>>> prevents you from converting it.  Could you clarify this?
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> On Fri, Sep 24, 2010 at 10:56 AM, Christian Decker <<
> decker.christian@gmail.com><de...@gmail.com>
> >>>>>> decker.christian@gmail.com> wrote:
> >>>>>>
> >>>>>>> Hi all,
> >>>>>>>
> >>>>>>> I'm having quite a dilemma with the CompareWith attribute. The
> >>>>>>> Problem is that I have numeric IDs that I'd like to use as row
> keys, only
> >>>>>>> that I also have to offer a possibility to let users input them
> from std
> >>>>>>> input. Since I cannot ask my users to input an 8byte sequence
> representing
> >>>>>>> the ID they'd like, I was about to turn to UTF8, when I remembered
> that they
> >>>>>>> are compared lexicographically, so that 100 actually comes before
> 2, which
> >>>>>>> kills key slices. Also I cannot just code a converter in since this
> is
> >>>>>>> supposed to be a used with multiple columnfamilies, so just
> converting an
> >>>>>>> integer read into 8bytes isn't going to work either.
> >>>>>>> Any tricks for this one?
> >>>>>>>
> >>>>>>> Regards,
> >>>>>>> Chris
> >>>>>>>
> >>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>
> >>
> >
>
>
>

Re: LongType from user input

Posted by Stu Hood <st...@rackspace.com>.
Take a look at the get_indexed_slices method in the 0.7.0-beta Thrift interface.

-----Original Message-----
From: "Christian Decker" <de...@gmail.com>
Sent: Thursday, September 30, 2010 4:38am
To: user@cassandra.apache.org
Subject: Re: LongType from user input

I just read through the tickets on Jira, and it appears that indices are
implemented in the 0.7 source tree, but I cannot find any pointer on how to
use them. I'll be trying to create a custom CassandraStorage that loads data
through the indices, anyone else interested?

Regards,
Chris

On Thu, Sep 30, 2010 at 10:56 AM, Aaron Morton <aa...@thelastpickle.com>wrote:

> AFAIK indexes are still in dev. The only example is in the system_tests.py
> in the source tree.
>
> Aaron
>
>
> On 30 Sep 2010, at 20:10, Christian Decker <de...@gmail.com>
> wrote:
>
> Apparently I have blanked the 0.7 completely out of my memory. I was trying
> to implement application layer indices and ignored the fact that Cassandra
> 0.7 is implementing them by default. I found ticket CASSANDRA-749 about the
> indices and am reading through the code right now, but is there a higher
> level overview and a tutorial on how to get things started with these
> indices (and maybe some inner workings)? This might actually solve all of my
> problems I'm having right now :-)
>
> Regards,
> Chris
>
>
> On Mon, Sep 27, 2010 at 3:45 AM, Aaron Morton < <aa...@thelastpickle.com>
> aaron@thelastpickle.com> wrote:
>
>> The only thing I can think of is that values need to be in the correct
>> byte format when used in indexes in 0.7. Take a look at the types.py module
>> in the pycassa client  <http://github.com/pycassa/pycassa>
>> http://github.com/pycassa/pycassa for an example of which values need to
>> be byte packed.
>>
>> How is your pig function working against cassandra? Is it using the
>> ColumnFamilyRecordReader? . The code in the internal RowIterator for that
>> class has an example calling the cluster to get to the comparators.
>>
>> Aaron
>>
>>
>> On 27 Sep, 2010,at 03:11 AM, Christian Decker <<d...@gmail.com>
>> decker.christian@gmail.com> wrote:
>>
>> Hi Aaron,
>>
>> what changes can I expect in the 0.7 release regarding Comparison and
>> Parameters? My problem is mainly that I want to take Strings from stdin (or
>> Pig Scripts for that matter) and convert them in such a way that they are
>> interpreted correctly and converted to the corresponding byte representation
>> to use them in column names and keys.
>>
>> Regards,
>> Chris
>>
>> On Sun, Sep 26, 2010 at 5:20 AM, Aaron Morton < <aa...@thelastpickle.com>
>> aaron@thelastpickle.com> wrote:
>>
>>> Things a changing in v0.7, the row keys are byte arrays.
>>>
>>> Not sure I understand your other concerns.
>>>
>>> Aaron
>>>
>>>
>>> On 25 Sep 2010, at 08:10, Christian Decker <<d...@gmail.com>
>>> decker.christian@gmail.com> wrote:
>>>
>>>
>>> Thanks for your quick answer, I think I'll use an affix to sort of cast
>>> the keys, ranges and others from their textual representation (from Pig) to
>>> the desired byte representation, since I just noticed that the keys for the
>>> rows themselfs are always UTF8 interpreted, and since I want to make
>>> key-range as well as slice queries, I'll be better off this way I think.
>>> I'll just add a 'L' for Long and 'U' for UUID (of any kind).
>>>  Or is there a better way that I just can't see from my beginners angle?
>>> :-)thing
>>>
>>> Regards,
>>> Chris
>>>
>>>
>>> On Fri, Sep 24, 2010 at 6:35 PM, Tyler Hobbs < <ty...@riptano.com>
>>> tyler@riptano.com> wrote:
>>>
>>>> Yes, you can use describe_keyspace() and then look through the results.
>>>> It's a little ugly in 0.6, but it works
>>>>
>>>> - Tyler
>>>>
>>>>
>>>>
>>>> On Fri, Sep 24, 2010 at 11:25 AM, Christian Decker <<d...@gmail.com>
>>>> decker.christian@gmail.com> wrote:
>>>>
>>>>> Well I'm writing a loading function for Pig, and as it happens I want
>>>>> to be able to load slices from cassandra which are specified in the pig
>>>>> script (thus the input from stdin) but the ColumnFamily from which to read
>>>>> the data is another parameter and some of the CFs have UTF8, UUID, TimeUUID
>>>>> or Long types for their keys and columns, so simply converting everything I
>>>>> get to an 8byte long would break compatibility with the others.
>>>>> Now thinking about it I attacked the whole problem in a weird way,
>>>>> since UUID types won't work either.
>>>>> So let me change my question slightly, is there a way in 0.6 to detect
>>>>> the compareWith type on a running cluster? That way I could convert it to
>>>>> the right type :D
>>>>>
>>>>> Regards,
>>>>> Chris
>>>>>
>>>>>
>>>>> On Fri, Sep 24, 2010 at 6:09 PM, Tyler Hobbs < <ty...@riptano.com>
>>>>> tyler@riptano.com> wrote:
>>>>>
>>>>>> I'm not sure I understand why using this with multiple column families
>>>>>> prevents you from converting it.  Could you clarify this?
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Fri, Sep 24, 2010 at 10:56 AM, Christian Decker <<d...@gmail.com>
>>>>>> decker.christian@gmail.com> wrote:
>>>>>>
>>>>>>> Hi all,
>>>>>>>
>>>>>>> I'm having quite a dilemma with the CompareWith attribute. The
>>>>>>> Problem is that I have numeric IDs that I'd like to use as row keys, only
>>>>>>> that I also have to offer a possibility to let users input them from std
>>>>>>> input. Since I cannot ask my users to input an 8byte sequence representing
>>>>>>> the ID they'd like, I was about to turn to UTF8, when I remembered that they
>>>>>>> are compared lexicographically, so that 100 actually comes before 2, which
>>>>>>> kills key slices. Also I cannot just code a converter in since this is
>>>>>>> supposed to be a used with multiple columnfamilies, so just converting an
>>>>>>> integer read into 8bytes isn't going to work either.
>>>>>>> Any tricks for this one?
>>>>>>>
>>>>>>> Regards,
>>>>>>> Chris
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>



Re: LongType from user input

Posted by Christian Decker <de...@gmail.com>.
I just read through the tickets on Jira, and it appears that indices are
implemented in the 0.7 source tree, but I cannot find any pointer on how to
use them. I'll be trying to create a custom CassandraStorage that loads data
through the indices, anyone else interested?

Regards,
Chris

On Thu, Sep 30, 2010 at 10:56 AM, Aaron Morton <aa...@thelastpickle.com>wrote:

> AFAIK indexes are still in dev. The only example is in the system_tests.py
> in the source tree.
>
> Aaron
>
>
> On 30 Sep 2010, at 20:10, Christian Decker <de...@gmail.com>
> wrote:
>
> Apparently I have blanked the 0.7 completely out of my memory. I was trying
> to implement application layer indices and ignored the fact that Cassandra
> 0.7 is implementing them by default. I found ticket CASSANDRA-749 about the
> indices and am reading through the code right now, but is there a higher
> level overview and a tutorial on how to get things started with these
> indices (and maybe some inner workings)? This might actually solve all of my
> problems I'm having right now :-)
>
> Regards,
> Chris
>
>
> On Mon, Sep 27, 2010 at 3:45 AM, Aaron Morton < <aa...@thelastpickle.com>
> aaron@thelastpickle.com> wrote:
>
>> The only thing I can think of is that values need to be in the correct
>> byte format when used in indexes in 0.7. Take a look at the types.py module
>> in the pycassa client  <http://github.com/pycassa/pycassa>
>> http://github.com/pycassa/pycassa for an example of which values need to
>> be byte packed.
>>
>> How is your pig function working against cassandra? Is it using the
>> ColumnFamilyRecordReader? . The code in the internal RowIterator for that
>> class has an example calling the cluster to get to the comparators.
>>
>> Aaron
>>
>>
>> On 27 Sep, 2010,at 03:11 AM, Christian Decker <<d...@gmail.com>
>> decker.christian@gmail.com> wrote:
>>
>> Hi Aaron,
>>
>> what changes can I expect in the 0.7 release regarding Comparison and
>> Parameters? My problem is mainly that I want to take Strings from stdin (or
>> Pig Scripts for that matter) and convert them in such a way that they are
>> interpreted correctly and converted to the corresponding byte representation
>> to use them in column names and keys.
>>
>> Regards,
>> Chris
>>
>> On Sun, Sep 26, 2010 at 5:20 AM, Aaron Morton < <aa...@thelastpickle.com>
>> aaron@thelastpickle.com> wrote:
>>
>>> Things a changing in v0.7, the row keys are byte arrays.
>>>
>>> Not sure I understand your other concerns.
>>>
>>> Aaron
>>>
>>>
>>> On 25 Sep 2010, at 08:10, Christian Decker <<d...@gmail.com>
>>> decker.christian@gmail.com> wrote:
>>>
>>>
>>> Thanks for your quick answer, I think I'll use an affix to sort of cast
>>> the keys, ranges and others from their textual representation (from Pig) to
>>> the desired byte representation, since I just noticed that the keys for the
>>> rows themselfs are always UTF8 interpreted, and since I want to make
>>> key-range as well as slice queries, I'll be better off this way I think.
>>> I'll just add a 'L' for Long and 'U' for UUID (of any kind).
>>>  Or is there a better way that I just can't see from my beginners angle?
>>> :-)thing
>>>
>>> Regards,
>>> Chris
>>>
>>>
>>> On Fri, Sep 24, 2010 at 6:35 PM, Tyler Hobbs < <ty...@riptano.com>
>>> tyler@riptano.com> wrote:
>>>
>>>> Yes, you can use describe_keyspace() and then look through the results.
>>>> It's a little ugly in 0.6, but it works
>>>>
>>>> - Tyler
>>>>
>>>>
>>>>
>>>> On Fri, Sep 24, 2010 at 11:25 AM, Christian Decker <<d...@gmail.com>
>>>> decker.christian@gmail.com> wrote:
>>>>
>>>>> Well I'm writing a loading function for Pig, and as it happens I want
>>>>> to be able to load slices from cassandra which are specified in the pig
>>>>> script (thus the input from stdin) but the ColumnFamily from which to read
>>>>> the data is another parameter and some of the CFs have UTF8, UUID, TimeUUID
>>>>> or Long types for their keys and columns, so simply converting everything I
>>>>> get to an 8byte long would break compatibility with the others.
>>>>> Now thinking about it I attacked the whole problem in a weird way,
>>>>> since UUID types won't work either.
>>>>> So let me change my question slightly, is there a way in 0.6 to detect
>>>>> the compareWith type on a running cluster? That way I could convert it to
>>>>> the right type :D
>>>>>
>>>>> Regards,
>>>>> Chris
>>>>>
>>>>>
>>>>> On Fri, Sep 24, 2010 at 6:09 PM, Tyler Hobbs < <ty...@riptano.com>
>>>>> tyler@riptano.com> wrote:
>>>>>
>>>>>> I'm not sure I understand why using this with multiple column families
>>>>>> prevents you from converting it.  Could you clarify this?
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Fri, Sep 24, 2010 at 10:56 AM, Christian Decker <<d...@gmail.com>
>>>>>> decker.christian@gmail.com> wrote:
>>>>>>
>>>>>>> Hi all,
>>>>>>>
>>>>>>> I'm having quite a dilemma with the CompareWith attribute. The
>>>>>>> Problem is that I have numeric IDs that I'd like to use as row keys, only
>>>>>>> that I also have to offer a possibility to let users input them from std
>>>>>>> input. Since I cannot ask my users to input an 8byte sequence representing
>>>>>>> the ID they'd like, I was about to turn to UTF8, when I remembered that they
>>>>>>> are compared lexicographically, so that 100 actually comes before 2, which
>>>>>>> kills key slices. Also I cannot just code a converter in since this is
>>>>>>> supposed to be a used with multiple columnfamilies, so just converting an
>>>>>>> integer read into 8bytes isn't going to work either.
>>>>>>> Any tricks for this one?
>>>>>>>
>>>>>>> Regards,
>>>>>>> Chris
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>

Re: LongType from user input

Posted by Aaron Morton <aa...@thelastpickle.com>.
AFAIK indexes are still in dev. The only example is in the system_tests.py in the source tree.

Aaron

On 30 Sep 2010, at 20:10, Christian Decker <de...@gmail.com> wrote:

> Apparently I have blanked the 0.7 completely out of my memory. I was trying to implement application layer indices and ignored the fact that Cassandra 0.7 is implementing them by default. I found ticket CASSANDRA-749 about the indices and am reading through the code right now, but is there a higher level overview and a tutorial on how to get things started with these indices (and maybe some inner workings)? This might actually solve all of my problems I'm having right now :-)
> 
> Regards,
> Chris
> 
> 
> On Mon, Sep 27, 2010 at 3:45 AM, Aaron Morton <aa...@thelastpickle.com> wrote:
> The only thing I can think of is that values need to be in the correct byte format when used in indexes in 0.7. Take a look at the types.py module in the pycassa client http://github.com/pycassa/pycassa for an example of which values need to be byte packed. 
> 
> How is your pig function working against cassandra? Is it using the ColumnFamilyRecordReader? . The code in the internal RowIterator for that class has an example calling the cluster to get to the comparators.  
> 
> Aaron
> 
> 
> On 27 Sep, 2010,at 03:11 AM, Christian Decker <de...@gmail.com> wrote:
> 
>> Hi Aaron,
>> 
>> what changes can I expect in the 0.7 release regarding Comparison and Parameters? My problem is mainly that I want to take Strings from stdin (or Pig Scripts for that matter) and convert them in such a way that they are interpreted correctly and converted to the corresponding byte representation to use them in column names and keys.
>> 
>> Regards,
>> Chris
>> 
>> On Sun, Sep 26, 2010 at 5:20 AM, Aaron Morton <aa...@thelastpickle.com> wrote:
>> Things a changing in v0.7, the row keys are byte arrays.
>> 
>> Not sure I understand your other concerns. 
>> 
>> Aaron
>> 
>> 
>> On 25 Sep 2010, at 08:10, Christian Decker <de...@gmail.com> wrote:
>> 
>> 
>>> Thanks for your quick answer, I think I'll use an affix to sort of cast the keys, ranges and others from their textual representation (from Pig) to the desired byte representation, since I just noticed that the keys for the rows themselfs are always UTF8 interpreted, and since I want to make key-range as well as slice queries, I'll be better off this way I think. I'll just add a 'L' for Long and 'U' for UUID (of any kind).
>>> Or is there a better way that I just can't see from my beginners angle? :-)thing
>>> 
>>> Regards,
>>> Chris
>>> 
>>> 
>>> On Fri, Sep 24, 2010 at 6:35 PM, Tyler Hobbs <ty...@riptano.com> wrote:
>>> Yes, you can use describe_keyspace() and then look through the results.  It's a little ugly in 0.6, but it works
>>> 
>>> - Tyler
>>> 
>>> 
>>> 
>>> On Fri, Sep 24, 2010 at 11:25 AM, Christian Decker <de...@gmail.com> wrote:
>>> Well I'm writing a loading function for Pig, and as it happens I want to be able to load slices from cassandra which are specified in the pig script (thus the input from stdin) but the ColumnFamily from which to read the data is another parameter and some of the CFs have UTF8, UUID, TimeUUID or Long types for their keys and columns, so simply converting everything I get to an 8byte long would break compatibility with the others.
>>> Now thinking about it I attacked the whole problem in a weird way, since UUID types won't work either.
>>> So let me change my question slightly, is there a way in 0.6 to detect the compareWith type on a running cluster? That way I could convert it to the right type :D
>>> 
>>> Regards,
>>> Chris
>>> 
>>> 
>>> On Fri, Sep 24, 2010 at 6:09 PM, Tyler Hobbs <ty...@riptano.com> wrote:
>>> I'm not sure I understand why using this with multiple column families prevents you from converting it.  Could you clarify this?
>>> 
>>> 
>>> 
>>> On Fri, Sep 24, 2010 at 10:56 AM, Christian Decker <de...@gmail.com> wrote:
>>> Hi all,
>>> 
>>> I'm having quite a dilemma with the CompareWith attribute. The Problem is that I have numeric IDs that I'd like to use as row keys, only that I also have to offer a possibility to let users input them from std input. Since I cannot ask my users to input an 8byte sequence representing the ID they'd like, I was about to turn to UTF8, when I remembered that they are compared lexicographically, so that 100 actually comes before 2, which kills key slices. Also I cannot just code a converter in since this is supposed to be a used with multiple columnfamilies, so just converting an integer read into 8bytes isn't going to work either.
>>> Any tricks for this one?
>>> 
>>> Regards,
>>> Chris
>>> 
>>> 
>>> 
>>> 
>> 
> 

Re: LongType from user input

Posted by Christian Decker <de...@gmail.com>.
Apparently I have blanked the 0.7 completely out of my memory. I was trying
to implement application layer indices and ignored the fact that Cassandra
0.7 is implementing them by default. I found ticket CASSANDRA-749 about the
indices and am reading through the code right now, but is there a higher
level overview and a tutorial on how to get things started with these
indices (and maybe some inner workings)? This might actually solve all of my
problems I'm having right now :-)

Regards,
Chris


On Mon, Sep 27, 2010 at 3:45 AM, Aaron Morton <aa...@thelastpickle.com>wrote:

> The only thing I can think of is that values need to be in the correct byte
> format when used in indexes in 0.7. Take a look at the types.py module in
> the pycassa client http://github.com/pycassa/pycassa for an example of
> which values need to be byte packed.
>
> How is your pig function working against cassandra? Is it using the
> ColumnFamilyRecordReader? . The code in the internal RowIterator for that
> class has an example calling the cluster to get to the comparators.
>
> Aaron
>
>
> On 27 Sep, 2010,at 03:11 AM, Christian Decker <de...@gmail.com>
> wrote:
>
> Hi Aaron,
>
> what changes can I expect in the 0.7 release regarding Comparison and
> Parameters? My problem is mainly that I want to take Strings from stdin (or
> Pig Scripts for that matter) and convert them in such a way that they are
> interpreted correctly and converted to the corresponding byte representation
> to use them in column names and keys.
>
> Regards,
> Chris
>
> On Sun, Sep 26, 2010 at 5:20 AM, Aaron Morton <aa...@thelastpickle.com>wrote:
>
>> Things a changing in v0.7, the row keys are byte arrays.
>>
>> Not sure I understand your other concerns.
>>
>> Aaron
>>
>>
>> On 25 Sep 2010, at 08:10, Christian Decker <de...@gmail.com>
>> wrote:
>>
>>
>> Thanks for your quick answer, I think I'll use an affix to sort of cast
>> the keys, ranges and others from their textual representation (from Pig) to
>> the desired byte representation, since I just noticed that the keys for the
>> rows themselfs are always UTF8 interpreted, and since I want to make
>> key-range as well as slice queries, I'll be better off this way I think.
>> I'll just add a 'L' for Long and 'U' for UUID (of any kind).
>>  Or is there a better way that I just can't see from my beginners angle?
>> :-)thing
>>
>> Regards,
>> Chris
>>
>>
>> On Fri, Sep 24, 2010 at 6:35 PM, Tyler Hobbs < <ty...@riptano.com>
>> tyler@riptano.com> wrote:
>>
>>> Yes, you can use describe_keyspace() and then look through the results.
>>> It's a little ugly in 0.6, but it works
>>>
>>> - Tyler
>>>
>>>
>>>
>>> On Fri, Sep 24, 2010 at 11:25 AM, Christian Decker <<d...@gmail.com>
>>> decker.christian@gmail.com> wrote:
>>>
>>>> Well I'm writing a loading function for Pig, and as it happens I want to
>>>> be able to load slices from cassandra which are specified in the pig script
>>>> (thus the input from stdin) but the ColumnFamily from which to read the data
>>>> is another parameter and some of the CFs have UTF8, UUID, TimeUUID or Long
>>>> types for their keys and columns, so simply converting everything I get to
>>>> an 8byte long would break compatibility with the others.
>>>> Now thinking about it I attacked the whole problem in a weird way, since
>>>> UUID types won't work either.
>>>> So let me change my question slightly, is there a way in 0.6 to detect
>>>> the compareWith type on a running cluster? That way I could convert it to
>>>> the right type :D
>>>>
>>>> Regards,
>>>> Chris
>>>>
>>>>
>>>> On Fri, Sep 24, 2010 at 6:09 PM, Tyler Hobbs < <ty...@riptano.com>
>>>> tyler@riptano.com> wrote:
>>>>
>>>>> I'm not sure I understand why using this with multiple column families
>>>>> prevents you from converting it.  Could you clarify this?
>>>>>
>>>>>
>>>>>
>>>>> On Fri, Sep 24, 2010 at 10:56 AM, Christian Decker <<d...@gmail.com>
>>>>> decker.christian@gmail.com> wrote:
>>>>>
>>>>>> Hi all,
>>>>>>
>>>>>> I'm having quite a dilemma with the CompareWith attribute. The Problem
>>>>>> is that I have numeric IDs that I'd like to use as row keys, only that I
>>>>>> also have to offer a possibility to let users input them from std input.
>>>>>> Since I cannot ask my users to input an 8byte sequence representing the ID
>>>>>> they'd like, I was about to turn to UTF8, when I remembered that they are
>>>>>> compared lexicographically, so that 100 actually comes before 2, which kills
>>>>>> key slices. Also I cannot just code a converter in since this is supposed to
>>>>>> be a used with multiple columnfamilies, so just converting an integer read
>>>>>> into 8bytes isn't going to work either.
>>>>>> Any tricks for this one?
>>>>>>
>>>>>> Regards,
>>>>>> Chris
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>

Re: LongType from user input

Posted by Aaron Morton <aa...@thelastpickle.com>.
The only thing I can think of is that values need to be in the correct byte format when used in indexes in 0.7. Take a look at the types.py module in the pycassa client http://github.com/pycassa/pycassa for an example of which values need to be byte packed. 

How is your pig function working against cassandra? Is it using the ColumnFamilyRecordReader? . The code in the internal RowIterator for that class has an example calling the cluster to get to the comparators.  

Aaron


On 27 Sep, 2010,at 03:11 AM, Christian Decker <de...@gmail.com> wrote:

Hi Aaron,

what changes can I expect in the 0.7 release regarding Comparison and Parameters? My problem is mainly that I want to take Strings from stdin (or Pig Scripts for that matter) and convert them in such a way that they are interpreted correctly and converted to the corresponding byte representation to use them in column names and keys.

Regards,
Chris

On Sun, Sep 26, 2010 at 5:20 AM, Aaron Morton <aa...@thelastpickle.com> wrote:
Things a changing in v0.7, the row keys are byte arrays.

Not sure I understand your other concerns. 

Aaron


On 25 Sep 2010, at 08:10, Christian Decker <de...@gmail.com> wrote:


Thanks for your quick answer, I think I'll use an affix to sort of cast the keys, ranges and others from their textual representation (from Pig) to the desired byte representation, since I just noticed that the keys for the rows themselfs are always UTF8 interpreted, and since I want to make key-range as well as slice queries, I'll be better off this way I think. I'll just add a 'L' for Long and 'U' for UUID (of any kind).
Or is there a better way that I just can't see from my beginners angle? :-)thing

Regards,
Chris


On Fri, Sep 24, 2010 at 6:35 PM, Tyler Hobbs <ty...@riptano.com> wrote:
Yes, you can use describe_keyspace() and then look through the results.  It's a little ugly in 0.6, but it works.

- Tyler



On Fri, Sep 24, 2010 at 11:25 AM, Christian Decker <de...@gmail.com> wrote:
Well I'm writing a loading function for Pig, and as it happens I want to be able to load slices from cassandra which are specified in the pig script (thus the input from stdin) but the ColumnFamily from which to read the data is another parameter and some of the CFs have UTF8, UUID, TimeUUID or Long types for their keys and columns, so simply converting everything I get to an 8byte long would break compatibility with the others.
Now thinking about it I attacked the whole problem in a weird way, since UUID types won't work either.
So let me change my question slightly, is there a way in 0.6 to detect the compareWith type on a running cluster? That way I could convert it to the right type :D

Regards,
Chris


On Fri, Sep 24, 2010 at 6:09 PM, Tyler Hobbs <ty...@riptano.com> wrote:
I'm not sure I understand why using this with multiple column families prevents you from converting it.  Could you clarify this?



On Fri, Sep 24, 2010 at 10:56 AM, Christian Decker <de...@gmail.com> wrote:
Hi all,

I'm having quite a dilemma with the CompareWith attribute. The Problem is that I have numeric IDs that I'd like to use as row keys, only that I also have to offer a possibility to let users input them from std input. Since I cannot ask my users to input an 8byte sequence representing the ID they'd like, I was about to turn to UTF8, when I remembered that they are compared lexicographically, so that 100 actually comes before 2, which kills key slices. Also I cannot just code a converter in since this is supposed to be a used with multiple columnfamilies, so just converting an integer read into 8bytes isn't going to work either.
Any tricks for this one?

Regards,
Chris






Re: LongType from user input

Posted by Christian Decker <de...@gmail.com>.
Hi Aaron,

what changes can I expect in the 0.7 release regarding Comparison and
Parameters? My problem is mainly that I want to take Strings from stdin (or
Pig Scripts for that matter) and convert them in such a way that they are
interpreted correctly and converted to the corresponding byte representation
to use them in column names and keys.

Regards,
Chris

On Sun, Sep 26, 2010 at 5:20 AM, Aaron Morton <aa...@thelastpickle.com>wrote:

> Things a changing in v0.7, the row keys are byte arrays.
>
> Not sure I understand your other concerns.
>
> Aaron
>
>
> On 25 Sep 2010, at 08:10, Christian Decker <de...@gmail.com>
> wrote:
>
> Thanks for your quick answer, I think I'll use an affix to sort of cast the
> keys, ranges and others from their textual representation (from Pig) to the
> desired byte representation, since I just noticed that the keys for the rows
> themselfs are always UTF8 interpreted, and since I want to make key-range as
> well as slice queries, I'll be better off this way I think. I'll just add a
> 'L' for Long and 'U' for UUID (of any kind).
>  Or is there a better way that I just can't see from my beginners angle?
> :-)thing
>
> Regards,
> Chris
>
>
> On Fri, Sep 24, 2010 at 6:35 PM, Tyler Hobbs < <ty...@riptano.com>
> tyler@riptano.com> wrote:
>
>> Yes, you can use describe_keyspace() and then look through the results.
>> It's a little ugly in 0.6, but it works.
>>
>> - Tyler
>>
>>
>> On Fri, Sep 24, 2010 at 11:25 AM, Christian Decker <<d...@gmail.com>
>> decker.christian@gmail.com> wrote:
>>
>>> Well I'm writing a loading function for Pig, and as it happens I want to
>>> be able to load slices from cassandra which are specified in the pig script
>>> (thus the input from stdin) but the ColumnFamily from which to read the data
>>> is another parameter and some of the CFs have UTF8, UUID, TimeUUID or Long
>>> types for their keys and columns, so simply converting everything I get to
>>> an 8byte long would break compatibility with the others.
>>> Now thinking about it I attacked the whole problem in a weird way, since
>>> UUID types won't work either.
>>> So let me change my question slightly, is there a way in 0.6 to detect
>>> the compareWith type on a running cluster? That way I could convert it to
>>> the right type :D
>>>
>>> Regards,
>>> Chris
>>>
>>> On Fri, Sep 24, 2010 at 6:09 PM, Tyler Hobbs < <ty...@riptano.com>
>>> tyler@riptano.com> wrote:
>>>
>>>> I'm not sure I understand why using this with multiple column families
>>>> prevents you from converting it.  Could you clarify this?
>>>>
>>>>
>>>> On Fri, Sep 24, 2010 at 10:56 AM, Christian Decker <<d...@gmail.com>
>>>> decker.christian@gmail.com> wrote:
>>>>
>>>>> Hi all,
>>>>>
>>>>> I'm having quite a dilemma with the CompareWith attribute. The Problem
>>>>> is that I have numeric IDs that I'd like to use as row keys, only that I
>>>>> also have to offer a possibility to let users input them from std input.
>>>>> Since I cannot ask my users to input an 8byte sequence representing the ID
>>>>> they'd like, I was about to turn to UTF8, when I remembered that they are
>>>>> compared lexicographically, so that 100 actually comes before 2, which kills
>>>>> key slices. Also I cannot just code a converter in since this is supposed to
>>>>> be a used with multiple columnfamilies, so just converting an integer read
>>>>> into 8bytes isn't going to work either.
>>>>> Any tricks for this one?
>>>>>
>>>>> Regards,
>>>>> Chris
>>>>>
>>>>
>>>>
>>>
>>
>

Re: LongType from user input

Posted by Aaron Morton <aa...@thelastpickle.com>.
Things a changing in v0.7, the row keys are byte arrays.

Not sure I understand your other concerns. 

Aaron

On 25 Sep 2010, at 08:10, Christian Decker <de...@gmail.com> wrote:

> Thanks for your quick answer, I think I'll use an affix to sort of cast the keys, ranges and others from their textual representation (from Pig) to the desired byte representation, since I just noticed that the keys for the rows themselfs are always UTF8 interpreted, and since I want to make key-range as well as slice queries, I'll be better off this way I think. I'll just add a 'L' for Long and 'U' for UUID (of any kind).
> Or is there a better way that I just can't see from my beginners angle? :-)thing
> 
> Regards,
> Chris
> 
> 
> On Fri, Sep 24, 2010 at 6:35 PM, Tyler Hobbs <ty...@riptano.com> wrote:
> Yes, you can use describe_keyspace() and then look through the results.  It's a little ugly in 0.6, but it works.
> 
> - Tyler
> 
> 
> On Fri, Sep 24, 2010 at 11:25 AM, Christian Decker <de...@gmail.com> wrote:
> Well I'm writing a loading function for Pig, and as it happens I want to be able to load slices from cassandra which are specified in the pig script (thus the input from stdin) but the ColumnFamily from which to read the data is another parameter and some of the CFs have UTF8, UUID, TimeUUID or Long types for their keys and columns, so simply converting everything I get to an 8byte long would break compatibility with the others.
> Now thinking about it I attacked the whole problem in a weird way, since UUID types won't work either.
> So let me change my question slightly, is there a way in 0.6 to detect the compareWith type on a running cluster? That way I could convert it to the right type :D
> 
> Regards,
> Chris
> 
> On Fri, Sep 24, 2010 at 6:09 PM, Tyler Hobbs <ty...@riptano.com> wrote:
> I'm not sure I understand why using this with multiple column families prevents you from converting it.  Could you clarify this?
> 
> 
> On Fri, Sep 24, 2010 at 10:56 AM, Christian Decker <de...@gmail.com> wrote:
> Hi all,
> 
> I'm having quite a dilemma with the CompareWith attribute. The Problem is that I have numeric IDs that I'd like to use as row keys, only that I also have to offer a possibility to let users input them from std input. Since I cannot ask my users to input an 8byte sequence representing the ID they'd like, I was about to turn to UTF8, when I remembered that they are compared lexicographically, so that 100 actually comes before 2, which kills key slices. Also I cannot just code a converter in since this is supposed to be a used with multiple columnfamilies, so just converting an integer read into 8bytes isn't going to work either.
> Any tricks for this one?
> 
> Regards,
> Chris
> 
> 
> 
> 

Re: LongType from user input

Posted by Christian Decker <de...@gmail.com>.
Thanks for your quick answer, I think I'll use an affix to sort of cast the
keys, ranges and others from their textual representation (from Pig) to the
desired byte representation, since I just noticed that the keys for the rows
themselfs are always UTF8 interpreted, and since I want to make key-range as
well as slice queries, I'll be better off this way I think. I'll just add a
'L' for Long and 'U' for UUID (of any kind).
Or is there a better way that I just can't see from my beginners angle? :-)

Regards,
Chris


On Fri, Sep 24, 2010 at 6:35 PM, Tyler Hobbs <ty...@riptano.com> wrote:

> Yes, you can use describe_keyspace() and then look through the results.
> It's a little ugly in 0.6, but it works.
>
> - Tyler
>
>
> On Fri, Sep 24, 2010 at 11:25 AM, Christian Decker <
> decker.christian@gmail.com> wrote:
>
>> Well I'm writing a loading function for Pig, and as it happens I want to
>> be able to load slices from cassandra which are specified in the pig script
>> (thus the input from stdin) but the ColumnFamily from which to read the data
>> is another parameter and some of the CFs have UTF8, UUID, TimeUUID or Long
>> types for their keys and columns, so simply converting everything I get to
>> an 8byte long would break compatibility with the others.
>> Now thinking about it I attacked the whole problem in a weird way, since
>> UUID types won't work either.
>> So let me change my question slightly, is there a way in 0.6 to detect the
>> compareWith type on a running cluster? That way I could convert it to the
>> right type :D
>>
>> Regards,
>> Chris
>>
>> On Fri, Sep 24, 2010 at 6:09 PM, Tyler Hobbs <ty...@riptano.com> wrote:
>>
>>> I'm not sure I understand why using this with multiple column families
>>> prevents you from converting it.  Could you clarify this?
>>>
>>>
>>> On Fri, Sep 24, 2010 at 10:56 AM, Christian Decker <
>>> decker.christian@gmail.com> wrote:
>>>
>>>> Hi all,
>>>>
>>>> I'm having quite a dilemma with the CompareWith attribute. The Problem
>>>> is that I have numeric IDs that I'd like to use as row keys, only that I
>>>> also have to offer a possibility to let users input them from std input.
>>>> Since I cannot ask my users to input an 8byte sequence representing the ID
>>>> they'd like, I was about to turn to UTF8, when I remembered that they are
>>>> compared lexicographically, so that 100 actually comes before 2, which kills
>>>> key slices. Also I cannot just code a converter in since this is supposed to
>>>> be a used with multiple columnfamilies, so just converting an integer read
>>>> into 8bytes isn't going to work either.
>>>> Any tricks for this one?
>>>>
>>>> Regards,
>>>> Chris
>>>>
>>>
>>>
>>
>

Re: LongType from user input

Posted by Tyler Hobbs <ty...@riptano.com>.
Yes, you can use describe_keyspace() and then look through the results.
It's a little ugly in 0.6, but it works.

- Tyler

On Fri, Sep 24, 2010 at 11:25 AM, Christian Decker <
decker.christian@gmail.com> wrote:

> Well I'm writing a loading function for Pig, and as it happens I want to be
> able to load slices from cassandra which are specified in the pig script
> (thus the input from stdin) but the ColumnFamily from which to read the data
> is another parameter and some of the CFs have UTF8, UUID, TimeUUID or Long
> types for their keys and columns, so simply converting everything I get to
> an 8byte long would break compatibility with the others.
> Now thinking about it I attacked the whole problem in a weird way, since
> UUID types won't work either.
> So let me change my question slightly, is there a way in 0.6 to detect the
> compareWith type on a running cluster? That way I could convert it to the
> right type :D
>
> Regards,
> Chris
>
> On Fri, Sep 24, 2010 at 6:09 PM, Tyler Hobbs <ty...@riptano.com> wrote:
>
>> I'm not sure I understand why using this with multiple column families
>> prevents you from converting it.  Could you clarify this?
>>
>>
>> On Fri, Sep 24, 2010 at 10:56 AM, Christian Decker <
>> decker.christian@gmail.com> wrote:
>>
>>> Hi all,
>>>
>>> I'm having quite a dilemma with the CompareWith attribute. The Problem is
>>> that I have numeric IDs that I'd like to use as row keys, only that I also
>>> have to offer a possibility to let users input them from std input. Since I
>>> cannot ask my users to input an 8byte sequence representing the ID they'd
>>> like, I was about to turn to UTF8, when I remembered that they are compared
>>> lexicographically, so that 100 actually comes before 2, which kills key
>>> slices. Also I cannot just code a converter in since this is supposed to be
>>> a used with multiple columnfamilies, so just converting an integer read into
>>> 8bytes isn't going to work either.
>>> Any tricks for this one?
>>>
>>> Regards,
>>> Chris
>>>
>>
>>
>

Re: LongType from user input

Posted by Christian Decker <de...@gmail.com>.
Well I'm writing a loading function for Pig, and as it happens I want to be
able to load slices from cassandra which are specified in the pig script
(thus the input from stdin) but the ColumnFamily from which to read the data
is another parameter and some of the CFs have UTF8, UUID, TimeUUID or Long
types for their keys and columns, so simply converting everything I get to
an 8byte long would break compatibility with the others.
Now thinking about it I attacked the whole problem in a weird way, since
UUID types won't work either.
So let me change my question slightly, is there a way in 0.6 to detect the
compareWith type on a running cluster? That way I could convert it to the
right type :D

Regards,
Chris

On Fri, Sep 24, 2010 at 6:09 PM, Tyler Hobbs <ty...@riptano.com> wrote:

> I'm not sure I understand why using this with multiple column families
> prevents you from converting it.  Could you clarify this?
>
>
> On Fri, Sep 24, 2010 at 10:56 AM, Christian Decker <
> decker.christian@gmail.com> wrote:
>
>> Hi all,
>>
>> I'm having quite a dilemma with the CompareWith attribute. The Problem is
>> that I have numeric IDs that I'd like to use as row keys, only that I also
>> have to offer a possibility to let users input them from std input. Since I
>> cannot ask my users to input an 8byte sequence representing the ID they'd
>> like, I was about to turn to UTF8, when I remembered that they are compared
>> lexicographically, so that 100 actually comes before 2, which kills key
>> slices. Also I cannot just code a converter in since this is supposed to be
>> a used with multiple columnfamilies, so just converting an integer read into
>> 8bytes isn't going to work either.
>> Any tricks for this one?
>>
>> Regards,
>> Chris
>>
>
>

Re: LongType from user input

Posted by Tyler Hobbs <ty...@riptano.com>.
I'm not sure I understand why using this with multiple column families
prevents you from converting it.  Could you clarify this?

On Fri, Sep 24, 2010 at 10:56 AM, Christian Decker <
decker.christian@gmail.com> wrote:

> Hi all,
>
> I'm having quite a dilemma with the CompareWith attribute. The Problem is
> that I have numeric IDs that I'd like to use as row keys, only that I also
> have to offer a possibility to let users input them from std input. Since I
> cannot ask my users to input an 8byte sequence representing the ID they'd
> like, I was about to turn to UTF8, when I remembered that they are compared
> lexicographically, so that 100 actually comes before 2, which kills key
> slices. Also I cannot just code a converter in since this is supposed to be
> a used with multiple columnfamilies, so just converting an integer read into
> 8bytes isn't going to work either.
> Any tricks for this one?
>
> Regards,
> Chris
>