You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Leonid Ilyevsky <li...@mooncapital.com> on 2012/07/06 00:40:56 UTC

Composite key in thrift java api

I need to create a ByteBuffer instance containing the proper composite key, based on the values of the components of the key. I am going to use it for update operation.
I tried to simply concatenate the buffers corresponding to the components, but I am not sure this is correct, because I am getting exception that comes from the server :

InvalidRequestException(why:Not enough bytes to read value of component 0)

In the server log I see this:

org.apache.thrift.transport.TTransportException: Cannot read. Remote side has closed. Tried to read 4 bytes, but only got 0 bytes. (This is often indicative of an internal error on the server side. Please check your server logs.)

(I believe here when it says "server side" it actually means client, because it is the server's log).

Seems like the buffer that my client sends is too short.  I suspect there is a way in thrift to do it properly, but I don't know how.
Looks like Hector has a Composite class that maybe can help, but at this point I would really prefer to do it in Cassandra itself, without Hector.

Thanks!

Leonid


________________________________
This email, along with any attachments, is confidential and may be legally privileged or otherwise protected from disclosure. Any unauthorized dissemination, copying or use of the contents of this email is strictly prohibited and may be in violation of law. If you are not the intended recipient, any disclosure, copying, forwarding or distribution of this email is strictly prohibited and this email and any attachments should be deleted immediately. This email and any attachments do not constitute an offer to sell or a solicitation of an offer to purchase any interest in any investment vehicle sponsored by Moon Capital Management LP ("Moon Capital"). Moon Capital does not provide legal, accounting or tax advice. Any statement regarding legal, accounting or tax matters was not intended or written to be relied upon by any person as advice. Moon Capital does not waive confidentiality or privilege as a result of this email.

RE: Dynamic CF

Posted by Derek Williams <de...@fyrie.net>.
To clarify, I haven't tested it with compact storage, but it will only use
the first part of the primary key without compact storage.
On Jul 6, 2012 4:50 PM, "Derek Williams" <de...@fyrie.net> wrote:

> Actually, my solution only makes a row for each unique value of the first
> part of the primary key.
> On Jul 6, 2012 3:07 PM, "Leonid Ilyevsky" <li...@mooncapital.com>
> wrote:
>
>>  Thanks Derek, I actually saw your post, but this is not a good
>> substitution for a real dynamic table. Your trick will make a table with
>> even more rows; I wanted wide rows so the reads and writes would be more
>> efficient.****
>>
>> ** **
>>
>> *From:* Derek Williams [mailto:derek@fyrie.net]
>> *Sent:* Friday, July 06, 2012 4:58 PM
>> *To:* user@cassandra.apache.org
>> *Subject:* Re: Dynamic CF****
>>
>> ** **
>>
>> On Fri, Jul 6, 2012 at 2:49 PM, Leonid Ilyevsky <
>> lilyevsky@mooncapital.com> wrote:****
>>
>> At this point I am really confused about what direction Cassandra is
>> going. CQL 3 has the benefit of composite keys, but no dynamic columns.
>> I thought, the whole point of Cassandra was to provide dynamic tables.***
>> *
>>
>> ** **
>>
>> See my earlier post on the subject:****
>>
>> http://www.mail-archive.com/user@cassandra.apache.org/msg23160.html****
>>
>> ** **
>>
>> You can have a dynamic table with CQL3, you just can't have a table with
>> a mix of dynamic/nondynamic columns.****
>>
>> ** **
>>
>> -- ****
>>
>> Derek Williams****
>>
>> ** **
>>
>> ------------------------------
>> This email, along with any attachments, is confidential and may be
>> legally privileged or otherwise protected from disclosure. Any unauthorized
>> dissemination, copying or use of the contents of this email is strictly
>> prohibited and may be in violation of law. If you are not the intended
>> recipient, any disclosure, copying, forwarding or distribution of this
>> email is strictly prohibited and this email and any attachments should be
>> deleted immediately. This email and any attachments do not constitute an
>> offer to sell or a solicitation of an offer to purchase any interest in any
>> investment vehicle sponsored by Moon Capital Management LP (“Moon
>> Capital”). Moon Capital does not provide legal, accounting or tax advice.
>> Any statement regarding legal, accounting or tax matters was not intended
>> or written to be relied upon by any person as advice. Moon Capital does not
>> waive confidentiality or privilege as a result of this email.
>>
>

RE: Dynamic CF

Posted by Derek Williams <de...@fyrie.net>.
Actually, my solution only makes a row for each unique value of the first
part of the primary key.
On Jul 6, 2012 3:07 PM, "Leonid Ilyevsky" <li...@mooncapital.com> wrote:

>  Thanks Derek, I actually saw your post, but this is not a good
> substitution for a real dynamic table. Your trick will make a table with
> even more rows; I wanted wide rows so the reads and writes would be more
> efficient.****
>
> ** **
>
> *From:* Derek Williams [mailto:derek@fyrie.net]
> *Sent:* Friday, July 06, 2012 4:58 PM
> *To:* user@cassandra.apache.org
> *Subject:* Re: Dynamic CF****
>
> ** **
>
> On Fri, Jul 6, 2012 at 2:49 PM, Leonid Ilyevsky <li...@mooncapital.com>
> wrote:****
>
> At this point I am really confused about what direction Cassandra is
> going. CQL 3 has the benefit of composite keys, but no dynamic columns.
> I thought, the whole point of Cassandra was to provide dynamic tables.****
>
> ** **
>
> See my earlier post on the subject:****
>
> http://www.mail-archive.com/user@cassandra.apache.org/msg23160.html****
>
> ** **
>
> You can have a dynamic table with CQL3, you just can't have a table with a
> mix of dynamic/nondynamic columns.****
>
> ** **
>
> -- ****
>
> Derek Williams****
>
> ** **
>
> ------------------------------
> This email, along with any attachments, is confidential and may be legally
> privileged or otherwise protected from disclosure. Any unauthorized
> dissemination, copying or use of the contents of this email is strictly
> prohibited and may be in violation of law. If you are not the intended
> recipient, any disclosure, copying, forwarding or distribution of this
> email is strictly prohibited and this email and any attachments should be
> deleted immediately. This email and any attachments do not constitute an
> offer to sell or a solicitation of an offer to purchase any interest in any
> investment vehicle sponsored by Moon Capital Management LP (“Moon
> Capital”). Moon Capital does not provide legal, accounting or tax advice.
> Any statement regarding legal, accounting or tax matters was not intended
> or written to be relied upon by any person as advice. Moon Capital does not
> waive confidentiality or privilege as a result of this email.
>

RE: Dynamic CF

Posted by Leonid Ilyevsky <li...@mooncapital.com>.
Thanks Derek, I actually saw your post, but this is not a good substitution for a real dynamic table. Your trick will make a table with even more rows; I wanted wide rows so the reads and writes would be more efficient.

From: Derek Williams [mailto:derek@fyrie.net]
Sent: Friday, July 06, 2012 4:58 PM
To: user@cassandra.apache.org
Subject: Re: Dynamic CF

On Fri, Jul 6, 2012 at 2:49 PM, Leonid Ilyevsky <li...@mooncapital.com>> wrote:
At this point I am really confused about what direction Cassandra is going. CQL 3 has the benefit of composite keys, but no dynamic columns.
I thought, the whole point of Cassandra was to provide dynamic tables.

See my earlier post on the subject:
http://www.mail-archive.com/user@cassandra.apache.org/msg23160.html

You can have a dynamic table with CQL3, you just can't have a table with a mix of dynamic/nondynamic columns.

--
Derek Williams


________________________________
This email, along with any attachments, is confidential and may be legally privileged or otherwise protected from disclosure. Any unauthorized dissemination, copying or use of the contents of this email is strictly prohibited and may be in violation of law. If you are not the intended recipient, any disclosure, copying, forwarding or distribution of this email is strictly prohibited and this email and any attachments should be deleted immediately. This email and any attachments do not constitute an offer to sell or a solicitation of an offer to purchase any interest in any investment vehicle sponsored by Moon Capital Management LP (“Moon Capital”). Moon Capital does not provide legal, accounting or tax advice. Any statement regarding legal, accounting or tax matters was not intended or written to be relied upon by any person as advice. Moon Capital does not waive confidentiality or privilege as a result of this email.

Re: Dynamic CF

Posted by Derek Williams <de...@fyrie.net>.
On Fri, Jul 6, 2012 at 2:49 PM, Leonid Ilyevsky
<li...@mooncapital.com>wrote:

> At this point I am really confused about what direction Cassandra is
> going. CQL 3 has the benefit of composite keys, but no dynamic columns.
> I thought, the whole point of Cassandra was to provide dynamic tables.
>

See my earlier post on the subject:
http://www.mail-archive.com/user@cassandra.apache.org/msg23160.html

You can have a dynamic table with CQL3, you just can't have a table with a
mix of dynamic/nondynamic columns.

-- 
Derek Williams

Re: Dynamic CF

Posted by Carlos Carrasco <ca...@groupalia.com>.
I am confused then. I remember reviewing the source for CQL3 and finding
that the row reader used the column count in the CF definition in order to
find how many columns it needed to read a single row. I guess I missed a
filter over the composited part or that I reviewed an old version.

On 10 July 2012 16:34, Sylvain Lebresne <sy...@datastax.com> wrote:

> On Tue, Jul 10, 2012 at 4:19 PM, Carlos Carrasco <
> carlos.carrasco@groupalia.com> wrote:
>
>> I think he means something like having a fixed set of coiumns in the
>> table definition, then in the actual rows having other columns not
>> specified in the defintion, indepentent of the composited part of the PK.
>> When I reviewed CQL3 for using in Gossie[1] I realized I couldn't have
>> this, and that it would complicate things like migrations or optional
>> columns. For this reason I didn't use CQL3 and instead wrote a row unmaper
>> that detects the discontinuities in the composited part and uses those as
>> the boundaries for the individual concrete rows stored in a wide row [2].
>> For example:
>>
>> Given a Timeline table defined as key validation UTF8Type, column name
>> validation CompositeType(LongType, AsciiType), value validation BytesType:
>>
>> Timeline: {
>>     user1: {
>>         1341933021000000: {
>>             Author: "Tom",
>>             Body: "Hey!"
>>         },
>>         1341933022000000: {
>>             Author: "Paul",
>>             Body: "Nice",
>>             Lat: 40.0,
>>             Lon: 20.0
>>         },
>>         1341933023000000: {
>>             Author: "Lana",
>>             Body: "Cool"
>>         }
>>     },
>>     ...
>> }
>>
>> Both of the following structs are valid and will be able to be unmaped
>> from the wide row "user1":
>>
>> type Tweet struct {
>>     UserID  string `cf:"Timeline" key:"UserID" cols:"When"`
>>     When    int64
>>     Author  string
>>     Body    string
>> }
>>
>> type GeoTweet struct {
>>     UserID  string `cf:"Timeline" key:"UserID" cols:"When"`
>>     When    int64
>>     Author  string
>>     Body    string
>>     Lat     float32
>>     Lon     float32
>> }
>>
>
> That's exactly how CQL3 works. In that example, you would declare:
> CREATE TABLE tweet (
>     UserID text,
>     When int,
>     Author text,
>     Body text,
>     Lat float,
>     Long float,
>     PRIMARY KEY (UserId, When)
> )
> and that would layout things *exactly* like your Timeline above, but with
> validation.
>
> The fact that you have to declare Lat and Long does not mean that every
> CQL row must have them.
>
>
>> much nicer behaviour for model changes and migrations.
>>
>
> Not sure what you mean by that since adding new columns to a CQL3
> definition is basically free.
>
> --
> Sylvain
>



-- 
<http://www.groupalia.com/>Carlos CarrascoIT - Software Architect
Llull, 95-97, 2º planta, 08005 BarcelonaSkype: carlos.carrasco.groupalia
www.groupalia.comcarlos.carrasco@groupalia.com

Re: Dynamic CF

Posted by Sylvain Lebresne <sy...@datastax.com>.
On Tue, Jul 10, 2012 at 4:19 PM, Carlos Carrasco <
carlos.carrasco@groupalia.com> wrote:

> I think he means something like having a fixed set of coiumns in the table
> definition, then in the actual rows having other columns not specified in
> the defintion, indepentent of the composited part of the PK. When I
> reviewed CQL3 for using in Gossie[1] I realized I couldn't have this, and
> that it would complicate things like migrations or optional columns. For
> this reason I didn't use CQL3 and instead wrote a row unmaper that detects
> the discontinuities in the composited part and uses those as the boundaries
> for the individual concrete rows stored in a wide row [2]. For example:
>
> Given a Timeline table defined as key validation UTF8Type, column name
> validation CompositeType(LongType, AsciiType), value validation BytesType:
>
> Timeline: {
>     user1: {
>         1341933021000000: {
>             Author: "Tom",
>             Body: "Hey!"
>         },
>         1341933022000000: {
>             Author: "Paul",
>             Body: "Nice",
>             Lat: 40.0,
>             Lon: 20.0
>         },
>         1341933023000000: {
>             Author: "Lana",
>             Body: "Cool"
>         }
>     },
>     ...
> }
>
> Both of the following structs are valid and will be able to be unmaped
> from the wide row "user1":
>
> type Tweet struct {
>     UserID  string `cf:"Timeline" key:"UserID" cols:"When"`
>     When    int64
>     Author  string
>     Body    string
> }
>
> type GeoTweet struct {
>     UserID  string `cf:"Timeline" key:"UserID" cols:"When"`
>     When    int64
>     Author  string
>     Body    string
>     Lat     float32
>     Lon     float32
> }
>

That's exactly how CQL3 works. In that example, you would declare:
CREATE TABLE tweet (
    UserID text,
    When int,
    Author text,
    Body text,
    Lat float,
    Long float,
    PRIMARY KEY (UserId, When)
)
and that would layout things *exactly* like your Timeline above, but with
validation.

The fact that you have to declare Lat and Long does not mean that every CQL
row must have them.


> much nicer behaviour for model changes and migrations.
>

Not sure what you mean by that since adding new columns to a CQL3
definition is basically free.

--
Sylvain

Re: Dynamic CF

Posted by Carlos Carrasco <ca...@groupalia.com>.
I think he means something like having a fixed set of coiumns in the table
definition, then in the actual rows having other columns not specified in
the defintion, indepentent of the composited part of the PK. When I
reviewed CQL3 for using in Gossie[1] I realized I couldn't have this, and
that it would complicate things like migrations or optional columns. For
this reason I didn't use CQL3 and instead wrote a row unmaper that detects
the discontinuities in the composited part and uses those as the boundaries
for the individual concrete rows stored in a wide row [2]. For example:

Given a Timeline table defined as key validation UTF8Type, column name
validation CompositeType(LongType, AsciiType), value validation BytesType:

Timeline: {
    user1: {
        1341933021000000: {
            Author: "Tom",
            Body: "Hey!"
        },
        1341933022000000: {
            Author: "Paul",
            Body: "Nice",
            Lat: 40.0,
            Lon: 20.0
        },
        1341933023000000: {
            Author: "Lana",
            Body: "Cool"
        }
    },
    ...
}

Both of the following structs are valid and will be able to be unmaped from
the wide row "user1":

type Tweet struct {
    UserID  string `cf:"Timeline" key:"UserID" cols:"When"`
    When    int64
    Author  string
    Body    string
}

type GeoTweet struct {
    UserID  string `cf:"Timeline" key:"UserID" cols:"When"`
    When    int64
    Author  string
    Body    string
    Lat     float32
    Lon     float32
}

Granted I lose database-side validation over the individual column values
(BytesType) but in exchange I get very flexible rows and much nicer
behaviour for model changes and migrations.

1: https://github.com/carloscm/gossie
2: https://github.com/carloscm/gossie/blob/master/src/gossie/mapping.go#L339

On 10 July 2012 14:23, Sylvain Lebresne <sy...@datastax.com> wrote:

> On Fri, Jul 6, 2012 at 10:49 PM, Leonid Ilyevsky
> <li...@mooncapital.com> wrote:
> > At this point I am really confused about what direction Cassandra is
> going. CQL 3 has the benefit of composite keys, but no dynamic columns.
> > I thought, the whole point of Cassandra was to provide dynamic tables.
>
> CQL3 absolutely provide "dynamic tables"/wide rows, the syntax is just
> different. The typical example for wide rows is a time serie, for
> instance keeping all the events for a given event_kind in the same C*
> row ordered by time. You declare that in CQL3 using:
>   CREATE TABLE events (
>     event_kind text,
>     time timestamp,
>     event_name text,
>     event_details text,
>     PRIMARY KEY (event_kind, time)
>   )
>
> The important part in such definition is that one CQL row (i.e a given
> event_kind, time, event_name, even_details) does not map to an internal
> Cassandra row. More precisely, all events sharing the same event_kind will
> be
> in the same internal row. This is a wide row/dynamic table in the sense of
> thrift.
>
>
> > I need to have a huge table to store market quotes, and be able to query
> it by name and timestamp (t1 <= t <= t2), therefore I wanted the composite
> key.
> > Loading data to such table using prepared statements (CQL 3-based) was
> very slow, because it makes a server call for each row.
>
> You should use a BATCH statement which is the equivalent to batch_mutate.
>
> --
> Sylvain
>



-- 
<http://www.groupalia.com/>Carlos CarrascoIT - Software Architect
Llull, 95-97, 2º planta, 08005 BarcelonaSkype: carlos.carrasco.groupalia
www.groupalia.comcarlos.carrasco@groupalia.com

RE: Dynamic CF

Posted by Leonid Ilyevsky <li...@mooncapital.com>.
I see now there is a package org.apache.cassandra.cql3.statements, with BatchStatement class. Is this what I should use?

-----Original Message-----
From: Leonid Ilyevsky [mailto:lilyevsky@mooncapital.com]
Sent: Tuesday, July 10, 2012 11:45 AM
To: user@cassandra.apache.org
Subject: RE: Dynamic CF

I see. I actually tried it, and it consistently throws an exception. Below is my test code. I have two tests; test1 is for the composite key case, and test2 is for the simple key. The test2 works fine, while test1 gives me:

Exception in thread "main" InvalidRequestException(why:Not enough bytes to read value of component 0)
        at org.apache.cassandra.thrift.Cassandra$batch_mutate_result.read(Cassandra.java:20253)
        at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:78)
        at org.apache.cassandra.thrift.Cassandra$Client.recv_batch_mutate(Cassandra.java:922)
        at org.apache.cassandra.thrift.Cassandra$Client.batch_mutate(Cassandra.java:908)
        at com.moon.cql.BatchTest.test1(BatchTest.java:99)
        at com.moon.cql.BatchTest.main(BatchTest.java:45)


So you suggest to use BATCH statement. Since I do it from Java, it means creating a huge string (I may need to update thousands records at once), and executing it. Does it even make sense? Why is this going to be any better than simply execute prepared statement multiple times? The only thing it does is reduce number of calls to the server, but I have to figure out if this is the bottle neck I need to optimize.
Or maybe I need to break all my updates in a number of batches.
By the way, can a batch statement be prepared? With thousands of question marks in it?


============================================================

public class BatchTest {

    /**
     * @param args the command line arguments
     */
    public static void main(String[] args) throws TTransportException,
            InvalidRequestException, TException, UnavailableException,
            TimedOutException {

        String host = args[0];
        int port = Integer.parseInt(args[1]);

        test1(host, port);
        //test2(host, port);
    }

    private static void test1(String host, int port) throws TTransportException,
            InvalidRequestException, TException, UnavailableException,
            TimedOutException {
        TTransport transport =
                new TFramedTransport(new org.apache.thrift.transport.TSocket(
                host, port));
        transport.open();
        TProtocol protocol = new TBinaryProtocol(transport);
        Cassandra.Client client = new Cassandra.Client(protocol);
        client.set_cql_version("3.0.0");
        client.set_keyspace("test");

        Map<ByteBuffer, Map<String, List<Mutation>>> mutationMap =
                new HashMap();

        Map<String, List<Mutation>> mutations = new HashMap();
        List<Mutation> columnsMutations = new ArrayList();

        // key
        ByteBuffer keyBuffer = AsciiType.instance.decompose("KEY1");

        // key1 as column
        Column key1 = new Column();
        key1.setName("key1".getBytes());
        key1.setValue(LongType.instance.decompose(System.nanoTime()));
        key1.setTimestamp(System.currentTimeMillis());
        ColumnOrSuperColumn cc = new ColumnOrSuperColumn();
        cc.setColumn(key1);
        Mutation m = new Mutation();
        m.setColumn_or_supercolumn(cc);
        columnsMutations.add(m);

        // value column
        Column value = new Column();
        value.setName("value".getBytes());
        value.setValue(DoubleType.instance.decompose(5.3));
        value.setTimestamp(System.currentTimeMillis());
        cc = new ColumnOrSuperColumn();
        cc.setColumn(value);
        m = new Mutation();
        m.setColumn_or_supercolumn(cc);
        columnsMutations.add(m);

        // Inner mutation map
        mutations.put("testtable1", columnsMutations);

        // outer map : use the partition key
        mutationMap.put(keyBuffer, mutations);

        // Execute
        client.batch_mutate(mutationMap, ConsistencyLevel.ANY);
    }

      private static void test2(String host, int port) throws TTransportException,
            InvalidRequestException, TException, UnavailableException,
            TimedOutException {
        TTransport transport =
                new TFramedTransport(new org.apache.thrift.transport.TSocket(
                host, port));
        transport.open();
        TProtocol protocol = new TBinaryProtocol(transport);
        Cassandra.Client client = new Cassandra.Client(protocol);
        client.set_cql_version("3.0.0");
        client.set_keyspace("test");

        Map<ByteBuffer, Map<String, List<Mutation>>> mutationMap =
                new HashMap();

        Map<String, List<Mutation>> mutations = new HashMap();
        List<Mutation> columnsMutations = new ArrayList();

        // key
        ByteBuffer keyBuffer = AsciiType.instance.decompose("KEY1");

        // value column
        Column value = new Column();
        value.setName("value".getBytes());
        value.setValue(DoubleType.instance.decompose(5.3));
        value.setTimestamp(System.currentTimeMillis());
        ColumnOrSuperColumn cc = new ColumnOrSuperColumn();
        cc.setColumn(value);
        Mutation m = new Mutation();
        m.setColumn_or_supercolumn(cc);
        columnsMutations.add(m);

        // Inner mutation map
        mutations.put("testtable2", columnsMutations);

        // outer map : use the partition key
        mutationMap.put(keyBuffer, mutations);

        // Execute
        client.batch_mutate(mutationMap, ConsistencyLevel.ANY);
    }
}

============================================================

-----Original Message-----
From: Sylvain Lebresne [mailto:sylvain@datastax.com]
Sent: Tuesday, July 10, 2012 10:37 AM
To: user@cassandra.apache.org
Subject: Re: Dynamic CF

On Tue, Jul 10, 2012 at 4:17 PM, Leonid Ilyevsky
<li...@mooncapital.com> wrote:
> So I guess, in the batch_mutate call, in the map that I pass to it, only the first element of the composite key should be used as a key (because it is the real key), and the other parts of the key should be passed as regular columns? Is this correct? While I am waiting for your confirmation, I am going to try it.

I would really advise you to use the BATCH statement of CQL3 rather
than the thrift batch_mutate call. If only because until
https://issues.apache.org/jira/browse/CASSANDRA-4377 is resolved it
won't work at all, but also because the whole point of CQL3 is to hide
that kind of complexity.

--
Sylvain

>
> -----Original Message-----
> From: Sylvain Lebresne [mailto:sylvain@datastax.com]
> Sent: Tuesday, July 10, 2012 8:24 AM
> To: user@cassandra.apache.org
> Subject: Re: Dynamic CF
>
> On Fri, Jul 6, 2012 at 10:49 PM, Leonid Ilyevsky
> <li...@mooncapital.com> wrote:
>> At this point I am really confused about what direction Cassandra is going. CQL 3 has the benefit of composite keys, but no dynamic columns.
>> I thought, the whole point of Cassandra was to provide dynamic tables.
>
> CQL3 absolutely provide "dynamic tables"/wide rows, the syntax is just
> different. The typical example for wide rows is a time serie, for
> instance keeping all the events for a given event_kind in the same C*
> row ordered by time. You declare that in CQL3 using:
>   CREATE TABLE events (
>     event_kind text,
>     time timestamp,
>     event_name text,
>     event_details text,
>     PRIMARY KEY (event_kind, time)
>   )
>
> The important part in such definition is that one CQL row (i.e a given
> event_kind, time, event_name, even_details) does not map to an internal
> Cassandra row. More precisely, all events sharing the same event_kind will be
> in the same internal row. This is a wide row/dynamic table in the sense of
> thrift.
>
>
>> I need to have a huge table to store market quotes, and be able to query it by name and timestamp (t1 <= t <= t2), therefore I wanted the composite key.
>> Loading data to such table using prepared statements (CQL 3-based) was very slow, because it makes a server call for each row.
>
> You should use a BATCH statement which is the equivalent to batch_mutate.
>
> --
> Sylvain
>
> This email, along with any attachments, is confidential and may be legally privileged or otherwise protected from disclosure. Any unauthorized dissemination, copying or use of the contents of this email is strictly prohibited and may be in violation of law. If you are not the intended recipient, any disclosure, copying, forwarding or distribution of this email is strictly prohibited and this email and any attachments should be deleted immediately.  This email and any attachments do not constitute an offer to sell or a solicitation of an offer to purchase any interest in any investment vehicle sponsored by Moon Capital Management LP ("Moon Capital"). Moon Capital does not provide legal, accounting or tax advice. Any statement regarding legal, accounting or tax matters was not intended or written to be relied upon by any person as advice. Moon Capital does not waive confidentiality or privilege as a result of this email.

This email, along with any attachments, is confidential and may be legally privileged or otherwise protected from disclosure. Any unauthorized dissemination, copying or use of the contents of this email is strictly prohibited and may be in violation of law. If you are not the intended recipient, any disclosure, copying, forwarding or distribution of this email is strictly prohibited and this email and any attachments should be deleted immediately.  This email and any attachments do not constitute an offer to sell or a solicitation of an offer to purchase any interest in any investment vehicle sponsored by Moon Capital Management LP ("Moon Capital"). Moon Capital does not provide legal, accounting or tax advice. Any statement regarding legal, accounting or tax matters was not intended or written to be relied upon by any person as advice. Moon Capital does not waive confidentiality or privilege as a result of this email.

This email, along with any attachments, is confidential and may be legally privileged or otherwise protected from disclosure. Any unauthorized dissemination, copying or use of the contents of this email is strictly prohibited and may be in violation of law. If you are not the intended recipient, any disclosure, copying, forwarding or distribution of this email is strictly prohibited and this email and any attachments should be deleted immediately.  This email and any attachments do not constitute an offer to sell or a solicitation of an offer to purchase any interest in any investment vehicle sponsored by Moon Capital Management LP ("Moon Capital"). Moon Capital does not provide legal, accounting or tax advice. Any statement regarding legal, accounting or tax matters was not intended or written to be relied upon by any person as advice. Moon Capital does not waive confidentiality or privilege as a result of this email.

RE: Dynamic CF

Posted by Leonid Ilyevsky <li...@mooncapital.com>.
I see. I actually tried it, and it consistently throws an exception. Below is my test code. I have two tests; test1 is for the composite key case, and test2 is for the simple key. The test2 works fine, while test1 gives me:

Exception in thread "main" InvalidRequestException(why:Not enough bytes to read value of component 0)
        at org.apache.cassandra.thrift.Cassandra$batch_mutate_result.read(Cassandra.java:20253)
        at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:78)
        at org.apache.cassandra.thrift.Cassandra$Client.recv_batch_mutate(Cassandra.java:922)
        at org.apache.cassandra.thrift.Cassandra$Client.batch_mutate(Cassandra.java:908)
        at com.moon.cql.BatchTest.test1(BatchTest.java:99)
        at com.moon.cql.BatchTest.main(BatchTest.java:45)


So you suggest to use BATCH statement. Since I do it from Java, it means creating a huge string (I may need to update thousands records at once), and executing it. Does it even make sense? Why is this going to be any better than simply execute prepared statement multiple times? The only thing it does is reduce number of calls to the server, but I have to figure out if this is the bottle neck I need to optimize.
Or maybe I need to break all my updates in a number of batches.
By the way, can a batch statement be prepared? With thousands of question marks in it?


============================================================

public class BatchTest {

    /**
     * @param args the command line arguments
     */
    public static void main(String[] args) throws TTransportException,
            InvalidRequestException, TException, UnavailableException,
            TimedOutException {

        String host = args[0];
        int port = Integer.parseInt(args[1]);

        test1(host, port);
        //test2(host, port);
    }

    private static void test1(String host, int port) throws TTransportException,
            InvalidRequestException, TException, UnavailableException,
            TimedOutException {
        TTransport transport =
                new TFramedTransport(new org.apache.thrift.transport.TSocket(
                host, port));
        transport.open();
        TProtocol protocol = new TBinaryProtocol(transport);
        Cassandra.Client client = new Cassandra.Client(protocol);
        client.set_cql_version("3.0.0");
        client.set_keyspace("test");

        Map<ByteBuffer, Map<String, List<Mutation>>> mutationMap =
                new HashMap();

        Map<String, List<Mutation>> mutations = new HashMap();
        List<Mutation> columnsMutations = new ArrayList();

        // key
        ByteBuffer keyBuffer = AsciiType.instance.decompose("KEY1");

        // key1 as column
        Column key1 = new Column();
        key1.setName("key1".getBytes());
        key1.setValue(LongType.instance.decompose(System.nanoTime()));
        key1.setTimestamp(System.currentTimeMillis());
        ColumnOrSuperColumn cc = new ColumnOrSuperColumn();
        cc.setColumn(key1);
        Mutation m = new Mutation();
        m.setColumn_or_supercolumn(cc);
        columnsMutations.add(m);

        // value column
        Column value = new Column();
        value.setName("value".getBytes());
        value.setValue(DoubleType.instance.decompose(5.3));
        value.setTimestamp(System.currentTimeMillis());
        cc = new ColumnOrSuperColumn();
        cc.setColumn(value);
        m = new Mutation();
        m.setColumn_or_supercolumn(cc);
        columnsMutations.add(m);

        // Inner mutation map
        mutations.put("testtable1", columnsMutations);

        // outer map : use the partition key
        mutationMap.put(keyBuffer, mutations);

        // Execute
        client.batch_mutate(mutationMap, ConsistencyLevel.ANY);
    }

      private static void test2(String host, int port) throws TTransportException,
            InvalidRequestException, TException, UnavailableException,
            TimedOutException {
        TTransport transport =
                new TFramedTransport(new org.apache.thrift.transport.TSocket(
                host, port));
        transport.open();
        TProtocol protocol = new TBinaryProtocol(transport);
        Cassandra.Client client = new Cassandra.Client(protocol);
        client.set_cql_version("3.0.0");
        client.set_keyspace("test");

        Map<ByteBuffer, Map<String, List<Mutation>>> mutationMap =
                new HashMap();

        Map<String, List<Mutation>> mutations = new HashMap();
        List<Mutation> columnsMutations = new ArrayList();

        // key
        ByteBuffer keyBuffer = AsciiType.instance.decompose("KEY1");

        // value column
        Column value = new Column();
        value.setName("value".getBytes());
        value.setValue(DoubleType.instance.decompose(5.3));
        value.setTimestamp(System.currentTimeMillis());
        ColumnOrSuperColumn cc = new ColumnOrSuperColumn();
        cc.setColumn(value);
        Mutation m = new Mutation();
        m.setColumn_or_supercolumn(cc);
        columnsMutations.add(m);

        // Inner mutation map
        mutations.put("testtable2", columnsMutations);

        // outer map : use the partition key
        mutationMap.put(keyBuffer, mutations);

        // Execute
        client.batch_mutate(mutationMap, ConsistencyLevel.ANY);
    }
}

============================================================

-----Original Message-----
From: Sylvain Lebresne [mailto:sylvain@datastax.com]
Sent: Tuesday, July 10, 2012 10:37 AM
To: user@cassandra.apache.org
Subject: Re: Dynamic CF

On Tue, Jul 10, 2012 at 4:17 PM, Leonid Ilyevsky
<li...@mooncapital.com> wrote:
> So I guess, in the batch_mutate call, in the map that I pass to it, only the first element of the composite key should be used as a key (because it is the real key), and the other parts of the key should be passed as regular columns? Is this correct? While I am waiting for your confirmation, I am going to try it.

I would really advise you to use the BATCH statement of CQL3 rather
than the thrift batch_mutate call. If only because until
https://issues.apache.org/jira/browse/CASSANDRA-4377 is resolved it
won't work at all, but also because the whole point of CQL3 is to hide
that kind of complexity.

--
Sylvain

>
> -----Original Message-----
> From: Sylvain Lebresne [mailto:sylvain@datastax.com]
> Sent: Tuesday, July 10, 2012 8:24 AM
> To: user@cassandra.apache.org
> Subject: Re: Dynamic CF
>
> On Fri, Jul 6, 2012 at 10:49 PM, Leonid Ilyevsky
> <li...@mooncapital.com> wrote:
>> At this point I am really confused about what direction Cassandra is going. CQL 3 has the benefit of composite keys, but no dynamic columns.
>> I thought, the whole point of Cassandra was to provide dynamic tables.
>
> CQL3 absolutely provide "dynamic tables"/wide rows, the syntax is just
> different. The typical example for wide rows is a time serie, for
> instance keeping all the events for a given event_kind in the same C*
> row ordered by time. You declare that in CQL3 using:
>   CREATE TABLE events (
>     event_kind text,
>     time timestamp,
>     event_name text,
>     event_details text,
>     PRIMARY KEY (event_kind, time)
>   )
>
> The important part in such definition is that one CQL row (i.e a given
> event_kind, time, event_name, even_details) does not map to an internal
> Cassandra row. More precisely, all events sharing the same event_kind will be
> in the same internal row. This is a wide row/dynamic table in the sense of
> thrift.
>
>
>> I need to have a huge table to store market quotes, and be able to query it by name and timestamp (t1 <= t <= t2), therefore I wanted the composite key.
>> Loading data to such table using prepared statements (CQL 3-based) was very slow, because it makes a server call for each row.
>
> You should use a BATCH statement which is the equivalent to batch_mutate.
>
> --
> Sylvain
>
> This email, along with any attachments, is confidential and may be legally privileged or otherwise protected from disclosure. Any unauthorized dissemination, copying or use of the contents of this email is strictly prohibited and may be in violation of law. If you are not the intended recipient, any disclosure, copying, forwarding or distribution of this email is strictly prohibited and this email and any attachments should be deleted immediately.  This email and any attachments do not constitute an offer to sell or a solicitation of an offer to purchase any interest in any investment vehicle sponsored by Moon Capital Management LP ("Moon Capital"). Moon Capital does not provide legal, accounting or tax advice. Any statement regarding legal, accounting or tax matters was not intended or written to be relied upon by any person as advice. Moon Capital does not waive confidentiality or privilege as a result of this email.

This email, along with any attachments, is confidential and may be legally privileged or otherwise protected from disclosure. Any unauthorized dissemination, copying or use of the contents of this email is strictly prohibited and may be in violation of law. If you are not the intended recipient, any disclosure, copying, forwarding or distribution of this email is strictly prohibited and this email and any attachments should be deleted immediately.  This email and any attachments do not constitute an offer to sell or a solicitation of an offer to purchase any interest in any investment vehicle sponsored by Moon Capital Management LP ("Moon Capital"). Moon Capital does not provide legal, accounting or tax advice. Any statement regarding legal, accounting or tax matters was not intended or written to be relied upon by any person as advice. Moon Capital does not waive confidentiality or privilege as a result of this email.

Re: Dynamic CF

Posted by Sylvain Lebresne <sy...@datastax.com>.
On Tue, Jul 10, 2012 at 4:17 PM, Leonid Ilyevsky
<li...@mooncapital.com> wrote:
> So I guess, in the batch_mutate call, in the map that I pass to it, only the first element of the composite key should be used as a key (because it is the real key), and the other parts of the key should be passed as regular columns? Is this correct? While I am waiting for your confirmation, I am going to try it.

I would really advise you to use the BATCH statement of CQL3 rather
than the thrift batch_mutate call. If only because until
https://issues.apache.org/jira/browse/CASSANDRA-4377 is resolved it
won't work at all, but also because the whole point of CQL3 is to hide
that kind of complexity.

--
Sylvain

>
> -----Original Message-----
> From: Sylvain Lebresne [mailto:sylvain@datastax.com]
> Sent: Tuesday, July 10, 2012 8:24 AM
> To: user@cassandra.apache.org
> Subject: Re: Dynamic CF
>
> On Fri, Jul 6, 2012 at 10:49 PM, Leonid Ilyevsky
> <li...@mooncapital.com> wrote:
>> At this point I am really confused about what direction Cassandra is going. CQL 3 has the benefit of composite keys, but no dynamic columns.
>> I thought, the whole point of Cassandra was to provide dynamic tables.
>
> CQL3 absolutely provide "dynamic tables"/wide rows, the syntax is just
> different. The typical example for wide rows is a time serie, for
> instance keeping all the events for a given event_kind in the same C*
> row ordered by time. You declare that in CQL3 using:
>   CREATE TABLE events (
>     event_kind text,
>     time timestamp,
>     event_name text,
>     event_details text,
>     PRIMARY KEY (event_kind, time)
>   )
>
> The important part in such definition is that one CQL row (i.e a given
> event_kind, time, event_name, even_details) does not map to an internal
> Cassandra row. More precisely, all events sharing the same event_kind will be
> in the same internal row. This is a wide row/dynamic table in the sense of
> thrift.
>
>
>> I need to have a huge table to store market quotes, and be able to query it by name and timestamp (t1 <= t <= t2), therefore I wanted the composite key.
>> Loading data to such table using prepared statements (CQL 3-based) was very slow, because it makes a server call for each row.
>
> You should use a BATCH statement which is the equivalent to batch_mutate.
>
> --
> Sylvain
>
> This email, along with any attachments, is confidential and may be legally privileged or otherwise protected from disclosure. Any unauthorized dissemination, copying or use of the contents of this email is strictly prohibited and may be in violation of law. If you are not the intended recipient, any disclosure, copying, forwarding or distribution of this email is strictly prohibited and this email and any attachments should be deleted immediately.  This email and any attachments do not constitute an offer to sell or a solicitation of an offer to purchase any interest in any investment vehicle sponsored by Moon Capital Management LP ("Moon Capital"). Moon Capital does not provide legal, accounting or tax advice. Any statement regarding legal, accounting or tax matters was not intended or written to be relied upon by any person as advice. Moon Capital does not waive confidentiality or privilege as a result of this email.

RE: Dynamic CF

Posted by Leonid Ilyevsky <li...@mooncapital.com>.
Thanks Sylvain, this is useful.
So I guess, in the batch_mutate call, in the map that I pass to it, only the first element of the composite key should be used as a key (because it is the real key), and the other parts of the key should be passed as regular columns? Is this correct? While I am waiting for your confirmation, I am going to try it.

-----Original Message-----
From: Sylvain Lebresne [mailto:sylvain@datastax.com]
Sent: Tuesday, July 10, 2012 8:24 AM
To: user@cassandra.apache.org
Subject: Re: Dynamic CF

On Fri, Jul 6, 2012 at 10:49 PM, Leonid Ilyevsky
<li...@mooncapital.com> wrote:
> At this point I am really confused about what direction Cassandra is going. CQL 3 has the benefit of composite keys, but no dynamic columns.
> I thought, the whole point of Cassandra was to provide dynamic tables.

CQL3 absolutely provide "dynamic tables"/wide rows, the syntax is just
different. The typical example for wide rows is a time serie, for
instance keeping all the events for a given event_kind in the same C*
row ordered by time. You declare that in CQL3 using:
  CREATE TABLE events (
    event_kind text,
    time timestamp,
    event_name text,
    event_details text,
    PRIMARY KEY (event_kind, time)
  )

The important part in such definition is that one CQL row (i.e a given
event_kind, time, event_name, even_details) does not map to an internal
Cassandra row. More precisely, all events sharing the same event_kind will be
in the same internal row. This is a wide row/dynamic table in the sense of
thrift.


> I need to have a huge table to store market quotes, and be able to query it by name and timestamp (t1 <= t <= t2), therefore I wanted the composite key.
> Loading data to such table using prepared statements (CQL 3-based) was very slow, because it makes a server call for each row.

You should use a BATCH statement which is the equivalent to batch_mutate.

--
Sylvain

This email, along with any attachments, is confidential and may be legally privileged or otherwise protected from disclosure. Any unauthorized dissemination, copying or use of the contents of this email is strictly prohibited and may be in violation of law. If you are not the intended recipient, any disclosure, copying, forwarding or distribution of this email is strictly prohibited and this email and any attachments should be deleted immediately.  This email and any attachments do not constitute an offer to sell or a solicitation of an offer to purchase any interest in any investment vehicle sponsored by Moon Capital Management LP ("Moon Capital"). Moon Capital does not provide legal, accounting or tax advice. Any statement regarding legal, accounting or tax matters was not intended or written to be relied upon by any person as advice. Moon Capital does not waive confidentiality or privilege as a result of this email.

Re: Dynamic CF

Posted by Greg Fausak <gr...@named.com>.
I thought I'd give this a try:

create columnfamily
    at_event__ac_c
(
    ac_c text,
    ac_creation bigint,
    ac_name text,
    ac_value text,
    PRIMARY KEY (ac_c, ac_creation)
) with compression_parameters:sstable_compression = '';

Then, insert a few columns:

begin batch using consistency quorum
insert into at_event__ac_c (ac_c, ac_creation, ac_name, ac_value)
values ('83433.361.t4.l1.cisco.com', 1303167920747402,
'ac_event_type', 'SERV.CPE.CONN')
insert into at_event__ac_c (ac_c, ac_creation, ac_name, ac_value)
values ('83433.361.t4.l1.cisco.com', 1303167920747402, 'ac_vid', '')
insert into at_event__ac_c (ac_c, ac_creation, ac_name, ac_value)
values ('83433.361.t4.l1.cisco.com', 1303167920747402, 'ac_state',
'UP')
apply batch;

then followed up with a query:
cqlsh:op2> select * from at_event__ac_c;
 ac_c                      | ac_creation      | ac_name  | ac_value
---------------------------+------------------+----------+----------
 83433.361.t4.l1.cisco.com | 1303167920747402 | ac_state |       UP

So, I can't get the same index behavior from the ac_creation column
as I get from the ac_c column.  That is, ac_c is additive, ac_creation
overwrites.  When I turn this around and create a real wide row, like this:

create columnfamily
    at_event__ac_c
(
    ac_c text,
    ac_creation bigint,
    ac_event_type text,
    ac_vid text,
    ac_state text,
    PRIMARY KEY (ac_c, ac_creation)
) with compression_parameters:sstable_compression = '';

I get the behavior I want, which is a composite row which is indexed
by ac_c and ac_creation.

If I have 100 columns defined as a static CF (like above), when I do an insert
does it just use the space for the columns inserted, or the columns declared?

-g


On Tue, Jul 10, 2012 at 7:23 AM, Sylvain Lebresne <sy...@datastax.com> wrote:
> On Fri, Jul 6, 2012 at 10:49 PM, Leonid Ilyevsky
> <li...@mooncapital.com> wrote:
>> At this point I am really confused about what direction Cassandra is going. CQL 3 has the benefit of composite keys, but no dynamic columns.
>> I thought, the whole point of Cassandra was to provide dynamic tables.
>
> CQL3 absolutely provide "dynamic tables"/wide rows, the syntax is just
> different. The typical example for wide rows is a time serie, for
> instance keeping all the events for a given event_kind in the same C*
> row ordered by time. You declare that in CQL3 using:
>   CREATE TABLE events (
>     event_kind text,
>     time timestamp,
>     event_name text,
>     event_details text,
>     PRIMARY KEY (event_kind, time)
>   )
>
> The important part in such definition is that one CQL row (i.e a given
> event_kind, time, event_name, even_details) does not map to an internal
> Cassandra row. More precisely, all events sharing the same event_kind will be
> in the same internal row. This is a wide row/dynamic table in the sense of
> thrift.
>
>
>> I need to have a huge table to store market quotes, and be able to query it by name and timestamp (t1 <= t <= t2), therefore I wanted the composite key.
>> Loading data to such table using prepared statements (CQL 3-based) was very slow, because it makes a server call for each row.
>
> You should use a BATCH statement which is the equivalent to batch_mutate.
>
> --
> Sylvain

Re: Dynamic CF

Posted by Sylvain Lebresne <sy...@datastax.com>.
On Fri, Jul 6, 2012 at 10:49 PM, Leonid Ilyevsky
<li...@mooncapital.com> wrote:
> At this point I am really confused about what direction Cassandra is going. CQL 3 has the benefit of composite keys, but no dynamic columns.
> I thought, the whole point of Cassandra was to provide dynamic tables.

CQL3 absolutely provide "dynamic tables"/wide rows, the syntax is just
different. The typical example for wide rows is a time serie, for
instance keeping all the events for a given event_kind in the same C*
row ordered by time. You declare that in CQL3 using:
  CREATE TABLE events (
    event_kind text,
    time timestamp,
    event_name text,
    event_details text,
    PRIMARY KEY (event_kind, time)
  )

The important part in such definition is that one CQL row (i.e a given
event_kind, time, event_name, even_details) does not map to an internal
Cassandra row. More precisely, all events sharing the same event_kind will be
in the same internal row. This is a wide row/dynamic table in the sense of
thrift.


> I need to have a huge table to store market quotes, and be able to query it by name and timestamp (t1 <= t <= t2), therefore I wanted the composite key.
> Loading data to such table using prepared statements (CQL 3-based) was very slow, because it makes a server call for each row.

You should use a BATCH statement which is the equivalent to batch_mutate.

--
Sylvain

RE: Dynamic CF

Posted by Leonid Ilyevsky <li...@mooncapital.com>.
Indeed the cql 3.0 reference in cassandra11.pdf does not mention the dynamic tables whatsoever.
Now I found the following posting:

http://www.datastax.com/support-forums/topic/flexible-schema-in-cql-3

where the moderator said " Yes, CQL 3 expects all columns to be defined in the metadata".

At this point I am really confused about what direction Cassandra is going. CQL 3 has the benefit of composite keys, but no dynamic columns.
I thought, the whole point of Cassandra was to provide dynamic tables.

I need to have a huge table to store market quotes, and be able to query it by name and timestamp (t1 <= t <= t2), therefore I wanted the composite key.
Loading data to such table using prepared statements (CQL 3-based) was very slow, because it makes a server call for each row.
So I looked at the API way of doing it, using batch_mutate. Unfortunately, it seems that batch_mutate would not support composite key.

Then I thought of making wide rows, so I store many quotes in one row (few thousand). For this I really need dynamic table.

I would appreciate any useful suggestion.

-----Original Message-----
From: Leonid Ilyevsky [mailto:lilyevsky@mooncapital.com]
Sent: Friday, July 06, 2012 4:16 PM
To: user@cassandra.apache.org
Subject: RE: Dynamic CF

Thanks Jeremy, but this doesn't work for me. I am using cql3, because I need new features like composite keys. The manual you pointed to is for 2.0.
I have suspicion that cql3 does not support dynamic tables at all. Is there a manual for cql3?

-----Original Message-----
From: Jeremy Hanna [mailto:jeremy.hanna1234@gmail.com]
Sent: Friday, July 06, 2012 4:06 PM
To: user@cassandra.apache.org
Subject: Re: Dynamic CF

you can use the cqlsh help but it will eventually refer you to a cql reference such as this one that says what the options are.  Looks like you need just 'default_validation'.

http://www.datastax.com/docs/1.0/references/cql/index#cql-column-family-storage-parameters

On Jul 6, 2012, at 2:13 PM, Leonid Ilyevsky wrote:

> Can I create a dynamic Cf using cql3? What is the syntax?
>
> Here is what I tried:
>
>
> cqlsh:test> create table mytest (
>         ... asset ascii, datetime int, count int, primary key(asset, datetime) )
>         ...  with default_validation_class = double;
> Bad Request: default_validation_class is not a valid keyword argument for CREATE TABLE
>
> This email, along with any attachments, is confidential and may be legally privileged or otherwise protected from disclosure. Any unauthorized dissemination, copying or use of the contents of this email is strictly prohibited and may be in violation of law. If you are not the intended recipient, any disclosure, copying, forwarding or distribution of this email is strictly prohibited and this email and any attachments should be deleted immediately. This email and any attachments do not constitute an offer to sell or a solicitation of an offer to purchase any interest in any investment vehicle sponsored by Moon Capital Management LP ("Moon Capital"). Moon Capital does not provide legal, accounting or tax advice. Any statement regarding legal, accounting or tax matters was not intended or written to be relied upon by any person as advice. Moon Capital does not waive confidentiality or privilege as a result of this email.


This email, along with any attachments, is confidential and may be legally privileged or otherwise protected from disclosure. Any unauthorized dissemination, copying or use of the contents of this email is strictly prohibited and may be in violation of law. If you are not the intended recipient, any disclosure, copying, forwarding or distribution of this email is strictly prohibited and this email and any attachments should be deleted immediately.  This email and any attachments do not constitute an offer to sell or a solicitation of an offer to purchase any interest in any investment vehicle sponsored by Moon Capital Management LP ("Moon Capital"). Moon Capital does not provide legal, accounting or tax advice. Any statement regarding legal, accounting or tax matters was not intended or written to be relied upon by any person as advice. Moon Capital does not waive confidentiality or privilege as a result of this email.

This email, along with any attachments, is confidential and may be legally privileged or otherwise protected from disclosure. Any unauthorized dissemination, copying or use of the contents of this email is strictly prohibited and may be in violation of law. If you are not the intended recipient, any disclosure, copying, forwarding or distribution of this email is strictly prohibited and this email and any attachments should be deleted immediately.  This email and any attachments do not constitute an offer to sell or a solicitation of an offer to purchase any interest in any investment vehicle sponsored by Moon Capital Management LP ("Moon Capital"). Moon Capital does not provide legal, accounting or tax advice. Any statement regarding legal, accounting or tax matters was not intended or written to be relied upon by any person as advice. Moon Capital does not waive confidentiality or privilege as a result of this email.

Re: Dynamic CF

Posted by Jeremy Hanna <je...@gmail.com>.
Oh you're right - that has changed in cql 3:
http://www.datastax.com/docs/1.1/configuration/storage_configuration#cf-storage-options

I'm not sure then why that wouldn't work as it seems valid with the option and its value (double).

On Jul 6, 2012, at 3:15 PM, Leonid Ilyevsky wrote:

> Thanks Jeremy, but this doesn't work for me. I am using cql3, because I need new features like composite keys. The manual you pointed to is for 2.0.
> I have suspicion that cql3 does not support dynamic tables at all. Is there a manual for cql3?
> 
> -----Original Message-----
> From: Jeremy Hanna [mailto:jeremy.hanna1234@gmail.com]
> Sent: Friday, July 06, 2012 4:06 PM
> To: user@cassandra.apache.org
> Subject: Re: Dynamic CF
> 
> you can use the cqlsh help but it will eventually refer you to a cql reference such as this one that says what the options are.  Looks like you need just 'default_validation'.
> 
> http://www.datastax.com/docs/1.0/references/cql/index#cql-column-family-storage-parameters
> 
> On Jul 6, 2012, at 2:13 PM, Leonid Ilyevsky wrote:
> 
>> Can I create a dynamic Cf using cql3? What is the syntax?
>> 
>> Here is what I tried:
>> 
>> 
>> cqlsh:test> create table mytest (
>>        ... asset ascii, datetime int, count int, primary key(asset, datetime) )
>>        ...  with default_validation_class = double;
>> Bad Request: default_validation_class is not a valid keyword argument for CREATE TABLE
>> 
>> This email, along with any attachments, is confidential and may be legally privileged or otherwise protected from disclosure. Any unauthorized dissemination, copying or use of the contents of this email is strictly prohibited and may be in violation of law. If you are not the intended recipient, any disclosure, copying, forwarding or distribution of this email is strictly prohibited and this email and any attachments should be deleted immediately. This email and any attachments do not constitute an offer to sell or a solicitation of an offer to purchase any interest in any investment vehicle sponsored by Moon Capital Management LP ("Moon Capital"). Moon Capital does not provide legal, accounting or tax advice. Any statement regarding legal, accounting or tax matters was not intended or written to be relied upon by any person as advice. Moon Capital does not waive confidentiality or privilege as a result of this email.
> 
> 
> This email, along with any attachments, is confidential and may be legally privileged or otherwise protected from disclosure. Any unauthorized dissemination, copying or use of the contents of this email is strictly prohibited and may be in violation of law. If you are not the intended recipient, any disclosure, copying, forwarding or distribution of this email is strictly prohibited and this email and any attachments should be deleted immediately.  This email and any attachments do not constitute an offer to sell or a solicitation of an offer to purchase any interest in any investment vehicle sponsored by Moon Capital Management LP ("Moon Capital"). Moon Capital does not provide legal, accounting or tax advice. Any statement regarding legal, accounting or tax matters was not intended or written to be relied upon by any person as advice. Moon Capital does not waive confidentiality or privilege as a result of this email.


RE: Dynamic CF

Posted by Leonid Ilyevsky <li...@mooncapital.com>.
Thanks Jeremy, but this doesn't work for me. I am using cql3, because I need new features like composite keys. The manual you pointed to is for 2.0.
I have suspicion that cql3 does not support dynamic tables at all. Is there a manual for cql3?

-----Original Message-----
From: Jeremy Hanna [mailto:jeremy.hanna1234@gmail.com]
Sent: Friday, July 06, 2012 4:06 PM
To: user@cassandra.apache.org
Subject: Re: Dynamic CF

you can use the cqlsh help but it will eventually refer you to a cql reference such as this one that says what the options are.  Looks like you need just 'default_validation'.

http://www.datastax.com/docs/1.0/references/cql/index#cql-column-family-storage-parameters

On Jul 6, 2012, at 2:13 PM, Leonid Ilyevsky wrote:

> Can I create a dynamic Cf using cql3? What is the syntax?
>
> Here is what I tried:
>
>
> cqlsh:test> create table mytest (
>         ... asset ascii, datetime int, count int, primary key(asset, datetime) )
>         ...  with default_validation_class = double;
> Bad Request: default_validation_class is not a valid keyword argument for CREATE TABLE
>
> This email, along with any attachments, is confidential and may be legally privileged or otherwise protected from disclosure. Any unauthorized dissemination, copying or use of the contents of this email is strictly prohibited and may be in violation of law. If you are not the intended recipient, any disclosure, copying, forwarding or distribution of this email is strictly prohibited and this email and any attachments should be deleted immediately. This email and any attachments do not constitute an offer to sell or a solicitation of an offer to purchase any interest in any investment vehicle sponsored by Moon Capital Management LP ("Moon Capital"). Moon Capital does not provide legal, accounting or tax advice. Any statement regarding legal, accounting or tax matters was not intended or written to be relied upon by any person as advice. Moon Capital does not waive confidentiality or privilege as a result of this email.


This email, along with any attachments, is confidential and may be legally privileged or otherwise protected from disclosure. Any unauthorized dissemination, copying or use of the contents of this email is strictly prohibited and may be in violation of law. If you are not the intended recipient, any disclosure, copying, forwarding or distribution of this email is strictly prohibited and this email and any attachments should be deleted immediately.  This email and any attachments do not constitute an offer to sell or a solicitation of an offer to purchase any interest in any investment vehicle sponsored by Moon Capital Management LP ("Moon Capital"). Moon Capital does not provide legal, accounting or tax advice. Any statement regarding legal, accounting or tax matters was not intended or written to be relied upon by any person as advice. Moon Capital does not waive confidentiality or privilege as a result of this email.

Re: Dynamic CF

Posted by Jeremy Hanna <je...@gmail.com>.
you can use the cqlsh help but it will eventually refer you to a cql reference such as this one that says what the options are.  Looks like you need just 'default_validation'.

http://www.datastax.com/docs/1.0/references/cql/index#cql-column-family-storage-parameters

On Jul 6, 2012, at 2:13 PM, Leonid Ilyevsky wrote:

> Can I create a dynamic Cf using cql3? What is the syntax?
>  
> Here is what I tried:
>  
>  
> cqlsh:test> create table mytest (
>         ... asset ascii, datetime int, count int, primary key(asset, datetime) )
>         ...  with default_validation_class = double;
> Bad Request: default_validation_class is not a valid keyword argument for CREATE TABLE
> 
> This email, along with any attachments, is confidential and may be legally privileged or otherwise protected from disclosure. Any unauthorized dissemination, copying or use of the contents of this email is strictly prohibited and may be in violation of law. If you are not the intended recipient, any disclosure, copying, forwarding or distribution of this email is strictly prohibited and this email and any attachments should be deleted immediately. This email and any attachments do not constitute an offer to sell or a solicitation of an offer to purchase any interest in any investment vehicle sponsored by Moon Capital Management LP (“Moon Capital”). Moon Capital does not provide legal, accounting or tax advice. Any statement regarding legal, accounting or tax matters was not intended or written to be relied upon by any person as advice. Moon Capital does not waive confidentiality or privilege as a result of this email.


Dynamic CF

Posted by Leonid Ilyevsky <li...@mooncapital.com>.
Can I create a dynamic Cf using cql3? What is the syntax?

Here is what I tried:


cqlsh:test> create table mytest (
        ... asset ascii, datetime int, count int, primary key(asset, datetime) )
        ...  with default_validation_class = double;
Bad Request: default_validation_class is not a valid keyword argument for CREATE TABLE

________________________________
This email, along with any attachments, is confidential and may be legally privileged or otherwise protected from disclosure. Any unauthorized dissemination, copying or use of the contents of this email is strictly prohibited and may be in violation of law. If you are not the intended recipient, any disclosure, copying, forwarding or distribution of this email is strictly prohibited and this email and any attachments should be deleted immediately. This email and any attachments do not constitute an offer to sell or a solicitation of an offer to purchase any interest in any investment vehicle sponsored by Moon Capital Management LP ("Moon Capital"). Moon Capital does not provide legal, accounting or tax advice. Any statement regarding legal, accounting or tax matters was not intended or written to be relied upon by any person as advice. Moon Capital does not waive confidentiality or privilege as a result of this email.

Re: Composite key in thrift java api - still need help!!

Posted by aaron morton <aa...@thelastpickle.com>.
AFAIK in all cases we consider values to be just bytes. They are passed to the ADT to turn into type aware instances. 

Can you try doing what you want to do with Hector or Astyanax? Then take a look at how they work if you still want to roll your own. 

Cheers

-----------------
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com

On 7/07/2012, at 3:48 AM, Leonid Ilyevsky wrote:

> I tried it, does not work for my batch_mutate (I tried both CompositeType.Builder and also my own implementation following the code examples).
> So it seems that the format with short length before the field and zero byte after does not work for the key.
> I found by experiment and debugger that in case of a simple key, the key buffer in the  mutation_map passed to the batch_mutate contains simple key bits, without length or terminating zero.
> The same thing (simple concatenation) does not work for composite key case. Short length with terminating zero does not work either.
> So the question is: what is the required format of the key buffer?
>  
> From: Leonid Ilyevsky [mailto:lilyevsky@mooncapital.com] 
> Sent: Friday, July 06, 2012 10:05 AM
> To: 'user@cassandra.apache.org'
> Subject: RE: Composite key in thrift java api
>  
> Thanks Aaron,
>  
> I guess, you suggest I just borrow the logic from that CompositeType.java, the “decompose” method, to comply with the protocol.
> I thought I can just call it, but I don’t see the “decompose(Object... objects)” in the current 1.1.2 version, so probably your snippet is from the older version.
>  
> On the other hand, in the new version I see CompositeType.Builder inner class, looks like I should use that.
> Could you please confirm?
>  
> Thanks,
>  
> Leonid
>  
>  
> From: aaron morton [mailto:aaron@thelastpickle.com] 
> Sent: Thursday, July 05, 2012 7:10 PM
> To: user@cassandra.apache.org
> Subject: Re: Composite key in thrift java api
>  
>  I would really prefer to do it in Cassandra itself,
> See https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/db/marshal/CompositeType.java
>  
> Cheers
>  
>  
> -----------------
> Aaron Morton
> Freelance Developer
> @aaronmorton
> http://www.thelastpickle.com
>  
> On 6/07/2012, at 10:40 AM, Leonid Ilyevsky wrote:
>  
> 
> I need to create a ByteBuffer instance containing the proper composite key, based on the values of the components of the key. I am going to use it for update operation.
> I tried to simply concatenate the buffers corresponding to the components, but I am not sure this is correct, because I am getting exception that comes from the server :
>  
> InvalidRequestException(why:Not enough bytes to read value of component 0)
>  
> In the server log I see this:
>  
> org.apache.thrift.transport.TTransportException: Cannot read. Remote side has closed. Tried to read 4 bytes, but only got 0 bytes. (This is often indicative of an internal error on the server side. Please check your server logs.)
>  
> (I believe here when it says “server side” it actually means client, because it is the server’s log).
>  
> Seems like the buffer that my client sends is too short.  I suspect there is a way in thrift to do it properly, but I don’t know how.
> Looks like Hector has a Composite class that maybe can help, but at this point I would really prefer to do it in Cassandra itself, without Hector.
>  
> Thanks!
>  
> Leonid
>  
>  
> This email, along with any attachments, is confidential and may be legally privileged or otherwise protected from disclosure. Any unauthorized dissemination, copying or use of the contents of this email is strictly prohibited and may be in violation of law. If you are not the intended recipient, any disclosure, copying, forwarding or distribution of this email is strictly prohibited and this email and any attachments should be deleted immediately. This email and any attachments do not constitute an offer to sell or a solicitation of an offer to purchase any interest in any investment vehicle sponsored by Moon Capital Management LP (“Moon Capital”). Moon Capital does not provide legal, accounting or tax advice. Any statement regarding legal, accounting or tax matters was not intended or written to be relied upon by any person as advice. Moon Capital does not waive confidentiality or privilege as a result of this email.
>  
>  
> This email, along with any attachments, is confidential and may be legally privileged or otherwise protected from disclosure. Any unauthorized dissemination, copying or use of the contents of this email is strictly prohibited and may be in violation of law. If you are not the intended recipient, any disclosure, copying, forwarding or distribution of this email is strictly prohibited and this email and any attachments should be deleted immediately. This email and any attachments do not constitute an offer to sell or a solicitation of an offer to purchase any interest in any investment vehicle sponsored by Moon Capital Management LP (“Moon Capital”). Moon Capital does not provide legal, accounting or tax advice. Any statement regarding legal, accounting or tax matters was not intended or written to be relied upon by any person as advice. Moon Capital does not waive confidentiality or privilege as a result of this email.
> 
> This email, along with any attachments, is confidential and may be legally privileged or otherwise protected from disclosure. Any unauthorized dissemination, copying or use of the contents of this email is strictly prohibited and may be in violation of law. If you are not the intended recipient, any disclosure, copying, forwarding or distribution of this email is strictly prohibited and this email and any attachments should be deleted immediately. This email and any attachments do not constitute an offer to sell or a solicitation of an offer to purchase any interest in any investment vehicle sponsored by Moon Capital Management LP (“Moon Capital”). Moon Capital does not provide legal, accounting or tax advice. Any statement regarding legal, accounting or tax matters was not intended or written to be relied upon by any person as advice. Moon Capital does not waive confidentiality or privilege as a result of this email.


RE: Composite key in thrift java api - still need help!!

Posted by Leonid Ilyevsky <li...@mooncapital.com>.
I tried it, does not work for my batch_mutate (I tried both CompositeType.Builder and also my own implementation following the code examples).
So it seems that the format with short length before the field and zero byte after does not work for the key.
I found by experiment and debugger that in case of a simple key, the key buffer in the  mutation_map passed to the batch_mutate contains simple key bits, without length or terminating zero.
The same thing (simple concatenation) does not work for composite key case. Short length with terminating zero does not work either.
So the question is: what is the required format of the key buffer?

From: Leonid Ilyevsky [mailto:lilyevsky@mooncapital.com]
Sent: Friday, July 06, 2012 10:05 AM
To: 'user@cassandra.apache.org'
Subject: RE: Composite key in thrift java api

Thanks Aaron,

I guess, you suggest I just borrow the logic from that CompositeType.java, the "decompose" method, to comply with the protocol.

I thought I can just call it, but I don't see the "decompose(Object... objects)" in the current 1.1.2 version, so probably your snippet is from the older version.



On the other hand, in the new version I see CompositeType.Builder inner class, looks like I should use that.

Could you please confirm?



Thanks,



Leonid


From: aaron morton [mailto:aaron@thelastpickle.com]
Sent: Thursday, July 05, 2012 7:10 PM
To: user@cassandra.apache.org
Subject: Re: Composite key in thrift java api

 I would really prefer to do it in Cassandra itself,
See https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/db/marshal/CompositeType.java

Cheers


-----------------
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com

On 6/07/2012, at 10:40 AM, Leonid Ilyevsky wrote:

I need to create a ByteBuffer instance containing the proper composite key, based on the values of the components of the key. I am going to use it for update operation.
I tried to simply concatenate the buffers corresponding to the components, but I am not sure this is correct, because I am getting exception that comes from the server :

InvalidRequestException(why:Not enough bytes to read value of component 0)

In the server log I see this:

org.apache.thrift.transport.TTransportException: Cannot read. Remote side has closed. Tried to read 4 bytes, but only got 0 bytes. (This is often indicative of an internal error on the server side. Please check your server logs.)

(I believe here when it says "server side" it actually means client, because it is the server's log).

Seems like the buffer that my client sends is too short.  I suspect there is a way in thrift to do it properly, but I don't know how.
Looks like Hector has a Composite class that maybe can help, but at this point I would really prefer to do it in Cassandra itself, without Hector.

Thanks!

Leonid


________________________________
This email, along with any attachments, is confidential and may be legally privileged or otherwise protected from disclosure. Any unauthorized dissemination, copying or use of the contents of this email is strictly prohibited and may be in violation of law. If you are not the intended recipient, any disclosure, copying, forwarding or distribution of this email is strictly prohibited and this email and any attachments should be deleted immediately. This email and any attachments do not constitute an offer to sell or a solicitation of an offer to purchase any interest in any investment vehicle sponsored by Moon Capital Management LP ("Moon Capital"). Moon Capital does not provide legal, accounting or tax advice. Any statement regarding legal, accounting or tax matters was not intended or written to be relied upon by any person as advice. Moon Capital does not waive confidentiality or privilege as a result of this email.


________________________________
This email, along with any attachments, is confidential and may be legally privileged or otherwise protected from disclosure. Any unauthorized dissemination, copying or use of the contents of this email is strictly prohibited and may be in violation of law. If you are not the intended recipient, any disclosure, copying, forwarding or distribution of this email is strictly prohibited and this email and any attachments should be deleted immediately. This email and any attachments do not constitute an offer to sell or a solicitation of an offer to purchase any interest in any investment vehicle sponsored by Moon Capital Management LP ("Moon Capital"). Moon Capital does not provide legal, accounting or tax advice. Any statement regarding legal, accounting or tax matters was not intended or written to be relied upon by any person as advice. Moon Capital does not waive confidentiality or privilege as a result of this email.

________________________________
This email, along with any attachments, is confidential and may be legally privileged or otherwise protected from disclosure. Any unauthorized dissemination, copying or use of the contents of this email is strictly prohibited and may be in violation of law. If you are not the intended recipient, any disclosure, copying, forwarding or distribution of this email is strictly prohibited and this email and any attachments should be deleted immediately. This email and any attachments do not constitute an offer to sell or a solicitation of an offer to purchase any interest in any investment vehicle sponsored by Moon Capital Management LP ("Moon Capital"). Moon Capital does not provide legal, accounting or tax advice. Any statement regarding legal, accounting or tax matters was not intended or written to be relied upon by any person as advice. Moon Capital does not waive confidentiality or privilege as a result of this email.

RE: Composite key in thrift java api

Posted by Leonid Ilyevsky <li...@mooncapital.com>.
Thanks Aaron,

I guess, you suggest I just borrow the logic from that CompositeType.java, the "decompose" method, to comply with the protocol.

I thought I can just call it, but I don't see the "decompose(Object... objects)" in the current 1.1.2 version, so probably your snippet is from the older version.



On the other hand, in the new version I see CompositeType.Builder inner class, looks like I should use that.

Could you please confirm?



Thanks,



Leonid


From: aaron morton [mailto:aaron@thelastpickle.com]
Sent: Thursday, July 05, 2012 7:10 PM
To: user@cassandra.apache.org
Subject: Re: Composite key in thrift java api

 I would really prefer to do it in Cassandra itself,
See https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/db/marshal/CompositeType.java

Cheers


-----------------
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com

On 6/07/2012, at 10:40 AM, Leonid Ilyevsky wrote:


I need to create a ByteBuffer instance containing the proper composite key, based on the values of the components of the key. I am going to use it for update operation.
I tried to simply concatenate the buffers corresponding to the components, but I am not sure this is correct, because I am getting exception that comes from the server :

InvalidRequestException(why:Not enough bytes to read value of component 0)

In the server log I see this:

org.apache.thrift.transport.TTransportException: Cannot read. Remote side has closed. Tried to read 4 bytes, but only got 0 bytes. (This is often indicative of an internal error on the server side. Please check your server logs.)

(I believe here when it says "server side" it actually means client, because it is the server's log).

Seems like the buffer that my client sends is too short.  I suspect there is a way in thrift to do it properly, but I don't know how.
Looks like Hector has a Composite class that maybe can help, but at this point I would really prefer to do it in Cassandra itself, without Hector.

Thanks!

Leonid


________________________________
This email, along with any attachments, is confidential and may be legally privileged or otherwise protected from disclosure. Any unauthorized dissemination, copying or use of the contents of this email is strictly prohibited and may be in violation of law. If you are not the intended recipient, any disclosure, copying, forwarding or distribution of this email is strictly prohibited and this email and any attachments should be deleted immediately. This email and any attachments do not constitute an offer to sell or a solicitation of an offer to purchase any interest in any investment vehicle sponsored by Moon Capital Management LP ("Moon Capital"). Moon Capital does not provide legal, accounting or tax advice. Any statement regarding legal, accounting or tax matters was not intended or written to be relied upon by any person as advice. Moon Capital does not waive confidentiality or privilege as a result of this email.


________________________________
This email, along with any attachments, is confidential and may be legally privileged or otherwise protected from disclosure. Any unauthorized dissemination, copying or use of the contents of this email is strictly prohibited and may be in violation of law. If you are not the intended recipient, any disclosure, copying, forwarding or distribution of this email is strictly prohibited and this email and any attachments should be deleted immediately. This email and any attachments do not constitute an offer to sell or a solicitation of an offer to purchase any interest in any investment vehicle sponsored by Moon Capital Management LP ("Moon Capital"). Moon Capital does not provide legal, accounting or tax advice. Any statement regarding legal, accounting or tax matters was not intended or written to be relied upon by any person as advice. Moon Capital does not waive confidentiality or privilege as a result of this email.

Re: Composite key in thrift java api

Posted by aaron morton <aa...@thelastpickle.com>.
>  I would really prefer to do it in Cassandra itself,
See https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/db/marshal/CompositeType.java

Cheers


-----------------
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com

On 6/07/2012, at 10:40 AM, Leonid Ilyevsky wrote:

> I need to create a ByteBuffer instance containing the proper composite key, based on the values of the components of the key. I am going to use it for update operation.
> I tried to simply concatenate the buffers corresponding to the components, but I am not sure this is correct, because I am getting exception that comes from the server :
>  
> InvalidRequestException(why:Not enough bytes to read value of component 0)
>  
> In the server log I see this:
>  
> org.apache.thrift.transport.TTransportException: Cannot read. Remote side has closed. Tried to read 4 bytes, but only got 0 bytes. (This is often indicative of an internal error on the server side. Please check your server logs.)
>  
> (I believe here when it says “server side” it actually means client, because it is the server’s log).
>  
> Seems like the buffer that my client sends is too short.  I suspect there is a way in thrift to do it properly, but I don’t know how.
> Looks like Hector has a Composite class that maybe can help, but at this point I would really prefer to do it in Cassandra itself, without Hector.
>  
> Thanks!
>  
> Leonid
>  
> 
> This email, along with any attachments, is confidential and may be legally privileged or otherwise protected from disclosure. Any unauthorized dissemination, copying or use of the contents of this email is strictly prohibited and may be in violation of law. If you are not the intended recipient, any disclosure, copying, forwarding or distribution of this email is strictly prohibited and this email and any attachments should be deleted immediately. This email and any attachments do not constitute an offer to sell or a solicitation of an offer to purchase any interest in any investment vehicle sponsored by Moon Capital Management LP (“Moon Capital”). Moon Capital does not provide legal, accounting or tax advice. Any statement regarding legal, accounting or tax matters was not intended or written to be relied upon by any person as advice. Moon Capital does not waive confidentiality or privilege as a result of this email.