You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Rüdiger Klaehn <rk...@gmail.com> on 2014/02/19 11:27:59 UTC
Performance problem with large wide row inserts using CQL
Hi all,
I am evaluating Cassandra for satellite telemetry storage and analysis. I
set up a little three node cluster on my local development machine and
wrote a few simple test programs.
My use case requires storing incoming telemetry updates in the database at
the same rate as they are coming in. A telemetry update is a map of
name/value pairs that arrives at a certain time.
The idea is that I want to store the data as quickly as possible, and then
later store it in an additional format that is more amenable to analysis.
The format I have chosen for my test is the following:
CREATE TABLE IF NOT EXISTS test.wide (
time varchar,
name varchar,
value varchar,
PRIMARY KEY (time,name))
WITH COMPACT STORAGE
The layout I want to achieve with this is something like this:
+-------+-------+-------+-------+-------+-------+
| | name1 | name2 | name3 | ... | nameN |
| time +-------+-------+-------+-------+-------+
| | val1 | val2 | val3 | ... | valN |
+-------+-------+-------+-------|-------+-------+
(Time will at some point be some kind of timestamp, and value will become a
blob. But this is just for initial testing)
The problem is the following: I am getting very low performance for bulk
inserts into the above table. In my test program, each insert has a new,
unique time and creates a row with 10000 name/value pairs. This should map
into creating a new row in the underlying storage engine, correct? I do
that 1000 times and measure both time per insert and total time.
I am getting about 0.5s for each insert of 10000 name/value pairs, which is
much lower than the rate at which the telemetry is coming in at my system.
I have read a few previous threads on this subject and am using batch
prepared statements for maximum performance (
https://issues.apache.org/jira/browse/CASSANDRA-4693 ). But that does not
help.
Here is the CQL benchmark:
https://gist.github.com/rklaehn/9089304#file-cassandratestminimized-scala
I have written the exact same thing using the thrift API of astyanax, and I
am getting much better performance. Each insert of 10000 name/values takes
0.04s using a ColumnListMutation. When I use async calls for both programs,
as suggested by somebody on Stackoverflow, the difference gets even larger.
The CQL insert remains at 0.5s per insert on average, whereas the
astyanax ColumnListMutation
approach takes 0.01s per insert on average, even on my test cluster. That's
the kind of performance I need.
Here is the thrift benchmark, modified from an ast example:
https://gist.github.com/rklaehn/9089304#file-astclient-java
I realize that running on a test cluster on localhost is not a 100%
realistic test. But nevertheless you would expect both tests to have
roughly similar performance.
I saw a few suggestions to create a table with CQL and fill it using the
thrift API. For example in this thread
http://mail-archives.apache.org/mod_mbox/cassandra-user/201309.mbox/%3C523334B8.8070802@gmail.com%3E.
But I would very much prefer to use pure CQL for this. It seems that
the
thrift API is considered deprecated, so I would not feel comfortable
starting a new project using a legacy API.
I already posted a question on SO about this, but did not get any
satisfactory answer. Just general performance tuning tips that do nothing
to explain the difference between the CQL and thrift approaches.
http://stackoverflow.com/questions/21778671/cassandra-how-to-insert-a-new-wide-row-with-good-performance-using-cql
Am I doing something wrong, or is this a fundamental limitation of CQL. If
the latter is the case, what's the plan to mitigate the issue?
There is a JIRA issue about this (
https://issues.apache.org/jira/browse/CASSANDRA-5959 ), but it is marked as
a duplicate of https://issues.apache.org/jira/browse/CASSANDRA-4693 . But
according to my benchmarks batch prepared statements do not solve this
issue!
I would really appreciate any help on this issue. The telemetry data I
would like to import into C* for testing contains ~2*10^12 samples, where
each sample consists of time, value and status. If quick batch insertion is
not possible, I would not even be able to insert it in an acceptable time.
best regards,
Rüdiger
Re: Performance problem with large wide row inserts using CQL
Posted by Yogi Nerella <yn...@gmail.com>.
Rüdiger,
I have tried CQL only, and it was failing after 127 records added. I have
to check what is wrong.
I have the keyspace and table definiton exactly as you.
I am new to scala, I do not know how to do this. I may try this in the
evening.
Yogi
On Wed, Feb 19, 2014 at 2:50 PM, Rüdiger Klaehn <rk...@gmail.com> wrote:
> This must be something to do with server side validation. If I define the
> table like this it does not happen:
>
> cqlsh> CREATE KEYSPACE IF NOT EXISTS test1 WITH REPLICATION = { 'class' :
> 'SimpleStrategy', 'replication_factor' : 1 };
> cqlsh> use test1;
> cqlsh:test1> create TABLE employees2 (time blob, name blob, value blob,
> PRIMARY KEY (name, value)) WITH COMPACT STORAGE;
>
> I should really update the AstClient benchmark so that it creates the
> table itself. But as I said I am not very familiar with astyanax (or with
> cassandra in general for that matter). It seems that I just happened to hit
> an issue the the first time I was playing with it...
>
> cheers,
>
> Rüdiger
>
> p.s. Did you also have a large performance difference between the Thrift
> and the CQL?
>
> On Wed, Feb 19, 2014 at 10:57 PM, Yogi Nerella <yn...@gmail.com>wrote:
>
>> I have a two node cluster. Tried with both 2.0.4 and 2.0.5.
>> I have tried your code, and exactly after inserting 127 rows, the next
>> insert fails.
>>
>> 10.566482102276002 123
>> 2.7760618708015863 124
>> 8.936212688296054 125
>> 9.532923906962095 126
>> 7.5081516753554505 127
>> java.lang.RuntimeException: failed to write data to C*
>> at demo.AstClient.insert(AstClient.java:75)
>> at demo.AstClient.loadData(AstClient.java:112)
>> at demo.AstClient.main(AstClient.java:127)
>> Caused by:
>> com.netflix.astyanax.connectionpool.exceptions.BadRequestException:
>> BadRequestException: [host=127.0.0.1(127.0.0.1):9160, latency=22(22),
>> attempts=1]InvalidRequestException(why:String didn't validate.)
>> at
>> com.netflix.astyanax.thrift.ThriftConverter.ToConnectionPoolException(ThriftConverter.java:159)
>> at
>> com.netflix.astyanax.thrift.AbstractOperationImpl.execute(AbstractOperationImpl.java:61)
>> at
>> com.netflix.astyanax.thrift.AbstractOperationImpl.execute(AbstractOperationImpl.java:28)
>> at
>> com.netflix.astyanax.thrift.ThriftSyncConnectionFactoryImpl$ThriftConnection.execute(ThriftSyncConnectionFactoryImpl.java:151)
>> at
>> com.netflix.astyanax.connectionpool.impl.AbstractExecuteWithFailoverImpl.tryOperation(AbstractExecuteWithFailoverImpl.java:69)
>> at
>> com.netflix.astyanax.connectionpool.impl.AbstractHostPartitionConnectionPool.executeWithFailover(AbstractHostPartitionConnectionPool.java:256)
>> at
>> com.netflix.astyanax.thrift.ThriftKeyspaceImpl.executeOperation(ThriftKeyspaceImpl.java:478)
>> at
>> com.netflix.astyanax.thrift.ThriftKeyspaceImpl.access$000(ThriftKeyspaceImpl.java:73)
>> at
>> com.netflix.astyanax.thrift.ThriftKeyspaceImpl$1.execute(ThriftKeyspaceImpl.java:116)
>> at demo.AstClient.insert(AstClient.java:72)
>> ... 2 more
>> Caused by: InvalidRequestException(why:String didn't validate.)
>> at
>> org.apache.cassandra.thrift.Cassandra$batch_mutate_result.read(Cassandra.java:20833)
>> at
>> org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:78)
>> at
>> org.apache.cassandra.thrift.Cassandra$Client.recv_batch_mutate(Cassandra.java:964)
>> at
>> org.apache.cassandra.thrift.Cassandra$Client.batch_mutate(Cassandra.java:950)
>> at
>> com.netflix.astyanax.thrift.ThriftKeyspaceImpl$1$1.internalExecute(ThriftKeyspaceImpl.java:122)
>> at
>> com.netflix.astyanax.thrift.ThriftKeyspaceImpl$1$1.internalExecute(ThriftKeyspaceImpl.java:119)
>> at
>> com.netflix.astyanax.thrift.AbstractOperationImpl.execute(AbstractOperationImpl.java:56)
>> ... 10 more
>>
>>
>>
>> On Wed, Feb 19, 2014 at 12:38 PM, Rüdiger Klaehn <rk...@gmail.com>wrote:
>>
>>> On Wed, Feb 19, 2014 at 7:49 PM, Sylvain Lebresne <sy...@datastax.com>wrote:
>>>
>>>> On Wed, Feb 19, 2014 at 11:27 AM, Rüdiger Klaehn <rk...@gmail.com>wrote:
>>>>
>>>>>
>>>>> Am I doing something wrong, or is this a fundamental limitation of CQL.
>>>>>
>>>>
>>>> Neither. I believe you are running into
>>>> https://issues.apache.org/jira/browse/CASSANDRA-6737, which is a bug,
>>>> a performance bug, which we should and will fix. So thanks for the report.
>>>> If you could give a shot to the patch on that issue and check if it helps,
>>>> that would definitively be much appreciated.
>>>>
>>>> Hi Sylvain,
>>>
>>> Yes, this issue looks almost identical to the problem I have been
>>> experiencing. Great to hear that you are aware of the issue and that there
>>> is a fix.
>>>
>>> I have cloned the cassandra repo, applied the patch, and built it. But
>>> when I want to run the bechmark I get an exception. See below. I tried with
>>> a non-managed dependency to
>>> cassandra-driver-core-2.0.0-rc3-SNAPSHOT-jar-with-dependencies.jar, which I
>>> compiled from source because I read that that might help. But that did not
>>> make a difference.
>>>
>>> So currently I don't know how to give the patch a try. Any ideas?
>>>
>>> cheers,
>>>
>>> Rüdiger
>>>
>>> Exception in thread "main" java.lang.IllegalArgumentException:
>>> replicate_on_write is not a column defined in this metadata
>>> at
>>> com.datastax.driver.core.ColumnDefinitions.getAllIdx(ColumnDefinitions.java:273)
>>> at
>>> com.datastax.driver.core.ColumnDefinitions.getFirstIdx(ColumnDefinitions.java:279)
>>> at com.datastax.driver.core.Row.getBool(Row.java:117)
>>> at
>>> com.datastax.driver.core.TableMetadata$Options.<init>(TableMetadata.java:474)
>>> at
>>> com.datastax.driver.core.TableMetadata.build(TableMetadata.java:107)
>>> at
>>> com.datastax.driver.core.Metadata.buildTableMetadata(Metadata.java:128)
>>> at com.datastax.driver.core.Metadata.rebuildSchema(Metadata.java:89)
>>> at
>>> com.datastax.driver.core.ControlConnection.refreshSchema(ControlConnection.java:259)
>>> at
>>> com.datastax.driver.core.ControlConnection.tryConnect(ControlConnection.java:214)
>>> at
>>> com.datastax.driver.core.ControlConnection.reconnectInternal(ControlConnection.java:161)
>>> at
>>> com.datastax.driver.core.ControlConnection.connect(ControlConnection.java:77)
>>> at com.datastax.driver.core.Cluster$Manager.init(Cluster.java:890)
>>> at
>>> com.datastax.driver.core.Cluster$Manager.newSession(Cluster.java:910)
>>> at
>>> com.datastax.driver.core.Cluster$Manager.access$200(Cluster.java:806)
>>> at com.datastax.driver.core.Cluster.connect(Cluster.java:158)
>>> at
>>> cassandra.CassandraTestMinimized$delayedInit$body.apply(CassandraTestMinimized.scala:31)
>>> at scala.Function0$class.apply$mcV$sp(Function0.scala:40)
>>> at
>>> scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:12)
>>> at scala.App$$anonfun$main$1.apply(App.scala:71)
>>> at scala.App$$anonfun$main$1.apply(App.scala:71)
>>> at scala.collection.immutable.List.foreach(List.scala:318)
>>> at
>>> scala.collection.generic.TraversableForwarder$class.foreach(TraversableForwarder.scala:32)
>>> at scala.App$class.main(App.scala:71)
>>> at
>>> cassandra.CassandraTestMinimized$.main(CassandraTestMinimized.scala:5)
>>> at
>>> cassandra.CassandraTestMinimized.main(CassandraTestMinimized.scala)
>>>
>>>
>>>> There is absolutely no fundamental reason why a CQL operation would
>>>> be more than 10 times slower than it's thrift equivalent, such dramatic
>>>> difference is indicative of a bug, something obviously wrong.
>>>>
>>>> --
>>>> Sylvain
>>>>
>>>
>>>
>>
>
Re: Performance problem with large wide row inserts using CQL
Posted by Rüdiger Klaehn <rk...@gmail.com>.
This must be something to do with server side validation. If I define the
table like this it does not happen:
cqlsh> CREATE KEYSPACE IF NOT EXISTS test1 WITH REPLICATION = { 'class' :
'SimpleStrategy', 'replication_factor' : 1 };
cqlsh> use test1;
cqlsh:test1> create TABLE employees2 (time blob, name blob, value blob,
PRIMARY KEY (name, value)) WITH COMPACT STORAGE;
I should really update the AstClient benchmark so that it creates the table
itself. But as I said I am not very familiar with astyanax (or with
cassandra in general for that matter). It seems that I just happened to hit
an issue the the first time I was playing with it...
cheers,
Rüdiger
p.s. Did you also have a large performance difference between the Thrift
and the CQL?
On Wed, Feb 19, 2014 at 10:57 PM, Yogi Nerella <yn...@gmail.com>wrote:
> I have a two node cluster. Tried with both 2.0.4 and 2.0.5.
> I have tried your code, and exactly after inserting 127 rows, the next
> insert fails.
>
> 10.566482102276002 123
> 2.7760618708015863 124
> 8.936212688296054 125
> 9.532923906962095 126
> 7.5081516753554505 127
> java.lang.RuntimeException: failed to write data to C*
> at demo.AstClient.insert(AstClient.java:75)
> at demo.AstClient.loadData(AstClient.java:112)
> at demo.AstClient.main(AstClient.java:127)
> Caused by:
> com.netflix.astyanax.connectionpool.exceptions.BadRequestException:
> BadRequestException: [host=127.0.0.1(127.0.0.1):9160, latency=22(22),
> attempts=1]InvalidRequestException(why:String didn't validate.)
> at
> com.netflix.astyanax.thrift.ThriftConverter.ToConnectionPoolException(ThriftConverter.java:159)
> at
> com.netflix.astyanax.thrift.AbstractOperationImpl.execute(AbstractOperationImpl.java:61)
> at
> com.netflix.astyanax.thrift.AbstractOperationImpl.execute(AbstractOperationImpl.java:28)
> at
> com.netflix.astyanax.thrift.ThriftSyncConnectionFactoryImpl$ThriftConnection.execute(ThriftSyncConnectionFactoryImpl.java:151)
> at
> com.netflix.astyanax.connectionpool.impl.AbstractExecuteWithFailoverImpl.tryOperation(AbstractExecuteWithFailoverImpl.java:69)
> at
> com.netflix.astyanax.connectionpool.impl.AbstractHostPartitionConnectionPool.executeWithFailover(AbstractHostPartitionConnectionPool.java:256)
> at
> com.netflix.astyanax.thrift.ThriftKeyspaceImpl.executeOperation(ThriftKeyspaceImpl.java:478)
> at
> com.netflix.astyanax.thrift.ThriftKeyspaceImpl.access$000(ThriftKeyspaceImpl.java:73)
> at
> com.netflix.astyanax.thrift.ThriftKeyspaceImpl$1.execute(ThriftKeyspaceImpl.java:116)
> at demo.AstClient.insert(AstClient.java:72)
> ... 2 more
> Caused by: InvalidRequestException(why:String didn't validate.)
> at
> org.apache.cassandra.thrift.Cassandra$batch_mutate_result.read(Cassandra.java:20833)
> at
> org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:78)
> at
> org.apache.cassandra.thrift.Cassandra$Client.recv_batch_mutate(Cassandra.java:964)
> at
> org.apache.cassandra.thrift.Cassandra$Client.batch_mutate(Cassandra.java:950)
> at
> com.netflix.astyanax.thrift.ThriftKeyspaceImpl$1$1.internalExecute(ThriftKeyspaceImpl.java:122)
> at
> com.netflix.astyanax.thrift.ThriftKeyspaceImpl$1$1.internalExecute(ThriftKeyspaceImpl.java:119)
> at
> com.netflix.astyanax.thrift.AbstractOperationImpl.execute(AbstractOperationImpl.java:56)
> ... 10 more
>
>
>
> On Wed, Feb 19, 2014 at 12:38 PM, Rüdiger Klaehn <rk...@gmail.com>wrote:
>
>> On Wed, Feb 19, 2014 at 7:49 PM, Sylvain Lebresne <sy...@datastax.com>wrote:
>>
>>> On Wed, Feb 19, 2014 at 11:27 AM, Rüdiger Klaehn <rk...@gmail.com>wrote:
>>>
>>>>
>>>> Am I doing something wrong, or is this a fundamental limitation of CQL.
>>>>
>>>
>>> Neither. I believe you are running into
>>> https://issues.apache.org/jira/browse/CASSANDRA-6737, which is a bug, a
>>> performance bug, which we should and will fix. So thanks for the report. If
>>> you could give a shot to the patch on that issue and check if it helps,
>>> that would definitively be much appreciated.
>>>
>>> Hi Sylvain,
>>
>> Yes, this issue looks almost identical to the problem I have been
>> experiencing. Great to hear that you are aware of the issue and that there
>> is a fix.
>>
>> I have cloned the cassandra repo, applied the patch, and built it. But
>> when I want to run the bechmark I get an exception. See below. I tried with
>> a non-managed dependency to
>> cassandra-driver-core-2.0.0-rc3-SNAPSHOT-jar-with-dependencies.jar, which I
>> compiled from source because I read that that might help. But that did not
>> make a difference.
>>
>> So currently I don't know how to give the patch a try. Any ideas?
>>
>> cheers,
>>
>> Rüdiger
>>
>> Exception in thread "main" java.lang.IllegalArgumentException:
>> replicate_on_write is not a column defined in this metadata
>> at
>> com.datastax.driver.core.ColumnDefinitions.getAllIdx(ColumnDefinitions.java:273)
>> at
>> com.datastax.driver.core.ColumnDefinitions.getFirstIdx(ColumnDefinitions.java:279)
>> at com.datastax.driver.core.Row.getBool(Row.java:117)
>> at
>> com.datastax.driver.core.TableMetadata$Options.<init>(TableMetadata.java:474)
>> at
>> com.datastax.driver.core.TableMetadata.build(TableMetadata.java:107)
>> at
>> com.datastax.driver.core.Metadata.buildTableMetadata(Metadata.java:128)
>> at com.datastax.driver.core.Metadata.rebuildSchema(Metadata.java:89)
>> at
>> com.datastax.driver.core.ControlConnection.refreshSchema(ControlConnection.java:259)
>> at
>> com.datastax.driver.core.ControlConnection.tryConnect(ControlConnection.java:214)
>> at
>> com.datastax.driver.core.ControlConnection.reconnectInternal(ControlConnection.java:161)
>> at
>> com.datastax.driver.core.ControlConnection.connect(ControlConnection.java:77)
>> at com.datastax.driver.core.Cluster$Manager.init(Cluster.java:890)
>> at
>> com.datastax.driver.core.Cluster$Manager.newSession(Cluster.java:910)
>> at
>> com.datastax.driver.core.Cluster$Manager.access$200(Cluster.java:806)
>> at com.datastax.driver.core.Cluster.connect(Cluster.java:158)
>> at
>> cassandra.CassandraTestMinimized$delayedInit$body.apply(CassandraTestMinimized.scala:31)
>> at scala.Function0$class.apply$mcV$sp(Function0.scala:40)
>> at
>> scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:12)
>> at scala.App$$anonfun$main$1.apply(App.scala:71)
>> at scala.App$$anonfun$main$1.apply(App.scala:71)
>> at scala.collection.immutable.List.foreach(List.scala:318)
>> at
>> scala.collection.generic.TraversableForwarder$class.foreach(TraversableForwarder.scala:32)
>> at scala.App$class.main(App.scala:71)
>> at
>> cassandra.CassandraTestMinimized$.main(CassandraTestMinimized.scala:5)
>> at cassandra.CassandraTestMinimized.main(CassandraTestMinimized.scala)
>>
>>
>>> There is absolutely no fundamental reason why a CQL operation would be
>>> more than 10 times slower than it's thrift equivalent, such dramatic
>>> difference is indicative of a bug, something obviously wrong.
>>>
>>> --
>>> Sylvain
>>>
>>
>>
>
Re: Performance problem with large wide row inserts using CQL
Posted by Yogi Nerella <yn...@gmail.com>.
I have a two node cluster. Tried with both 2.0.4 and 2.0.5.
I have tried your code, and exactly after inserting 127 rows, the next
insert fails.
10.566482102276002 123
2.7760618708015863 124
8.936212688296054 125
9.532923906962095 126
7.5081516753554505 127
java.lang.RuntimeException: failed to write data to C*
at demo.AstClient.insert(AstClient.java:75)
at demo.AstClient.loadData(AstClient.java:112)
at demo.AstClient.main(AstClient.java:127)
Caused by:
com.netflix.astyanax.connectionpool.exceptions.BadRequestException:
BadRequestException: [host=127.0.0.1(127.0.0.1):9160, latency=22(22),
attempts=1]InvalidRequestException(why:String didn't validate.)
at
com.netflix.astyanax.thrift.ThriftConverter.ToConnectionPoolException(ThriftConverter.java:159)
at
com.netflix.astyanax.thrift.AbstractOperationImpl.execute(AbstractOperationImpl.java:61)
at
com.netflix.astyanax.thrift.AbstractOperationImpl.execute(AbstractOperationImpl.java:28)
at
com.netflix.astyanax.thrift.ThriftSyncConnectionFactoryImpl$ThriftConnection.execute(ThriftSyncConnectionFactoryImpl.java:151)
at
com.netflix.astyanax.connectionpool.impl.AbstractExecuteWithFailoverImpl.tryOperation(AbstractExecuteWithFailoverImpl.java:69)
at
com.netflix.astyanax.connectionpool.impl.AbstractHostPartitionConnectionPool.executeWithFailover(AbstractHostPartitionConnectionPool.java:256)
at
com.netflix.astyanax.thrift.ThriftKeyspaceImpl.executeOperation(ThriftKeyspaceImpl.java:478)
at
com.netflix.astyanax.thrift.ThriftKeyspaceImpl.access$000(ThriftKeyspaceImpl.java:73)
at
com.netflix.astyanax.thrift.ThriftKeyspaceImpl$1.execute(ThriftKeyspaceImpl.java:116)
at demo.AstClient.insert(AstClient.java:72)
... 2 more
Caused by: InvalidRequestException(why:String didn't validate.)
at
org.apache.cassandra.thrift.Cassandra$batch_mutate_result.read(Cassandra.java:20833)
at
org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:78)
at
org.apache.cassandra.thrift.Cassandra$Client.recv_batch_mutate(Cassandra.java:964)
at
org.apache.cassandra.thrift.Cassandra$Client.batch_mutate(Cassandra.java:950)
at
com.netflix.astyanax.thrift.ThriftKeyspaceImpl$1$1.internalExecute(ThriftKeyspaceImpl.java:122)
at
com.netflix.astyanax.thrift.ThriftKeyspaceImpl$1$1.internalExecute(ThriftKeyspaceImpl.java:119)
at
com.netflix.astyanax.thrift.AbstractOperationImpl.execute(AbstractOperationImpl.java:56)
... 10 more
On Wed, Feb 19, 2014 at 12:38 PM, Rüdiger Klaehn <rk...@gmail.com> wrote:
> On Wed, Feb 19, 2014 at 7:49 PM, Sylvain Lebresne <sy...@datastax.com>wrote:
>
>> On Wed, Feb 19, 2014 at 11:27 AM, Rüdiger Klaehn <rk...@gmail.com>wrote:
>>
>>>
>>> Am I doing something wrong, or is this a fundamental limitation of CQL.
>>>
>>
>> Neither. I believe you are running into
>> https://issues.apache.org/jira/browse/CASSANDRA-6737, which is a bug, a
>> performance bug, which we should and will fix. So thanks for the report. If
>> you could give a shot to the patch on that issue and check if it helps,
>> that would definitively be much appreciated.
>>
>> Hi Sylvain,
>
> Yes, this issue looks almost identical to the problem I have been
> experiencing. Great to hear that you are aware of the issue and that there
> is a fix.
>
> I have cloned the cassandra repo, applied the patch, and built it. But
> when I want to run the bechmark I get an exception. See below. I tried with
> a non-managed dependency to
> cassandra-driver-core-2.0.0-rc3-SNAPSHOT-jar-with-dependencies.jar, which I
> compiled from source because I read that that might help. But that did not
> make a difference.
>
> So currently I don't know how to give the patch a try. Any ideas?
>
> cheers,
>
> Rüdiger
>
> Exception in thread "main" java.lang.IllegalArgumentException:
> replicate_on_write is not a column defined in this metadata
> at
> com.datastax.driver.core.ColumnDefinitions.getAllIdx(ColumnDefinitions.java:273)
> at
> com.datastax.driver.core.ColumnDefinitions.getFirstIdx(ColumnDefinitions.java:279)
> at com.datastax.driver.core.Row.getBool(Row.java:117)
> at
> com.datastax.driver.core.TableMetadata$Options.<init>(TableMetadata.java:474)
> at com.datastax.driver.core.TableMetadata.build(TableMetadata.java:107)
> at
> com.datastax.driver.core.Metadata.buildTableMetadata(Metadata.java:128)
> at com.datastax.driver.core.Metadata.rebuildSchema(Metadata.java:89)
> at
> com.datastax.driver.core.ControlConnection.refreshSchema(ControlConnection.java:259)
> at
> com.datastax.driver.core.ControlConnection.tryConnect(ControlConnection.java:214)
> at
> com.datastax.driver.core.ControlConnection.reconnectInternal(ControlConnection.java:161)
> at
> com.datastax.driver.core.ControlConnection.connect(ControlConnection.java:77)
> at com.datastax.driver.core.Cluster$Manager.init(Cluster.java:890)
> at
> com.datastax.driver.core.Cluster$Manager.newSession(Cluster.java:910)
> at
> com.datastax.driver.core.Cluster$Manager.access$200(Cluster.java:806)
> at com.datastax.driver.core.Cluster.connect(Cluster.java:158)
> at
> cassandra.CassandraTestMinimized$delayedInit$body.apply(CassandraTestMinimized.scala:31)
> at scala.Function0$class.apply$mcV$sp(Function0.scala:40)
> at
> scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:12)
> at scala.App$$anonfun$main$1.apply(App.scala:71)
> at scala.App$$anonfun$main$1.apply(App.scala:71)
> at scala.collection.immutable.List.foreach(List.scala:318)
> at
> scala.collection.generic.TraversableForwarder$class.foreach(TraversableForwarder.scala:32)
> at scala.App$class.main(App.scala:71)
> at
> cassandra.CassandraTestMinimized$.main(CassandraTestMinimized.scala:5)
> at cassandra.CassandraTestMinimized.main(CassandraTestMinimized.scala)
>
>
>> There is absolutely no fundamental reason why a CQL operation would be
>> more than 10 times slower than it's thrift equivalent, such dramatic
>> difference is indicative of a bug, something obviously wrong.
>>
>> --
>> Sylvain
>>
>
>
Re: Performance problem with large wide row inserts using CQL
Posted by Mohit Anchlia <mo...@gmail.com>.
+1
I like hector client that uses thrift interface and exposes APIs that is
similar to how Cassandra physically stores the values.
On Thu, Feb 20, 2014 at 9:26 AM, Peter Lin <wo...@gmail.com> wrote:
>
> I disagree with the sentiment that "thrift is not worth the trouble".
>
> CQL and all SQL inspired dialects limit one's ability to use arbitrary
> typed data in dynamic columns. With thrift it's easy and straight forward.
> With CQL there is no way to tell Cassandra the type of the name and value
> for a dynamic column. You can only set the default type. That means using a
> "pure cql" approach you can deviate from the default type. Cassandra will
> throw an exception indicating the type is different than the default type.
>
> Until such time that CQL abandons the shackles of SQL and adds the ability
> to indicate the column and value type. Something like this
>
> insert into myColumnFamily(staticColumn1, staticColumn2, 20 as int,
> dynamicColumn as string) into ('text1','text2',30.55 as double, 3500 as
> long)
>
> This is one area where Thrift is superior to CQL. Having said that, it's
> valid to use Cassandra "as if" it was a relational database, but then you'd
> miss out on some of the unique features.
>
>
>
>
> On Thu, Feb 20, 2014 at 12:12 PM, Sylvain Lebresne <sy...@datastax.com>wrote:
>
>> On Thu, Feb 20, 2014 at 2:16 PM, Edward Capriolo <ed...@gmail.com>wrote:
>>
>>> For what it is worth you schema is simple and uses compact storage. Thus
>>> you really dont need anything in cassandra 2.0 as far as i can tell. You
>>> might be happier with a stable release like 1.2.something and just hector
>>> or astyanax. You are really dealing with many issues you should not have to
>>> just to protoype a simple cassandra app.
>>
>>
>>
>> Of course, if everyone was using that reasoning, no-one would ever test
>> new features and report problems/suggest improvement. So thanks to anyone
>> like Rüdiger that actually tries stuff and take the time to report problems
>> when they think they encounter one. Keep at it, *you* are the one helping
>> Cassandra to get better everyday.
>>
>> And you are also right Rüdiger that it's probably not worth the trouble
>> to start with thrift if you're gonna use CQL later. And you definitively
>> should use CQL, it is Cassandra's future.
>>
>> --
>> Sylvain
>>
>>
>>
>>>
>>> On Thursday, February 20, 2014, Sylvain Lebresne <sy...@datastax.com>
>>> wrote:
>>> >
>>> >
>>> >
>>> > On Wed, Feb 19, 2014 at 9:38 PM, Rüdiger Klaehn <rk...@gmail.com>
>>> wrote:
>>> >>
>>> >> I have cloned the cassandra repo, applied the patch, and built it.
>>> But when I want to run the bechmark I get an exception. See below. I tried
>>> with a non-managed dependency to
>>> cassandra-driver-core-2.0.0-rc3-SNAPSHOT-jar-with-dependencies.jar, which I
>>> compiled from source because I read that that might help. But that did not
>>> make a difference.
>>> >>
>>> >> So currently I don't know how to give the patch a try. Any ideas?
>>> >>
>>> >> cheers,
>>> >>
>>> >> Rüdiger
>>> >>
>>> >> Exception in thread "main" java.lang.IllegalArgumentException:
>>> replicate_on_write is not a column defined in this metadata
>>> >> at
>>> com.datastax.driver.core.ColumnDefinitions.getAllIdx(ColumnDefinitions.java:273)
>>> >> at
>>> com.datastax.driver.core.ColumnDefinitions.getFirstIdx(ColumnDefinitions.java:279)
>>> >> at com.datastax.driver.core.Row.getBool(Row.java:117)
>>> >> at
>>> com.datastax.driver.core.TableMetadata$Options.<init>(TableMetadata.java:474)
>>> >> at
>>> com.datastax.driver.core.TableMetadata.build(TableMetadata.java:107)
>>> >> at
>>> com.datastax.driver.core.Metadata.buildTableMetadata(Metadata.java:128)
>>> >> at
>>> com.datastax.driver.core.Metadata.rebuildSchema(Metadata.java:89)
>>> >> at
>>> com.datastax.driver.core.ControlConnection.refreshSchema(ControlConnection.java:259)
>>> >> at
>>> com.datastax.driver.core.ControlConnection.tryConnect(ControlConnection.java:214)
>>> >> at
>>> com.datastax.driver.core.ControlConnection.reconnectInternal(ControlConnection.java:161)
>>> >> at
>>> com.datastax.driver.core.ControlConnection.connect(ControlConnection.java:77)
>>> >> at com.datastax.driver.core.Cluster$Manager.init(Cluster.java:890)
>>> >> at
>>> com.datastax.driver.core.Cluster$Manager.newSession(Cluster.java:910)
>>> >> at
>>> com.datastax.driver.core.Cluster$Manager.access$200(Cluster.java:806)
>>> >> at com.datastax.driver.core.Cluster.connect(Cluster.java:158)
>>> >> at
>>> cassandra.CassandraTestMinimized$delayedInit$body.apply(CassandraTestMinimized.scala:31)
>>> >> at scala.Function0$class.apply$mcV$sp(Function0.scala:40)
>>> >> at
>>> scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:12)
>>> >> at scala.App$$anonfun$main$1.apply(App.scala:71)
>>> >> at scala.App$$anonfun$main$1.apply(App.scala:71)
>>> >> at scala.collection.immutable.List.foreach(List.scala:318)
>>> >> at
>>> scala.collection.generic.TraversableForwarder$class.foreach(TraversableForwarder.scala:32)
>>> >> at scala.App$class.main(App.scala:71)
>>> >> at
>>> cassandra.CassandraTestMinimized$.main(CassandraTestMinimized.scala:5)
>>> >> at
>>> cassandra.CassandraTestMinimized.main(CassandraTestMinimized.scala)
>>> >
>>> > I believe you've tried the cassandra trunk branch? trunk is basically
>>> the future Cassandra 2.1 and the driver is currently unhappy because the
>>> replicate_on_write option has been removed in that version. I'm supposed to
>>> have fixed that on the driver 2.0 branch like 2 days ago so maybe you're
>>> also using a slightly old version of the driver sources in there? Or maybe
>>> I've screwed up my fix, I'll double check. But anyway, it would be overall
>>> simpler to test with the cassandra-2.0 branch of Cassandra, with which you
>>> shouldn't run into that.
>>> > --
>>> > Sylvain
>>>
>>> --
>>> Sorry this was sent from mobile. Will do less grammar and spell check
>>> than usual.
>>>
>>
>>
>
Re: Performance problem with large wide row inserts using CQL
Posted by Benedict Elliott Smith <be...@datastax.com>.
>
> Cassandra will throw an exception indicating the type is different than
> the default type.
If you want untyped data, store blobs. Or store in a different column
(they're free when empty, after all). Type safety is considered a good
thing by many.
On 20 February 2014 17:26, Peter Lin <wo...@gmail.com> wrote:
>
> I disagree with the sentiment that "thrift is not worth the trouble".
>
> CQL and all SQL inspired dialects limit one's ability to use arbitrary
> typed data in dynamic columns. With thrift it's easy and straight forward.
> With CQL there is no way to tell Cassandra the type of the name and value
> for a dynamic column. You can only set the default type. That means using a
> "pure cql" approach you can deviate from the default type. Cassandra will
> throw an exception indicating the type is different than the default type.
>
> Until such time that CQL abandons the shackles of SQL and adds the ability
> to indicate the column and value type. Something like this
>
> insert into myColumnFamily(staticColumn1, staticColumn2, 20 as int,
> dynamicColumn as string) into ('text1','text2',30.55 as double, 3500 as
> long)
>
> This is one area where Thrift is superior to CQL. Having said that, it's
> valid to use Cassandra "as if" it was a relational database, but then you'd
> miss out on some of the unique features.
>
>
>
>
> On Thu, Feb 20, 2014 at 12:12 PM, Sylvain Lebresne <sy...@datastax.com>wrote:
>
>> On Thu, Feb 20, 2014 at 2:16 PM, Edward Capriolo <ed...@gmail.com>wrote:
>>
>>> For what it is worth you schema is simple and uses compact storage. Thus
>>> you really dont need anything in cassandra 2.0 as far as i can tell. You
>>> might be happier with a stable release like 1.2.something and just hector
>>> or astyanax. You are really dealing with many issues you should not have to
>>> just to protoype a simple cassandra app.
>>
>>
>>
>> Of course, if everyone was using that reasoning, no-one would ever test
>> new features and report problems/suggest improvement. So thanks to anyone
>> like Rüdiger that actually tries stuff and take the time to report problems
>> when they think they encounter one. Keep at it, *you* are the one helping
>> Cassandra to get better everyday.
>>
>> And you are also right Rüdiger that it's probably not worth the trouble
>> to start with thrift if you're gonna use CQL later. And you definitively
>> should use CQL, it is Cassandra's future.
>>
>> --
>> Sylvain
>>
>>
>>
>>>
>>> On Thursday, February 20, 2014, Sylvain Lebresne <sy...@datastax.com>
>>> wrote:
>>> >
>>> >
>>> >
>>> > On Wed, Feb 19, 2014 at 9:38 PM, Rüdiger Klaehn <rk...@gmail.com>
>>> wrote:
>>> >>
>>> >> I have cloned the cassandra repo, applied the patch, and built it.
>>> But when I want to run the bechmark I get an exception. See below. I tried
>>> with a non-managed dependency to
>>> cassandra-driver-core-2.0.0-rc3-SNAPSHOT-jar-with-dependencies.jar, which I
>>> compiled from source because I read that that might help. But that did not
>>> make a difference.
>>> >>
>>> >> So currently I don't know how to give the patch a try. Any ideas?
>>> >>
>>> >> cheers,
>>> >>
>>> >> Rüdiger
>>> >>
>>> >> Exception in thread "main" java.lang.IllegalArgumentException:
>>> replicate_on_write is not a column defined in this metadata
>>> >> at
>>> com.datastax.driver.core.ColumnDefinitions.getAllIdx(ColumnDefinitions.java:273)
>>> >> at
>>> com.datastax.driver.core.ColumnDefinitions.getFirstIdx(ColumnDefinitions.java:279)
>>> >> at com.datastax.driver.core.Row.getBool(Row.java:117)
>>> >> at
>>> com.datastax.driver.core.TableMetadata$Options.<init>(TableMetadata.java:474)
>>> >> at
>>> com.datastax.driver.core.TableMetadata.build(TableMetadata.java:107)
>>> >> at
>>> com.datastax.driver.core.Metadata.buildTableMetadata(Metadata.java:128)
>>> >> at
>>> com.datastax.driver.core.Metadata.rebuildSchema(Metadata.java:89)
>>> >> at
>>> com.datastax.driver.core.ControlConnection.refreshSchema(ControlConnection.java:259)
>>> >> at
>>> com.datastax.driver.core.ControlConnection.tryConnect(ControlConnection.java:214)
>>> >> at
>>> com.datastax.driver.core.ControlConnection.reconnectInternal(ControlConnection.java:161)
>>> >> at
>>> com.datastax.driver.core.ControlConnection.connect(ControlConnection.java:77)
>>> >> at com.datastax.driver.core.Cluster$Manager.init(Cluster.java:890)
>>> >> at
>>> com.datastax.driver.core.Cluster$Manager.newSession(Cluster.java:910)
>>> >> at
>>> com.datastax.driver.core.Cluster$Manager.access$200(Cluster.java:806)
>>> >> at com.datastax.driver.core.Cluster.connect(Cluster.java:158)
>>> >> at
>>> cassandra.CassandraTestMinimized$delayedInit$body.apply(CassandraTestMinimized.scala:31)
>>> >> at scala.Function0$class.apply$mcV$sp(Function0.scala:40)
>>> >> at
>>> scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:12)
>>> >> at scala.App$$anonfun$main$1.apply(App.scala:71)
>>> >> at scala.App$$anonfun$main$1.apply(App.scala:71)
>>> >> at scala.collection.immutable.List.foreach(List.scala:318)
>>> >> at
>>> scala.collection.generic.TraversableForwarder$class.foreach(TraversableForwarder.scala:32)
>>> >> at scala.App$class.main(App.scala:71)
>>> >> at
>>> cassandra.CassandraTestMinimized$.main(CassandraTestMinimized.scala:5)
>>> >> at
>>> cassandra.CassandraTestMinimized.main(CassandraTestMinimized.scala)
>>> >
>>> > I believe you've tried the cassandra trunk branch? trunk is basically
>>> the future Cassandra 2.1 and the driver is currently unhappy because the
>>> replicate_on_write option has been removed in that version. I'm supposed to
>>> have fixed that on the driver 2.0 branch like 2 days ago so maybe you're
>>> also using a slightly old version of the driver sources in there? Or maybe
>>> I've screwed up my fix, I'll double check. But anyway, it would be overall
>>> simpler to test with the cassandra-2.0 branch of Cassandra, with which you
>>> shouldn't run into that.
>>> > --
>>> > Sylvain
>>>
>>> --
>>> Sorry this was sent from mobile. Will do less grammar and spell check
>>> than usual.
>>>
>>
>>
>
Re: Performance problem with large wide row inserts using CQL
Posted by Peter Lin <wo...@gmail.com>.
Hi Ed,
you're definitely not mad. I've seen this all over the place. We have
several large retail customers and they all suffer the EAV horror. Having
built EAV horrors in the past and guilty of inflicting that pain on people,
mixing static and dynamic is "Totally Freaking awesome!"
I know many large shops buy mainframes just to make EAV queries fast. Big
retail shops can fork over millions for a big box, but everyone else
"probably" shouldn't. I'm totally biased, to me the ability to use both in
a single columnFamily is the gem of Cassandra. Without it, we're stuck
using old techniques that create a painful nightmare. I've had to fix old
systems using EAV and it's so painful to just figure out what properties a
damn record.
On Thu, Feb 20, 2014 at 4:03 PM, Edward Capriolo <ed...@gmail.com>wrote:
> Peter,
>
> I must meet you and shake your hand. I was actually having a debate with a
> number of people about a week back claiming there was "no reason to mix
> static and dynamic". We do it all the time I am glad someone else besides
> me "gets it" and I am not totally mad.
>
> Ed
>
>
>
> On Thu, Feb 20, 2014 at 3:26 PM, Peter Lin <wo...@gmail.com> wrote:
>
>>
>> Hi Duyhai,
>>
>> yes, I am talking about mixing static and dynamic columns in a single
>> column family. Let me give you an example from retail.
>>
>> Say you're amazon and you sell over 10K different products. How do you
>> store all those products with all the different properties like color,
>> size, dimensions, etc. With relational databases people use EAV (entity
>> attribute value) tables. This means querying for data the system has to
>> reconstruct the object by pivot a bunch of rows and flattening it out to
>> populate the java object. Typically there are common fields to a product
>> like SKU, price, and category.
>>
>> Using both static and dynamic columns, data can be stored in 1 row and
>> queried by 1 row. Anyone that has used EAV approach to build product
>> databases will tell you how much that sucks. Another example is from auto
>> insurance. Typically a policy database will allow 1 or more types of items
>> for property insurance. Property insurance is home/auto insurance.
>>
>> Each insurance carrier supports different number of insurable items,
>> coverages and endorsements. Many systems use the same EAV approach, but the
>> problem is bigger. Typically a commercial auto policy may have hundreds of
>> drivers and vehicles. Each policy may have dozens or hundreds of coverages
>> and endorsements. It is common for an auto insurance model to have hundreds
>> of coverage and endorsements with different properties. Using the old ORM
>> approach, it's usually mapped table-per-class. Problem is, that results in
>> query explosion for polymorphic queries. This is a known problem with
>> polymorphic queries using traditional techniques.
>>
>> Given that Cassandra + thrift gives developers the ability to store
>> dynamic columns of different types, it solves the performance issues
>> inherent in EAV technique.
>>
>> The point I was trying to make in my first response is that going with
>> pure CQL makes it much harder to take advantage of the COOL features of
>> Cassandra. It does require building a framework to make it "mostly"
>> transparent to developers, but it is worth it in my opinion to learn and
>> understand both thrift and cql. I use annotations in my framework and
>> delegates to handle the serialization. This way, the developer only needs
>> annotate the class and the framework handles serialization and
>> deserialization.
>>
>>
>>
>>
>>
>> On Thu, Feb 20, 2014 at 3:05 PM, DuyHai Doan <do...@gmail.com>wrote:
>>
>>> "Developers can use what ever type they want for the name or value in a
>>> dynamic column and the framework will handle it appropriately."
>>>
>>> What do you mean by "dynamic" column ? If you want to be able to insert
>>> an arbitrary number of columns in one physical row, CQL3 clustering is
>>> there and does pretty well the job.
>>>
>>> If by "dynamic" you mean a column whose validation type can change at
>>> runtime (like the dynamic composite type :
>>> http://hector-client.github.io/hector/build/html/content/composite_with_templates.html)
>>> then why don't you just use blob type and serialize it yourself at client
>>> side ?
>>>
>>> More pratically, in your previous example :
>>>
>>> - insert into myColumnFamily(staticColumn1, staticColumn2, 20 as int,
>>> dynamicColumn as string) into ('text1','text2',30.55 as double, 3500 as
>>> long)
>>>
>>> I can't see real sensible use-case where you need to mix static and
>>> dynamic columns in the same column family. If you need to save domain
>>> model, use skinny row with a fixed number of columns known before hand. If
>>> you want to store time series or timeline of data, wide row is there.
>>>
>>>
>>> On Thu, Feb 20, 2014 at 8:55 PM, Peter Lin <wo...@gmail.com> wrote:
>>>
>>>>
>>>> my apologies Sylvain, I didn't mean to misquote you. I still feel that
>>>> even if someone is only going to use CQL, it is "worth it" to learn thrift.
>>>>
>>>> In the interest of discussion, I looked at both jira tickets and I
>>>> don't see how that makes it so a developer can specify the name and value
>>>> type for a dynamic column.
>>>>
>>>> https://issues.apache.org/jira/browse/CASSANDRA-6561
>>>> https://issues.apache.org/jira/browse/CASSANDRA-4851
>>>>
>>>> Am I missing something? If the grammar for insert statements doesn't
>>>> give users the ability declare the name and value type, it means the
>>>> developer has to default name and value to bytes. In their code, they have
>>>> to handle that manually or build their own framework. I built my own
>>>> framework, which handles this for me. Developers can use what ever type
>>>> they want for the name or value in a dynamic column and the framework will
>>>> handle it appropriately.
>>>>
>>>> To me, developers should take time to learn both and use both. I
>>>> realize it's more work to understand both and take time to read the code.
>>>> Not everyone is crazy enough spend time reading cassandra code base or
>>>> spend hundreds of hours studying hector and other cassandra clients. I will
>>>> say this, if I hadn't spend time studying cassandra and reading Hector
>>>> code, I wouldn't have been able to help one of DataStax customer port
>>>> Hector to .Net. I also wouldn't have been able to port Hector to C#
>>>> natively in 3 months.
>>>>
>>>> Rather than recommend people be lazy, it would be more useful to list
>>>> the pros/cons. To my knowledge, there isn't a good writeup on the pros/cons
>>>> of thrift and cql on cassandra.apache.org. I don't know if the
>>>> DataStax docs have a detailed write up of it, does it?
>>>>
>>>>
>>>>
>>>>
>>>> On Thu, Feb 20, 2014 at 12:46 PM, Sylvain Lebresne <
>>>> sylvain@datastax.com> wrote:
>>>>
>>>>> On Thu, Feb 20, 2014 at 6:26 PM, Peter Lin <wo...@gmail.com> wrote:
>>>>>
>>>>>>
>>>>>> I disagree with the sentiment that "thrift is not worth the trouble".
>>>>>>
>>>>>
>>>>> Way to quote only part of my sentence and get mental on it. My full
>>>>> sentence was "it's probably not worth the trouble to start with thrift if
>>>>> you're gonna use CQL later".
>>>>>
>>>>>
>>>>>>
>>>>>> CQL and all SQL inspired dialects limit one's ability to use
>>>>>> arbitrary typed data in dynamic columns. With thrift it's easy and straight
>>>>>> forward. With CQL there is no way to tell Cassandra the type of the name
>>>>>> and value for a dynamic column. You can only set the default type. That
>>>>>> means using a "pure cql" approach you can deviate from the default type.
>>>>>> Cassandra will throw an exception indicating the type is different than the
>>>>>> default type.
>>>>>>
>>>>>
>>>>>> Until such time that CQL abandons the shackles of SQL and adds the
>>>>>> ability to indicate the column and value type. Something like this
>>>>>>
>>>>>
>>>>>> insert into myColumnFamily(staticColumn1, staticColumn2, 20 as int,
>>>>>> dynamicColumn as string) into ('text1','text2',30.55 as double, 3500 as
>>>>>> long)
>>>>>>
>>>>>> This is one area where Thrift is superior to CQL. Having said that,
>>>>>> it's valid to use Cassandra "as if" it was a relational database, but then
>>>>>> you'd miss out on some of the unique features.
>>>>>>
>>>>>
>>>>> Man, if I had a nickel every time someone came on that mailing list
>>>>> pretending that something was possible with thrift and not CQL ... I will
>>>>> claim this: with CASSANDRA-6561 and CASSANDRA-4851 that just got in, there
>>>>> is *nothing* that thrift can do that CQL cannot. But well, what do I know
>>>>> about Cassandra.
>>>>>
>>>>> --
>>>>> Sylvain
>>>>>
>>>>>
>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Thu, Feb 20, 2014 at 12:12 PM, Sylvain Lebresne <
>>>>>> sylvain@datastax.com> wrote:
>>>>>>
>>>>>>> On Thu, Feb 20, 2014 at 2:16 PM, Edward Capriolo <
>>>>>>> edlinuxguru@gmail.com> wrote:
>>>>>>>
>>>>>>>> For what it is worth you schema is simple and uses compact storage.
>>>>>>>> Thus you really dont need anything in cassandra 2.0 as far as i can tell.
>>>>>>>> You might be happier with a stable release like 1.2.something and just
>>>>>>>> hector or astyanax. You are really dealing with many issues you should not
>>>>>>>> have to just to protoype a simple cassandra app.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Of course, if everyone was using that reasoning, no-one would ever
>>>>>>> test new features and report problems/suggest improvement. So thanks to
>>>>>>> anyone like Rüdiger that actually tries stuff and take the time to report
>>>>>>> problems when they think they encounter one. Keep at it, *you* are the one
>>>>>>> helping Cassandra to get better everyday.
>>>>>>>
>>>>>>> And you are also right Rüdiger that it's probably not worth the
>>>>>>> trouble to start with thrift if you're gonna use CQL later. And you
>>>>>>> definitively should use CQL, it is Cassandra's future.
>>>>>>>
>>>>>>> --
>>>>>>> Sylvain
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>> On Thursday, February 20, 2014, Sylvain Lebresne <
>>>>>>>> sylvain@datastax.com> wrote:
>>>>>>>> >
>>>>>>>> >
>>>>>>>> >
>>>>>>>> > On Wed, Feb 19, 2014 at 9:38 PM, Rüdiger Klaehn <
>>>>>>>> rklaehn@gmail.com> wrote:
>>>>>>>> >>
>>>>>>>> >> I have cloned the cassandra repo, applied the patch, and built
>>>>>>>> it. But when I want to run the bechmark I get an exception. See below. I
>>>>>>>> tried with a non-managed dependency to
>>>>>>>> cassandra-driver-core-2.0.0-rc3-SNAPSHOT-jar-with-dependencies.jar, which I
>>>>>>>> compiled from source because I read that that might help. But that did not
>>>>>>>> make a difference.
>>>>>>>> >>
>>>>>>>> >> So currently I don't know how to give the patch a try. Any ideas?
>>>>>>>> >>
>>>>>>>> >> cheers,
>>>>>>>> >>
>>>>>>>> >> Rüdiger
>>>>>>>> >>
>>>>>>>> >> Exception in thread "main" java.lang.IllegalArgumentException:
>>>>>>>> replicate_on_write is not a column defined in this metadata
>>>>>>>> >> at
>>>>>>>> com.datastax.driver.core.ColumnDefinitions.getAllIdx(ColumnDefinitions.java:273)
>>>>>>>> >> at
>>>>>>>> com.datastax.driver.core.ColumnDefinitions.getFirstIdx(ColumnDefinitions.java:279)
>>>>>>>> >> at com.datastax.driver.core.Row.getBool(Row.java:117)
>>>>>>>> >> at
>>>>>>>> com.datastax.driver.core.TableMetadata$Options.<init>(TableMetadata.java:474)
>>>>>>>> >> at
>>>>>>>> com.datastax.driver.core.TableMetadata.build(TableMetadata.java:107)
>>>>>>>> >> at
>>>>>>>> com.datastax.driver.core.Metadata.buildTableMetadata(Metadata.java:128)
>>>>>>>> >> at
>>>>>>>> com.datastax.driver.core.Metadata.rebuildSchema(Metadata.java:89)
>>>>>>>> >> at
>>>>>>>> com.datastax.driver.core.ControlConnection.refreshSchema(ControlConnection.java:259)
>>>>>>>> >> at
>>>>>>>> com.datastax.driver.core.ControlConnection.tryConnect(ControlConnection.java:214)
>>>>>>>> >> at
>>>>>>>> com.datastax.driver.core.ControlConnection.reconnectInternal(ControlConnection.java:161)
>>>>>>>> >> at
>>>>>>>> com.datastax.driver.core.ControlConnection.connect(ControlConnection.java:77)
>>>>>>>> >> at
>>>>>>>> com.datastax.driver.core.Cluster$Manager.init(Cluster.java:890)
>>>>>>>> >> at
>>>>>>>> com.datastax.driver.core.Cluster$Manager.newSession(Cluster.java:910)
>>>>>>>> >> at
>>>>>>>> com.datastax.driver.core.Cluster$Manager.access$200(Cluster.java:806)
>>>>>>>> >> at com.datastax.driver.core.Cluster.connect(Cluster.java:158)
>>>>>>>> >> at
>>>>>>>> cassandra.CassandraTestMinimized$delayedInit$body.apply(CassandraTestMinimized.scala:31)
>>>>>>>> >> at scala.Function0$class.apply$mcV$sp(Function0.scala:40)
>>>>>>>> >> at
>>>>>>>> scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:12)
>>>>>>>> >> at scala.App$$anonfun$main$1.apply(App.scala:71)
>>>>>>>> >> at scala.App$$anonfun$main$1.apply(App.scala:71)
>>>>>>>> >> at scala.collection.immutable.List.foreach(List.scala:318)
>>>>>>>> >> at
>>>>>>>> scala.collection.generic.TraversableForwarder$class.foreach(TraversableForwarder.scala:32)
>>>>>>>> >> at scala.App$class.main(App.scala:71)
>>>>>>>> >> at
>>>>>>>> cassandra.CassandraTestMinimized$.main(CassandraTestMinimized.scala:5)
>>>>>>>> >> at
>>>>>>>> cassandra.CassandraTestMinimized.main(CassandraTestMinimized.scala)
>>>>>>>> >
>>>>>>>> > I believe you've tried the cassandra trunk branch? trunk is
>>>>>>>> basically the future Cassandra 2.1 and the driver is currently unhappy
>>>>>>>> because the replicate_on_write option has been removed in that version. I'm
>>>>>>>> supposed to have fixed that on the driver 2.0 branch like 2 days ago so
>>>>>>>> maybe you're also using a slightly old version of the driver sources in
>>>>>>>> there? Or maybe I've screwed up my fix, I'll double check. But anyway, it
>>>>>>>> would be overall simpler to test with the cassandra-2.0 branch of
>>>>>>>> Cassandra, with which you shouldn't run into that.
>>>>>>>> > --
>>>>>>>> > Sylvain
>>>>>>>>
>>>>>>>> --
>>>>>>>> Sorry this was sent from mobile. Will do less grammar and spell
>>>>>>>> check than usual.
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
Re: Performance problem with large wide row inserts using CQL
Posted by Edward Capriolo <ed...@gmail.com>.
Peter,
I must meet you and shake your hand. I was actually having a debate with a
number of people about a week back claiming there was "no reason to mix
static and dynamic". We do it all the time I am glad someone else besides
me "gets it" and I am not totally mad.
Ed
On Thu, Feb 20, 2014 at 3:26 PM, Peter Lin <wo...@gmail.com> wrote:
>
> Hi Duyhai,
>
> yes, I am talking about mixing static and dynamic columns in a single
> column family. Let me give you an example from retail.
>
> Say you're amazon and you sell over 10K different products. How do you
> store all those products with all the different properties like color,
> size, dimensions, etc. With relational databases people use EAV (entity
> attribute value) tables. This means querying for data the system has to
> reconstruct the object by pivot a bunch of rows and flattening it out to
> populate the java object. Typically there are common fields to a product
> like SKU, price, and category.
>
> Using both static and dynamic columns, data can be stored in 1 row and
> queried by 1 row. Anyone that has used EAV approach to build product
> databases will tell you how much that sucks. Another example is from auto
> insurance. Typically a policy database will allow 1 or more types of items
> for property insurance. Property insurance is home/auto insurance.
>
> Each insurance carrier supports different number of insurable items,
> coverages and endorsements. Many systems use the same EAV approach, but the
> problem is bigger. Typically a commercial auto policy may have hundreds of
> drivers and vehicles. Each policy may have dozens or hundreds of coverages
> and endorsements. It is common for an auto insurance model to have hundreds
> of coverage and endorsements with different properties. Using the old ORM
> approach, it's usually mapped table-per-class. Problem is, that results in
> query explosion for polymorphic queries. This is a known problem with
> polymorphic queries using traditional techniques.
>
> Given that Cassandra + thrift gives developers the ability to store
> dynamic columns of different types, it solves the performance issues
> inherent in EAV technique.
>
> The point I was trying to make in my first response is that going with
> pure CQL makes it much harder to take advantage of the COOL features of
> Cassandra. It does require building a framework to make it "mostly"
> transparent to developers, but it is worth it in my opinion to learn and
> understand both thrift and cql. I use annotations in my framework and
> delegates to handle the serialization. This way, the developer only needs
> annotate the class and the framework handles serialization and
> deserialization.
>
>
>
>
>
> On Thu, Feb 20, 2014 at 3:05 PM, DuyHai Doan <do...@gmail.com> wrote:
>
>> "Developers can use what ever type they want for the name or value in a
>> dynamic column and the framework will handle it appropriately."
>>
>> What do you mean by "dynamic" column ? If you want to be able to insert
>> an arbitrary number of columns in one physical row, CQL3 clustering is
>> there and does pretty well the job.
>>
>> If by "dynamic" you mean a column whose validation type can change at
>> runtime (like the dynamic composite type :
>> http://hector-client.github.io/hector/build/html/content/composite_with_templates.html)
>> then why don't you just use blob type and serialize it yourself at client
>> side ?
>>
>> More pratically, in your previous example :
>>
>> - insert into myColumnFamily(staticColumn1, staticColumn2, 20 as int,
>> dynamicColumn as string) into ('text1','text2',30.55 as double, 3500 as
>> long)
>>
>> I can't see real sensible use-case where you need to mix static and
>> dynamic columns in the same column family. If you need to save domain
>> model, use skinny row with a fixed number of columns known before hand. If
>> you want to store time series or timeline of data, wide row is there.
>>
>>
>> On Thu, Feb 20, 2014 at 8:55 PM, Peter Lin <wo...@gmail.com> wrote:
>>
>>>
>>> my apologies Sylvain, I didn't mean to misquote you. I still feel that
>>> even if someone is only going to use CQL, it is "worth it" to learn thrift.
>>>
>>> In the interest of discussion, I looked at both jira tickets and I don't
>>> see how that makes it so a developer can specify the name and value type
>>> for a dynamic column.
>>>
>>> https://issues.apache.org/jira/browse/CASSANDRA-6561
>>> https://issues.apache.org/jira/browse/CASSANDRA-4851
>>>
>>> Am I missing something? If the grammar for insert statements doesn't
>>> give users the ability declare the name and value type, it means the
>>> developer has to default name and value to bytes. In their code, they have
>>> to handle that manually or build their own framework. I built my own
>>> framework, which handles this for me. Developers can use what ever type
>>> they want for the name or value in a dynamic column and the framework will
>>> handle it appropriately.
>>>
>>> To me, developers should take time to learn both and use both. I realize
>>> it's more work to understand both and take time to read the code. Not
>>> everyone is crazy enough spend time reading cassandra code base or spend
>>> hundreds of hours studying hector and other cassandra clients. I will say
>>> this, if I hadn't spend time studying cassandra and reading Hector code, I
>>> wouldn't have been able to help one of DataStax customer port Hector to
>>> .Net. I also wouldn't have been able to port Hector to C# natively in 3
>>> months.
>>>
>>> Rather than recommend people be lazy, it would be more useful to list
>>> the pros/cons. To my knowledge, there isn't a good writeup on the pros/cons
>>> of thrift and cql on cassandra.apache.org. I don't know if the DataStax
>>> docs have a detailed write up of it, does it?
>>>
>>>
>>>
>>>
>>> On Thu, Feb 20, 2014 at 12:46 PM, Sylvain Lebresne <sylvain@datastax.com
>>> > wrote:
>>>
>>>> On Thu, Feb 20, 2014 at 6:26 PM, Peter Lin <wo...@gmail.com> wrote:
>>>>
>>>>>
>>>>> I disagree with the sentiment that "thrift is not worth the trouble".
>>>>>
>>>>
>>>> Way to quote only part of my sentence and get mental on it. My full
>>>> sentence was "it's probably not worth the trouble to start with thrift if
>>>> you're gonna use CQL later".
>>>>
>>>>
>>>>>
>>>>> CQL and all SQL inspired dialects limit one's ability to use arbitrary
>>>>> typed data in dynamic columns. With thrift it's easy and straight forward.
>>>>> With CQL there is no way to tell Cassandra the type of the name and value
>>>>> for a dynamic column. You can only set the default type. That means using a
>>>>> "pure cql" approach you can deviate from the default type. Cassandra will
>>>>> throw an exception indicating the type is different than the default type.
>>>>>
>>>>
>>>>> Until such time that CQL abandons the shackles of SQL and adds the
>>>>> ability to indicate the column and value type. Something like this
>>>>>
>>>>
>>>>> insert into myColumnFamily(staticColumn1, staticColumn2, 20 as int,
>>>>> dynamicColumn as string) into ('text1','text2',30.55 as double, 3500 as
>>>>> long)
>>>>>
>>>>> This is one area where Thrift is superior to CQL. Having said that,
>>>>> it's valid to use Cassandra "as if" it was a relational database, but then
>>>>> you'd miss out on some of the unique features.
>>>>>
>>>>
>>>> Man, if I had a nickel every time someone came on that mailing list
>>>> pretending that something was possible with thrift and not CQL ... I will
>>>> claim this: with CASSANDRA-6561 and CASSANDRA-4851 that just got in, there
>>>> is *nothing* that thrift can do that CQL cannot. But well, what do I know
>>>> about Cassandra.
>>>>
>>>> --
>>>> Sylvain
>>>>
>>>>
>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Thu, Feb 20, 2014 at 12:12 PM, Sylvain Lebresne <
>>>>> sylvain@datastax.com> wrote:
>>>>>
>>>>>> On Thu, Feb 20, 2014 at 2:16 PM, Edward Capriolo <
>>>>>> edlinuxguru@gmail.com> wrote:
>>>>>>
>>>>>>> For what it is worth you schema is simple and uses compact storage.
>>>>>>> Thus you really dont need anything in cassandra 2.0 as far as i can tell.
>>>>>>> You might be happier with a stable release like 1.2.something and just
>>>>>>> hector or astyanax. You are really dealing with many issues you should not
>>>>>>> have to just to protoype a simple cassandra app.
>>>>>>
>>>>>>
>>>>>>
>>>>>> Of course, if everyone was using that reasoning, no-one would ever
>>>>>> test new features and report problems/suggest improvement. So thanks to
>>>>>> anyone like Rüdiger that actually tries stuff and take the time to report
>>>>>> problems when they think they encounter one. Keep at it, *you* are the one
>>>>>> helping Cassandra to get better everyday.
>>>>>>
>>>>>> And you are also right Rüdiger that it's probably not worth the
>>>>>> trouble to start with thrift if you're gonna use CQL later. And you
>>>>>> definitively should use CQL, it is Cassandra's future.
>>>>>>
>>>>>> --
>>>>>> Sylvain
>>>>>>
>>>>>>
>>>>>>
>>>>>>>
>>>>>>> On Thursday, February 20, 2014, Sylvain Lebresne <
>>>>>>> sylvain@datastax.com> wrote:
>>>>>>> >
>>>>>>> >
>>>>>>> >
>>>>>>> > On Wed, Feb 19, 2014 at 9:38 PM, Rüdiger Klaehn <rk...@gmail.com>
>>>>>>> wrote:
>>>>>>> >>
>>>>>>> >> I have cloned the cassandra repo, applied the patch, and built
>>>>>>> it. But when I want to run the bechmark I get an exception. See below. I
>>>>>>> tried with a non-managed dependency to
>>>>>>> cassandra-driver-core-2.0.0-rc3-SNAPSHOT-jar-with-dependencies.jar, which I
>>>>>>> compiled from source because I read that that might help. But that did not
>>>>>>> make a difference.
>>>>>>> >>
>>>>>>> >> So currently I don't know how to give the patch a try. Any ideas?
>>>>>>> >>
>>>>>>> >> cheers,
>>>>>>> >>
>>>>>>> >> Rüdiger
>>>>>>> >>
>>>>>>> >> Exception in thread "main" java.lang.IllegalArgumentException:
>>>>>>> replicate_on_write is not a column defined in this metadata
>>>>>>> >> at
>>>>>>> com.datastax.driver.core.ColumnDefinitions.getAllIdx(ColumnDefinitions.java:273)
>>>>>>> >> at
>>>>>>> com.datastax.driver.core.ColumnDefinitions.getFirstIdx(ColumnDefinitions.java:279)
>>>>>>> >> at com.datastax.driver.core.Row.getBool(Row.java:117)
>>>>>>> >> at
>>>>>>> com.datastax.driver.core.TableMetadata$Options.<init>(TableMetadata.java:474)
>>>>>>> >> at
>>>>>>> com.datastax.driver.core.TableMetadata.build(TableMetadata.java:107)
>>>>>>> >> at
>>>>>>> com.datastax.driver.core.Metadata.buildTableMetadata(Metadata.java:128)
>>>>>>> >> at
>>>>>>> com.datastax.driver.core.Metadata.rebuildSchema(Metadata.java:89)
>>>>>>> >> at
>>>>>>> com.datastax.driver.core.ControlConnection.refreshSchema(ControlConnection.java:259)
>>>>>>> >> at
>>>>>>> com.datastax.driver.core.ControlConnection.tryConnect(ControlConnection.java:214)
>>>>>>> >> at
>>>>>>> com.datastax.driver.core.ControlConnection.reconnectInternal(ControlConnection.java:161)
>>>>>>> >> at
>>>>>>> com.datastax.driver.core.ControlConnection.connect(ControlConnection.java:77)
>>>>>>> >> at
>>>>>>> com.datastax.driver.core.Cluster$Manager.init(Cluster.java:890)
>>>>>>> >> at
>>>>>>> com.datastax.driver.core.Cluster$Manager.newSession(Cluster.java:910)
>>>>>>> >> at
>>>>>>> com.datastax.driver.core.Cluster$Manager.access$200(Cluster.java:806)
>>>>>>> >> at com.datastax.driver.core.Cluster.connect(Cluster.java:158)
>>>>>>> >> at
>>>>>>> cassandra.CassandraTestMinimized$delayedInit$body.apply(CassandraTestMinimized.scala:31)
>>>>>>> >> at scala.Function0$class.apply$mcV$sp(Function0.scala:40)
>>>>>>> >> at
>>>>>>> scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:12)
>>>>>>> >> at scala.App$$anonfun$main$1.apply(App.scala:71)
>>>>>>> >> at scala.App$$anonfun$main$1.apply(App.scala:71)
>>>>>>> >> at scala.collection.immutable.List.foreach(List.scala:318)
>>>>>>> >> at
>>>>>>> scala.collection.generic.TraversableForwarder$class.foreach(TraversableForwarder.scala:32)
>>>>>>> >> at scala.App$class.main(App.scala:71)
>>>>>>> >> at
>>>>>>> cassandra.CassandraTestMinimized$.main(CassandraTestMinimized.scala:5)
>>>>>>> >> at
>>>>>>> cassandra.CassandraTestMinimized.main(CassandraTestMinimized.scala)
>>>>>>> >
>>>>>>> > I believe you've tried the cassandra trunk branch? trunk is
>>>>>>> basically the future Cassandra 2.1 and the driver is currently unhappy
>>>>>>> because the replicate_on_write option has been removed in that version. I'm
>>>>>>> supposed to have fixed that on the driver 2.0 branch like 2 days ago so
>>>>>>> maybe you're also using a slightly old version of the driver sources in
>>>>>>> there? Or maybe I've screwed up my fix, I'll double check. But anyway, it
>>>>>>> would be overall simpler to test with the cassandra-2.0 branch of
>>>>>>> Cassandra, with which you shouldn't run into that.
>>>>>>> > --
>>>>>>> > Sylvain
>>>>>>>
>>>>>>> --
>>>>>>> Sorry this was sent from mobile. Will do less grammar and spell
>>>>>>> check than usual.
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
Re: Performance problem with large wide row inserts using CQL
Posted by DuyHai Doan <do...@gmail.com>.
Ok I see what you mean Peter. After reading
CASSANDRA-6561<https://issues.apache.org/jira/browse/CASSANDRA-6561>the
use case is pretty clear.
On Thu, Feb 20, 2014 at 9:26 PM, Peter Lin <wo...@gmail.com> wrote:
>
> Hi Duyhai,
>
> yes, I am talking about mixing static and dynamic columns in a single
> column family. Let me give you an example from retail.
>
> Say you're amazon and you sell over 10K different products. How do you
> store all those products with all the different properties like color,
> size, dimensions, etc. With relational databases people use EAV (entity
> attribute value) tables. This means querying for data the system has to
> reconstruct the object by pivot a bunch of rows and flattening it out to
> populate the java object. Typically there are common fields to a product
> like SKU, price, and category.
>
> Using both static and dynamic columns, data can be stored in 1 row and
> queried by 1 row. Anyone that has used EAV approach to build product
> databases will tell you how much that sucks. Another example is from auto
> insurance. Typically a policy database will allow 1 or more types of items
> for property insurance. Property insurance is home/auto insurance.
>
> Each insurance carrier supports different number of insurable items,
> coverages and endorsements. Many systems use the same EAV approach, but the
> problem is bigger. Typically a commercial auto policy may have hundreds of
> drivers and vehicles. Each policy may have dozens or hundreds of coverages
> and endorsements. It is common for an auto insurance model to have hundreds
> of coverage and endorsements with different properties. Using the old ORM
> approach, it's usually mapped table-per-class. Problem is, that results in
> query explosion for polymorphic queries. This is a known problem with
> polymorphic queries using traditional techniques.
>
> Given that Cassandra + thrift gives developers the ability to store
> dynamic columns of different types, it solves the performance issues
> inherent in EAV technique.
>
> The point I was trying to make in my first response is that going with
> pure CQL makes it much harder to take advantage of the COOL features of
> Cassandra. It does require building a framework to make it "mostly"
> transparent to developers, but it is worth it in my opinion to learn and
> understand both thrift and cql. I use annotations in my framework and
> delegates to handle the serialization. This way, the developer only needs
> annotate the class and the framework handles serialization and
> deserialization.
>
>
>
>
>
> On Thu, Feb 20, 2014 at 3:05 PM, DuyHai Doan <do...@gmail.com> wrote:
>
>> "Developers can use what ever type they want for the name or value in a
>> dynamic column and the framework will handle it appropriately."
>>
>> What do you mean by "dynamic" column ? If you want to be able to insert
>> an arbitrary number of columns in one physical row, CQL3 clustering is
>> there and does pretty well the job.
>>
>> If by "dynamic" you mean a column whose validation type can change at
>> runtime (like the dynamic composite type :
>> http://hector-client.github.io/hector/build/html/content/composite_with_templates.html)
>> then why don't you just use blob type and serialize it yourself at client
>> side ?
>>
>> More pratically, in your previous example :
>>
>> - insert into myColumnFamily(staticColumn1, staticColumn2, 20 as int,
>> dynamicColumn as string) into ('text1','text2',30.55 as double, 3500 as
>> long)
>>
>> I can't see real sensible use-case where you need to mix static and
>> dynamic columns in the same column family. If you need to save domain
>> model, use skinny row with a fixed number of columns known before hand. If
>> you want to store time series or timeline of data, wide row is there.
>>
>>
>> On Thu, Feb 20, 2014 at 8:55 PM, Peter Lin <wo...@gmail.com> wrote:
>>
>>>
>>> my apologies Sylvain, I didn't mean to misquote you. I still feel that
>>> even if someone is only going to use CQL, it is "worth it" to learn thrift.
>>>
>>> In the interest of discussion, I looked at both jira tickets and I don't
>>> see how that makes it so a developer can specify the name and value type
>>> for a dynamic column.
>>>
>>> https://issues.apache.org/jira/browse/CASSANDRA-6561
>>> https://issues.apache.org/jira/browse/CASSANDRA-4851
>>>
>>> Am I missing something? If the grammar for insert statements doesn't
>>> give users the ability declare the name and value type, it means the
>>> developer has to default name and value to bytes. In their code, they have
>>> to handle that manually or build their own framework. I built my own
>>> framework, which handles this for me. Developers can use what ever type
>>> they want for the name or value in a dynamic column and the framework will
>>> handle it appropriately.
>>>
>>> To me, developers should take time to learn both and use both. I realize
>>> it's more work to understand both and take time to read the code. Not
>>> everyone is crazy enough spend time reading cassandra code base or spend
>>> hundreds of hours studying hector and other cassandra clients. I will say
>>> this, if I hadn't spend time studying cassandra and reading Hector code, I
>>> wouldn't have been able to help one of DataStax customer port Hector to
>>> .Net. I also wouldn't have been able to port Hector to C# natively in 3
>>> months.
>>>
>>> Rather than recommend people be lazy, it would be more useful to list
>>> the pros/cons. To my knowledge, there isn't a good writeup on the pros/cons
>>> of thrift and cql on cassandra.apache.org. I don't know if the DataStax
>>> docs have a detailed write up of it, does it?
>>>
>>>
>>>
>>>
>>> On Thu, Feb 20, 2014 at 12:46 PM, Sylvain Lebresne <sylvain@datastax.com
>>> > wrote:
>>>
>>>> On Thu, Feb 20, 2014 at 6:26 PM, Peter Lin <wo...@gmail.com> wrote:
>>>>
>>>>>
>>>>> I disagree with the sentiment that "thrift is not worth the trouble".
>>>>>
>>>>
>>>> Way to quote only part of my sentence and get mental on it. My full
>>>> sentence was "it's probably not worth the trouble to start with thrift if
>>>> you're gonna use CQL later".
>>>>
>>>>
>>>>>
>>>>> CQL and all SQL inspired dialects limit one's ability to use arbitrary
>>>>> typed data in dynamic columns. With thrift it's easy and straight forward.
>>>>> With CQL there is no way to tell Cassandra the type of the name and value
>>>>> for a dynamic column. You can only set the default type. That means using a
>>>>> "pure cql" approach you can deviate from the default type. Cassandra will
>>>>> throw an exception indicating the type is different than the default type.
>>>>>
>>>>
>>>>> Until such time that CQL abandons the shackles of SQL and adds the
>>>>> ability to indicate the column and value type. Something like this
>>>>>
>>>>
>>>>> insert into myColumnFamily(staticColumn1, staticColumn2, 20 as int,
>>>>> dynamicColumn as string) into ('text1','text2',30.55 as double, 3500 as
>>>>> long)
>>>>>
>>>>> This is one area where Thrift is superior to CQL. Having said that,
>>>>> it's valid to use Cassandra "as if" it was a relational database, but then
>>>>> you'd miss out on some of the unique features.
>>>>>
>>>>
>>>> Man, if I had a nickel every time someone came on that mailing list
>>>> pretending that something was possible with thrift and not CQL ... I will
>>>> claim this: with CASSANDRA-6561 and CASSANDRA-4851 that just got in, there
>>>> is *nothing* that thrift can do that CQL cannot. But well, what do I know
>>>> about Cassandra.
>>>>
>>>> --
>>>> Sylvain
>>>>
>>>>
>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Thu, Feb 20, 2014 at 12:12 PM, Sylvain Lebresne <
>>>>> sylvain@datastax.com> wrote:
>>>>>
>>>>>> On Thu, Feb 20, 2014 at 2:16 PM, Edward Capriolo <
>>>>>> edlinuxguru@gmail.com> wrote:
>>>>>>
>>>>>>> For what it is worth you schema is simple and uses compact storage.
>>>>>>> Thus you really dont need anything in cassandra 2.0 as far as i can tell.
>>>>>>> You might be happier with a stable release like 1.2.something and just
>>>>>>> hector or astyanax. You are really dealing with many issues you should not
>>>>>>> have to just to protoype a simple cassandra app.
>>>>>>
>>>>>>
>>>>>>
>>>>>> Of course, if everyone was using that reasoning, no-one would ever
>>>>>> test new features and report problems/suggest improvement. So thanks to
>>>>>> anyone like Rüdiger that actually tries stuff and take the time to report
>>>>>> problems when they think they encounter one. Keep at it, *you* are the one
>>>>>> helping Cassandra to get better everyday.
>>>>>>
>>>>>> And you are also right Rüdiger that it's probably not worth the
>>>>>> trouble to start with thrift if you're gonna use CQL later. And you
>>>>>> definitively should use CQL, it is Cassandra's future.
>>>>>>
>>>>>> --
>>>>>> Sylvain
>>>>>>
>>>>>>
>>>>>>
>>>>>>>
>>>>>>> On Thursday, February 20, 2014, Sylvain Lebresne <
>>>>>>> sylvain@datastax.com> wrote:
>>>>>>> >
>>>>>>> >
>>>>>>> >
>>>>>>> > On Wed, Feb 19, 2014 at 9:38 PM, Rüdiger Klaehn <rk...@gmail.com>
>>>>>>> wrote:
>>>>>>> >>
>>>>>>> >> I have cloned the cassandra repo, applied the patch, and built
>>>>>>> it. But when I want to run the bechmark I get an exception. See below. I
>>>>>>> tried with a non-managed dependency to
>>>>>>> cassandra-driver-core-2.0.0-rc3-SNAPSHOT-jar-with-dependencies.jar, which I
>>>>>>> compiled from source because I read that that might help. But that did not
>>>>>>> make a difference.
>>>>>>> >>
>>>>>>> >> So currently I don't know how to give the patch a try. Any ideas?
>>>>>>> >>
>>>>>>> >> cheers,
>>>>>>> >>
>>>>>>> >> Rüdiger
>>>>>>> >>
>>>>>>> >> Exception in thread "main" java.lang.IllegalArgumentException:
>>>>>>> replicate_on_write is not a column defined in this metadata
>>>>>>> >> at
>>>>>>> com.datastax.driver.core.ColumnDefinitions.getAllIdx(ColumnDefinitions.java:273)
>>>>>>> >> at
>>>>>>> com.datastax.driver.core.ColumnDefinitions.getFirstIdx(ColumnDefinitions.java:279)
>>>>>>> >> at com.datastax.driver.core.Row.getBool(Row.java:117)
>>>>>>> >> at
>>>>>>> com.datastax.driver.core.TableMetadata$Options.<init>(TableMetadata.java:474)
>>>>>>> >> at
>>>>>>> com.datastax.driver.core.TableMetadata.build(TableMetadata.java:107)
>>>>>>> >> at
>>>>>>> com.datastax.driver.core.Metadata.buildTableMetadata(Metadata.java:128)
>>>>>>> >> at
>>>>>>> com.datastax.driver.core.Metadata.rebuildSchema(Metadata.java:89)
>>>>>>> >> at
>>>>>>> com.datastax.driver.core.ControlConnection.refreshSchema(ControlConnection.java:259)
>>>>>>> >> at
>>>>>>> com.datastax.driver.core.ControlConnection.tryConnect(ControlConnection.java:214)
>>>>>>> >> at
>>>>>>> com.datastax.driver.core.ControlConnection.reconnectInternal(ControlConnection.java:161)
>>>>>>> >> at
>>>>>>> com.datastax.driver.core.ControlConnection.connect(ControlConnection.java:77)
>>>>>>> >> at
>>>>>>> com.datastax.driver.core.Cluster$Manager.init(Cluster.java:890)
>>>>>>> >> at
>>>>>>> com.datastax.driver.core.Cluster$Manager.newSession(Cluster.java:910)
>>>>>>> >> at
>>>>>>> com.datastax.driver.core.Cluster$Manager.access$200(Cluster.java:806)
>>>>>>> >> at com.datastax.driver.core.Cluster.connect(Cluster.java:158)
>>>>>>> >> at
>>>>>>> cassandra.CassandraTestMinimized$delayedInit$body.apply(CassandraTestMinimized.scala:31)
>>>>>>> >> at scala.Function0$class.apply$mcV$sp(Function0.scala:40)
>>>>>>> >> at
>>>>>>> scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:12)
>>>>>>> >> at scala.App$$anonfun$main$1.apply(App.scala:71)
>>>>>>> >> at scala.App$$anonfun$main$1.apply(App.scala:71)
>>>>>>> >> at scala.collection.immutable.List.foreach(List.scala:318)
>>>>>>> >> at
>>>>>>> scala.collection.generic.TraversableForwarder$class.foreach(TraversableForwarder.scala:32)
>>>>>>> >> at scala.App$class.main(App.scala:71)
>>>>>>> >> at
>>>>>>> cassandra.CassandraTestMinimized$.main(CassandraTestMinimized.scala:5)
>>>>>>> >> at
>>>>>>> cassandra.CassandraTestMinimized.main(CassandraTestMinimized.scala)
>>>>>>> >
>>>>>>> > I believe you've tried the cassandra trunk branch? trunk is
>>>>>>> basically the future Cassandra 2.1 and the driver is currently unhappy
>>>>>>> because the replicate_on_write option has been removed in that version. I'm
>>>>>>> supposed to have fixed that on the driver 2.0 branch like 2 days ago so
>>>>>>> maybe you're also using a slightly old version of the driver sources in
>>>>>>> there? Or maybe I've screwed up my fix, I'll double check. But anyway, it
>>>>>>> would be overall simpler to test with the cassandra-2.0 branch of
>>>>>>> Cassandra, with which you shouldn't run into that.
>>>>>>> > --
>>>>>>> > Sylvain
>>>>>>>
>>>>>>> --
>>>>>>> Sorry this was sent from mobile. Will do less grammar and spell
>>>>>>> check than usual.
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
Re: Performance problem with large wide row inserts using CQL
Posted by Peter Lin <wo...@gmail.com>.
Hi Duyhai,
yes, I am talking about mixing static and dynamic columns in a single
column family. Let me give you an example from retail.
Say you're amazon and you sell over 10K different products. How do you
store all those products with all the different properties like color,
size, dimensions, etc. With relational databases people use EAV (entity
attribute value) tables. This means querying for data the system has to
reconstruct the object by pivot a bunch of rows and flattening it out to
populate the java object. Typically there are common fields to a product
like SKU, price, and category.
Using both static and dynamic columns, data can be stored in 1 row and
queried by 1 row. Anyone that has used EAV approach to build product
databases will tell you how much that sucks. Another example is from auto
insurance. Typically a policy database will allow 1 or more types of items
for property insurance. Property insurance is home/auto insurance.
Each insurance carrier supports different number of insurable items,
coverages and endorsements. Many systems use the same EAV approach, but the
problem is bigger. Typically a commercial auto policy may have hundreds of
drivers and vehicles. Each policy may have dozens or hundreds of coverages
and endorsements. It is common for an auto insurance model to have hundreds
of coverage and endorsements with different properties. Using the old ORM
approach, it's usually mapped table-per-class. Problem is, that results in
query explosion for polymorphic queries. This is a known problem with
polymorphic queries using traditional techniques.
Given that Cassandra + thrift gives developers the ability to store dynamic
columns of different types, it solves the performance issues inherent in
EAV technique.
The point I was trying to make in my first response is that going with pure
CQL makes it much harder to take advantage of the COOL features of
Cassandra. It does require building a framework to make it "mostly"
transparent to developers, but it is worth it in my opinion to learn and
understand both thrift and cql. I use annotations in my framework and
delegates to handle the serialization. This way, the developer only needs
annotate the class and the framework handles serialization and
deserialization.
On Thu, Feb 20, 2014 at 3:05 PM, DuyHai Doan <do...@gmail.com> wrote:
> "Developers can use what ever type they want for the name or value in a
> dynamic column and the framework will handle it appropriately."
>
> What do you mean by "dynamic" column ? If you want to be able to insert
> an arbitrary number of columns in one physical row, CQL3 clustering is
> there and does pretty well the job.
>
> If by "dynamic" you mean a column whose validation type can change at
> runtime (like the dynamic composite type :
> http://hector-client.github.io/hector/build/html/content/composite_with_templates.html)
> then why don't you just use blob type and serialize it yourself at client
> side ?
>
> More pratically, in your previous example :
>
> - insert into myColumnFamily(staticColumn1, staticColumn2, 20 as int,
> dynamicColumn as string) into ('text1','text2',30.55 as double, 3500 as
> long)
>
> I can't see real sensible use-case where you need to mix static and
> dynamic columns in the same column family. If you need to save domain
> model, use skinny row with a fixed number of columns known before hand. If
> you want to store time series or timeline of data, wide row is there.
>
>
> On Thu, Feb 20, 2014 at 8:55 PM, Peter Lin <wo...@gmail.com> wrote:
>
>>
>> my apologies Sylvain, I didn't mean to misquote you. I still feel that
>> even if someone is only going to use CQL, it is "worth it" to learn thrift.
>>
>> In the interest of discussion, I looked at both jira tickets and I don't
>> see how that makes it so a developer can specify the name and value type
>> for a dynamic column.
>>
>> https://issues.apache.org/jira/browse/CASSANDRA-6561
>> https://issues.apache.org/jira/browse/CASSANDRA-4851
>>
>> Am I missing something? If the grammar for insert statements doesn't give
>> users the ability declare the name and value type, it means the developer
>> has to default name and value to bytes. In their code, they have to handle
>> that manually or build their own framework. I built my own framework, which
>> handles this for me. Developers can use what ever type they want for the
>> name or value in a dynamic column and the framework will handle it
>> appropriately.
>>
>> To me, developers should take time to learn both and use both. I realize
>> it's more work to understand both and take time to read the code. Not
>> everyone is crazy enough spend time reading cassandra code base or spend
>> hundreds of hours studying hector and other cassandra clients. I will say
>> this, if I hadn't spend time studying cassandra and reading Hector code, I
>> wouldn't have been able to help one of DataStax customer port Hector to
>> .Net. I also wouldn't have been able to port Hector to C# natively in 3
>> months.
>>
>> Rather than recommend people be lazy, it would be more useful to list the
>> pros/cons. To my knowledge, there isn't a good writeup on the pros/cons of
>> thrift and cql on cassandra.apache.org. I don't know if the DataStax
>> docs have a detailed write up of it, does it?
>>
>>
>>
>>
>> On Thu, Feb 20, 2014 at 12:46 PM, Sylvain Lebresne <sy...@datastax.com>wrote:
>>
>>> On Thu, Feb 20, 2014 at 6:26 PM, Peter Lin <wo...@gmail.com> wrote:
>>>
>>>>
>>>> I disagree with the sentiment that "thrift is not worth the trouble".
>>>>
>>>
>>> Way to quote only part of my sentence and get mental on it. My full
>>> sentence was "it's probably not worth the trouble to start with thrift if
>>> you're gonna use CQL later".
>>>
>>>
>>>>
>>>> CQL and all SQL inspired dialects limit one's ability to use arbitrary
>>>> typed data in dynamic columns. With thrift it's easy and straight forward.
>>>> With CQL there is no way to tell Cassandra the type of the name and value
>>>> for a dynamic column. You can only set the default type. That means using a
>>>> "pure cql" approach you can deviate from the default type. Cassandra will
>>>> throw an exception indicating the type is different than the default type.
>>>>
>>>
>>>> Until such time that CQL abandons the shackles of SQL and adds the
>>>> ability to indicate the column and value type. Something like this
>>>>
>>>
>>>> insert into myColumnFamily(staticColumn1, staticColumn2, 20 as int,
>>>> dynamicColumn as string) into ('text1','text2',30.55 as double, 3500 as
>>>> long)
>>>>
>>>> This is one area where Thrift is superior to CQL. Having said that,
>>>> it's valid to use Cassandra "as if" it was a relational database, but then
>>>> you'd miss out on some of the unique features.
>>>>
>>>
>>> Man, if I had a nickel every time someone came on that mailing list
>>> pretending that something was possible with thrift and not CQL ... I will
>>> claim this: with CASSANDRA-6561 and CASSANDRA-4851 that just got in, there
>>> is *nothing* that thrift can do that CQL cannot. But well, what do I know
>>> about Cassandra.
>>>
>>> --
>>> Sylvain
>>>
>>>
>>>
>>>>
>>>>
>>>>
>>>>
>>>> On Thu, Feb 20, 2014 at 12:12 PM, Sylvain Lebresne <
>>>> sylvain@datastax.com> wrote:
>>>>
>>>>> On Thu, Feb 20, 2014 at 2:16 PM, Edward Capriolo <
>>>>> edlinuxguru@gmail.com> wrote:
>>>>>
>>>>>> For what it is worth you schema is simple and uses compact storage.
>>>>>> Thus you really dont need anything in cassandra 2.0 as far as i can tell.
>>>>>> You might be happier with a stable release like 1.2.something and just
>>>>>> hector or astyanax. You are really dealing with many issues you should not
>>>>>> have to just to protoype a simple cassandra app.
>>>>>
>>>>>
>>>>>
>>>>> Of course, if everyone was using that reasoning, no-one would ever
>>>>> test new features and report problems/suggest improvement. So thanks to
>>>>> anyone like Rüdiger that actually tries stuff and take the time to report
>>>>> problems when they think they encounter one. Keep at it, *you* are the one
>>>>> helping Cassandra to get better everyday.
>>>>>
>>>>> And you are also right Rüdiger that it's probably not worth the
>>>>> trouble to start with thrift if you're gonna use CQL later. And you
>>>>> definitively should use CQL, it is Cassandra's future.
>>>>>
>>>>> --
>>>>> Sylvain
>>>>>
>>>>>
>>>>>
>>>>>>
>>>>>> On Thursday, February 20, 2014, Sylvain Lebresne <
>>>>>> sylvain@datastax.com> wrote:
>>>>>> >
>>>>>> >
>>>>>> >
>>>>>> > On Wed, Feb 19, 2014 at 9:38 PM, Rüdiger Klaehn <rk...@gmail.com>
>>>>>> wrote:
>>>>>> >>
>>>>>> >> I have cloned the cassandra repo, applied the patch, and built it.
>>>>>> But when I want to run the bechmark I get an exception. See below. I tried
>>>>>> with a non-managed dependency to
>>>>>> cassandra-driver-core-2.0.0-rc3-SNAPSHOT-jar-with-dependencies.jar, which I
>>>>>> compiled from source because I read that that might help. But that did not
>>>>>> make a difference.
>>>>>> >>
>>>>>> >> So currently I don't know how to give the patch a try. Any ideas?
>>>>>> >>
>>>>>> >> cheers,
>>>>>> >>
>>>>>> >> Rüdiger
>>>>>> >>
>>>>>> >> Exception in thread "main" java.lang.IllegalArgumentException:
>>>>>> replicate_on_write is not a column defined in this metadata
>>>>>> >> at
>>>>>> com.datastax.driver.core.ColumnDefinitions.getAllIdx(ColumnDefinitions.java:273)
>>>>>> >> at
>>>>>> com.datastax.driver.core.ColumnDefinitions.getFirstIdx(ColumnDefinitions.java:279)
>>>>>> >> at com.datastax.driver.core.Row.getBool(Row.java:117)
>>>>>> >> at
>>>>>> com.datastax.driver.core.TableMetadata$Options.<init>(TableMetadata.java:474)
>>>>>> >> at
>>>>>> com.datastax.driver.core.TableMetadata.build(TableMetadata.java:107)
>>>>>> >> at
>>>>>> com.datastax.driver.core.Metadata.buildTableMetadata(Metadata.java:128)
>>>>>> >> at
>>>>>> com.datastax.driver.core.Metadata.rebuildSchema(Metadata.java:89)
>>>>>> >> at
>>>>>> com.datastax.driver.core.ControlConnection.refreshSchema(ControlConnection.java:259)
>>>>>> >> at
>>>>>> com.datastax.driver.core.ControlConnection.tryConnect(ControlConnection.java:214)
>>>>>> >> at
>>>>>> com.datastax.driver.core.ControlConnection.reconnectInternal(ControlConnection.java:161)
>>>>>> >> at
>>>>>> com.datastax.driver.core.ControlConnection.connect(ControlConnection.java:77)
>>>>>> >> at
>>>>>> com.datastax.driver.core.Cluster$Manager.init(Cluster.java:890)
>>>>>> >> at
>>>>>> com.datastax.driver.core.Cluster$Manager.newSession(Cluster.java:910)
>>>>>> >> at
>>>>>> com.datastax.driver.core.Cluster$Manager.access$200(Cluster.java:806)
>>>>>> >> at com.datastax.driver.core.Cluster.connect(Cluster.java:158)
>>>>>> >> at
>>>>>> cassandra.CassandraTestMinimized$delayedInit$body.apply(CassandraTestMinimized.scala:31)
>>>>>> >> at scala.Function0$class.apply$mcV$sp(Function0.scala:40)
>>>>>> >> at
>>>>>> scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:12)
>>>>>> >> at scala.App$$anonfun$main$1.apply(App.scala:71)
>>>>>> >> at scala.App$$anonfun$main$1.apply(App.scala:71)
>>>>>> >> at scala.collection.immutable.List.foreach(List.scala:318)
>>>>>> >> at
>>>>>> scala.collection.generic.TraversableForwarder$class.foreach(TraversableForwarder.scala:32)
>>>>>> >> at scala.App$class.main(App.scala:71)
>>>>>> >> at
>>>>>> cassandra.CassandraTestMinimized$.main(CassandraTestMinimized.scala:5)
>>>>>> >> at
>>>>>> cassandra.CassandraTestMinimized.main(CassandraTestMinimized.scala)
>>>>>> >
>>>>>> > I believe you've tried the cassandra trunk branch? trunk is
>>>>>> basically the future Cassandra 2.1 and the driver is currently unhappy
>>>>>> because the replicate_on_write option has been removed in that version. I'm
>>>>>> supposed to have fixed that on the driver 2.0 branch like 2 days ago so
>>>>>> maybe you're also using a slightly old version of the driver sources in
>>>>>> there? Or maybe I've screwed up my fix, I'll double check. But anyway, it
>>>>>> would be overall simpler to test with the cassandra-2.0 branch of
>>>>>> Cassandra, with which you shouldn't run into that.
>>>>>> > --
>>>>>> > Sylvain
>>>>>>
>>>>>> --
>>>>>> Sorry this was sent from mobile. Will do less grammar and spell check
>>>>>> than usual.
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>
Re: Performance problem with large wide row inserts using CQL
Posted by DuyHai Doan <do...@gmail.com>.
"Developers can use what ever type they want for the name or value in a
dynamic column and the framework will handle it appropriately."
What do you mean by "dynamic" column ? If you want to be able to insert an
arbitrary number of columns in one physical row, CQL3 clustering is there
and does pretty well the job.
If by "dynamic" you mean a column whose validation type can change at
runtime (like the dynamic composite type :
http://hector-client.github.io/hector/build/html/content/composite_with_templates.html)
then why don't you just use blob type and serialize it yourself at client
side ?
More pratically, in your previous example :
- insert into myColumnFamily(staticColumn1, staticColumn2, 20 as int,
dynamicColumn as string) into ('text1','text2',30.55 as double, 3500 as
long)
I can't see real sensible use-case where you need to mix static and
dynamic columns in the same column family. If you need to save domain
model, use skinny row with a fixed number of columns known before hand. If
you want to store time series or timeline of data, wide row is there.
On Thu, Feb 20, 2014 at 8:55 PM, Peter Lin <wo...@gmail.com> wrote:
>
> my apologies Sylvain, I didn't mean to misquote you. I still feel that
> even if someone is only going to use CQL, it is "worth it" to learn thrift.
>
> In the interest of discussion, I looked at both jira tickets and I don't
> see how that makes it so a developer can specify the name and value type
> for a dynamic column.
>
> https://issues.apache.org/jira/browse/CASSANDRA-6561
> https://issues.apache.org/jira/browse/CASSANDRA-4851
>
> Am I missing something? If the grammar for insert statements doesn't give
> users the ability declare the name and value type, it means the developer
> has to default name and value to bytes. In their code, they have to handle
> that manually or build their own framework. I built my own framework, which
> handles this for me. Developers can use what ever type they want for the
> name or value in a dynamic column and the framework will handle it
> appropriately.
>
> To me, developers should take time to learn both and use both. I realize
> it's more work to understand both and take time to read the code. Not
> everyone is crazy enough spend time reading cassandra code base or spend
> hundreds of hours studying hector and other cassandra clients. I will say
> this, if I hadn't spend time studying cassandra and reading Hector code, I
> wouldn't have been able to help one of DataStax customer port Hector to
> .Net. I also wouldn't have been able to port Hector to C# natively in 3
> months.
>
> Rather than recommend people be lazy, it would be more useful to list the
> pros/cons. To my knowledge, there isn't a good writeup on the pros/cons of
> thrift and cql on cassandra.apache.org. I don't know if the DataStax docs
> have a detailed write up of it, does it?
>
>
>
>
> On Thu, Feb 20, 2014 at 12:46 PM, Sylvain Lebresne <sy...@datastax.com>wrote:
>
>> On Thu, Feb 20, 2014 at 6:26 PM, Peter Lin <wo...@gmail.com> wrote:
>>
>>>
>>> I disagree with the sentiment that "thrift is not worth the trouble".
>>>
>>
>> Way to quote only part of my sentence and get mental on it. My full
>> sentence was "it's probably not worth the trouble to start with thrift if
>> you're gonna use CQL later".
>>
>>
>>>
>>> CQL and all SQL inspired dialects limit one's ability to use arbitrary
>>> typed data in dynamic columns. With thrift it's easy and straight forward.
>>> With CQL there is no way to tell Cassandra the type of the name and value
>>> for a dynamic column. You can only set the default type. That means using a
>>> "pure cql" approach you can deviate from the default type. Cassandra will
>>> throw an exception indicating the type is different than the default type.
>>>
>>
>>> Until such time that CQL abandons the shackles of SQL and adds the
>>> ability to indicate the column and value type. Something like this
>>>
>>
>>> insert into myColumnFamily(staticColumn1, staticColumn2, 20 as int,
>>> dynamicColumn as string) into ('text1','text2',30.55 as double, 3500 as
>>> long)
>>>
>>> This is one area where Thrift is superior to CQL. Having said that, it's
>>> valid to use Cassandra "as if" it was a relational database, but then you'd
>>> miss out on some of the unique features.
>>>
>>
>> Man, if I had a nickel every time someone came on that mailing list
>> pretending that something was possible with thrift and not CQL ... I will
>> claim this: with CASSANDRA-6561 and CASSANDRA-4851 that just got in, there
>> is *nothing* that thrift can do that CQL cannot. But well, what do I know
>> about Cassandra.
>>
>> --
>> Sylvain
>>
>>
>>
>>>
>>>
>>>
>>>
>>> On Thu, Feb 20, 2014 at 12:12 PM, Sylvain Lebresne <sylvain@datastax.com
>>> > wrote:
>>>
>>>> On Thu, Feb 20, 2014 at 2:16 PM, Edward Capriolo <edlinuxguru@gmail.com
>>>> > wrote:
>>>>
>>>>> For what it is worth you schema is simple and uses compact storage.
>>>>> Thus you really dont need anything in cassandra 2.0 as far as i can tell.
>>>>> You might be happier with a stable release like 1.2.something and just
>>>>> hector or astyanax. You are really dealing with many issues you should not
>>>>> have to just to protoype a simple cassandra app.
>>>>
>>>>
>>>>
>>>> Of course, if everyone was using that reasoning, no-one would ever test
>>>> new features and report problems/suggest improvement. So thanks to anyone
>>>> like Rüdiger that actually tries stuff and take the time to report problems
>>>> when they think they encounter one. Keep at it, *you* are the one helping
>>>> Cassandra to get better everyday.
>>>>
>>>> And you are also right Rüdiger that it's probably not worth the trouble
>>>> to start with thrift if you're gonna use CQL later. And you definitively
>>>> should use CQL, it is Cassandra's future.
>>>>
>>>> --
>>>> Sylvain
>>>>
>>>>
>>>>
>>>>>
>>>>> On Thursday, February 20, 2014, Sylvain Lebresne <sy...@datastax.com>
>>>>> wrote:
>>>>> >
>>>>> >
>>>>> >
>>>>> > On Wed, Feb 19, 2014 at 9:38 PM, Rüdiger Klaehn <rk...@gmail.com>
>>>>> wrote:
>>>>> >>
>>>>> >> I have cloned the cassandra repo, applied the patch, and built it.
>>>>> But when I want to run the bechmark I get an exception. See below. I tried
>>>>> with a non-managed dependency to
>>>>> cassandra-driver-core-2.0.0-rc3-SNAPSHOT-jar-with-dependencies.jar, which I
>>>>> compiled from source because I read that that might help. But that did not
>>>>> make a difference.
>>>>> >>
>>>>> >> So currently I don't know how to give the patch a try. Any ideas?
>>>>> >>
>>>>> >> cheers,
>>>>> >>
>>>>> >> Rüdiger
>>>>> >>
>>>>> >> Exception in thread "main" java.lang.IllegalArgumentException:
>>>>> replicate_on_write is not a column defined in this metadata
>>>>> >> at
>>>>> com.datastax.driver.core.ColumnDefinitions.getAllIdx(ColumnDefinitions.java:273)
>>>>> >> at
>>>>> com.datastax.driver.core.ColumnDefinitions.getFirstIdx(ColumnDefinitions.java:279)
>>>>> >> at com.datastax.driver.core.Row.getBool(Row.java:117)
>>>>> >> at
>>>>> com.datastax.driver.core.TableMetadata$Options.<init>(TableMetadata.java:474)
>>>>> >> at
>>>>> com.datastax.driver.core.TableMetadata.build(TableMetadata.java:107)
>>>>> >> at
>>>>> com.datastax.driver.core.Metadata.buildTableMetadata(Metadata.java:128)
>>>>> >> at
>>>>> com.datastax.driver.core.Metadata.rebuildSchema(Metadata.java:89)
>>>>> >> at
>>>>> com.datastax.driver.core.ControlConnection.refreshSchema(ControlConnection.java:259)
>>>>> >> at
>>>>> com.datastax.driver.core.ControlConnection.tryConnect(ControlConnection.java:214)
>>>>> >> at
>>>>> com.datastax.driver.core.ControlConnection.reconnectInternal(ControlConnection.java:161)
>>>>> >> at
>>>>> com.datastax.driver.core.ControlConnection.connect(ControlConnection.java:77)
>>>>> >> at
>>>>> com.datastax.driver.core.Cluster$Manager.init(Cluster.java:890)
>>>>> >> at
>>>>> com.datastax.driver.core.Cluster$Manager.newSession(Cluster.java:910)
>>>>> >> at
>>>>> com.datastax.driver.core.Cluster$Manager.access$200(Cluster.java:806)
>>>>> >> at com.datastax.driver.core.Cluster.connect(Cluster.java:158)
>>>>> >> at
>>>>> cassandra.CassandraTestMinimized$delayedInit$body.apply(CassandraTestMinimized.scala:31)
>>>>> >> at scala.Function0$class.apply$mcV$sp(Function0.scala:40)
>>>>> >> at
>>>>> scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:12)
>>>>> >> at scala.App$$anonfun$main$1.apply(App.scala:71)
>>>>> >> at scala.App$$anonfun$main$1.apply(App.scala:71)
>>>>> >> at scala.collection.immutable.List.foreach(List.scala:318)
>>>>> >> at
>>>>> scala.collection.generic.TraversableForwarder$class.foreach(TraversableForwarder.scala:32)
>>>>> >> at scala.App$class.main(App.scala:71)
>>>>> >> at
>>>>> cassandra.CassandraTestMinimized$.main(CassandraTestMinimized.scala:5)
>>>>> >> at
>>>>> cassandra.CassandraTestMinimized.main(CassandraTestMinimized.scala)
>>>>> >
>>>>> > I believe you've tried the cassandra trunk branch? trunk is
>>>>> basically the future Cassandra 2.1 and the driver is currently unhappy
>>>>> because the replicate_on_write option has been removed in that version. I'm
>>>>> supposed to have fixed that on the driver 2.0 branch like 2 days ago so
>>>>> maybe you're also using a slightly old version of the driver sources in
>>>>> there? Or maybe I've screwed up my fix, I'll double check. But anyway, it
>>>>> would be overall simpler to test with the cassandra-2.0 branch of
>>>>> Cassandra, with which you shouldn't run into that.
>>>>> > --
>>>>> > Sylvain
>>>>>
>>>>> --
>>>>> Sorry this was sent from mobile. Will do less grammar and spell check
>>>>> than usual.
>>>>>
>>>>
>>>>
>>>
>>
>
Re: Performance problem with large wide row inserts using CQL
Posted by Peter Lin <wo...@gmail.com>.
my apologies Sylvain, I didn't mean to misquote you. I still feel that even
if someone is only going to use CQL, it is "worth it" to learn thrift.
In the interest of discussion, I looked at both jira tickets and I don't
see how that makes it so a developer can specify the name and value type
for a dynamic column.
https://issues.apache.org/jira/browse/CASSANDRA-6561
https://issues.apache.org/jira/browse/CASSANDRA-4851
Am I missing something? If the grammar for insert statements doesn't give
users the ability declare the name and value type, it means the developer
has to default name and value to bytes. In their code, they have to handle
that manually or build their own framework. I built my own framework, which
handles this for me. Developers can use what ever type they want for the
name or value in a dynamic column and the framework will handle it
appropriately.
To me, developers should take time to learn both and use both. I realize
it's more work to understand both and take time to read the code. Not
everyone is crazy enough spend time reading cassandra code base or spend
hundreds of hours studying hector and other cassandra clients. I will say
this, if I hadn't spend time studying cassandra and reading Hector code, I
wouldn't have been able to help one of DataStax customer port Hector to
.Net. I also wouldn't have been able to port Hector to C# natively in 3
months.
Rather than recommend people be lazy, it would be more useful to list the
pros/cons. To my knowledge, there isn't a good writeup on the pros/cons of
thrift and cql on cassandra.apache.org. I don't know if the DataStax docs
have a detailed write up of it, does it?
On Thu, Feb 20, 2014 at 12:46 PM, Sylvain Lebresne <sy...@datastax.com>wrote:
> On Thu, Feb 20, 2014 at 6:26 PM, Peter Lin <wo...@gmail.com> wrote:
>
>>
>> I disagree with the sentiment that "thrift is not worth the trouble".
>>
>
> Way to quote only part of my sentence and get mental on it. My full
> sentence was "it's probably not worth the trouble to start with thrift if
> you're gonna use CQL later".
>
>
>>
>> CQL and all SQL inspired dialects limit one's ability to use arbitrary
>> typed data in dynamic columns. With thrift it's easy and straight forward.
>> With CQL there is no way to tell Cassandra the type of the name and value
>> for a dynamic column. You can only set the default type. That means using a
>> "pure cql" approach you can deviate from the default type. Cassandra will
>> throw an exception indicating the type is different than the default type.
>>
>
>> Until such time that CQL abandons the shackles of SQL and adds the
>> ability to indicate the column and value type. Something like this
>>
>
>> insert into myColumnFamily(staticColumn1, staticColumn2, 20 as int,
>> dynamicColumn as string) into ('text1','text2',30.55 as double, 3500 as
>> long)
>>
>> This is one area where Thrift is superior to CQL. Having said that, it's
>> valid to use Cassandra "as if" it was a relational database, but then you'd
>> miss out on some of the unique features.
>>
>
> Man, if I had a nickel every time someone came on that mailing list
> pretending that something was possible with thrift and not CQL ... I will
> claim this: with CASSANDRA-6561 and CASSANDRA-4851 that just got in, there
> is *nothing* that thrift can do that CQL cannot. But well, what do I know
> about Cassandra.
>
> --
> Sylvain
>
>
>
>>
>>
>>
>>
>> On Thu, Feb 20, 2014 at 12:12 PM, Sylvain Lebresne <sy...@datastax.com>wrote:
>>
>>> On Thu, Feb 20, 2014 at 2:16 PM, Edward Capriolo <ed...@gmail.com>wrote:
>>>
>>>> For what it is worth you schema is simple and uses compact storage.
>>>> Thus you really dont need anything in cassandra 2.0 as far as i can tell.
>>>> You might be happier with a stable release like 1.2.something and just
>>>> hector or astyanax. You are really dealing with many issues you should not
>>>> have to just to protoype a simple cassandra app.
>>>
>>>
>>>
>>> Of course, if everyone was using that reasoning, no-one would ever test
>>> new features and report problems/suggest improvement. So thanks to anyone
>>> like Rüdiger that actually tries stuff and take the time to report problems
>>> when they think they encounter one. Keep at it, *you* are the one helping
>>> Cassandra to get better everyday.
>>>
>>> And you are also right Rüdiger that it's probably not worth the trouble
>>> to start with thrift if you're gonna use CQL later. And you definitively
>>> should use CQL, it is Cassandra's future.
>>>
>>> --
>>> Sylvain
>>>
>>>
>>>
>>>>
>>>> On Thursday, February 20, 2014, Sylvain Lebresne <sy...@datastax.com>
>>>> wrote:
>>>> >
>>>> >
>>>> >
>>>> > On Wed, Feb 19, 2014 at 9:38 PM, Rüdiger Klaehn <rk...@gmail.com>
>>>> wrote:
>>>> >>
>>>> >> I have cloned the cassandra repo, applied the patch, and built it.
>>>> But when I want to run the bechmark I get an exception. See below. I tried
>>>> with a non-managed dependency to
>>>> cassandra-driver-core-2.0.0-rc3-SNAPSHOT-jar-with-dependencies.jar, which I
>>>> compiled from source because I read that that might help. But that did not
>>>> make a difference.
>>>> >>
>>>> >> So currently I don't know how to give the patch a try. Any ideas?
>>>> >>
>>>> >> cheers,
>>>> >>
>>>> >> Rüdiger
>>>> >>
>>>> >> Exception in thread "main" java.lang.IllegalArgumentException:
>>>> replicate_on_write is not a column defined in this metadata
>>>> >> at
>>>> com.datastax.driver.core.ColumnDefinitions.getAllIdx(ColumnDefinitions.java:273)
>>>> >> at
>>>> com.datastax.driver.core.ColumnDefinitions.getFirstIdx(ColumnDefinitions.java:279)
>>>> >> at com.datastax.driver.core.Row.getBool(Row.java:117)
>>>> >> at
>>>> com.datastax.driver.core.TableMetadata$Options.<init>(TableMetadata.java:474)
>>>> >> at
>>>> com.datastax.driver.core.TableMetadata.build(TableMetadata.java:107)
>>>> >> at
>>>> com.datastax.driver.core.Metadata.buildTableMetadata(Metadata.java:128)
>>>> >> at
>>>> com.datastax.driver.core.Metadata.rebuildSchema(Metadata.java:89)
>>>> >> at
>>>> com.datastax.driver.core.ControlConnection.refreshSchema(ControlConnection.java:259)
>>>> >> at
>>>> com.datastax.driver.core.ControlConnection.tryConnect(ControlConnection.java:214)
>>>> >> at
>>>> com.datastax.driver.core.ControlConnection.reconnectInternal(ControlConnection.java:161)
>>>> >> at
>>>> com.datastax.driver.core.ControlConnection.connect(ControlConnection.java:77)
>>>> >> at
>>>> com.datastax.driver.core.Cluster$Manager.init(Cluster.java:890)
>>>> >> at
>>>> com.datastax.driver.core.Cluster$Manager.newSession(Cluster.java:910)
>>>> >> at
>>>> com.datastax.driver.core.Cluster$Manager.access$200(Cluster.java:806)
>>>> >> at com.datastax.driver.core.Cluster.connect(Cluster.java:158)
>>>> >> at
>>>> cassandra.CassandraTestMinimized$delayedInit$body.apply(CassandraTestMinimized.scala:31)
>>>> >> at scala.Function0$class.apply$mcV$sp(Function0.scala:40)
>>>> >> at
>>>> scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:12)
>>>> >> at scala.App$$anonfun$main$1.apply(App.scala:71)
>>>> >> at scala.App$$anonfun$main$1.apply(App.scala:71)
>>>> >> at scala.collection.immutable.List.foreach(List.scala:318)
>>>> >> at
>>>> scala.collection.generic.TraversableForwarder$class.foreach(TraversableForwarder.scala:32)
>>>> >> at scala.App$class.main(App.scala:71)
>>>> >> at
>>>> cassandra.CassandraTestMinimized$.main(CassandraTestMinimized.scala:5)
>>>> >> at
>>>> cassandra.CassandraTestMinimized.main(CassandraTestMinimized.scala)
>>>> >
>>>> > I believe you've tried the cassandra trunk branch? trunk is basically
>>>> the future Cassandra 2.1 and the driver is currently unhappy because the
>>>> replicate_on_write option has been removed in that version. I'm supposed to
>>>> have fixed that on the driver 2.0 branch like 2 days ago so maybe you're
>>>> also using a slightly old version of the driver sources in there? Or maybe
>>>> I've screwed up my fix, I'll double check. But anyway, it would be overall
>>>> simpler to test with the cassandra-2.0 branch of Cassandra, with which you
>>>> shouldn't run into that.
>>>> > --
>>>> > Sylvain
>>>>
>>>> --
>>>> Sorry this was sent from mobile. Will do less grammar and spell check
>>>> than usual.
>>>>
>>>
>>>
>>
>
Re: Performance problem with large wide row inserts using CQL
Posted by Edward Capriolo <ed...@gmail.com>.
The only thing you really can not do CQL3 loses some of the concept of CQL2
metadata, namedly the default validation and then column specific
validation.
In cassandra-cql we can say (butchering the syntax)
create column family x
DEFAULT_VALIDATOR = UTF8Type
columns named y are int
columns named z are string
You can do this in CQL.:
create table x (
rowkey blob,
column blob,
value blob,
primary key(rowkey,column) using compact storage ;
But you lost the concept of columns named y validate as int. Everything is
just blob as far as CQL understands it. That being said in the schema
presented nothing stops the user from implementing their design in either
"system"
On Thu, Feb 20, 2014 at 12:46 PM, Sylvain Lebresne <sy...@datastax.com>wrote:
> On Thu, Feb 20, 2014 at 6:26 PM, Peter Lin <wo...@gmail.com> wrote:
>
>>
>> I disagree with the sentiment that "thrift is not worth the trouble".
>>
>
> Way to quote only part of my sentence and get mental on it. My full
> sentence was "it's probably not worth the trouble to start with thrift if
> you're gonna use CQL later".
>
>
>>
>> CQL and all SQL inspired dialects limit one's ability to use arbitrary
>> typed data in dynamic columns. With thrift it's easy and straight forward.
>> With CQL there is no way to tell Cassandra the type of the name and value
>> for a dynamic column. You can only set the default type. That means using a
>> "pure cql" approach you can deviate from the default type. Cassandra will
>> throw an exception indicating the type is different than the default type.
>>
>
>> Until such time that CQL abandons the shackles of SQL and adds the
>> ability to indicate the column and value type. Something like this
>>
>
>> insert into myColumnFamily(staticColumn1, staticColumn2, 20 as int,
>> dynamicColumn as string) into ('text1','text2',30.55 as double, 3500 as
>> long)
>>
>> This is one area where Thrift is superior to CQL. Having said that, it's
>> valid to use Cassandra "as if" it was a relational database, but then you'd
>> miss out on some of the unique features.
>>
>
> Man, if I had a nickel every time someone came on that mailing list
> pretending that something was possible with thrift and not CQL ... I will
> claim this: with CASSANDRA-6561 and CASSANDRA-4851 that just got in, there
> is *nothing* that thrift can do that CQL cannot. But well, what do I know
> about Cassandra.
>
> --
> Sylvain
>
>
>
>>
>>
>>
>>
>> On Thu, Feb 20, 2014 at 12:12 PM, Sylvain Lebresne <sy...@datastax.com>wrote:
>>
>>> On Thu, Feb 20, 2014 at 2:16 PM, Edward Capriolo <ed...@gmail.com>wrote:
>>>
>>>> For what it is worth you schema is simple and uses compact storage.
>>>> Thus you really dont need anything in cassandra 2.0 as far as i can tell.
>>>> You might be happier with a stable release like 1.2.something and just
>>>> hector or astyanax. You are really dealing with many issues you should not
>>>> have to just to protoype a simple cassandra app.
>>>
>>>
>>>
>>> Of course, if everyone was using that reasoning, no-one would ever test
>>> new features and report problems/suggest improvement. So thanks to anyone
>>> like Rüdiger that actually tries stuff and take the time to report problems
>>> when they think they encounter one. Keep at it, *you* are the one helping
>>> Cassandra to get better everyday.
>>>
>>> And you are also right Rüdiger that it's probably not worth the trouble
>>> to start with thrift if you're gonna use CQL later. And you definitively
>>> should use CQL, it is Cassandra's future.
>>>
>>> --
>>> Sylvain
>>>
>>>
>>>
>>>>
>>>> On Thursday, February 20, 2014, Sylvain Lebresne <sy...@datastax.com>
>>>> wrote:
>>>> >
>>>> >
>>>> >
>>>> > On Wed, Feb 19, 2014 at 9:38 PM, Rüdiger Klaehn <rk...@gmail.com>
>>>> wrote:
>>>> >>
>>>> >> I have cloned the cassandra repo, applied the patch, and built it.
>>>> But when I want to run the bechmark I get an exception. See below. I tried
>>>> with a non-managed dependency to
>>>> cassandra-driver-core-2.0.0-rc3-SNAPSHOT-jar-with-dependencies.jar, which I
>>>> compiled from source because I read that that might help. But that did not
>>>> make a difference.
>>>> >>
>>>> >> So currently I don't know how to give the patch a try. Any ideas?
>>>> >>
>>>> >> cheers,
>>>> >>
>>>> >> Rüdiger
>>>> >>
>>>> >> Exception in thread "main" java.lang.IllegalArgumentException:
>>>> replicate_on_write is not a column defined in this metadata
>>>> >> at
>>>> com.datastax.driver.core.ColumnDefinitions.getAllIdx(ColumnDefinitions.java:273)
>>>> >> at
>>>> com.datastax.driver.core.ColumnDefinitions.getFirstIdx(ColumnDefinitions.java:279)
>>>> >> at com.datastax.driver.core.Row.getBool(Row.java:117)
>>>> >> at
>>>> com.datastax.driver.core.TableMetadata$Options.<init>(TableMetadata.java:474)
>>>> >> at
>>>> com.datastax.driver.core.TableMetadata.build(TableMetadata.java:107)
>>>> >> at
>>>> com.datastax.driver.core.Metadata.buildTableMetadata(Metadata.java:128)
>>>> >> at
>>>> com.datastax.driver.core.Metadata.rebuildSchema(Metadata.java:89)
>>>> >> at
>>>> com.datastax.driver.core.ControlConnection.refreshSchema(ControlConnection.java:259)
>>>> >> at
>>>> com.datastax.driver.core.ControlConnection.tryConnect(ControlConnection.java:214)
>>>> >> at
>>>> com.datastax.driver.core.ControlConnection.reconnectInternal(ControlConnection.java:161)
>>>> >> at
>>>> com.datastax.driver.core.ControlConnection.connect(ControlConnection.java:77)
>>>> >> at
>>>> com.datastax.driver.core.Cluster$Manager.init(Cluster.java:890)
>>>> >> at
>>>> com.datastax.driver.core.Cluster$Manager.newSession(Cluster.java:910)
>>>> >> at
>>>> com.datastax.driver.core.Cluster$Manager.access$200(Cluster.java:806)
>>>> >> at com.datastax.driver.core.Cluster.connect(Cluster.java:158)
>>>> >> at
>>>> cassandra.CassandraTestMinimized$delayedInit$body.apply(CassandraTestMinimized.scala:31)
>>>> >> at scala.Function0$class.apply$mcV$sp(Function0.scala:40)
>>>> >> at
>>>> scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:12)
>>>> >> at scala.App$$anonfun$main$1.apply(App.scala:71)
>>>> >> at scala.App$$anonfun$main$1.apply(App.scala:71)
>>>> >> at scala.collection.immutable.List.foreach(List.scala:318)
>>>> >> at
>>>> scala.collection.generic.TraversableForwarder$class.foreach(TraversableForwarder.scala:32)
>>>> >> at scala.App$class.main(App.scala:71)
>>>> >> at
>>>> cassandra.CassandraTestMinimized$.main(CassandraTestMinimized.scala:5)
>>>> >> at
>>>> cassandra.CassandraTestMinimized.main(CassandraTestMinimized.scala)
>>>> >
>>>> > I believe you've tried the cassandra trunk branch? trunk is basically
>>>> the future Cassandra 2.1 and the driver is currently unhappy because the
>>>> replicate_on_write option has been removed in that version. I'm supposed to
>>>> have fixed that on the driver 2.0 branch like 2 days ago so maybe you're
>>>> also using a slightly old version of the driver sources in there? Or maybe
>>>> I've screwed up my fix, I'll double check. But anyway, it would be overall
>>>> simpler to test with the cassandra-2.0 branch of Cassandra, with which you
>>>> shouldn't run into that.
>>>> > --
>>>> > Sylvain
>>>>
>>>> --
>>>> Sorry this was sent from mobile. Will do less grammar and spell check
>>>> than usual.
>>>>
>>>
>>>
>>
>
Re: Performance problem with large wide row inserts using CQL
Posted by Sylvain Lebresne <sy...@datastax.com>.
On Thu, Feb 20, 2014 at 6:26 PM, Peter Lin <wo...@gmail.com> wrote:
>
> I disagree with the sentiment that "thrift is not worth the trouble".
>
Way to quote only part of my sentence and get mental on it. My full
sentence was "it's probably not worth the trouble to start with thrift if
you're gonna use CQL later".
>
> CQL and all SQL inspired dialects limit one's ability to use arbitrary
> typed data in dynamic columns. With thrift it's easy and straight forward.
> With CQL there is no way to tell Cassandra the type of the name and value
> for a dynamic column. You can only set the default type. That means using a
> "pure cql" approach you can deviate from the default type. Cassandra will
> throw an exception indicating the type is different than the default type.
>
> Until such time that CQL abandons the shackles of SQL and adds the ability
> to indicate the column and value type. Something like this
>
> insert into myColumnFamily(staticColumn1, staticColumn2, 20 as int,
> dynamicColumn as string) into ('text1','text2',30.55 as double, 3500 as
> long)
>
> This is one area where Thrift is superior to CQL. Having said that, it's
> valid to use Cassandra "as if" it was a relational database, but then you'd
> miss out on some of the unique features.
>
Man, if I had a nickel every time someone came on that mailing list
pretending that something was possible with thrift and not CQL ... I will
claim this: with CASSANDRA-6561 and CASSANDRA-4851 that just got in, there
is *nothing* that thrift can do that CQL cannot. But well, what do I know
about Cassandra.
--
Sylvain
>
>
>
>
> On Thu, Feb 20, 2014 at 12:12 PM, Sylvain Lebresne <sy...@datastax.com>wrote:
>
>> On Thu, Feb 20, 2014 at 2:16 PM, Edward Capriolo <ed...@gmail.com>wrote:
>>
>>> For what it is worth you schema is simple and uses compact storage. Thus
>>> you really dont need anything in cassandra 2.0 as far as i can tell. You
>>> might be happier with a stable release like 1.2.something and just hector
>>> or astyanax. You are really dealing with many issues you should not have to
>>> just to protoype a simple cassandra app.
>>
>>
>>
>> Of course, if everyone was using that reasoning, no-one would ever test
>> new features and report problems/suggest improvement. So thanks to anyone
>> like Rüdiger that actually tries stuff and take the time to report problems
>> when they think they encounter one. Keep at it, *you* are the one helping
>> Cassandra to get better everyday.
>>
>> And you are also right Rüdiger that it's probably not worth the trouble
>> to start with thrift if you're gonna use CQL later. And you definitively
>> should use CQL, it is Cassandra's future.
>>
>> --
>> Sylvain
>>
>>
>>
>>>
>>> On Thursday, February 20, 2014, Sylvain Lebresne <sy...@datastax.com>
>>> wrote:
>>> >
>>> >
>>> >
>>> > On Wed, Feb 19, 2014 at 9:38 PM, Rüdiger Klaehn <rk...@gmail.com>
>>> wrote:
>>> >>
>>> >> I have cloned the cassandra repo, applied the patch, and built it.
>>> But when I want to run the bechmark I get an exception. See below. I tried
>>> with a non-managed dependency to
>>> cassandra-driver-core-2.0.0-rc3-SNAPSHOT-jar-with-dependencies.jar, which I
>>> compiled from source because I read that that might help. But that did not
>>> make a difference.
>>> >>
>>> >> So currently I don't know how to give the patch a try. Any ideas?
>>> >>
>>> >> cheers,
>>> >>
>>> >> Rüdiger
>>> >>
>>> >> Exception in thread "main" java.lang.IllegalArgumentException:
>>> replicate_on_write is not a column defined in this metadata
>>> >> at
>>> com.datastax.driver.core.ColumnDefinitions.getAllIdx(ColumnDefinitions.java:273)
>>> >> at
>>> com.datastax.driver.core.ColumnDefinitions.getFirstIdx(ColumnDefinitions.java:279)
>>> >> at com.datastax.driver.core.Row.getBool(Row.java:117)
>>> >> at
>>> com.datastax.driver.core.TableMetadata$Options.<init>(TableMetadata.java:474)
>>> >> at
>>> com.datastax.driver.core.TableMetadata.build(TableMetadata.java:107)
>>> >> at
>>> com.datastax.driver.core.Metadata.buildTableMetadata(Metadata.java:128)
>>> >> at
>>> com.datastax.driver.core.Metadata.rebuildSchema(Metadata.java:89)
>>> >> at
>>> com.datastax.driver.core.ControlConnection.refreshSchema(ControlConnection.java:259)
>>> >> at
>>> com.datastax.driver.core.ControlConnection.tryConnect(ControlConnection.java:214)
>>> >> at
>>> com.datastax.driver.core.ControlConnection.reconnectInternal(ControlConnection.java:161)
>>> >> at
>>> com.datastax.driver.core.ControlConnection.connect(ControlConnection.java:77)
>>> >> at com.datastax.driver.core.Cluster$Manager.init(Cluster.java:890)
>>> >> at
>>> com.datastax.driver.core.Cluster$Manager.newSession(Cluster.java:910)
>>> >> at
>>> com.datastax.driver.core.Cluster$Manager.access$200(Cluster.java:806)
>>> >> at com.datastax.driver.core.Cluster.connect(Cluster.java:158)
>>> >> at
>>> cassandra.CassandraTestMinimized$delayedInit$body.apply(CassandraTestMinimized.scala:31)
>>> >> at scala.Function0$class.apply$mcV$sp(Function0.scala:40)
>>> >> at
>>> scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:12)
>>> >> at scala.App$$anonfun$main$1.apply(App.scala:71)
>>> >> at scala.App$$anonfun$main$1.apply(App.scala:71)
>>> >> at scala.collection.immutable.List.foreach(List.scala:318)
>>> >> at
>>> scala.collection.generic.TraversableForwarder$class.foreach(TraversableForwarder.scala:32)
>>> >> at scala.App$class.main(App.scala:71)
>>> >> at
>>> cassandra.CassandraTestMinimized$.main(CassandraTestMinimized.scala:5)
>>> >> at
>>> cassandra.CassandraTestMinimized.main(CassandraTestMinimized.scala)
>>> >
>>> > I believe you've tried the cassandra trunk branch? trunk is basically
>>> the future Cassandra 2.1 and the driver is currently unhappy because the
>>> replicate_on_write option has been removed in that version. I'm supposed to
>>> have fixed that on the driver 2.0 branch like 2 days ago so maybe you're
>>> also using a slightly old version of the driver sources in there? Or maybe
>>> I've screwed up my fix, I'll double check. But anyway, it would be overall
>>> simpler to test with the cassandra-2.0 branch of Cassandra, with which you
>>> shouldn't run into that.
>>> > --
>>> > Sylvain
>>>
>>> --
>>> Sorry this was sent from mobile. Will do less grammar and spell check
>>> than usual.
>>>
>>
>>
>
Re: Performance problem with large wide row inserts using CQL
Posted by Peter Lin <wo...@gmail.com>.
I disagree with the sentiment that "thrift is not worth the trouble".
CQL and all SQL inspired dialects limit one's ability to use arbitrary
typed data in dynamic columns. With thrift it's easy and straight forward.
With CQL there is no way to tell Cassandra the type of the name and value
for a dynamic column. You can only set the default type. That means using a
"pure cql" approach you can deviate from the default type. Cassandra will
throw an exception indicating the type is different than the default type.
Until such time that CQL abandons the shackles of SQL and adds the ability
to indicate the column and value type. Something like this
insert into myColumnFamily(staticColumn1, staticColumn2, 20 as int,
dynamicColumn as string) into ('text1','text2',30.55 as double, 3500 as
long)
This is one area where Thrift is superior to CQL. Having said that, it's
valid to use Cassandra "as if" it was a relational database, but then you'd
miss out on some of the unique features.
On Thu, Feb 20, 2014 at 12:12 PM, Sylvain Lebresne <sy...@datastax.com>wrote:
> On Thu, Feb 20, 2014 at 2:16 PM, Edward Capriolo <ed...@gmail.com>wrote:
>
>> For what it is worth you schema is simple and uses compact storage. Thus
>> you really dont need anything in cassandra 2.0 as far as i can tell. You
>> might be happier with a stable release like 1.2.something and just hector
>> or astyanax. You are really dealing with many issues you should not have to
>> just to protoype a simple cassandra app.
>
>
>
> Of course, if everyone was using that reasoning, no-one would ever test
> new features and report problems/suggest improvement. So thanks to anyone
> like Rüdiger that actually tries stuff and take the time to report problems
> when they think they encounter one. Keep at it, *you* are the one helping
> Cassandra to get better everyday.
>
> And you are also right Rüdiger that it's probably not worth the trouble to
> start with thrift if you're gonna use CQL later. And you definitively
> should use CQL, it is Cassandra's future.
>
> --
> Sylvain
>
>
>
>>
>> On Thursday, February 20, 2014, Sylvain Lebresne <sy...@datastax.com>
>> wrote:
>> >
>> >
>> >
>> > On Wed, Feb 19, 2014 at 9:38 PM, Rüdiger Klaehn <rk...@gmail.com>
>> wrote:
>> >>
>> >> I have cloned the cassandra repo, applied the patch, and built it. But
>> when I want to run the bechmark I get an exception. See below. I tried with
>> a non-managed dependency to
>> cassandra-driver-core-2.0.0-rc3-SNAPSHOT-jar-with-dependencies.jar, which I
>> compiled from source because I read that that might help. But that did not
>> make a difference.
>> >>
>> >> So currently I don't know how to give the patch a try. Any ideas?
>> >>
>> >> cheers,
>> >>
>> >> Rüdiger
>> >>
>> >> Exception in thread "main" java.lang.IllegalArgumentException:
>> replicate_on_write is not a column defined in this metadata
>> >> at
>> com.datastax.driver.core.ColumnDefinitions.getAllIdx(ColumnDefinitions.java:273)
>> >> at
>> com.datastax.driver.core.ColumnDefinitions.getFirstIdx(ColumnDefinitions.java:279)
>> >> at com.datastax.driver.core.Row.getBool(Row.java:117)
>> >> at
>> com.datastax.driver.core.TableMetadata$Options.<init>(TableMetadata.java:474)
>> >> at
>> com.datastax.driver.core.TableMetadata.build(TableMetadata.java:107)
>> >> at
>> com.datastax.driver.core.Metadata.buildTableMetadata(Metadata.java:128)
>> >> at
>> com.datastax.driver.core.Metadata.rebuildSchema(Metadata.java:89)
>> >> at
>> com.datastax.driver.core.ControlConnection.refreshSchema(ControlConnection.java:259)
>> >> at
>> com.datastax.driver.core.ControlConnection.tryConnect(ControlConnection.java:214)
>> >> at
>> com.datastax.driver.core.ControlConnection.reconnectInternal(ControlConnection.java:161)
>> >> at
>> com.datastax.driver.core.ControlConnection.connect(ControlConnection.java:77)
>> >> at com.datastax.driver.core.Cluster$Manager.init(Cluster.java:890)
>> >> at
>> com.datastax.driver.core.Cluster$Manager.newSession(Cluster.java:910)
>> >> at
>> com.datastax.driver.core.Cluster$Manager.access$200(Cluster.java:806)
>> >> at com.datastax.driver.core.Cluster.connect(Cluster.java:158)
>> >> at
>> cassandra.CassandraTestMinimized$delayedInit$body.apply(CassandraTestMinimized.scala:31)
>> >> at scala.Function0$class.apply$mcV$sp(Function0.scala:40)
>> >> at
>> scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:12)
>> >> at scala.App$$anonfun$main$1.apply(App.scala:71)
>> >> at scala.App$$anonfun$main$1.apply(App.scala:71)
>> >> at scala.collection.immutable.List.foreach(List.scala:318)
>> >> at
>> scala.collection.generic.TraversableForwarder$class.foreach(TraversableForwarder.scala:32)
>> >> at scala.App$class.main(App.scala:71)
>> >> at
>> cassandra.CassandraTestMinimized$.main(CassandraTestMinimized.scala:5)
>> >> at
>> cassandra.CassandraTestMinimized.main(CassandraTestMinimized.scala)
>> >
>> > I believe you've tried the cassandra trunk branch? trunk is basically
>> the future Cassandra 2.1 and the driver is currently unhappy because the
>> replicate_on_write option has been removed in that version. I'm supposed to
>> have fixed that on the driver 2.0 branch like 2 days ago so maybe you're
>> also using a slightly old version of the driver sources in there? Or maybe
>> I've screwed up my fix, I'll double check. But anyway, it would be overall
>> simpler to test with the cassandra-2.0 branch of Cassandra, with which you
>> shouldn't run into that.
>> > --
>> > Sylvain
>>
>> --
>> Sorry this was sent from mobile. Will do less grammar and spell check
>> than usual.
>>
>
>
Re: Performance problem with large wide row inserts using CQL
Posted by Edward Capriolo <ed...@gmail.com>.
Hopefully in 3 years no one will be calling your schema 'legacy' and 'not
suggested' like they do with mine.
On Thursday, February 20, 2014, Laing, Michael <mi...@nytimes.com>
wrote:
> Just to add my 2 cents...
> We are very happy CQL users, running in production.
> I have had no problems modeling whatever I have needed to, including
problems similar to the examples set forth previously, in CQL.
> Personally I think it is an excellent improvement to Cassandra, and we
have no intentions to ever look back to thrift.
> Michael Laing
> Systems Architect
> NYTimes
>
> On Thu, Feb 20, 2014 at 7:49 PM, Edward Capriolo <ed...@gmail.com>
wrote:
>>
>>
>> On Thursday, February 20, 2014, Robert Coli <rc...@eventbrite.com> wrote:
>> > On Thu, Feb 20, 2014 at 9:12 AM, Sylvain Lebresne <sy...@datastax.com>
wrote:
>> >>
>> >> Of course, if everyone was using that reasoning, no-one would ever
test new features and report problems/suggest improvement. So thanks to
anyone like Rüdiger that actually tries stuff and take the time to report
problems when they think they encounter one. Keep at it, *you* are the one
helping Cassandra to get better everyday.
>> >
>> >
>> > Perhaps people who are prototyping their first application with a
piece of software are not the ideal people to beta test it?
>> >
>> > The people catching new version bullets for the community should be
experienced operators choosing to do so in development and staging
environments.
>> > The current paradigm ensures that new users have to deal with
Cassandra problems that interfere with their prototyping process and
initial production deploy, presumably getting a very bad initial impression
of Cassandra in the process.
>> > =Rob
>> >
>>
>> You would be surprised how many people pick software a of software b
based on initial impressions.
>>
>> The reason I ended up choosing cassandra over hbase mostly boilded down
to c* being easy to set up and not crashing. If it took us say 3 days to
stand up a cassandra cluster and do the hello world thing i might very well
be a voldemort user!
>>
>>
>>
>>
>>
>> --
>> Sorry this was sent from mobile. Will do less grammar and spell check
than usual.
>
>
--
Sorry this was sent from mobile. Will do less grammar and spell check than
usual.
Re: Performance problem with large wide row inserts using CQL
Posted by "Laing, Michael" <mi...@nytimes.com>.
Just to add my 2 cents...
We are very happy CQL users, running in production.
I have had no problems modeling whatever I have needed to, including
problems similar to the examples set forth previously, in CQL.
Personally I think it is an excellent improvement to Cassandra, and we have
no intentions to ever look back to thrift.
Michael Laing
Systems Architect
NYTimes
On Thu, Feb 20, 2014 at 7:49 PM, Edward Capriolo <ed...@gmail.com>wrote:
>
>
> On Thursday, February 20, 2014, Robert Coli <rc...@eventbrite.com> wrote:
> > On Thu, Feb 20, 2014 at 9:12 AM, Sylvain Lebresne <sy...@datastax.com>
> wrote:
> >>
> >> Of course, if everyone was using that reasoning, no-one would ever test
> new features and report problems/suggest improvement. So thanks to anyone
> like Rüdiger that actually tries stuff and take the time to report problems
> when they think they encounter one. Keep at it, *you* are the one helping
> Cassandra to get better everyday.
> >
> >
> > Perhaps people who are prototyping their first application with a piece
> of software are not the ideal people to beta test it?
> >
> > The people catching new version bullets for the community should be
> experienced operators choosing to do so in development and staging
> environments.
> > The current paradigm ensures that new users have to deal with Cassandra
> problems that interfere with their prototyping process and initial
> production deploy, presumably getting a very bad initial impression of
> Cassandra in the process.
> > =Rob
> >
>
> You would be surprised how many people pick software a of software b based
> on initial impressions.
>
> The reason I ended up choosing cassandra over hbase mostly boilded down to
> c* being easy to set up and not crashing. If it took us say 3 days to stand
> up a cassandra cluster and do the hello world thing i might very well be a
> voldemort user!
>
>
>
>
>
>
> --
> Sorry this was sent from mobile. Will do less grammar and spell check than
> usual.
>
Re: Performance problem with large wide row inserts using CQL
Posted by Edward Capriolo <ed...@gmail.com>.
On Thursday, February 20, 2014, Robert Coli <rc...@eventbrite.com> wrote:
> On Thu, Feb 20, 2014 at 9:12 AM, Sylvain Lebresne <sy...@datastax.com>
wrote:
>>
>> Of course, if everyone was using that reasoning, no-one would ever test
new features and report problems/suggest improvement. So thanks to anyone
like Rüdiger that actually tries stuff and take the time to report problems
when they think they encounter one. Keep at it, *you* are the one helping
Cassandra to get better everyday.
>
>
> Perhaps people who are prototyping their first application with a piece
of software are not the ideal people to beta test it?
>
> The people catching new version bullets for the community should be
experienced operators choosing to do so in development and staging
environments.
> The current paradigm ensures that new users have to deal with Cassandra
problems that interfere with their prototyping process and initial
production deploy, presumably getting a very bad initial impression of
Cassandra in the process.
> =Rob
>
You would be surprised how many people pick software a of software b based
on initial impressions.
The reason I ended up choosing cassandra over hbase mostly boilded down to
c* being easy to set up and not crashing. If it took us say 3 days to stand
up a cassandra cluster and do the hello world thing i might very well be a
voldemort user!
--
Sorry this was sent from mobile. Will do less grammar and spell check than
usual.
Re: Performance problem with large wide row inserts using CQL
Posted by Robert Coli <rc...@eventbrite.com>.
On Thu, Feb 20, 2014 at 9:12 AM, Sylvain Lebresne <sy...@datastax.com>wrote:
> Of course, if everyone was using that reasoning, no-one would ever test
> new features and report problems/suggest improvement. So thanks to anyone
> like Rüdiger that actually tries stuff and take the time to report problems
> when they think they encounter one. Keep at it, *you* are the one helping
> Cassandra to get better everyday.
>
Perhaps people who are prototyping their first application with a piece of
software are not the ideal people to beta test it?
The people catching new version bullets for the community should be
experienced operators choosing to do so in development and staging
environments.
The current paradigm ensures that new users have to deal with Cassandra
problems that interfere with their prototyping process and initial
production deploy, presumably getting a very bad initial impression of
Cassandra in the process.
=Rob
Re: Performance problem with large wide row inserts using CQL
Posted by Edward Capriolo <ed...@gmail.com>.
Just read this. i did not mean to offend or start a debate. Generally when
people ask me for help I give them the simplest option I know that works.
It pains be to watch new users struggling with incompatible drivers and
bugs.
On Thursday, February 20, 2014, Sylvain Lebresne <sy...@datastax.com>
wrote:
> On Thu, Feb 20, 2014 at 2:16 PM, Edward Capriolo <ed...@gmail.com>
wrote:
>>
>> For what it is worth you schema is simple and uses compact storage. Thus
you really dont need anything in cassandra 2.0 as far as i can tell. You
might be happier with a stable release like 1.2.something and just hector
or astyanax. You are really dealing with many issues you should not have to
just to protoype a simple cassandra app.
>
> Of course, if everyone was using that reasoning, no-one would ever test
new features and report problems/suggest improvement. So thanks to anyone
like Rüdiger that actually tries stuff and take the time to report problems
when they think they encounter one. Keep at it, *you* are the one helping
Cassandra to get better everyday.
> And you are also right Rüdiger that it's probably not worth the trouble
to start with thrift if you're gonna use CQL later. And you definitively
should use CQL, it is Cassandra's future.
> --
> Sylvain
>
>
>
> On Thursday, February 20, 2014, Sylvain Lebresne <sy...@datastax.com>
wrote:
>>
>>
>>
>> On Wed, Feb 19, 2014 at 9:38 PM, Rüdiger Klaehn <rk...@gmail.com>
wrote:
>>>
>>> I have cloned the cassandra repo, applied the patch, and built it. But
when I want to run the bechmark I get an exception. See below. I tried with
a non-managed dependency to
cassandra-driver-core-2.0.0-rc3-SNAPSHOT-jar-with-dependencies.jar, which I
compiled from source because I read that that might help. But that did not
make a difference.
>>>
>>> So currently I don't know how to give the patch a try. Any ideas?
>>>
>>> cheers,
>>>
>>> Rüdiger
>>>
>>> Exception in thread "main" java.lang.IllegalArgumentException:
replicate_on_write is not a column defined in this metadata
>>> at
com.datastax.driver.core.ColumnDefinitions.getAllIdx(ColumnDefinitions.java:273)
>>> at
com.datastax.driver.core.ColumnDefinitions.getFirstIdx(ColumnDefinitions.java:279)
>>> at com.datastax.driver.core.Row.getBool(Row.java:117)
>>> at
com.datastax.driver.core.TableMetadata$Options.<init>(TableMetadata.java:474)
>>> at
com.datastax.driver.core.TableMetadata.build(TableMetadata.java:107)
>>> at
com.datastax.driver.core.Metadata.buildTableMetadata(Metadata.java:128)
>>> at com.datastax.driver.core.Metadata.rebuildSchema(Metadata.java:89)
>>> at
com.datastax.driver.core.ControlConnection.refreshSchema(ControlConnection.java:259)
>>> at
com.datastax.driver.core.ControlConnection.tryConnect(ControlConnection.java:214)
>>> at
com.datastax.driver.core.ControlConnection.reconnectInternal(ControlConnection.java:161)
>>> at
com.datastax.driver.core.ControlConnection.connect(ControlConnection.java:77)
>>> at com.datastax.driver.core.Cluster$Manager.init(Cluster.java:890)
>>> at
com.datastax.driver.core.Cluster$Manager.newSession(Cluster.java:910)
>>> at
com.datastax.driver.core.Cluster$Manager.access$200(Cluster.java:806)
>>> at com.datastax.driver.core.Cluster.connect(Cluster.java:158)
>>> at
cassandra.CassandraTestMinimized$delayedInit$body.apply(CassandraTestMinimized.scala:31)
>>> at scala.Function0$class.apply$mcV$sp(Function0.scala:40)
>>> at
scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:12)
>>> at scala.App$$anonfun$main$1.apply(App.scala:71)
>>> at scala.App$$anonfun$main$1.apply(App.scala:71)
>>> at scala.collection.immutable.List.foreach(List.scala:318)
>>> at
scala.collection.generic.TraversableForwarder$class.foreach(TraversableForwarder.scala:32)
>>> at scala.App$class.main(App.scala:71)
>>> at cassandra.CassandraTestMinimized$.main(Cassa
>
--
Sorry this was sent from mobile. Will do less grammar and spell check than
usual.
Re: Performance problem with large wide row inserts using CQL
Posted by Sylvain Lebresne <sy...@datastax.com>.
On Thu, Feb 20, 2014 at 2:16 PM, Edward Capriolo <ed...@gmail.com>wrote:
> For what it is worth you schema is simple and uses compact storage. Thus
> you really dont need anything in cassandra 2.0 as far as i can tell. You
> might be happier with a stable release like 1.2.something and just hector
> or astyanax. You are really dealing with many issues you should not have to
> just to protoype a simple cassandra app.
Of course, if everyone was using that reasoning, no-one would ever test new
features and report problems/suggest improvement. So thanks to anyone like
Rüdiger that actually tries stuff and take the time to report problems when
they think they encounter one. Keep at it, *you* are the one helping
Cassandra to get better everyday.
And you are also right Rüdiger that it's probably not worth the trouble to
start with thrift if you're gonna use CQL later. And you definitively
should use CQL, it is Cassandra's future.
--
Sylvain
>
> On Thursday, February 20, 2014, Sylvain Lebresne <sy...@datastax.com>
> wrote:
> >
> >
> >
> > On Wed, Feb 19, 2014 at 9:38 PM, Rüdiger Klaehn <rk...@gmail.com>
> wrote:
> >>
> >> I have cloned the cassandra repo, applied the patch, and built it. But
> when I want to run the bechmark I get an exception. See below. I tried with
> a non-managed dependency to
> cassandra-driver-core-2.0.0-rc3-SNAPSHOT-jar-with-dependencies.jar, which I
> compiled from source because I read that that might help. But that did not
> make a difference.
> >>
> >> So currently I don't know how to give the patch a try. Any ideas?
> >>
> >> cheers,
> >>
> >> Rüdiger
> >>
> >> Exception in thread "main" java.lang.IllegalArgumentException:
> replicate_on_write is not a column defined in this metadata
> >> at
> com.datastax.driver.core.ColumnDefinitions.getAllIdx(ColumnDefinitions.java:273)
> >> at
> com.datastax.driver.core.ColumnDefinitions.getFirstIdx(ColumnDefinitions.java:279)
> >> at com.datastax.driver.core.Row.getBool(Row.java:117)
> >> at
> com.datastax.driver.core.TableMetadata$Options.<init>(TableMetadata.java:474)
> >> at
> com.datastax.driver.core.TableMetadata.build(TableMetadata.java:107)
> >> at
> com.datastax.driver.core.Metadata.buildTableMetadata(Metadata.java:128)
> >> at com.datastax.driver.core.Metadata.rebuildSchema(Metadata.java:89)
> >> at
> com.datastax.driver.core.ControlConnection.refreshSchema(ControlConnection.java:259)
> >> at
> com.datastax.driver.core.ControlConnection.tryConnect(ControlConnection.java:214)
> >> at
> com.datastax.driver.core.ControlConnection.reconnectInternal(ControlConnection.java:161)
> >> at
> com.datastax.driver.core.ControlConnection.connect(ControlConnection.java:77)
> >> at com.datastax.driver.core.Cluster$Manager.init(Cluster.java:890)
> >> at
> com.datastax.driver.core.Cluster$Manager.newSession(Cluster.java:910)
> >> at
> com.datastax.driver.core.Cluster$Manager.access$200(Cluster.java:806)
> >> at com.datastax.driver.core.Cluster.connect(Cluster.java:158)
> >> at
> cassandra.CassandraTestMinimized$delayedInit$body.apply(CassandraTestMinimized.scala:31)
> >> at scala.Function0$class.apply$mcV$sp(Function0.scala:40)
> >> at
> scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:12)
> >> at scala.App$$anonfun$main$1.apply(App.scala:71)
> >> at scala.App$$anonfun$main$1.apply(App.scala:71)
> >> at scala.collection.immutable.List.foreach(List.scala:318)
> >> at
> scala.collection.generic.TraversableForwarder$class.foreach(TraversableForwarder.scala:32)
> >> at scala.App$class.main(App.scala:71)
> >> at
> cassandra.CassandraTestMinimized$.main(CassandraTestMinimized.scala:5)
> >> at
> cassandra.CassandraTestMinimized.main(CassandraTestMinimized.scala)
> >
> > I believe you've tried the cassandra trunk branch? trunk is basically
> the future Cassandra 2.1 and the driver is currently unhappy because the
> replicate_on_write option has been removed in that version. I'm supposed to
> have fixed that on the driver 2.0 branch like 2 days ago so maybe you're
> also using a slightly old version of the driver sources in there? Or maybe
> I've screwed up my fix, I'll double check. But anyway, it would be overall
> simpler to test with the cassandra-2.0 branch of Cassandra, with which you
> shouldn't run into that.
> > --
> > Sylvain
>
> --
> Sorry this was sent from mobile. Will do less grammar and spell check than
> usual.
>
Re: Performance problem with large wide row inserts using CQL
Posted by Edward Capriolo <ed...@gmail.com>.
Dont worry there will be plenty of time to upgrade to 2.0 or 2.1 later. It
is an easy upgrade path an you will likely do it 2-4 tmes a year. Dont
chose the latest and gteatest now thnking that you are future proofing. In
reality you are volunteering as a beta tester.
On Thursday, February 20, 2014, Edward Capriolo <ed...@gmail.com>
wrote:
> For what it is worth you schema is simple and uses compact storage. Thus
you really dont need anything in cassandra 2.0 as far as i can tell. You
might be happier with a stable release like 1.2.something and just hector
or astyanax. You are really dealing with many issues you should not have to
just to protoype a simple cassandra app.
>
> On Thursday, February 20, 2014, Sylvain Lebresne <sy...@datastax.com>
wrote:
>>
>>
>>
>> On Wed, Feb 19, 2014 at 9:38 PM, Rüdiger Klaehn <rk...@gmail.com>
wrote:
>>>
>>> I have cloned the cassandra repo, applied the patch, and built it. But
when I want to run the bechmark I get an exception. See below. I tried with
a non-managed dependency to
cassandra-driver-core-2.0.0-rc3-SNAPSHOT-jar-with-dependencies.jar, which I
compiled from source because I read that that might help. But that did not
make a difference.
>>>
>>> So currently I don't know how to give the patch a try. Any ideas?
>>>
>>> cheers,
>>>
>>> Rüdiger
>>>
>>> Exception in thread "main" java.lang.IllegalArgumentException:
replicate_on_write is not a column defined in this metadata
>>> at
com.datastax.driver.core.ColumnDefinitions.getAllIdx(ColumnDefinitions.java:273)
>>> at
com.datastax.driver.core.ColumnDefinitions.getFirstIdx(ColumnDefinitions.java:279)
>>> at com.datastax.driver.core.Row.getBool(Row.java:117)
>>> at
com.datastax.driver.core.TableMetadata$Options.<init>(TableMetadata.java:474)
>>> at
com.datastax.driver.core.TableMetadata.build(TableMetadata.java:107)
>>> at
com.datastax.driver.core.Metadata.buildTableMetadata(Metadata.java:128)
>>> at com.datastax.driver.core.Metadata.rebuildSchema(Metadata.java:89)
>>> at
com.datastax.driver.core.ControlConnection.refreshSchema(ControlConnection.java:259)
>>> at
com.datastax.driver.core.ControlConnection.tryConnect(ControlConnection.java:214)
>>> at
com.datastax.driver.core.ControlConnection.reconnectInternal(ControlConnection.java:161)
>>> at
com.datastax.driver.core.ControlConnection.connect(ControlConnection.java:77)
>>> at com.datastax.driver.core.Cluster$Manager.init(Cluster.java:890)
>>> at
com.datastax.driver.core.Cluster$Manager.newSession(Cluster.java:910)
>>> at
com.datastax.driver.core.Cluster$Manager.access$200(Cluster.java:806)
>>> at com.datastax.driver.core.Cluster.connect(Cluster.java:158)
>>> at
cassandra.CassandraTestMinimized$delayedInit$body.apply(CassandraTestMinimized.scala:31)
>>> at scala.Function0$class.apply$mcV$sp(Function0.scala:40)
>>> at
scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:12)
>>> at scala.App$$anonfun$main$1.apply(App.scala:71)
>>> at scala.App$$anonfun$main$1.apply(App.scala:71)
>>> at scala.collection.immutable.List.foreach(List.scala:318)
>>> at
scala.collection.generic.TraversableForwarder$class.foreach(TraversableForwarder.scala:32)
>>> at scala.App$class.main(App.scala:71)
>>> at
cassandra.CassandraTestMinimized$.main(CassandraTestMinimized.scala:5)
>>> at
cassandra.CassandraTestMinimized.main(CassandraTestMinimized.scala)
>>
>> I believe you've tried the cassandra trunk branch? trunk is basically
the future Cassandra 2.1 and the driver is currently unhappy because the
replicate_on_write option has been removed in that version. I'm supposed to
have fixed that on the driver 2.0 branch like 2 days ago so maybe you're
also using a slightly old version of the driver sources in there? Or maybe
I've screwed up my fix, I'll double check. But anyway, it would be overall
simpler to test with the cassandra-2.0 branch of Cassandra, with which you
shouldn't run into that.
>> --
>> Sylvain
>
> --
> Sorry this was sent from mobile. Will do less grammar and spell check
than usual.
>
--
Sorry this was sent from mobile. Will do less grammar and spell check than
usual.
Re: Performance problem with large wide row inserts using CQL
Posted by Edward Capriolo <ed...@gmail.com>.
For what it is worth you schema is simple and uses compact storage. Thus
you really dont need anything in cassandra 2.0 as far as i can tell. You
might be happier with a stable release like 1.2.something and just hector
or astyanax. You are really dealing with many issues you should not have to
just to protoype a simple cassandra app.
On Thursday, February 20, 2014, Sylvain Lebresne <sy...@datastax.com>
wrote:
>
>
>
> On Wed, Feb 19, 2014 at 9:38 PM, Rüdiger Klaehn <rk...@gmail.com> wrote:
>>
>> I have cloned the cassandra repo, applied the patch, and built it. But
when I want to run the bechmark I get an exception. See below. I tried with
a non-managed dependency to
cassandra-driver-core-2.0.0-rc3-SNAPSHOT-jar-with-dependencies.jar, which I
compiled from source because I read that that might help. But that did not
make a difference.
>>
>> So currently I don't know how to give the patch a try. Any ideas?
>>
>> cheers,
>>
>> Rüdiger
>>
>> Exception in thread "main" java.lang.IllegalArgumentException:
replicate_on_write is not a column defined in this metadata
>> at
com.datastax.driver.core.ColumnDefinitions.getAllIdx(ColumnDefinitions.java:273)
>> at
com.datastax.driver.core.ColumnDefinitions.getFirstIdx(ColumnDefinitions.java:279)
>> at com.datastax.driver.core.Row.getBool(Row.java:117)
>> at
com.datastax.driver.core.TableMetadata$Options.<init>(TableMetadata.java:474)
>> at
com.datastax.driver.core.TableMetadata.build(TableMetadata.java:107)
>> at
com.datastax.driver.core.Metadata.buildTableMetadata(Metadata.java:128)
>> at com.datastax.driver.core.Metadata.rebuildSchema(Metadata.java:89)
>> at
com.datastax.driver.core.ControlConnection.refreshSchema(ControlConnection.java:259)
>> at
com.datastax.driver.core.ControlConnection.tryConnect(ControlConnection.java:214)
>> at
com.datastax.driver.core.ControlConnection.reconnectInternal(ControlConnection.java:161)
>> at
com.datastax.driver.core.ControlConnection.connect(ControlConnection.java:77)
>> at com.datastax.driver.core.Cluster$Manager.init(Cluster.java:890)
>> at
com.datastax.driver.core.Cluster$Manager.newSession(Cluster.java:910)
>> at
com.datastax.driver.core.Cluster$Manager.access$200(Cluster.java:806)
>> at com.datastax.driver.core.Cluster.connect(Cluster.java:158)
>> at
cassandra.CassandraTestMinimized$delayedInit$body.apply(CassandraTestMinimized.scala:31)
>> at scala.Function0$class.apply$mcV$sp(Function0.scala:40)
>> at
scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:12)
>> at scala.App$$anonfun$main$1.apply(App.scala:71)
>> at scala.App$$anonfun$main$1.apply(App.scala:71)
>> at scala.collection.immutable.List.foreach(List.scala:318)
>> at
scala.collection.generic.TraversableForwarder$class.foreach(TraversableForwarder.scala:32)
>> at scala.App$class.main(App.scala:71)
>> at
cassandra.CassandraTestMinimized$.main(CassandraTestMinimized.scala:5)
>> at
cassandra.CassandraTestMinimized.main(CassandraTestMinimized.scala)
>
> I believe you've tried the cassandra trunk branch? trunk is basically the
future Cassandra 2.1 and the driver is currently unhappy because the
replicate_on_write option has been removed in that version. I'm supposed to
have fixed that on the driver 2.0 branch like 2 days ago so maybe you're
also using a slightly old version of the driver sources in there? Or maybe
I've screwed up my fix, I'll double check. But anyway, it would be overall
simpler to test with the cassandra-2.0 branch of Cassandra, with which you
shouldn't run into that.
> --
> Sylvain
--
Sorry this was sent from mobile. Will do less grammar and spell check than
usual.
Re: Performance problem with large wide row inserts using CQL
Posted by Mohit Anchlia <mo...@gmail.com>.
On Thu, Feb 20, 2014 at 4:37 PM, Edward Capriolo <ed...@gmail.com>wrote:
> Recomendations in cassandra have a shelf life of about 1 to 2 years. If
> you try to assert a recomendation from year ago you stand a solid chance of
> someone telling you there is now a better way.
>
> Casaandra once loved being a schemaless datastore. Imagine that?
>
> >> I agree with that. I also think that using CQL hides the basics of how
> Cassandra really stores the columns underneath and much of it's
> capabilities. People often confuse CQL with SQL and not completely
> understand that purpose of CQL is simply make it easy for users to
> understand and use Cassandra. Some of the things like INSERT doesn't make
> sense from DB standpoint since everything in Cassandra essentially is
> MUTATION.
>
> On Thursday, February 20, 2014, Peter Lin <wo...@gmail.com> wrote:
> >
> > good example Ed.
> >
> > I'm so happy to see other people doing things like this. Even if the
> official DataStax docs recommend don't mix static and dynamic, to me that's
> a huge disservice to Cassandra users.
> >
> > If someone really wants to stick to relational model, then NewSql is a
> better fit, plus gives users the full power of SQL with subqueries, like,
> and joins. NewSql can't handle these kinds of use cases due to static
> nature of relational tables, row size limit and column limit.
> >
> >
> >
> > On Thu, Feb 20, 2014 at 6:18 PM, Edward Capriolo <ed...@gmail.com>
> wrote:
> >
> > CASSANDRA-6561 is interesting. Though having statically defined columns
> are not exactly a solution to do everything in "thrift".
> >
> >
> http://planetcassandra.org/blog/post/poking-around-with-an-idea-ranged-metadata/
> >
> > Before collections or CQL existed I did some of these concepts myself.
> >
> > Say you have a column family named AllMyStuff
> >
> > columns named "friends_" would be a string and they would be a "Map" of
> friends to age
> >
> > set AllMySuff[edward][friends_bob]=34
> >
> > set AllMySuff[edward][friends_sara]=33
> >
> > Column name password could be a string
> >
> > set AllMySuff[edward][password]='mother'
> >
> > Columns named phone[00] phone[100] would be an array of phone numbers
> >
> > set AllMySuff[edward][phone[00]]=555-5555'
> >
> > It was quite easy for me to slice all the phone numbers
> >
> > startkey: phone
> > endkey: phone[100]
> >
> > But then every column starting with "action_xxxx" could be a page hit
> and i could have thousands / ten thousands of these
> >
> > In many cases CQL has nice/nicer abstractions for some of these things.
> But its largest detraction for me is that I can not take this already
> existing column family AllMyStuff and 'explain' it to CQL. Its a perfectly
> valid way to design something, and might be (probably) is more space
> efficient then the system of using composites CQL uses to pack things. I
> feel that as a data access language it dictates too much schema, not only
> what is in row schema, but it controls the format of the data on disk as
> well. Also schema's like mine above are very valid but selecting them into
> a table of fixed rows and columns does not map well.
> >
> > The way hive handles tackles this problem, is that the metadata is
> interpreted by a SerDe so that the physical data and the logical definition
> are not coupled.
> >
> >
> >
> >
> > On Thu, Feb 20, 2014 at 5:23 PM, DuyHai Doan <do...@gmail.com>
> wrote:
> >
> > Rüdiger
> >
> > "SortedMap<byte[], SortedMap<byte[], Pair<Long, byte[]>>"
> >
> > When using a RandomPartitioner or Murmur3Partitioner, the outer map is
> a simple Map, not SortedMap.
> >
> > The only case you have a SortedMap for row key is when using
> OrderPreservingPartitioner, which is clearly not advised for most cases
> because of hot spots in the cluster.
> >
> >
> >
> > On Thu, Feb 2
>
> --
> Sorry this was sent from mobile. Will do less grammar and spell check than
> usual.
>
Re: Performance problem with large wide row inserts using CQL
Posted by Peter Lin <wo...@gmail.com>.
Yeah
Slowly nosql products are adding schema :)
At least Cassandra is ahead of the curve
Sent from my iPhone
> On Feb 20, 2014, at 7:37 PM, Edward Capriolo <ed...@gmail.com> wrote:
>
> Recomendations in cassandra have a shelf life of about 1 to 2 years. If you try to assert a recomendation from year ago you stand a solid chance of someone telling you there is now a better way.
>
> Casaandra once loved being a schemaless datastore. Imagine that?
>
>
> On Thursday, February 20, 2014, Peter Lin <wo...@gmail.com> wrote:
> >
> > good example Ed.
> >
> > I'm so happy to see other people doing things like this. Even if the official DataStax docs recommend don't mix static and dynamic, to me that's a huge disservice to Cassandra users.
> >
> > If someone really wants to stick to relational model, then NewSql is a better fit, plus gives users the full power of SQL with subqueries, like, and joins. NewSql can't handle these kinds of use cases due to static nature of relational tables, row size limit and column limit.
> >
> >
> >
> > On Thu, Feb 20, 2014 at 6:18 PM, Edward Capriolo <ed...@gmail.com> wrote:
> >
> > CASSANDRA-6561 is interesting. Though having statically defined columns are not exactly a solution to do everything in "thrift".
> >
> > http://planetcassandra.org/blog/post/poking-around-with-an-idea-ranged-metadata/
> >
> > Before collections or CQL existed I did some of these concepts myself.
> >
> > Say you have a column family named AllMyStuff
> >
> > columns named "friends_" would be a string and they would be a "Map" of friends to age
> >
> > set AllMySuff[edward][friends_bob]=34
> >
> > set AllMySuff[edward][friends_sara]=33
> >
> > Column name password could be a string
> >
> > set AllMySuff[edward][password]='mother'
> >
> > Columns named phone[00] phone[100] would be an array of phone numbers
> >
> > set AllMySuff[edward][phone[00]]=555-5555'
> >
> > It was quite easy for me to slice all the phone numbers
> >
> > startkey: phone
> > endkey: phone[100]
> >
> > But then every column starting with "action_xxxx" could be a page hit and i could have thousands / ten thousands of these
> >
> > In many cases CQL has nice/nicer abstractions for some of these things. But its largest detraction for me is that I can not take this already existing column family AllMyStuff and 'explain' it to CQL. Its a perfectly valid way to design something, and might be (probably) is more space efficient then the system of using composites CQL uses to pack things. I feel that as a data access language it dictates too much schema, not only what is in row schema, but it controls the format of the data on disk as well. Also schema's like mine above are very valid but selecting them into a table of fixed rows and columns does not map well.
> >
> > The way hive handles tackles this problem, is that the metadata is interpreted by a SerDe so that the physical data and the logical definition are not coupled.
> >
> >
> >
> >
> > On Thu, Feb 20, 2014 at 5:23 PM, DuyHai Doan <do...@gmail.com> wrote:
> >
> > R��diger
> >
> > "SortedMap<byte[], SortedMap<byte[], Pair<Long, byte[]>>"
> >
> > When using a RandomPartitioner or Murmur3Partitioner, the outer map is a simple Map, not SortedMap.
> >
> > The only case you have a SortedMap for row key is when using OrderPreservingPartitioner, which is clearly not advised for most cases because of hot spots in the cluster.
> >
> >
> >
> > On Thu, Feb 2
>
> --
> Sorry this was sent from mobile. Will do less grammar and spell check than usual.
Re: Performance problem with large wide row inserts using CQL
Posted by Edward Capriolo <ed...@gmail.com>.
Recomendations in cassandra have a shelf life of about 1 to 2 years. If you
try to assert a recomendation from year ago you stand a solid chance of
someone telling you there is now a better way.
Casaandra once loved being a schemaless datastore. Imagine that?
On Thursday, February 20, 2014, Peter Lin <wo...@gmail.com> wrote:
>
> good example Ed.
>
> I'm so happy to see other people doing things like this. Even if the
official DataStax docs recommend don't mix static and dynamic, to me that's
a huge disservice to Cassandra users.
>
> If someone really wants to stick to relational model, then NewSql is a
better fit, plus gives users the full power of SQL with subqueries, like,
and joins. NewSql can't handle these kinds of use cases due to static
nature of relational tables, row size limit and column limit.
>
>
>
> On Thu, Feb 20, 2014 at 6:18 PM, Edward Capriolo <ed...@gmail.com>
wrote:
>
> CASSANDRA-6561 is interesting. Though having statically defined columns
are not exactly a solution to do everything in "thrift".
>
>
http://planetcassandra.org/blog/post/poking-around-with-an-idea-ranged-metadata/
>
> Before collections or CQL existed I did some of these concepts myself.
>
> Say you have a column family named AllMyStuff
>
> columns named "friends_" would be a string and they would be a "Map" of
friends to age
>
> set AllMySuff[edward][friends_bob]=34
>
> set AllMySuff[edward][friends_sara]=33
>
> Column name password could be a string
>
> set AllMySuff[edward][password]='mother'
>
> Columns named phone[00] phone[100] would be an array of phone numbers
>
> set AllMySuff[edward][phone[00]]=555-5555'
>
> It was quite easy for me to slice all the phone numbers
>
> startkey: phone
> endkey: phone[100]
>
> But then every column starting with "action_xxxx" could be a page hit and
i could have thousands / ten thousands of these
>
> In many cases CQL has nice/nicer abstractions for some of these things.
But its largest detraction for me is that I can not take this already
existing column family AllMyStuff and 'explain' it to CQL. Its a perfectly
valid way to design something, and might be (probably) is more space
efficient then the system of using composites CQL uses to pack things. I
feel that as a data access language it dictates too much schema, not only
what is in row schema, but it controls the format of the data on disk as
well. Also schema's like mine above are very valid but selecting them into
a table of fixed rows and columns does not map well.
>
> The way hive handles tackles this problem, is that the metadata is
interpreted by a SerDe so that the physical data and the logical definition
are not coupled.
>
>
>
>
> On Thu, Feb 20, 2014 at 5:23 PM, DuyHai Doan <do...@gmail.com> wrote:
>
> Rüdiger
>
> "SortedMap<byte[], SortedMap<byte[], Pair<Long, byte[]>>"
>
> When using a RandomPartitioner or Murmur3Partitioner, the outer map is a
simple Map, not SortedMap.
>
> The only case you have a SortedMap for row key is when using
OrderPreservingPartitioner, which is clearly not advised for most cases
because of hot spots in the cluster.
>
>
>
> On Thu, Feb 2
--
Sorry this was sent from mobile. Will do less grammar and spell check than
usual.
Re: Performance problem with large wide row inserts using CQL
Posted by Peter Lin <wo...@gmail.com>.
good example Ed.
I'm so happy to see other people doing things like this. Even if the
official DataStax docs recommend don't mix static and dynamic, to me that's
a huge disservice to Cassandra users.
If someone really wants to stick to relational model, then NewSql is a
better fit, plus gives users the full power of SQL with subqueries, like,
and joins. NewSql can't handle these kinds of use cases due to static
nature of relational tables, row size limit and column limit.
On Thu, Feb 20, 2014 at 6:18 PM, Edward Capriolo <ed...@gmail.com>wrote:
> CASSANDRA-6561 is interesting. Though having statically defined columns
> are not exactly a solution to do everything in "thrift".
>
>
> http://planetcassandra.org/blog/post/poking-around-with-an-idea-ranged-metadata/
>
> Before collections or CQL existed I did some of these concepts myself.
>
> Say you have a column family named AllMyStuff
>
> columns named "friends_" would be a string and they would be a "Map" of
> friends to age
>
> set AllMySuff[edward][friends_bob]=34
>
> set AllMySuff[edward][friends_sara]=33
>
> Column name password could be a string
>
> set AllMySuff[edward][password]='mother'
>
> Columns named phone[00] phone[100] would be an array of phone numbers
>
> set AllMySuff[edward][phone[00]]=555-5555'
>
> It was quite easy for me to slice all the phone numbers
>
> startkey: phone
> endkey: phone[100]
>
> But then every column starting with "action_xxxx" could be a page hit and
> i could have thousands / ten thousands of these
>
> In many cases CQL has nice/nicer abstractions for some of these things.
> But its largest detraction for me is that I can not take this already
> existing column family AllMyStuff and 'explain' it to CQL. Its a perfectly
> valid way to design something, and might be (probably) is more space
> efficient then the system of using composites CQL uses to pack things. I
> feel that as a data access language it dictates too much schema, not only
> what is in row schema, but it controls the format of the data on disk as
> well. Also schema's like mine above are very valid but selecting them into
> a table of fixed rows and columns does not map well.
>
> The way hive handles tackles this problem, is that the metadata is
> interpreted by a SerDe so that the physical data and the logical definition
> are not coupled.
>
>
>
>
> On Thu, Feb 20, 2014 at 5:23 PM, DuyHai Doan <do...@gmail.com> wrote:
>
>> Rüdiger
>>
>> "SortedMap<byte[], SortedMap<byte[], Pair<Long, byte[]>>"
>>
>> When using a RandomPartitioner or Murmur3Partitioner, the outer map is a
>> simple Map, not SortedMap.
>>
>> The only case you have a SortedMap for row key is when using
>> OrderPreservingPartitioner, which is clearly not advised for most cases
>> because of hot spots in the cluster.
>>
>>
>>
>> On Thu, Feb 20, 2014 at 10:49 PM, Rüdiger Klaehn <rk...@gmail.com>wrote:
>>
>>> Hi Sylvain,
>>>
>>> I applied the patch to the cassandra-2.0 branch (this required some
>>> manual work since I could not figure out which commit it was supposed to
>>> apply for, and it did not apply to the head of cassandra-2.0).
>>>
>>> The benchmark now runs in pretty much identical time to the thrift based
>>> benchmark. ~30s for 1000 inserts of 10000 key/value pairs each. Great work!
>>>
>>>
>>> I still have some questions regarding the mapping. Please bear with me
>>> if these are stupid questions. I am quite new to Cassandra.
>>>
>>> The basic cassandra data model for a keyspace is something like this,
>>> right?
>>>
>>> SortedMap<byte[], SortedMap<byte[], Pair<Long, byte[]>>
>>> ^ row key. determines which server(s) the rest is
>>> stored on
>>> ^ column key
>>> ^
>>> timestamp (latest one wins)
>>>
>>> ^ value (can be size 0)
>>>
>>> So if I have a table like the one in my benchmark (using blobs)
>>>
>>> CREATE TABLE IF NOT EXISTS test.wide (
>>> time blob,
>>> name blob,
>>> value blob,
>>> PRIMARY KEY (time,name))
>>> WITH COMPACT STORAGE
>>>
>>> From reading http://www.datastax.com/dev/blog/thrift-to-cql3 it seems
>>> that
>>>
>>> - time maps to the row key and name maps to the column key without any
>>> overhead
>>> - value directly maps to value in the model above without any prefix
>>>
>>> is that correct, or is there some overhead involved in CQL over the raw
>>> model as described above? If so, where exactly?
>>>
>>> kind regards and many thanks for your help,
>>>
>>> Rüdiger
>>>
>>>
>>> On Thu, Feb 20, 2014 at 8:36 AM, Sylvain Lebresne <sy...@datastax.com>wrote:
>>>
>>>>
>>>>
>>>>
>>>> On Wed, Feb 19, 2014 at 9:38 PM, Rüdiger Klaehn <rk...@gmail.com>wrote:
>>>>
>>>>>
>>>>> I have cloned the cassandra repo, applied the patch, and built it. But
>>>>> when I want to run the bechmark I get an exception. See below. I tried with
>>>>> a non-managed dependency to
>>>>> cassandra-driver-core-2.0.0-rc3-SNAPSHOT-jar-with-dependencies.jar, which I
>>>>> compiled from source because I read that that might help. But that did not
>>>>> make a difference.
>>>>>
>>>>> So currently I don't know how to give the patch a try. Any ideas?
>>>>>
>>>>> cheers,
>>>>>
>>>>> Rüdiger
>>>>>
>>>>> Exception in thread "main" java.lang.IllegalArgumentException:
>>>>> replicate_on_write is not a column defined in this metadata
>>>>> at
>>>>> com.datastax.driver.core.ColumnDefinitions.getAllIdx(ColumnDefinitions.java:273)
>>>>> at
>>>>> com.datastax.driver.core.ColumnDefinitions.getFirstIdx(ColumnDefinitions.java:279)
>>>>> at com.datastax.driver.core.Row.getBool(Row.java:117)
>>>>> at
>>>>> com.datastax.driver.core.TableMetadata$Options.<init>(TableMetadata.java:474)
>>>>> at
>>>>> com.datastax.driver.core.TableMetadata.build(TableMetadata.java:107)
>>>>> at
>>>>> com.datastax.driver.core.Metadata.buildTableMetadata(Metadata.java:128)
>>>>> at
>>>>> com.datastax.driver.core.Metadata.rebuildSchema(Metadata.java:89)
>>>>> at
>>>>> com.datastax.driver.core.ControlConnection.refreshSchema(ControlConnection.java:259)
>>>>> at
>>>>> com.datastax.driver.core.ControlConnection.tryConnect(ControlConnection.java:214)
>>>>> at
>>>>> com.datastax.driver.core.ControlConnection.reconnectInternal(ControlConnection.java:161)
>>>>> at
>>>>> com.datastax.driver.core.ControlConnection.connect(ControlConnection.java:77)
>>>>> at com.datastax.driver.core.Cluster$Manager.init(Cluster.java:890)
>>>>> at
>>>>> com.datastax.driver.core.Cluster$Manager.newSession(Cluster.java:910)
>>>>> at
>>>>> com.datastax.driver.core.Cluster$Manager.access$200(Cluster.java:806)
>>>>> at com.datastax.driver.core.Cluster.connect(Cluster.java:158)
>>>>> at
>>>>> cassandra.CassandraTestMinimized$delayedInit$body.apply(CassandraTestMinimized.scala:31)
>>>>> at scala.Function0$class.apply$mcV$sp(Function0.scala:40)
>>>>> at
>>>>> scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:12)
>>>>> at scala.App$$anonfun$main$1.apply(App.scala:71)
>>>>> at scala.App$$anonfun$main$1.apply(App.scala:71)
>>>>> at scala.collection.immutable.List.foreach(List.scala:318)
>>>>> at
>>>>> scala.collection.generic.TraversableForwarder$class.foreach(TraversableForwarder.scala:32)
>>>>> at scala.App$class.main(App.scala:71)
>>>>> at
>>>>> cassandra.CassandraTestMinimized$.main(CassandraTestMinimized.scala:5)
>>>>> at
>>>>> cassandra.CassandraTestMinimized.main(CassandraTestMinimized.scala)
>>>>>
>>>>
>>>> I believe you've tried the cassandra trunk branch? trunk is basically
>>>> the future Cassandra 2.1 and the driver is currently unhappy because the
>>>> replicate_on_write option has been removed in that version. I'm supposed to
>>>> have fixed that on the driver 2.0 branch like 2 days ago so maybe you're
>>>> also using a slightly old version of the driver sources in there? Or maybe
>>>> I've screwed up my fix, I'll double check. But anyway, it would be overall
>>>> simpler to test with the cassandra-2.0 branch of Cassandra, with which you
>>>> shouldn't run into that.
>>>>
>>>> --
>>>> Sylvain
>>>>
>>>
>>>
>>
>
Re: Performance problem with large wide row inserts using CQL
Posted by Edward Capriolo <ed...@gmail.com>.
CASSANDRA-6561 is interesting. Though having statically defined columns are
not exactly a solution to do everything in "thrift".
http://planetcassandra.org/blog/post/poking-around-with-an-idea-ranged-metadata/
Before collections or CQL existed I did some of these concepts myself.
Say you have a column family named AllMyStuff
columns named "friends_" would be a string and they would be a "Map" of
friends to age
set AllMySuff[edward][friends_bob]=34
set AllMySuff[edward][friends_sara]=33
Column name password could be a string
set AllMySuff[edward][password]='mother'
Columns named phone[00] phone[100] would be an array of phone numbers
set AllMySuff[edward][phone[00]]=555-5555'
It was quite easy for me to slice all the phone numbers
startkey: phone
endkey: phone[100]
But then every column starting with "action_xxxx" could be a page hit and i
could have thousands / ten thousands of these
In many cases CQL has nice/nicer abstractions for some of these things. But
its largest detraction for me is that I can not take this already existing
column family AllMyStuff and 'explain' it to CQL. Its a perfectly valid way
to design something, and might be (probably) is more space efficient then
the system of using composites CQL uses to pack things. I feel that as a
data access language it dictates too much schema, not only what is in row
schema, but it controls the format of the data on disk as well. Also
schema's like mine above are very valid but selecting them into a table of
fixed rows and columns does not map well.
The way hive handles tackles this problem, is that the metadata is
interpreted by a SerDe so that the physical data and the logical definition
are not coupled.
On Thu, Feb 20, 2014 at 5:23 PM, DuyHai Doan <do...@gmail.com> wrote:
> Rüdiger
>
> "SortedMap<byte[], SortedMap<byte[], Pair<Long, byte[]>>"
>
> When using a RandomPartitioner or Murmur3Partitioner, the outer map is a
> simple Map, not SortedMap.
>
> The only case you have a SortedMap for row key is when using
> OrderPreservingPartitioner, which is clearly not advised for most cases
> because of hot spots in the cluster.
>
>
>
> On Thu, Feb 20, 2014 at 10:49 PM, Rüdiger Klaehn <rk...@gmail.com>wrote:
>
>> Hi Sylvain,
>>
>> I applied the patch to the cassandra-2.0 branch (this required some
>> manual work since I could not figure out which commit it was supposed to
>> apply for, and it did not apply to the head of cassandra-2.0).
>>
>> The benchmark now runs in pretty much identical time to the thrift based
>> benchmark. ~30s for 1000 inserts of 10000 key/value pairs each. Great work!
>>
>>
>> I still have some questions regarding the mapping. Please bear with me if
>> these are stupid questions. I am quite new to Cassandra.
>>
>> The basic cassandra data model for a keyspace is something like this,
>> right?
>>
>> SortedMap<byte[], SortedMap<byte[], Pair<Long, byte[]>>
>> ^ row key. determines which server(s) the rest is stored
>> on
>> ^ column key
>> ^
>> timestamp (latest one wins)
>> ^
>> value (can be size 0)
>>
>> So if I have a table like the one in my benchmark (using blobs)
>>
>> CREATE TABLE IF NOT EXISTS test.wide (
>> time blob,
>> name blob,
>> value blob,
>> PRIMARY KEY (time,name))
>> WITH COMPACT STORAGE
>>
>> From reading http://www.datastax.com/dev/blog/thrift-to-cql3 it seems
>> that
>>
>> - time maps to the row key and name maps to the column key without any
>> overhead
>> - value directly maps to value in the model above without any prefix
>>
>> is that correct, or is there some overhead involved in CQL over the raw
>> model as described above? If so, where exactly?
>>
>> kind regards and many thanks for your help,
>>
>> Rüdiger
>>
>>
>> On Thu, Feb 20, 2014 at 8:36 AM, Sylvain Lebresne <sy...@datastax.com>wrote:
>>
>>>
>>>
>>>
>>> On Wed, Feb 19, 2014 at 9:38 PM, Rüdiger Klaehn <rk...@gmail.com>wrote:
>>>
>>>>
>>>> I have cloned the cassandra repo, applied the patch, and built it. But
>>>> when I want to run the bechmark I get an exception. See below. I tried with
>>>> a non-managed dependency to
>>>> cassandra-driver-core-2.0.0-rc3-SNAPSHOT-jar-with-dependencies.jar, which I
>>>> compiled from source because I read that that might help. But that did not
>>>> make a difference.
>>>>
>>>> So currently I don't know how to give the patch a try. Any ideas?
>>>>
>>>> cheers,
>>>>
>>>> Rüdiger
>>>>
>>>> Exception in thread "main" java.lang.IllegalArgumentException:
>>>> replicate_on_write is not a column defined in this metadata
>>>> at
>>>> com.datastax.driver.core.ColumnDefinitions.getAllIdx(ColumnDefinitions.java:273)
>>>> at
>>>> com.datastax.driver.core.ColumnDefinitions.getFirstIdx(ColumnDefinitions.java:279)
>>>> at com.datastax.driver.core.Row.getBool(Row.java:117)
>>>> at
>>>> com.datastax.driver.core.TableMetadata$Options.<init>(TableMetadata.java:474)
>>>> at
>>>> com.datastax.driver.core.TableMetadata.build(TableMetadata.java:107)
>>>> at
>>>> com.datastax.driver.core.Metadata.buildTableMetadata(Metadata.java:128)
>>>> at com.datastax.driver.core.Metadata.rebuildSchema(Metadata.java:89)
>>>> at
>>>> com.datastax.driver.core.ControlConnection.refreshSchema(ControlConnection.java:259)
>>>> at
>>>> com.datastax.driver.core.ControlConnection.tryConnect(ControlConnection.java:214)
>>>> at
>>>> com.datastax.driver.core.ControlConnection.reconnectInternal(ControlConnection.java:161)
>>>> at
>>>> com.datastax.driver.core.ControlConnection.connect(ControlConnection.java:77)
>>>> at com.datastax.driver.core.Cluster$Manager.init(Cluster.java:890)
>>>> at
>>>> com.datastax.driver.core.Cluster$Manager.newSession(Cluster.java:910)
>>>> at
>>>> com.datastax.driver.core.Cluster$Manager.access$200(Cluster.java:806)
>>>> at com.datastax.driver.core.Cluster.connect(Cluster.java:158)
>>>> at
>>>> cassandra.CassandraTestMinimized$delayedInit$body.apply(CassandraTestMinimized.scala:31)
>>>> at scala.Function0$class.apply$mcV$sp(Function0.scala:40)
>>>> at
>>>> scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:12)
>>>> at scala.App$$anonfun$main$1.apply(App.scala:71)
>>>> at scala.App$$anonfun$main$1.apply(App.scala:71)
>>>> at scala.collection.immutable.List.foreach(List.scala:318)
>>>> at
>>>> scala.collection.generic.TraversableForwarder$class.foreach(TraversableForwarder.scala:32)
>>>> at scala.App$class.main(App.scala:71)
>>>> at
>>>> cassandra.CassandraTestMinimized$.main(CassandraTestMinimized.scala:5)
>>>> at
>>>> cassandra.CassandraTestMinimized.main(CassandraTestMinimized.scala)
>>>>
>>>
>>> I believe you've tried the cassandra trunk branch? trunk is basically
>>> the future Cassandra 2.1 and the driver is currently unhappy because the
>>> replicate_on_write option has been removed in that version. I'm supposed to
>>> have fixed that on the driver 2.0 branch like 2 days ago so maybe you're
>>> also using a slightly old version of the driver sources in there? Or maybe
>>> I've screwed up my fix, I'll double check. But anyway, it would be overall
>>> simpler to test with the cassandra-2.0 branch of Cassandra, with which you
>>> shouldn't run into that.
>>>
>>> --
>>> Sylvain
>>>
>>
>>
>
Re: Performance problem with large wide row inserts using CQL
Posted by Peter Lin <wo...@gmail.com>.
thanks Erick.
hopefully sylvain will forgive me for misquoting him. My goal was to share
knowledge and get people thinking about how best to use both thrift and
cql. Whenever I hear people say "cql is the future" I get annoyed. My bias
feeling is they compliment each other very well and users should learn
both. It increases the learning curve, but it helps to avoid making dumb
mistakes in the future by digging deep.
On Thu, Feb 20, 2014 at 6:00 PM, Erick Ramirez <er...@ramirez.com.au> wrote:
> Wow! What a fantastic robust discussion. I've just been educated.
>
> Peter --- Thanks for providing those use cases. They are great examples.
>
> Rudiger --- From what you've done so far, I wouldn't have said your are
> new to Cassandra. Well done.
>
> Cheers,
> Erick
>
>
Re: Performance problem with large wide row inserts using CQL
Posted by Erick Ramirez <er...@ramirez.com.au>.
Wow! What a fantastic robust discussion. I've just been educated.
Peter --- Thanks for providing those use cases. They are great examples.
Rudiger --- From what you've done so far, I wouldn't have said your are new
to Cassandra. Well done.
Cheers,
Erick
Re: Performance problem with large wide row inserts using CQL
Posted by DuyHai Doan <do...@gmail.com>.
Rüdiger
"SortedMap<byte[], SortedMap<byte[], Pair<Long, byte[]>>"
When using a RandomPartitioner or Murmur3Partitioner, the outer map is a
simple Map, not SortedMap.
The only case you have a SortedMap for row key is when using
OrderPreservingPartitioner, which is clearly not advised for most cases
because of hot spots in the cluster.
On Thu, Feb 20, 2014 at 10:49 PM, Rüdiger Klaehn <rk...@gmail.com> wrote:
> Hi Sylvain,
>
> I applied the patch to the cassandra-2.0 branch (this required some manual
> work since I could not figure out which commit it was supposed to apply
> for, and it did not apply to the head of cassandra-2.0).
>
> The benchmark now runs in pretty much identical time to the thrift based
> benchmark. ~30s for 1000 inserts of 10000 key/value pairs each. Great work!
>
>
> I still have some questions regarding the mapping. Please bear with me if
> these are stupid questions. I am quite new to Cassandra.
>
> The basic cassandra data model for a keyspace is something like this,
> right?
>
> SortedMap<byte[], SortedMap<byte[], Pair<Long, byte[]>>
> ^ row key. determines which server(s) the rest is stored
> on
> ^ column key
> ^ timestamp
> (latest one wins)
> ^
> value (can be size 0)
>
> So if I have a table like the one in my benchmark (using blobs)
>
> CREATE TABLE IF NOT EXISTS test.wide (
> time blob,
> name blob,
> value blob,
> PRIMARY KEY (time,name))
> WITH COMPACT STORAGE
>
> From reading http://www.datastax.com/dev/blog/thrift-to-cql3 it seems
> that
>
> - time maps to the row key and name maps to the column key without any
> overhead
> - value directly maps to value in the model above without any prefix
>
> is that correct, or is there some overhead involved in CQL over the raw
> model as described above? If so, where exactly?
>
> kind regards and many thanks for your help,
>
> Rüdiger
>
>
> On Thu, Feb 20, 2014 at 8:36 AM, Sylvain Lebresne <sy...@datastax.com>wrote:
>
>>
>>
>>
>> On Wed, Feb 19, 2014 at 9:38 PM, Rüdiger Klaehn <rk...@gmail.com>wrote:
>>
>>>
>>> I have cloned the cassandra repo, applied the patch, and built it. But
>>> when I want to run the bechmark I get an exception. See below. I tried with
>>> a non-managed dependency to
>>> cassandra-driver-core-2.0.0-rc3-SNAPSHOT-jar-with-dependencies.jar, which I
>>> compiled from source because I read that that might help. But that did not
>>> make a difference.
>>>
>>> So currently I don't know how to give the patch a try. Any ideas?
>>>
>>> cheers,
>>>
>>> Rüdiger
>>>
>>> Exception in thread "main" java.lang.IllegalArgumentException:
>>> replicate_on_write is not a column defined in this metadata
>>> at
>>> com.datastax.driver.core.ColumnDefinitions.getAllIdx(ColumnDefinitions.java:273)
>>> at
>>> com.datastax.driver.core.ColumnDefinitions.getFirstIdx(ColumnDefinitions.java:279)
>>> at com.datastax.driver.core.Row.getBool(Row.java:117)
>>> at
>>> com.datastax.driver.core.TableMetadata$Options.<init>(TableMetadata.java:474)
>>> at
>>> com.datastax.driver.core.TableMetadata.build(TableMetadata.java:107)
>>> at
>>> com.datastax.driver.core.Metadata.buildTableMetadata(Metadata.java:128)
>>> at com.datastax.driver.core.Metadata.rebuildSchema(Metadata.java:89)
>>> at
>>> com.datastax.driver.core.ControlConnection.refreshSchema(ControlConnection.java:259)
>>> at
>>> com.datastax.driver.core.ControlConnection.tryConnect(ControlConnection.java:214)
>>> at
>>> com.datastax.driver.core.ControlConnection.reconnectInternal(ControlConnection.java:161)
>>> at
>>> com.datastax.driver.core.ControlConnection.connect(ControlConnection.java:77)
>>> at com.datastax.driver.core.Cluster$Manager.init(Cluster.java:890)
>>> at
>>> com.datastax.driver.core.Cluster$Manager.newSession(Cluster.java:910)
>>> at
>>> com.datastax.driver.core.Cluster$Manager.access$200(Cluster.java:806)
>>> at com.datastax.driver.core.Cluster.connect(Cluster.java:158)
>>> at
>>> cassandra.CassandraTestMinimized$delayedInit$body.apply(CassandraTestMinimized.scala:31)
>>> at scala.Function0$class.apply$mcV$sp(Function0.scala:40)
>>> at
>>> scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:12)
>>> at scala.App$$anonfun$main$1.apply(App.scala:71)
>>> at scala.App$$anonfun$main$1.apply(App.scala:71)
>>> at scala.collection.immutable.List.foreach(List.scala:318)
>>> at
>>> scala.collection.generic.TraversableForwarder$class.foreach(TraversableForwarder.scala:32)
>>> at scala.App$class.main(App.scala:71)
>>> at
>>> cassandra.CassandraTestMinimized$.main(CassandraTestMinimized.scala:5)
>>> at
>>> cassandra.CassandraTestMinimized.main(CassandraTestMinimized.scala)
>>>
>>
>> I believe you've tried the cassandra trunk branch? trunk is basically the
>> future Cassandra 2.1 and the driver is currently unhappy because the
>> replicate_on_write option has been removed in that version. I'm supposed to
>> have fixed that on the driver 2.0 branch like 2 days ago so maybe you're
>> also using a slightly old version of the driver sources in there? Or maybe
>> I've screwed up my fix, I'll double check. But anyway, it would be overall
>> simpler to test with the cassandra-2.0 branch of Cassandra, with which you
>> shouldn't run into that.
>>
>> --
>> Sylvain
>>
>
>
Re: Performance problem with large wide row inserts using CQL
Posted by Peter Lin <wo...@gmail.com>.
I'm with you Ed.
I love that Cassandra is moving forward at a very fast pace, but often I
get the impression some people want to avoid thrift at all cost whether it
makes sense or not. I totally understand the commiters don't have enough
bandwidth to make sure all existing cases transition smoothly to CQL3 and
beyond, but to dismiss existing users doesn't foster good will. I'd go as
far as to say it hurts the community. Saying things like "cql3 is the
future" is a smack in the face of existing users of thrift.
I've been contributing to various apache projects for over 10 years, so I
know that pain first hand. I've made similar blunders over the years, so
I'm definitely guilty of doing the same thing on other projects. At the end
of the day though, I feel it's important for people to speak out, even if
it may be painful. Clearly there are use cases that don't fit CQL3 well and
is easier with thrift.
I'm hopeful that as CQL matures, it will get better at handling more
complex uses that currently don't transition well.
On Fri, Feb 21, 2014 at 9:34 AM, Edward Capriolo <ed...@gmail.com>wrote:
> The main issue is that cassandra has two of everything. Two access apis,
> two meta data systems, and two groups of users.
>
> Those groups of users using the original systems thrift, cfmetadata, and
> following the advice of three years ago have been labled obsolete (did you
> ever see that twighlight zone episode?).
>
> If you suggest a thrift only feature get ready to fight. People seem
> oblivious to the fact that you may have a 38 node cluster with 12 tb of
> data under compact storage, and that you can't just snap your fingers and
> adopt whatever new system to pack data that someone comes up with.
>
> Earlier in the thread I detailed a potential way to store collection like
> things in compact storage. You would just assume that with all the
> collective brain power in the project, that somehow, some way collections
> could make their way into compact storage. Or the new language would offer
> similiar features regardless of storage chosen (say like innodb and
> mariadb).
>
> The shelf life of codd's normal form has been what? 30 or 40 years and
> still going strong? Im always rather pisswd that 3 years after i start
> using cassandra everything has changed, that im not the future, and that no
> one is really interested in supporting anything i used the datastore for.
>
>
>
> On Friday, February 21, 2014, Sylvain Lebresne <sy...@datastax.com>
> wrote:
> > On Thu, Feb 20, 2014 at 10:49 PM, Rüdiger Klaehn <rk...@gmail.com>
> wrote:
> >>
> >> Hi Sylvain,
> >>
> >> I applied the patch to the cassandra-2.0 branch (this required some
> manual work since I could not figure out which commit it was supposed to
> apply for, and it did not apply to the head of cassandra-2.0).
> >
> > Yeah, some commit yesterday made the patch not apply cleanly anymore. In
> any case, It's not committed to the cassandra-2.0 branch and will be part
> of 2.0.6.
> >>
> >> The benchmark now runs in pretty much identical time to the thrift
> based benchmark. ~30s for 1000 inserts of 10000 key/value pairs each. Great
> work!
> >
> > Glad that it helped.
> >
> >>
> >> I still have some questions regarding the mapping. Please bear with me
> if these are stupid questions. I am quite new to Cassandra.
> >>
> >> The basic cassandra data model for a keyspace is something like this,
> right?
> >>
> >> SortedMap<byte[], SortedMap<byte[], Pair<Long, byte[]>>
> >> ^ row key. determines which server(s) the rest is
> stored on
> >> ^ column key
> >> ^
> timestamp (latest one wins)
> >>
> ^ value (can be size 0)
> >
> > It's a reasonable way to think of how things are stored internally, yes.
> Though as DuyHai mentioned, the first map is really sorting by token and in
> general that means you use mostly the sorting of the second map concretely.
> >
> >>
> >> So if I have a table like the one in my benchmark (using blobs)
> >>
> >> CREATE TABLE IF NOT EXISTS test.wide (
> >> time blob,
> >> name blob,
> >> value blob,
> >> PRIMARY KEY (time,name))
> >> WITH COMPACT STORAGE
> >>
> >> From reading http://www.datastax.com/dev/blog/thrift-to-cql3 it seems
> that
> >>
> >> - time maps to the row key and name maps to the column key without any
> overhead
> >> - value directly maps to value in the model above without any prefix
> >>
> >> is that correct, or is there some overhead involved in CQL over the raw
> model as described above? If so, where exactly?
> >
> > That's correct.
> > For completeness sake, if you were to remove the COMPACT STORAGE, there
> would be some overhead in how it maps to the underlying column key, but
> that overhead would buy you much more flexibility in how you could evolve
> this table schema (you could add more CQL columns later if needs be, have
> collections or have static columns following CASSANDRA-6561 that comes in
> 2.0.6; none of which you can have with COMPACT STORAGE). Note that it's
> perfectly fine to use COMPACT STORAGE if you know you don't and won't need
> the additional flexibility, but I generally advise people to actually check
> first that using COMPACT STORAGE does make a concrete and meaningful
> difference for their use case (be careful with premature optimization
> really). The difference in performance/storage space used is not always all
> that noticeable in practice (note that I didn't said it's never
> noticeable!) and is narrowing with Cassandra evolution (it's not impossible
> at all that we will get to "never noticeable" someday, while COMPACT
> STORAGE tables will never get the flexibility of normal tables because
> there is backwards compatibility issues). It's also my experience that more
> often that not (again, not always), flexibility turns out to be more
> important that squeezing every bit of performance you can (if it comes at
> the price of that flexibility that is) in the long run. Do what you want of
> that advise :)
> > --
> > Sylvain
> >
> >>
> >> kind regards and many thanks for your help,
> >>
> >> Rüdiger
> >>
> >>
> >> On Thu, Feb 20, 2014 at 8:36 AM, Sylvain Lebresne <sy...@datastax.com>
> wrote:
> >>>
> >>>
> >>>
> >>> On Wed, Feb 19, 2014 at 9:38 PM, Rüdiger Klaehn <rk...@gmail.com>
> wrote:
> >>>>
> >>>> I have cloned the cassandra repo, applied the patch, and built it.
> But when I want to run the bechmark I get an exception. See below. I tried
> with a non-managed dependency to
> cassandra-driver-core-2.0.0-rc3-SNAPSHOT-jar-with-dependencies.jar, which I
> compiled from source because I read that that might help. But that did not
> make a difference.
> >>>>
> >>>> So currently I don't know how to give the patch a try. Any ideas?
> >>>>
> >>>> cheers,
> >>>>
> >>>> Rüdiger
> >>>>
> >>>> Exception in thread "main" java.lang.IllegalArgumentException:
> replicate_on_write is not a column defined in this metadata
> >>>> at
> com.datastax.driver.core.ColumnDefinitions.getAllIdx(ColumnDefinitions.java:273)
> >>>> at
> com.datastax.driver.core.ColumnDefinitions.getFirstIdx(ColumnDefinitions.java:279)
> >>>> at com.datastax.driver.core.Row.getBool(Row.java:117)
> >>>> at
> com.datastax.driver.core.TableMetadata$Options.<init>(TableMetadata.java:474)
> >>>> at
> com.datastax.driver.core.TableMetadata.build(TableMetadata.java:107)
> >>>> at
> com.datastax.driver.core.Metadata.buildTableMetadata(Metadata.java:128)
> >>>> at
> com.datastax.driver.core.Metadata.rebuildSchema(Metadata.java:89)
> >>>> at
> com.datastax.driver.core.ControlConnection.refreshSchema(ControlConnection.java:259)
> >>>> at
> com.datastax.driver.core.ControlConnection.tryConnect(ControlConnection.java:214)
> >>>> at
> com.datastax.driver.core.ControlConnection.reconnectInternal(ControlConnection.java:161)
> >>>> at
> com.datastax.driver.core.ControlConnection.connect(ControlConnection.java:77)
> >>>> at com.datastax.driver.core.Cluster$Manager.init(Cluster.java:890)
> >>>> at
> com.datastax.driver.core.Cluster$Manager.newSession(Cluster.java:910)
> >>>> at
> com.datastax.driver.core.Cluster$Manager.access$200(Cluster.java:806)
> >>>> at com.datastax.driver.core.Cluster.connect(Cluster.java:158)
> >>>> at
> cassandra.CassandraTestMinimized$delayedInit$body.apply(CassandraTestMinimized.scala:31)
> >>>> at scala.Function0$class.apply$mcV$sp(Function0.scala:40)
> >>>> at
> scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:12)
> >>>> at scala.App$$anonfun$main$1.apply(App.scala:71)
> >>>> at scala.App$$anonfun$main$1.apply(App.scala:71)
> >>>> at scala.collection.immutable.List.foreach(List.scala:318)
> >>>> at
> scala.collection.generic.TraversableForwarder$class.foreach(TraversableForwarder.scala:32)
> >>>> at scala.App$class.main(App.scala:71)
> >>>> at
> cassandra.CassandraTestMinimized$.main(CassandraTestMinimized.scala:5)
> >>>> at
> cassandra.CassandraTestMinimized.main(CassandraTestMinimized.scala)
> >>>
> >>> I believe you've tried the cassandra trunk branch? trunk is basically
> the future Cassandra 2.1 and the driver is currently unhappy because the
> replicate_on_write option has been removed in that version. I'm supposed to
> have fixed that on the driver 2.0 branch like 2 days ago so maybe you're
> also using a slightly old version of the driver sources in there? Or maybe
> I've screwed up my fix, I'll double check. But anyway, it would be overall
> simpler to test with the cassandra-2.0 branch of Cassandra, with which you
> shouldn't run into that.
> >>> --
> >>> Sylvain
> >
> >
>
> --
> Sorry this was sent from mobile. Will do less grammar and spell check than
> usual.
>
Re: Performance problem with large wide row inserts using CQL
Posted by Edward Capriolo <ed...@gmail.com>.
The main issue is that cassandra has two of everything. Two access apis,
two meta data systems, and two groups of users.
Those groups of users using the original systems thrift, cfmetadata, and
following the advice of three years ago have been labled obsolete (did you
ever see that twighlight zone episode?).
If you suggest a thrift only feature get ready to fight. People seem
oblivious to the fact that you may have a 38 node cluster with 12 tb of
data under compact storage, and that you can't just snap your fingers and
adopt whatever new system to pack data that someone comes up with.
Earlier in the thread I detailed a potential way to store collection like
things in compact storage. You would just assume that with all the
collective brain power in the project, that somehow, some way collections
could make their way into compact storage. Or the new language would offer
similiar features regardless of storage chosen (say like innodb and
mariadb).
The shelf life of codd's normal form has been what? 30 or 40 years and
still going strong? Im always rather pisswd that 3 years after i start
using cassandra everything has changed, that im not the future, and that no
one is really interested in supporting anything i used the datastore for.
On Friday, February 21, 2014, Sylvain Lebresne <sy...@datastax.com> wrote:
> On Thu, Feb 20, 2014 at 10:49 PM, Rüdiger Klaehn <rk...@gmail.com>
wrote:
>>
>> Hi Sylvain,
>>
>> I applied the patch to the cassandra-2.0 branch (this required some
manual work since I could not figure out which commit it was supposed to
apply for, and it did not apply to the head of cassandra-2.0).
>
> Yeah, some commit yesterday made the patch not apply cleanly anymore. In
any case, It's not committed to the cassandra-2.0 branch and will be part
of 2.0.6.
>>
>> The benchmark now runs in pretty much identical time to the thrift based
benchmark. ~30s for 1000 inserts of 10000 key/value pairs each. Great work!
>
> Glad that it helped.
>
>>
>> I still have some questions regarding the mapping. Please bear with me
if these are stupid questions. I am quite new to Cassandra.
>>
>> The basic cassandra data model for a keyspace is something like this,
right?
>>
>> SortedMap<byte[], SortedMap<byte[], Pair<Long, byte[]>>
>> ^ row key. determines which server(s) the rest is
stored on
>> ^ column key
>> ^
timestamp (latest one wins)
>>
^ value (can be size 0)
>
> It's a reasonable way to think of how things are stored internally, yes.
Though as DuyHai mentioned, the first map is really sorting by token and in
general that means you use mostly the sorting of the second map concretely.
>
>>
>> So if I have a table like the one in my benchmark (using blobs)
>>
>> CREATE TABLE IF NOT EXISTS test.wide (
>> time blob,
>> name blob,
>> value blob,
>> PRIMARY KEY (time,name))
>> WITH COMPACT STORAGE
>>
>> From reading http://www.datastax.com/dev/blog/thrift-to-cql3 it seems
that
>>
>> - time maps to the row key and name maps to the column key without any
overhead
>> - value directly maps to value in the model above without any prefix
>>
>> is that correct, or is there some overhead involved in CQL over the raw
model as described above? If so, where exactly?
>
> That's correct.
> For completeness sake, if you were to remove the COMPACT STORAGE, there
would be some overhead in how it maps to the underlying column key, but
that overhead would buy you much more flexibility in how you could evolve
this table schema (you could add more CQL columns later if needs be, have
collections or have static columns following CASSANDRA-6561 that comes in
2.0.6; none of which you can have with COMPACT STORAGE). Note that it's
perfectly fine to use COMPACT STORAGE if you know you don't and won't need
the additional flexibility, but I generally advise people to actually check
first that using COMPACT STORAGE does make a concrete and meaningful
difference for their use case (be careful with premature optimization
really). The difference in performance/storage space used is not always all
that noticeable in practice (note that I didn't said it's never
noticeable!) and is narrowing with Cassandra evolution (it's not impossible
at all that we will get to "never noticeable" someday, while COMPACT
STORAGE tables will never get the flexibility of normal tables because
there is backwards compatibility issues). It's also my experience that more
often that not (again, not always), flexibility turns out to be more
important that squeezing every bit of performance you can (if it comes at
the price of that flexibility that is) in the long run. Do what you want of
that advise :)
> --
> Sylvain
>
>>
>> kind regards and many thanks for your help,
>>
>> Rüdiger
>>
>>
>> On Thu, Feb 20, 2014 at 8:36 AM, Sylvain Lebresne <sy...@datastax.com>
wrote:
>>>
>>>
>>>
>>> On Wed, Feb 19, 2014 at 9:38 PM, Rüdiger Klaehn <rk...@gmail.com>
wrote:
>>>>
>>>> I have cloned the cassandra repo, applied the patch, and built it. But
when I want to run the bechmark I get an exception. See below. I tried with
a non-managed dependency to
cassandra-driver-core-2.0.0-rc3-SNAPSHOT-jar-with-dependencies.jar, which I
compiled from source because I read that that might help. But that did not
make a difference.
>>>>
>>>> So currently I don't know how to give the patch a try. Any ideas?
>>>>
>>>> cheers,
>>>>
>>>> Rüdiger
>>>>
>>>> Exception in thread "main" java.lang.IllegalArgumentException:
replicate_on_write is not a column defined in this metadata
>>>> at
com.datastax.driver.core.ColumnDefinitions.getAllIdx(ColumnDefinitions.java:273)
>>>> at
com.datastax.driver.core.ColumnDefinitions.getFirstIdx(ColumnDefinitions.java:279)
>>>> at com.datastax.driver.core.Row.getBool(Row.java:117)
>>>> at
com.datastax.driver.core.TableMetadata$Options.<init>(TableMetadata.java:474)
>>>> at
com.datastax.driver.core.TableMetadata.build(TableMetadata.java:107)
>>>> at
com.datastax.driver.core.Metadata.buildTableMetadata(Metadata.java:128)
>>>> at
com.datastax.driver.core.Metadata.rebuildSchema(Metadata.java:89)
>>>> at
com.datastax.driver.core.ControlConnection.refreshSchema(ControlConnection.java:259)
>>>> at
com.datastax.driver.core.ControlConnection.tryConnect(ControlConnection.java:214)
>>>> at
com.datastax.driver.core.ControlConnection.reconnectInternal(ControlConnection.java:161)
>>>> at
com.datastax.driver.core.ControlConnection.connect(ControlConnection.java:77)
>>>> at com.datastax.driver.core.Cluster$Manager.init(Cluster.java:890)
>>>> at
com.datastax.driver.core.Cluster$Manager.newSession(Cluster.java:910)
>>>> at
com.datastax.driver.core.Cluster$Manager.access$200(Cluster.java:806)
>>>> at com.datastax.driver.core.Cluster.connect(Cluster.java:158)
>>>> at
cassandra.CassandraTestMinimized$delayedInit$body.apply(CassandraTestMinimized.scala:31)
>>>> at scala.Function0$class.apply$mcV$sp(Function0.scala:40)
>>>> at
scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:12)
>>>> at scala.App$$anonfun$main$1.apply(App.scala:71)
>>>> at scala.App$$anonfun$main$1.apply(App.scala:71)
>>>> at scala.collection.immutable.List.foreach(List.scala:318)
>>>> at
scala.collection.generic.TraversableForwarder$class.foreach(TraversableForwarder.scala:32)
>>>> at scala.App$class.main(App.scala:71)
>>>> at
cassandra.CassandraTestMinimized$.main(CassandraTestMinimized.scala:5)
>>>> at
cassandra.CassandraTestMinimized.main(CassandraTestMinimized.scala)
>>>
>>> I believe you've tried the cassandra trunk branch? trunk is basically
the future Cassandra 2.1 and the driver is currently unhappy because the
replicate_on_write option has been removed in that version. I'm supposed to
have fixed that on the driver 2.0 branch like 2 days ago so maybe you're
also using a slightly old version of the driver sources in there? Or maybe
I've screwed up my fix, I'll double check. But anyway, it would be overall
simpler to test with the cassandra-2.0 branch of Cassandra, with which you
shouldn't run into that.
>>> --
>>> Sylvain
>
>
--
Sorry this was sent from mobile. Will do less grammar and spell check than
usual.
Re: Performance problem with large wide row inserts using CQL
Posted by Rüdiger Klaehn <rk...@gmail.com>.
On Mon, Feb 24, 2014 at 11:47 AM, Sylvain Lebresne <sy...@datastax.com>wrote:
>
>>
>>>
>>>> I still have some questions regarding the mapping. Please bear with me
>>>> if these are stupid questions. I am quite new to Cassandra.
>>>>
>>>> The basic cassandra data model for a keyspace is something like this,
>>>> right?
>>>>
>>>> SortedMap<byte[], SortedMap<byte[], Pair<Long, byte[]>>
>>>> ^ row key. determines which server(s) the rest is
>>>> stored on
>>>> ^ column key
>>>> ^
>>>> timestamp (latest one wins)
>>>>
>>>> ^ value (can be size 0)
>>>>
>>>
>>> It's a reasonable way to think of how things are stored internally, yes.
>>> Though as DuyHai mentioned, the first map is really sorting by token and in
>>> general that means you use mostly the sorting of the second map concretely.
>>>
>>>
>> Yes, understood.
>>
>> So the first SortedMap is sorted on some kind of hash of the actual key
>> to make sure the data gets evenly distributed along the nodes? What if my
>> key is already a good hash: is there a way to use an identity function as a
>> hash function (in CQL)?
>>
>
> It's possible, yes. The hash function we're talking about is what
> Cassandra calls "the partitioner". You configure the partitioner in the
> yaml config file and there is one partitioner, ByteOrderedPartitioner, that
> is basically the identify function.
> We however usually discourage user for using it because the partitioner is
> global to a cluster and cannot be changed (you basically pick it at cluster
> creation time and are stuck with it until the end of time), and since
> ByteOrderedPartitioner can easily lead to hotspot in the data distribution
> if you're not careful...For those reasons, the default partitioner is also
> much more tested, and I can't remember anyone mentioning the partitioner
> has been a bottleneck.
>
> Thanks for the info. I thought that this might be possible to adjust on a
per-keyspace level.
But if you can only do this globally, then I will leave it alone. Other
than the (probably negibile) performance impact of hashing the hash again,
there is nothing wrong with doing so. Hashing a SHA1-hash will give a good
distribution.
anyway, this is getting a bit off-topic.
cheers,
Rüdiger
Re: Performance problem with large wide row inserts using CQL
Posted by Sylvain Lebresne <sy...@datastax.com>.
>
>
>
>>
>>> I still have some questions regarding the mapping. Please bear with me
>>> if these are stupid questions. I am quite new to Cassandra.
>>>
>>> The basic cassandra data model for a keyspace is something like this,
>>> right?
>>>
>>> SortedMap<byte[], SortedMap<byte[], Pair<Long, byte[]>>
>>> ^ row key. determines which server(s) the rest is
>>> stored on
>>> ^ column key
>>> ^
>>> timestamp (latest one wins)
>>>
>>> ^ value (can be size 0)
>>>
>>
>> It's a reasonable way to think of how things are stored internally, yes.
>> Though as DuyHai mentioned, the first map is really sorting by token and in
>> general that means you use mostly the sorting of the second map concretely.
>>
>>
> Yes, understood.
>
> So the first SortedMap is sorted on some kind of hash of the actual key to
> make sure the data gets evenly distributed along the nodes? What if my key
> is already a good hash: is there a way to use an identity function as a
> hash function (in CQL)?
>
It's possible, yes. The hash function we're talking about is what Cassandra
calls "the partitioner". You configure the partitioner in the yaml config
file and there is one partitioner, ByteOrderedPartitioner, that is
basically the identify function.
We however usually discourage user for using it because the partitioner is
global to a cluster and cannot be changed (you basically pick it at cluster
creation time and are stuck with it until the end of time), and since
ByteOrderedPartitioner can easily lead to hotspot in the data distribution
if you're not careful...For those reasons, the default partitioner is also
much more tested, and I can't remember anyone mentioning the partitioner
has been a bottleneck.
--
Sylvain
>
>
>>
>>> So if I have a table like the one in my benchmark (using blobs)
>>>
>>> CREATE TABLE IF NOT EXISTS test.wide (
>>> time blob,
>>> name blob,
>>> value blob,
>>> PRIMARY KEY (time,name))
>>> WITH COMPACT STORAGE
>>>
>>> From reading http://www.datastax.com/dev/blog/thrift-to-cql3 it seems
>>> that
>>>
>>> - time maps to the row key and name maps to the column key without any
>>> overhead
>>> - value directly maps to value in the model above without any prefix
>>>
>>> is that correct, or is there some overhead involved in CQL over the raw
>>> model as described above? If so, where exactly?
>>>
>>
>> That's correct.
>> For completeness sake, if you were to remove the COMPACT STORAGE, there
>> would be some overhead in how it maps to the underlying column key, but
>> that overhead would buy you much more flexibility in how you could evolve
>> this table schema (you could add more CQL columns later if needs be, have
>> collections or have static columns following CASSANDRA-6561 that comes in
>> 2.0.6; none of which you can have with COMPACT STORAGE). Note that it's
>> perfectly fine to use COMPACT STORAGE if you know you don't and won't need
>> the additional flexibility, but I generally advise people to actually check
>> first that using COMPACT STORAGE does make a concrete and meaningful
>> difference for their use case (be careful with premature optimization
>> really).
>>
>
> In this case I am confident that the schema will not change. But there
> will be other tables built from the same data where I am not going to use
> compact storage.
>
> cheers,
>
> Rüdiger
>
Re: Performance problem with large wide row inserts using CQL
Posted by Rüdiger Klaehn <rk...@gmail.com>.
On Fri, Feb 21, 2014 at 11:51 AM, Sylvain Lebresne <sy...@datastax.com>wrote:
> On Thu, Feb 20, 2014 at 10:49 PM, Rüdiger Klaehn <rk...@gmail.com>wrote:
>
>> Hi Sylvain,
>>
>> I applied the patch to the cassandra-2.0 branch (this required some
>> manual work since I could not figure out which commit it was supposed to
>> apply for, and it did not apply to the head of cassandra-2.0).
>>
>
> Yeah, some commit yesterday made the patch not apply cleanly anymore. In
> any case, It's not committed to the cassandra-2.0 branch and will be part
> of 2.0.6.
>
>
>> The benchmark now runs in pretty much identical time to the thrift based
>> benchmark. ~30s for 1000 inserts of 10000 key/value pairs each. Great work!
>>
>
> Glad that it helped.
>
>
Thanks for the quick fix. I was really starting to get irritated when the
people at SO basically told me that there is something wrong in my code.
>
>> I still have some questions regarding the mapping. Please bear with me if
>> these are stupid questions. I am quite new to Cassandra.
>>
>> The basic cassandra data model for a keyspace is something like this,
>> right?
>>
>> SortedMap<byte[], SortedMap<byte[], Pair<Long, byte[]>>
>> ^ row key. determines which server(s) the rest is stored
>> on
>> ^ column key
>> ^
>> timestamp (latest one wins)
>> ^
>> value (can be size 0)
>>
>
> It's a reasonable way to think of how things are stored internally, yes.
> Though as DuyHai mentioned, the first map is really sorting by token and in
> general that means you use mostly the sorting of the second map concretely.
>
>
Yes, understood.
So the first SortedMap is sorted on some kind of hash of the actual key to
make sure the data gets evenly distributed along the nodes? What if my key
is already a good hash: is there a way to use an identity function as a
hash function (in CQL)? I am thinking about some kind of content addressed
storage, where the key is a 20 byte SHA1 hash of the data (like in git).
Obviously this is already a pretty good hash.
>
>> So if I have a table like the one in my benchmark (using blobs)
>>
>> CREATE TABLE IF NOT EXISTS test.wide (
>> time blob,
>> name blob,
>> value blob,
>> PRIMARY KEY (time,name))
>> WITH COMPACT STORAGE
>>
>> From reading http://www.datastax.com/dev/blog/thrift-to-cql3 it seems
>> that
>>
>> - time maps to the row key and name maps to the column key without any
>> overhead
>> - value directly maps to value in the model above without any prefix
>>
>> is that correct, or is there some overhead involved in CQL over the raw
>> model as described above? If so, where exactly?
>>
>
> That's correct.
> For completeness sake, if you were to remove the COMPACT STORAGE, there
> would be some overhead in how it maps to the underlying column key, but
> that overhead would buy you much more flexibility in how you could evolve
> this table schema (you could add more CQL columns later if needs be, have
> collections or have static columns following CASSANDRA-6561 that comes in
> 2.0.6; none of which you can have with COMPACT STORAGE). Note that it's
> perfectly fine to use COMPACT STORAGE if you know you don't and won't need
> the additional flexibility, but I generally advise people to actually check
> first that using COMPACT STORAGE does make a concrete and meaningful
> difference for their use case (be careful with premature optimization
> really).
>
In this case I am confident that the schema will not change. But there will
be other tables built from the same data where I am not going to use
compact storage.
cheers,
Rüdiger
Re: Performance problem with large wide row inserts using CQL
Posted by Sylvain Lebresne <sy...@datastax.com>.
On Thu, Feb 20, 2014 at 10:49 PM, Rüdiger Klaehn <rk...@gmail.com> wrote:
> Hi Sylvain,
>
> I applied the patch to the cassandra-2.0 branch (this required some manual
> work since I could not figure out which commit it was supposed to apply
> for, and it did not apply to the head of cassandra-2.0).
>
Yeah, some commit yesterday made the patch not apply cleanly anymore. In
any case, It's not committed to the cassandra-2.0 branch and will be part
of 2.0.6.
> The benchmark now runs in pretty much identical time to the thrift based
> benchmark. ~30s for 1000 inserts of 10000 key/value pairs each. Great work!
>
Glad that it helped.
>
> I still have some questions regarding the mapping. Please bear with me if
> these are stupid questions. I am quite new to Cassandra.
>
> The basic cassandra data model for a keyspace is something like this,
> right?
>
> SortedMap<byte[], SortedMap<byte[], Pair<Long, byte[]>>
> ^ row key. determines which server(s) the rest is stored
> on
> ^ column key
> ^ timestamp
> (latest one wins)
> ^
> value (can be size 0)
>
It's a reasonable way to think of how things are stored internally, yes.
Though as DuyHai mentioned, the first map is really sorting by token and in
general that means you use mostly the sorting of the second map concretely.
>
> So if I have a table like the one in my benchmark (using blobs)
>
> CREATE TABLE IF NOT EXISTS test.wide (
> time blob,
> name blob,
> value blob,
> PRIMARY KEY (time,name))
> WITH COMPACT STORAGE
>
> From reading http://www.datastax.com/dev/blog/thrift-to-cql3 it seems
> that
>
> - time maps to the row key and name maps to the column key without any
> overhead
> - value directly maps to value in the model above without any prefix
>
> is that correct, or is there some overhead involved in CQL over the raw
> model as described above? If so, where exactly?
>
That's correct.
For completeness sake, if you were to remove the COMPACT STORAGE, there
would be some overhead in how it maps to the underlying column key, but
that overhead would buy you much more flexibility in how you could evolve
this table schema (you could add more CQL columns later if needs be, have
collections or have static columns following CASSANDRA-6561 that comes in
2.0.6; none of which you can have with COMPACT STORAGE). Note that it's
perfectly fine to use COMPACT STORAGE if you know you don't and won't need
the additional flexibility, but I generally advise people to actually check
first that using COMPACT STORAGE does make a concrete and meaningful
difference for their use case (be careful with premature optimization
really). The difference in performance/storage space used is not always all
that noticeable in practice (note that I didn't said it's never
noticeable!) and is narrowing with Cassandra evolution (it's not impossible
at all that we will get to "never noticeable" someday, while COMPACT
STORAGE tables will never get the flexibility of normal tables because
there is backwards compatibility issues). It's also my experience that more
often that not (again, not always), flexibility turns out to be more
important that squeezing every bit of performance you can (if it comes at
the price of that flexibility that is) in the long run. Do what you want of
that advise :)
--
Sylvain
>
> kind regards and many thanks for your help,
>
> Rüdiger
>
>
> On Thu, Feb 20, 2014 at 8:36 AM, Sylvain Lebresne <sy...@datastax.com>wrote:
>
>>
>>
>>
>> On Wed, Feb 19, 2014 at 9:38 PM, Rüdiger Klaehn <rk...@gmail.com>wrote:
>>
>>>
>>> I have cloned the cassandra repo, applied the patch, and built it. But
>>> when I want to run the bechmark I get an exception. See below. I tried with
>>> a non-managed dependency to
>>> cassandra-driver-core-2.0.0-rc3-SNAPSHOT-jar-with-dependencies.jar, which I
>>> compiled from source because I read that that might help. But that did not
>>> make a difference.
>>>
>>> So currently I don't know how to give the patch a try. Any ideas?
>>>
>>> cheers,
>>>
>>> Rüdiger
>>>
>>> Exception in thread "main" java.lang.IllegalArgumentException:
>>> replicate_on_write is not a column defined in this metadata
>>> at
>>> com.datastax.driver.core.ColumnDefinitions.getAllIdx(ColumnDefinitions.java:273)
>>> at
>>> com.datastax.driver.core.ColumnDefinitions.getFirstIdx(ColumnDefinitions.java:279)
>>> at com.datastax.driver.core.Row.getBool(Row.java:117)
>>> at
>>> com.datastax.driver.core.TableMetadata$Options.<init>(TableMetadata.java:474)
>>> at
>>> com.datastax.driver.core.TableMetadata.build(TableMetadata.java:107)
>>> at
>>> com.datastax.driver.core.Metadata.buildTableMetadata(Metadata.java:128)
>>> at com.datastax.driver.core.Metadata.rebuildSchema(Metadata.java:89)
>>> at
>>> com.datastax.driver.core.ControlConnection.refreshSchema(ControlConnection.java:259)
>>> at
>>> com.datastax.driver.core.ControlConnection.tryConnect(ControlConnection.java:214)
>>> at
>>> com.datastax.driver.core.ControlConnection.reconnectInternal(ControlConnection.java:161)
>>> at
>>> com.datastax.driver.core.ControlConnection.connect(ControlConnection.java:77)
>>> at com.datastax.driver.core.Cluster$Manager.init(Cluster.java:890)
>>> at
>>> com.datastax.driver.core.Cluster$Manager.newSession(Cluster.java:910)
>>> at
>>> com.datastax.driver.core.Cluster$Manager.access$200(Cluster.java:806)
>>> at com.datastax.driver.core.Cluster.connect(Cluster.java:158)
>>> at
>>> cassandra.CassandraTestMinimized$delayedInit$body.apply(CassandraTestMinimized.scala:31)
>>> at scala.Function0$class.apply$mcV$sp(Function0.scala:40)
>>> at
>>> scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:12)
>>> at scala.App$$anonfun$main$1.apply(App.scala:71)
>>> at scala.App$$anonfun$main$1.apply(App.scala:71)
>>> at scala.collection.immutable.List.foreach(List.scala:318)
>>> at
>>> scala.collection.generic.TraversableForwarder$class.foreach(TraversableForwarder.scala:32)
>>> at scala.App$class.main(App.scala:71)
>>> at
>>> cassandra.CassandraTestMinimized$.main(CassandraTestMinimized.scala:5)
>>> at
>>> cassandra.CassandraTestMinimized.main(CassandraTestMinimized.scala)
>>>
>>
>> I believe you've tried the cassandra trunk branch? trunk is basically the
>> future Cassandra 2.1 and the driver is currently unhappy because the
>> replicate_on_write option has been removed in that version. I'm supposed to
>> have fixed that on the driver 2.0 branch like 2 days ago so maybe you're
>> also using a slightly old version of the driver sources in there? Or maybe
>> I've screwed up my fix, I'll double check. But anyway, it would be overall
>> simpler to test with the cassandra-2.0 branch of Cassandra, with which you
>> shouldn't run into that.
>>
>> --
>> Sylvain
>>
>
>
Re: Performance problem with large wide row inserts using CQL
Posted by Rüdiger Klaehn <rk...@gmail.com>.
Hi Sylvain,
I applied the patch to the cassandra-2.0 branch (this required some manual
work since I could not figure out which commit it was supposed to apply
for, and it did not apply to the head of cassandra-2.0).
The benchmark now runs in pretty much identical time to the thrift based
benchmark. ~30s for 1000 inserts of 10000 key/value pairs each. Great work!
I still have some questions regarding the mapping. Please bear with me if
these are stupid questions. I am quite new to Cassandra.
The basic cassandra data model for a keyspace is something like this, right?
SortedMap<byte[], SortedMap<byte[], Pair<Long, byte[]>>
^ row key. determines which server(s) the rest is stored on
^ column key
^ timestamp
(latest one wins)
^
value (can be size 0)
So if I have a table like the one in my benchmark (using blobs)
CREATE TABLE IF NOT EXISTS test.wide (
time blob,
name blob,
value blob,
PRIMARY KEY (time,name))
WITH COMPACT STORAGE
>From reading http://www.datastax.com/dev/blog/thrift-to-cql3 it seems that
- time maps to the row key and name maps to the column key without any
overhead
- value directly maps to value in the model above without any prefix
is that correct, or is there some overhead involved in CQL over the raw
model as described above? If so, where exactly?
kind regards and many thanks for your help,
Rüdiger
On Thu, Feb 20, 2014 at 8:36 AM, Sylvain Lebresne <sy...@datastax.com>wrote:
>
>
>
> On Wed, Feb 19, 2014 at 9:38 PM, Rüdiger Klaehn <rk...@gmail.com> wrote:
>
>>
>> I have cloned the cassandra repo, applied the patch, and built it. But
>> when I want to run the bechmark I get an exception. See below. I tried with
>> a non-managed dependency to
>> cassandra-driver-core-2.0.0-rc3-SNAPSHOT-jar-with-dependencies.jar, which I
>> compiled from source because I read that that might help. But that did not
>> make a difference.
>>
>> So currently I don't know how to give the patch a try. Any ideas?
>>
>> cheers,
>>
>> Rüdiger
>>
>> Exception in thread "main" java.lang.IllegalArgumentException:
>> replicate_on_write is not a column defined in this metadata
>> at
>> com.datastax.driver.core.ColumnDefinitions.getAllIdx(ColumnDefinitions.java:273)
>> at
>> com.datastax.driver.core.ColumnDefinitions.getFirstIdx(ColumnDefinitions.java:279)
>> at com.datastax.driver.core.Row.getBool(Row.java:117)
>> at
>> com.datastax.driver.core.TableMetadata$Options.<init>(TableMetadata.java:474)
>> at
>> com.datastax.driver.core.TableMetadata.build(TableMetadata.java:107)
>> at
>> com.datastax.driver.core.Metadata.buildTableMetadata(Metadata.java:128)
>> at com.datastax.driver.core.Metadata.rebuildSchema(Metadata.java:89)
>> at
>> com.datastax.driver.core.ControlConnection.refreshSchema(ControlConnection.java:259)
>> at
>> com.datastax.driver.core.ControlConnection.tryConnect(ControlConnection.java:214)
>> at
>> com.datastax.driver.core.ControlConnection.reconnectInternal(ControlConnection.java:161)
>> at
>> com.datastax.driver.core.ControlConnection.connect(ControlConnection.java:77)
>> at com.datastax.driver.core.Cluster$Manager.init(Cluster.java:890)
>> at
>> com.datastax.driver.core.Cluster$Manager.newSession(Cluster.java:910)
>> at
>> com.datastax.driver.core.Cluster$Manager.access$200(Cluster.java:806)
>> at com.datastax.driver.core.Cluster.connect(Cluster.java:158)
>> at
>> cassandra.CassandraTestMinimized$delayedInit$body.apply(CassandraTestMinimized.scala:31)
>> at scala.Function0$class.apply$mcV$sp(Function0.scala:40)
>> at
>> scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:12)
>> at scala.App$$anonfun$main$1.apply(App.scala:71)
>> at scala.App$$anonfun$main$1.apply(App.scala:71)
>> at scala.collection.immutable.List.foreach(List.scala:318)
>> at
>> scala.collection.generic.TraversableForwarder$class.foreach(TraversableForwarder.scala:32)
>> at scala.App$class.main(App.scala:71)
>> at
>> cassandra.CassandraTestMinimized$.main(CassandraTestMinimized.scala:5)
>> at cassandra.CassandraTestMinimized.main(CassandraTestMinimized.scala)
>>
>
> I believe you've tried the cassandra trunk branch? trunk is basically the
> future Cassandra 2.1 and the driver is currently unhappy because the
> replicate_on_write option has been removed in that version. I'm supposed to
> have fixed that on the driver 2.0 branch like 2 days ago so maybe you're
> also using a slightly old version of the driver sources in there? Or maybe
> I've screwed up my fix, I'll double check. But anyway, it would be overall
> simpler to test with the cassandra-2.0 branch of Cassandra, with which you
> shouldn't run into that.
>
> --
> Sylvain
>
Re: Performance problem with large wide row inserts using CQL
Posted by Sylvain Lebresne <sy...@datastax.com>.
On Fri, Feb 21, 2014 at 8:53 PM, Yogi Nerella <yn...@gmail.com> wrote:
> I am using CCM to install the servers, it is bringing in the source code,
> is there any option for CCM which I can set only to download the binary,
> just to make sure it is not bringing in the working copy of the code.
>
No there isn't. And though I'm happy to help you with CCM problems, it
would be nice to at least create a separate thread for this as it's pretty
much unrelated to the initial email of this thread.
>
> I am using the following statements to create Keyspace and table
> definition.
>
> create keyspace test1 with replication = { 'class':'SimpleStrategy',
> 'replication_factor':1};
>
> CREATE TABLE IF NOT EXISTS wide (
> time varchar,
> name varchar,
> value varchar,
> PRIMARY KEY (time,name))
> WITH COMPACT STORAGE;
>
>
> On Fri, Feb 21, 2014 at 11:47 AM, Yogi Nerella <yn...@gmail.com>wrote:
>
>> Sylvain,
>>
>> I am trying ccm to install and it does from source directory, I have
>> tried 2.0.4/3/2/1 and 1.2.15, all of them are reporting the same failure
>> after 127 records inserted.
>>
>> I am using 1.56.34 and 1.56.38 client both reports the same issue.
>>
>> Is something wrong with the client or the server, none of the server logs
>> show any error.
>>
>> Thanks,
>> Yogi
>>
>>
>> On Wed, Feb 19, 2014 at 11:36 PM, Sylvain Lebresne <sy...@datastax.com>wrote:
>>
>>>
>>>
>>>
>>> On Wed, Feb 19, 2014 at 9:38 PM, Rüdiger Klaehn <rk...@gmail.com>wrote:
>>>
>>>>
>>>> I have cloned the cassandra repo, applied the patch, and built it. But
>>>> when I want to run the bechmark I get an exception. See below. I tried with
>>>> a non-managed dependency to
>>>> cassandra-driver-core-2.0.0-rc3-SNAPSHOT-jar-with-dependencies.jar, which I
>>>> compiled from source because I read that that might help. But that did not
>>>> make a difference.
>>>>
>>>> So currently I don't know how to give the patch a try. Any ideas?
>>>>
>>>> cheers,
>>>>
>>>> Rüdiger
>>>>
>>>> Exception in thread "main" java.lang.IllegalArgumentException:
>>>> replicate_on_write is not a column defined in this metadata
>>>> at
>>>> com.datastax.driver.core.ColumnDefinitions.getAllIdx(ColumnDefinitions.java:273)
>>>> at
>>>> com.datastax.driver.core.ColumnDefinitions.getFirstIdx(ColumnDefinitions.java:279)
>>>> at com.datastax.driver.core.Row.getBool(Row.java:117)
>>>> at
>>>> com.datastax.driver.core.TableMetadata$Options.<init>(TableMetadata.java:474)
>>>> at
>>>> com.datastax.driver.core.TableMetadata.build(TableMetadata.java:107)
>>>> at
>>>> com.datastax.driver.core.Metadata.buildTableMetadata(Metadata.java:128)
>>>> at com.datastax.driver.core.Metadata.rebuildSchema(Metadata.java:89)
>>>> at
>>>> com.datastax.driver.core.ControlConnection.refreshSchema(ControlConnection.java:259)
>>>> at
>>>> com.datastax.driver.core.ControlConnection.tryConnect(ControlConnection.java:214)
>>>> at
>>>> com.datastax.driver.core.ControlConnection.reconnectInternal(ControlConnection.java:161)
>>>> at
>>>> com.datastax.driver.core.ControlConnection.connect(ControlConnection.java:77)
>>>> at com.datastax.driver.core.Cluster$Manager.init(Cluster.java:890)
>>>> at
>>>> com.datastax.driver.core.Cluster$Manager.newSession(Cluster.java:910)
>>>> at
>>>> com.datastax.driver.core.Cluster$Manager.access$200(Cluster.java:806)
>>>> at com.datastax.driver.core.Cluster.connect(Cluster.java:158)
>>>> at
>>>> cassandra.CassandraTestMinimized$delayedInit$body.apply(CassandraTestMinimized.scala:31)
>>>> at scala.Function0$class.apply$mcV$sp(Function0.scala:40)
>>>> at
>>>> scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:12)
>>>> at scala.App$$anonfun$main$1.apply(App.scala:71)
>>>> at scala.App$$anonfun$main$1.apply(App.scala:71)
>>>> at scala.collection.immutable.List.foreach(List.scala:318)
>>>> at
>>>> scala.collection.generic.TraversableForwarder$class.foreach(TraversableForwarder.scala:32)
>>>> at scala.App$class.main(App.scala:71)
>>>> at
>>>> cassandra.CassandraTestMinimized$.main(CassandraTestMinimized.scala:5)
>>>> at
>>>> cassandra.CassandraTestMinimized.main(CassandraTestMinimized.scala)
>>>>
>>>
>>> I believe you've tried the cassandra trunk branch? trunk is basically
>>> the future Cassandra 2.1 and the driver is currently unhappy because the
>>> replicate_on_write option has been removed in that version. I'm supposed to
>>> have fixed that on the driver 2.0 branch like 2 days ago so maybe you're
>>> also using a slightly old version of the driver sources in there? Or maybe
>>> I've screwed up my fix, I'll double check. But anyway, it would be overall
>>> simpler to test with the cassandra-2.0 branch of Cassandra, with which you
>>> shouldn't run into that.
>>>
>>> --
>>> Sylvain
>>>
>>
>>
>
Re: Performance problem with large wide row inserts using CQL
Posted by Yogi Nerella <yn...@gmail.com>.
I am using CCM to install the servers, it is bringing in the source code,
is there any option for CCM which I can set only to download the binary,
just to make sure it is not bringing in the working copy of the code.
I am using the following statements to create Keyspace and table definition.
create keyspace test1 with replication = { 'class':'SimpleStrategy',
'replication_factor':1};
CREATE TABLE IF NOT EXISTS wide (
time varchar,
name varchar,
value varchar,
PRIMARY KEY (time,name))
WITH COMPACT STORAGE;
On Fri, Feb 21, 2014 at 11:47 AM, Yogi Nerella <yn...@gmail.com>wrote:
> Sylvain,
>
> I am trying ccm to install and it does from source directory, I have tried
> 2.0.4/3/2/1 and 1.2.15, all of them are reporting the same failure after
> 127 records inserted.
>
> I am using 1.56.34 and 1.56.38 client both reports the same issue.
>
> Is something wrong with the client or the server, none of the server logs
> show any error.
>
> Thanks,
> Yogi
>
>
> On Wed, Feb 19, 2014 at 11:36 PM, Sylvain Lebresne <sy...@datastax.com>wrote:
>
>>
>>
>>
>> On Wed, Feb 19, 2014 at 9:38 PM, Rüdiger Klaehn <rk...@gmail.com>wrote:
>>
>>>
>>> I have cloned the cassandra repo, applied the patch, and built it. But
>>> when I want to run the bechmark I get an exception. See below. I tried with
>>> a non-managed dependency to
>>> cassandra-driver-core-2.0.0-rc3-SNAPSHOT-jar-with-dependencies.jar, which I
>>> compiled from source because I read that that might help. But that did not
>>> make a difference.
>>>
>>> So currently I don't know how to give the patch a try. Any ideas?
>>>
>>> cheers,
>>>
>>> Rüdiger
>>>
>>> Exception in thread "main" java.lang.IllegalArgumentException:
>>> replicate_on_write is not a column defined in this metadata
>>> at
>>> com.datastax.driver.core.ColumnDefinitions.getAllIdx(ColumnDefinitions.java:273)
>>> at
>>> com.datastax.driver.core.ColumnDefinitions.getFirstIdx(ColumnDefinitions.java:279)
>>> at com.datastax.driver.core.Row.getBool(Row.java:117)
>>> at
>>> com.datastax.driver.core.TableMetadata$Options.<init>(TableMetadata.java:474)
>>> at
>>> com.datastax.driver.core.TableMetadata.build(TableMetadata.java:107)
>>> at
>>> com.datastax.driver.core.Metadata.buildTableMetadata(Metadata.java:128)
>>> at com.datastax.driver.core.Metadata.rebuildSchema(Metadata.java:89)
>>> at
>>> com.datastax.driver.core.ControlConnection.refreshSchema(ControlConnection.java:259)
>>> at
>>> com.datastax.driver.core.ControlConnection.tryConnect(ControlConnection.java:214)
>>> at
>>> com.datastax.driver.core.ControlConnection.reconnectInternal(ControlConnection.java:161)
>>> at
>>> com.datastax.driver.core.ControlConnection.connect(ControlConnection.java:77)
>>> at com.datastax.driver.core.Cluster$Manager.init(Cluster.java:890)
>>> at
>>> com.datastax.driver.core.Cluster$Manager.newSession(Cluster.java:910)
>>> at
>>> com.datastax.driver.core.Cluster$Manager.access$200(Cluster.java:806)
>>> at com.datastax.driver.core.Cluster.connect(Cluster.java:158)
>>> at
>>> cassandra.CassandraTestMinimized$delayedInit$body.apply(CassandraTestMinimized.scala:31)
>>> at scala.Function0$class.apply$mcV$sp(Function0.scala:40)
>>> at
>>> scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:12)
>>> at scala.App$$anonfun$main$1.apply(App.scala:71)
>>> at scala.App$$anonfun$main$1.apply(App.scala:71)
>>> at scala.collection.immutable.List.foreach(List.scala:318)
>>> at
>>> scala.collection.generic.TraversableForwarder$class.foreach(TraversableForwarder.scala:32)
>>> at scala.App$class.main(App.scala:71)
>>> at
>>> cassandra.CassandraTestMinimized$.main(CassandraTestMinimized.scala:5)
>>> at
>>> cassandra.CassandraTestMinimized.main(CassandraTestMinimized.scala)
>>>
>>
>> I believe you've tried the cassandra trunk branch? trunk is basically the
>> future Cassandra 2.1 and the driver is currently unhappy because the
>> replicate_on_write option has been removed in that version. I'm supposed to
>> have fixed that on the driver 2.0 branch like 2 days ago so maybe you're
>> also using a slightly old version of the driver sources in there? Or maybe
>> I've screwed up my fix, I'll double check. But anyway, it would be overall
>> simpler to test with the cassandra-2.0 branch of Cassandra, with which you
>> shouldn't run into that.
>>
>> --
>> Sylvain
>>
>
>
Re: Performance problem with large wide row inserts using CQL
Posted by Yogi Nerella <yn...@gmail.com>.
Sylvain,
I am trying ccm to install and it does from source directory, I have tried
2.0.4/3/2/1 and 1.2.15, all of them are reporting the same failure after
127 records inserted.
I am using 1.56.34 and 1.56.38 client both reports the same issue.
Is something wrong with the client or the server, none of the server logs
show any error.
Thanks,
Yogi
On Wed, Feb 19, 2014 at 11:36 PM, Sylvain Lebresne <sy...@datastax.com>wrote:
>
>
>
> On Wed, Feb 19, 2014 at 9:38 PM, Rüdiger Klaehn <rk...@gmail.com> wrote:
>
>>
>> I have cloned the cassandra repo, applied the patch, and built it. But
>> when I want to run the bechmark I get an exception. See below. I tried with
>> a non-managed dependency to
>> cassandra-driver-core-2.0.0-rc3-SNAPSHOT-jar-with-dependencies.jar, which I
>> compiled from source because I read that that might help. But that did not
>> make a difference.
>>
>> So currently I don't know how to give the patch a try. Any ideas?
>>
>> cheers,
>>
>> Rüdiger
>>
>> Exception in thread "main" java.lang.IllegalArgumentException:
>> replicate_on_write is not a column defined in this metadata
>> at
>> com.datastax.driver.core.ColumnDefinitions.getAllIdx(ColumnDefinitions.java:273)
>> at
>> com.datastax.driver.core.ColumnDefinitions.getFirstIdx(ColumnDefinitions.java:279)
>> at com.datastax.driver.core.Row.getBool(Row.java:117)
>> at
>> com.datastax.driver.core.TableMetadata$Options.<init>(TableMetadata.java:474)
>> at
>> com.datastax.driver.core.TableMetadata.build(TableMetadata.java:107)
>> at
>> com.datastax.driver.core.Metadata.buildTableMetadata(Metadata.java:128)
>> at com.datastax.driver.core.Metadata.rebuildSchema(Metadata.java:89)
>> at
>> com.datastax.driver.core.ControlConnection.refreshSchema(ControlConnection.java:259)
>> at
>> com.datastax.driver.core.ControlConnection.tryConnect(ControlConnection.java:214)
>> at
>> com.datastax.driver.core.ControlConnection.reconnectInternal(ControlConnection.java:161)
>> at
>> com.datastax.driver.core.ControlConnection.connect(ControlConnection.java:77)
>> at com.datastax.driver.core.Cluster$Manager.init(Cluster.java:890)
>> at
>> com.datastax.driver.core.Cluster$Manager.newSession(Cluster.java:910)
>> at
>> com.datastax.driver.core.Cluster$Manager.access$200(Cluster.java:806)
>> at com.datastax.driver.core.Cluster.connect(Cluster.java:158)
>> at
>> cassandra.CassandraTestMinimized$delayedInit$body.apply(CassandraTestMinimized.scala:31)
>> at scala.Function0$class.apply$mcV$sp(Function0.scala:40)
>> at
>> scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:12)
>> at scala.App$$anonfun$main$1.apply(App.scala:71)
>> at scala.App$$anonfun$main$1.apply(App.scala:71)
>> at scala.collection.immutable.List.foreach(List.scala:318)
>> at
>> scala.collection.generic.TraversableForwarder$class.foreach(TraversableForwarder.scala:32)
>> at scala.App$class.main(App.scala:71)
>> at
>> cassandra.CassandraTestMinimized$.main(CassandraTestMinimized.scala:5)
>> at cassandra.CassandraTestMinimized.main(CassandraTestMinimized.scala)
>>
>
> I believe you've tried the cassandra trunk branch? trunk is basically the
> future Cassandra 2.1 and the driver is currently unhappy because the
> replicate_on_write option has been removed in that version. I'm supposed to
> have fixed that on the driver 2.0 branch like 2 days ago so maybe you're
> also using a slightly old version of the driver sources in there? Or maybe
> I've screwed up my fix, I'll double check. But anyway, it would be overall
> simpler to test with the cassandra-2.0 branch of Cassandra, with which you
> shouldn't run into that.
>
> --
> Sylvain
>
Re: Performance problem with large wide row inserts using CQL
Posted by Sylvain Lebresne <sy...@datastax.com>.
On Wed, Feb 19, 2014 at 9:38 PM, Rüdiger Klaehn <rk...@gmail.com> wrote:
>
> I have cloned the cassandra repo, applied the patch, and built it. But
> when I want to run the bechmark I get an exception. See below. I tried with
> a non-managed dependency to
> cassandra-driver-core-2.0.0-rc3-SNAPSHOT-jar-with-dependencies.jar, which I
> compiled from source because I read that that might help. But that did not
> make a difference.
>
> So currently I don't know how to give the patch a try. Any ideas?
>
> cheers,
>
> Rüdiger
>
> Exception in thread "main" java.lang.IllegalArgumentException:
> replicate_on_write is not a column defined in this metadata
> at
> com.datastax.driver.core.ColumnDefinitions.getAllIdx(ColumnDefinitions.java:273)
> at
> com.datastax.driver.core.ColumnDefinitions.getFirstIdx(ColumnDefinitions.java:279)
> at com.datastax.driver.core.Row.getBool(Row.java:117)
> at
> com.datastax.driver.core.TableMetadata$Options.<init>(TableMetadata.java:474)
> at com.datastax.driver.core.TableMetadata.build(TableMetadata.java:107)
> at
> com.datastax.driver.core.Metadata.buildTableMetadata(Metadata.java:128)
> at com.datastax.driver.core.Metadata.rebuildSchema(Metadata.java:89)
> at
> com.datastax.driver.core.ControlConnection.refreshSchema(ControlConnection.java:259)
> at
> com.datastax.driver.core.ControlConnection.tryConnect(ControlConnection.java:214)
> at
> com.datastax.driver.core.ControlConnection.reconnectInternal(ControlConnection.java:161)
> at
> com.datastax.driver.core.ControlConnection.connect(ControlConnection.java:77)
> at com.datastax.driver.core.Cluster$Manager.init(Cluster.java:890)
> at
> com.datastax.driver.core.Cluster$Manager.newSession(Cluster.java:910)
> at
> com.datastax.driver.core.Cluster$Manager.access$200(Cluster.java:806)
> at com.datastax.driver.core.Cluster.connect(Cluster.java:158)
> at
> cassandra.CassandraTestMinimized$delayedInit$body.apply(CassandraTestMinimized.scala:31)
> at scala.Function0$class.apply$mcV$sp(Function0.scala:40)
> at
> scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:12)
> at scala.App$$anonfun$main$1.apply(App.scala:71)
> at scala.App$$anonfun$main$1.apply(App.scala:71)
> at scala.collection.immutable.List.foreach(List.scala:318)
> at
> scala.collection.generic.TraversableForwarder$class.foreach(TraversableForwarder.scala:32)
> at scala.App$class.main(App.scala:71)
> at
> cassandra.CassandraTestMinimized$.main(CassandraTestMinimized.scala:5)
> at cassandra.CassandraTestMinimized.main(CassandraTestMinimized.scala)
>
I believe you've tried the cassandra trunk branch? trunk is basically the
future Cassandra 2.1 and the driver is currently unhappy because the
replicate_on_write option has been removed in that version. I'm supposed to
have fixed that on the driver 2.0 branch like 2 days ago so maybe you're
also using a slightly old version of the driver sources in there? Or maybe
I've screwed up my fix, I'll double check. But anyway, it would be overall
simpler to test with the cassandra-2.0 branch of Cassandra, with which you
shouldn't run into that.
--
Sylvain
Re: Performance problem with large wide row inserts using CQL
Posted by Rüdiger Klaehn <rk...@gmail.com>.
On Wed, Feb 19, 2014 at 7:49 PM, Sylvain Lebresne <sy...@datastax.com>wrote:
> On Wed, Feb 19, 2014 at 11:27 AM, Rüdiger Klaehn <rk...@gmail.com>wrote:
>
>>
>> Am I doing something wrong, or is this a fundamental limitation of CQL.
>>
>
> Neither. I believe you are running into
> https://issues.apache.org/jira/browse/CASSANDRA-6737, which is a bug, a
> performance bug, which we should and will fix. So thanks for the report. If
> you could give a shot to the patch on that issue and check if it helps,
> that would definitively be much appreciated.
>
> Hi Sylvain,
Yes, this issue looks almost identical to the problem I have been
experiencing. Great to hear that you are aware of the issue and that there
is a fix.
I have cloned the cassandra repo, applied the patch, and built it. But when
I want to run the bechmark I get an exception. See below. I tried with a
non-managed dependency to
cassandra-driver-core-2.0.0-rc3-SNAPSHOT-jar-with-dependencies.jar, which I
compiled from source because I read that that might help. But that did not
make a difference.
So currently I don't know how to give the patch a try. Any ideas?
cheers,
Rüdiger
Exception in thread "main" java.lang.IllegalArgumentException:
replicate_on_write is not a column defined in this metadata
at
com.datastax.driver.core.ColumnDefinitions.getAllIdx(ColumnDefinitions.java:273)
at
com.datastax.driver.core.ColumnDefinitions.getFirstIdx(ColumnDefinitions.java:279)
at com.datastax.driver.core.Row.getBool(Row.java:117)
at
com.datastax.driver.core.TableMetadata$Options.<init>(TableMetadata.java:474)
at com.datastax.driver.core.TableMetadata.build(TableMetadata.java:107)
at
com.datastax.driver.core.Metadata.buildTableMetadata(Metadata.java:128)
at com.datastax.driver.core.Metadata.rebuildSchema(Metadata.java:89)
at
com.datastax.driver.core.ControlConnection.refreshSchema(ControlConnection.java:259)
at
com.datastax.driver.core.ControlConnection.tryConnect(ControlConnection.java:214)
at
com.datastax.driver.core.ControlConnection.reconnectInternal(ControlConnection.java:161)
at
com.datastax.driver.core.ControlConnection.connect(ControlConnection.java:77)
at com.datastax.driver.core.Cluster$Manager.init(Cluster.java:890)
at com.datastax.driver.core.Cluster$Manager.newSession(Cluster.java:910)
at com.datastax.driver.core.Cluster$Manager.access$200(Cluster.java:806)
at com.datastax.driver.core.Cluster.connect(Cluster.java:158)
at
cassandra.CassandraTestMinimized$delayedInit$body.apply(CassandraTestMinimized.scala:31)
at scala.Function0$class.apply$mcV$sp(Function0.scala:40)
at
scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:12)
at scala.App$$anonfun$main$1.apply(App.scala:71)
at scala.App$$anonfun$main$1.apply(App.scala:71)
at scala.collection.immutable.List.foreach(List.scala:318)
at
scala.collection.generic.TraversableForwarder$class.foreach(TraversableForwarder.scala:32)
at scala.App$class.main(App.scala:71)
at
cassandra.CassandraTestMinimized$.main(CassandraTestMinimized.scala:5)
at cassandra.CassandraTestMinimized.main(CassandraTestMinimized.scala)
> There is absolutely no fundamental reason why a CQL operation would be
> more than 10 times slower than it's thrift equivalent, such dramatic
> difference is indicative of a bug, something obviously wrong.
>
> --
> Sylvain
>
Re: Performance problem with large wide row inserts using CQL
Posted by Rüdiger Klaehn <rk...@gmail.com>.
Hi Yogi,
Both benchmarks go to different tables. I originally wanted to just write a
lot of data into an empty table and then evaluate what compression ratio I
can expect when I ran into the performance problem.
I am sorry, I forgot to mention this: I did not figure out how to create a
table using AST (it wasn't given in the example that I modified
https://github.com/Netflix/astyanax/blob/master/astyanax-examples/src/main/java/com/netflix/astyanax/examples/AstClient.java).
So the AstClient.java benchmark assumes that the table already exists.
cheers,
Rüdiger
On Wed, Feb 19, 2014 at 7:56 PM, Yogi Nerella <yn...@gmail.com> wrote:
> Rudger,
>
> I am trying this on 2.0.5 to see, but both Scala code and AST code are
> going to different tables? Can you give the exact AST code you are trying?
>
> Yogi
>
>
> On Wed, Feb 19, 2014 at 10:49 AM, Sylvain Lebresne <sy...@datastax.com>wrote:
>
>> On Wed, Feb 19, 2014 at 11:27 AM, Rüdiger Klaehn <rk...@gmail.com>wrote:
>>
>>>
>>> Am I doing something wrong, or is this a fundamental limitation of CQL.
>>>
>>
>> Neither. I believe you are running into
>> https://issues.apache.org/jira/browse/CASSANDRA-6737, which is a bug, a
>> performance bug, which we should and will fix. So thanks for the report. If
>> you could give a shot to the patch on that issue and check if it helps,
>> that would definitively be much appreciated.
>>
>> There is absolutely no fundamental reason why a CQL operation would be
>> more than 10 times slower than it's thrift equivalent, such dramatic
>> difference is indicative of a bug, something obviously wrong.
>>
>> --
>> Sylvain
>>
>
>
Re: Performance problem with large wide row inserts using CQL
Posted by Yogi Nerella <yn...@gmail.com>.
Rudger,
I am trying this on 2.0.5 to see, but both Scala code and AST code are
going to different tables? Can you give the exact AST code you are trying?
Yogi
On Wed, Feb 19, 2014 at 10:49 AM, Sylvain Lebresne <sy...@datastax.com>wrote:
> On Wed, Feb 19, 2014 at 11:27 AM, Rüdiger Klaehn <rk...@gmail.com>wrote:
>
>>
>> Am I doing something wrong, or is this a fundamental limitation of CQL.
>>
>
> Neither. I believe you are running into
> https://issues.apache.org/jira/browse/CASSANDRA-6737, which is a bug, a
> performance bug, which we should and will fix. So thanks for the report. If
> you could give a shot to the patch on that issue and check if it helps,
> that would definitively be much appreciated.
>
> There is absolutely no fundamental reason why a CQL operation would be
> more than 10 times slower than it's thrift equivalent, such dramatic
> difference is indicative of a bug, something obviously wrong.
>
> --
> Sylvain
>
Re: Performance problem with large wide row inserts using CQL
Posted by Sylvain Lebresne <sy...@datastax.com>.
On Wed, Feb 19, 2014 at 11:27 AM, Rüdiger Klaehn <rk...@gmail.com> wrote:
>
> Am I doing something wrong, or is this a fundamental limitation of CQL.
>
Neither. I believe you are running into
https://issues.apache.org/jira/browse/CASSANDRA-6737, which is a bug, a
performance bug, which we should and will fix. So thanks for the report. If
you could give a shot to the patch on that issue and check if it helps,
that would definitively be much appreciated.
There is absolutely no fundamental reason why a CQL operation would be more
than 10 times slower than it's thrift equivalent, such dramatic difference
is indicative of a bug, something obviously wrong.
--
Sylvain
Re: Performance problem with large wide row inserts using CQL
Posted by DuyHai Doan <do...@gmail.com>.
Agree with John
Preparing a statement follows this process:
1) send the statement to the server
2) statement validation on server side
3) if validation is ok, the C* node will assign an UUID to this prepared
statement
4) send back the UUID to the java driver core
Now, you can re-use this same prepared statement millions of time with
BoundStatement bs = preparedStatement.bind(values ....)
Please note that there will be a maximum of 100 000 prepared statements
retained per node
On Wed, Feb 19, 2014 at 3:57 PM, John Sanda <jo...@gmail.com> wrote:
> From a quick glance at your code, it looks like you are preparing your
> insert statement multiple times. You only need to prepare it once. I would
> expect to see some improvement with that change.
>
>
> On Wed, Feb 19, 2014 at 5:27 AM, Rüdiger Klaehn <rk...@gmail.com> wrote:
>
>> Hi all,
>>
>> I am evaluating Cassandra for satellite telemetry storage and analysis. I
>> set up a little three node cluster on my local development machine and
>> wrote a few simple test programs.
>>
>> My use case requires storing incoming telemetry updates in the database
>> at the same rate as they are coming in. A telemetry update is a map of
>> name/value pairs that arrives at a certain time.
>>
>> The idea is that I want to store the data as quickly as possible, and
>> then later store it in an additional format that is more amenable to
>> analysis.
>>
>> The format I have chosen for my test is the following:
>>
>> CREATE TABLE IF NOT EXISTS test.wide (
>> time varchar,
>> name varchar,
>> value varchar,
>> PRIMARY KEY (time,name))
>> WITH COMPACT STORAGE
>>
>> The layout I want to achieve with this is something like this:
>>
>> +-------+-------+-------+-------+-------+-------+
>> | | name1 | name2 | name3 | ... | nameN |
>> | time +-------+-------+-------+-------+-------+
>> | | val1 | val2 | val3 | ... | valN |
>> +-------+-------+-------+-------|-------+-------+
>>
>> (Time will at some point be some kind of timestamp, and value will become
>> a blob. But this is just for initial testing)
>>
>> The problem is the following: I am getting very low performance for bulk
>> inserts into the above table. In my test program, each insert has a new,
>> unique time and creates a row with 10000 name/value pairs. This should map
>> into creating a new row in the underlying storage engine, correct? I do
>> that 1000 times and measure both time per insert and total time.
>>
>> I am getting about 0.5s for each insert of 10000 name/value pairs, which
>> is much lower than the rate at which the telemetry is coming in at my
>> system. I have read a few previous threads on this subject and am using
>> batch prepared statements for maximum performance (
>> https://issues.apache.org/jira/browse/CASSANDRA-4693 ). But that does
>> not help.
>>
>> Here is the CQL benchmark:
>> https://gist.github.com/rklaehn/9089304#file-cassandratestminimized-scala
>>
>> I have written the exact same thing using the thrift API of astyanax, and
>> I am getting much better performance. Each insert of 10000 name/values
>> takes 0.04s using a ColumnListMutation. When I use async calls for both
>> programs, as suggested by somebody on Stackoverflow, the difference gets
>> even larger. The CQL insert remains at 0.5s per insert on average, whereas
>> the astyanax ColumnListMutation approach takes 0.01s per insert on
>> average, even on my test cluster. That's the kind of performance I need.
>>
>> Here is the thrift benchmark, modified from an ast example:
>> https://gist.github.com/rklaehn/9089304#file-astclient-java
>>
>> I realize that running on a test cluster on localhost is not a 100%
>> realistic test. But nevertheless you would expect both tests to have
>> roughly similar performance.
>>
>> I saw a few suggestions to create a table with CQL and fill it using the
>> thrift API. For example in this thread
>> http://mail-archives.apache.org/mod_mbox/cassandra-user/201309.mbox/%3C523334B8.8070802@gmail.com%3E. But I would very much prefer to use pure CQL for this. It seems that the
>> thrift API is considered deprecated, so I would not feel comfortable
>> starting a new project using a legacy API.
>>
>> I already posted a question on SO about this, but did not get any
>> satisfactory answer. Just general performance tuning tips that do nothing
>> to explain the difference between the CQL and thrift approaches.
>>
>> http://stackoverflow.com/questions/21778671/cassandra-how-to-insert-a-new-wide-row-with-good-performance-using-cql
>>
>> Am I doing something wrong, or is this a fundamental limitation of CQL.
>> If the latter is the case, what's the plan to mitigate the issue?
>>
>> There is a JIRA issue about this (
>> https://issues.apache.org/jira/browse/CASSANDRA-5959 ), but it is marked
>> as a duplicate of https://issues.apache.org/jira/browse/CASSANDRA-4693 .
>> But according to my benchmarks batch prepared statements do not solve this
>> issue!
>>
>> I would really appreciate any help on this issue. The telemetry data I
>> would like to import into C* for testing contains ~2*10^12 samples, where
>> each sample consists of time, value and status. If quick batch insertion is
>> not possible, I would not even be able to insert it in an acceptable time.
>>
>> best regards,
>>
>> Rüdiger
>>
>
>
>
> --
>
> - John
>
Re: Performance problem with large wide row inserts using CQL
Posted by Nate McCall <na...@thelastpickle.com>.
Hi Rüdiger,
I just saw this after I answered on the SO thread:
http://stackoverflow.com/questions/21778671/cassandra-how-to-insert-a-new-wide-row-with-good-performance-using-cql/21884943#21884943
On Wed, Feb 19, 2014 at 8:57 AM, John Sanda <jo...@gmail.com> wrote:
> From a quick glance at your code, it looks like you are preparing your
> insert statement multiple times. You only need to prepare it once. I would
> expect to see some improvement with that change.
>
>
> On Wed, Feb 19, 2014 at 5:27 AM, Rüdiger Klaehn <rk...@gmail.com> wrote:
>
>> Hi all,
>>
>> I am evaluating Cassandra for satellite telemetry storage and analysis. I
>> set up a little three node cluster on my local development machine and
>> wrote a few simple test programs.
>>
>> My use case requires storing incoming telemetry updates in the database
>> at the same rate as they are coming in. A telemetry update is a map of
>> name/value pairs that arrives at a certain time.
>>
>> The idea is that I want to store the data as quickly as possible, and
>> then later store it in an additional format that is more amenable to
>> analysis.
>>
>> The format I have chosen for my test is the following:
>>
>> CREATE TABLE IF NOT EXISTS test.wide (
>> time varchar,
>> name varchar,
>> value varchar,
>> PRIMARY KEY (time,name))
>> WITH COMPACT STORAGE
>>
>> The layout I want to achieve with this is something like this:
>>
>> +-------+-------+-------+-------+-------+-------+
>> | | name1 | name2 | name3 | ... | nameN |
>> | time +-------+-------+-------+-------+-------+
>> | | val1 | val2 | val3 | ... | valN |
>> +-------+-------+-------+-------|-------+-------+
>>
>> (Time will at some point be some kind of timestamp, and value will become
>> a blob. But this is just for initial testing)
>>
>> The problem is the following: I am getting very low performance for bulk
>> inserts into the above table. In my test program, each insert has a new,
>> unique time and creates a row with 10000 name/value pairs. This should map
>> into creating a new row in the underlying storage engine, correct? I do
>> that 1000 times and measure both time per insert and total time.
>>
>> I am getting about 0.5s for each insert of 10000 name/value pairs, which
>> is much lower than the rate at which the telemetry is coming in at my
>> system. I have read a few previous threads on this subject and am using
>> batch prepared statements for maximum performance (
>> https://issues.apache.org/jira/browse/CASSANDRA-4693 ). But that does
>> not help.
>>
>> Here is the CQL benchmark:
>> https://gist.github.com/rklaehn/9089304#file-cassandratestminimized-scala
>>
>> I have written the exact same thing using the thrift API of astyanax, and
>> I am getting much better performance. Each insert of 10000 name/values
>> takes 0.04s using a ColumnListMutation. When I use async calls for both
>> programs, as suggested by somebody on Stackoverflow, the difference gets
>> even larger. The CQL insert remains at 0.5s per insert on average, whereas
>> the astyanax ColumnListMutation approach takes 0.01s per insert on
>> average, even on my test cluster. That's the kind of performance I need.
>>
>> Here is the thrift benchmark, modified from an ast example:
>> https://gist.github.com/rklaehn/9089304#file-astclient-java
>>
>> I realize that running on a test cluster on localhost is not a 100%
>> realistic test. But nevertheless you would expect both tests to have
>> roughly similar performance.
>>
>> I saw a few suggestions to create a table with CQL and fill it using the
>> thrift API. For example in this thread
>> http://mail-archives.apache.org/mod_mbox/cassandra-user/201309.mbox/%3C523334B8.8070802@gmail.com%3E. But I would very much prefer to use pure CQL for this. It seems that the
>> thrift API is considered deprecated, so I would not feel comfortable
>> starting a new project using a legacy API.
>>
>> I already posted a question on SO about this, but did not get any
>> satisfactory answer. Just general performance tuning tips that do nothing
>> to explain the difference between the CQL and thrift approaches.
>>
>> http://stackoverflow.com/questions/21778671/cassandra-how-to-insert-a-new-wide-row-with-good-performance-using-cql
>>
>> Am I doing something wrong, or is this a fundamental limitation of CQL.
>> If the latter is the case, what's the plan to mitigate the issue?
>>
>> There is a JIRA issue about this (
>> https://issues.apache.org/jira/browse/CASSANDRA-5959 ), but it is marked
>> as a duplicate of https://issues.apache.org/jira/browse/CASSANDRA-4693 .
>> But according to my benchmarks batch prepared statements do not solve this
>> issue!
>>
>> I would really appreciate any help on this issue. The telemetry data I
>> would like to import into C* for testing contains ~2*10^12 samples, where
>> each sample consists of time, value and status. If quick batch insertion is
>> not possible, I would not even be able to insert it in an acceptable time.
>>
>> best regards,
>>
>> Rüdiger
>>
>
>
>
> --
>
> - John
>
--
-----------------
Nate McCall
Austin, TX
@zznate
Co-Founder & Sr. Technical Consultant
Apache Cassandra Consulting
http://www.thelastpickle.com
Re: Performance problem with large wide row inserts using CQL
Posted by John Sanda <jo...@gmail.com>.
>From a quick glance at your code, it looks like you are preparing your
insert statement multiple times. You only need to prepare it once. I would
expect to see some improvement with that change.
On Wed, Feb 19, 2014 at 5:27 AM, Rüdiger Klaehn <rk...@gmail.com> wrote:
> Hi all,
>
> I am evaluating Cassandra for satellite telemetry storage and analysis. I
> set up a little three node cluster on my local development machine and
> wrote a few simple test programs.
>
> My use case requires storing incoming telemetry updates in the database at
> the same rate as they are coming in. A telemetry update is a map of
> name/value pairs that arrives at a certain time.
>
> The idea is that I want to store the data as quickly as possible, and then
> later store it in an additional format that is more amenable to analysis.
>
> The format I have chosen for my test is the following:
>
> CREATE TABLE IF NOT EXISTS test.wide (
> time varchar,
> name varchar,
> value varchar,
> PRIMARY KEY (time,name))
> WITH COMPACT STORAGE
>
> The layout I want to achieve with this is something like this:
>
> +-------+-------+-------+-------+-------+-------+
> | | name1 | name2 | name3 | ... | nameN |
> | time +-------+-------+-------+-------+-------+
> | | val1 | val2 | val3 | ... | valN |
> +-------+-------+-------+-------|-------+-------+
>
> (Time will at some point be some kind of timestamp, and value will become
> a blob. But this is just for initial testing)
>
> The problem is the following: I am getting very low performance for bulk
> inserts into the above table. In my test program, each insert has a new,
> unique time and creates a row with 10000 name/value pairs. This should map
> into creating a new row in the underlying storage engine, correct? I do
> that 1000 times and measure both time per insert and total time.
>
> I am getting about 0.5s for each insert of 10000 name/value pairs, which
> is much lower than the rate at which the telemetry is coming in at my
> system. I have read a few previous threads on this subject and am using
> batch prepared statements for maximum performance (
> https://issues.apache.org/jira/browse/CASSANDRA-4693 ). But that does not
> help.
>
> Here is the CQL benchmark:
> https://gist.github.com/rklaehn/9089304#file-cassandratestminimized-scala
>
> I have written the exact same thing using the thrift API of astyanax, and
> I am getting much better performance. Each insert of 10000 name/values
> takes 0.04s using a ColumnListMutation. When I use async calls for both
> programs, as suggested by somebody on Stackoverflow, the difference gets
> even larger. The CQL insert remains at 0.5s per insert on average, whereas
> the astyanax ColumnListMutation approach takes 0.01s per insert on
> average, even on my test cluster. That's the kind of performance I need.
>
> Here is the thrift benchmark, modified from an ast example:
> https://gist.github.com/rklaehn/9089304#file-astclient-java
>
> I realize that running on a test cluster on localhost is not a 100%
> realistic test. But nevertheless you would expect both tests to have
> roughly similar performance.
>
> I saw a few suggestions to create a table with CQL and fill it using the
> thrift API. For example in this thread
> http://mail-archives.apache.org/mod_mbox/cassandra-user/201309.mbox/%3C523334B8.8070802@gmail.com%3E. But I would very much prefer to use pure CQL for this. It seems that the
> thrift API is considered deprecated, so I would not feel comfortable
> starting a new project using a legacy API.
>
> I already posted a question on SO about this, but did not get any
> satisfactory answer. Just general performance tuning tips that do nothing
> to explain the difference between the CQL and thrift approaches.
>
> http://stackoverflow.com/questions/21778671/cassandra-how-to-insert-a-new-wide-row-with-good-performance-using-cql
>
> Am I doing something wrong, or is this a fundamental limitation of CQL. If
> the latter is the case, what's the plan to mitigate the issue?
>
> There is a JIRA issue about this (
> https://issues.apache.org/jira/browse/CASSANDRA-5959 ), but it is marked
> as a duplicate of https://issues.apache.org/jira/browse/CASSANDRA-4693 .
> But according to my benchmarks batch prepared statements do not solve this
> issue!
>
> I would really appreciate any help on this issue. The telemetry data I
> would like to import into C* for testing contains ~2*10^12 samples, where
> each sample consists of time, value and status. If quick batch insertion is
> not possible, I would not even be able to insert it in an acceptable time.
>
> best regards,
>
> Rüdiger
>
--
- John