Slow write speeds

[Alexandre Beaulne <al...@gmail.com>]

Hi everyone,

First and foremost thanks to everyone involved with making C* available to
the world, it is a great technology to have access to.

I'm experimenting with C* for one of our projects and I cannot reproduce
the write speeds C* is lauded for. I would appreciate some guidance as to
what I'm doing wrong.

*Setup*: I have one, single-threaded, python client (using Datastax's
python driver), writing (no reads) to a C* cluster. All C* nodes are
launched by running the official Docker container. There's a single
keyspace with replication factor of 1 and client is set to consistency
level LOCAL ONE. In that keyspace there is a single table with ~40 columns
of mixed types. Two columns are set as primary key and two more as
clustering columns. The primary key is close to uniformly distributed in
the dataset. The writer is in a tight-loop, building CQL 3 insert
statements one by one and executing them against the C* cluster.

*Specs*: Cassandra v3.0.1, python-driver v3.0.0, host is CentOS 7 with 40
cores @ 3GHz and 66Gb of RAM.

In the course of my experimentation I came up with 7 scenarios trying to
isolate the performance bottleneck:

*Scenario 1*: the writer simply build the insert statement strings without
doing anything with them.

Results: sample size: 200002, percentiles (ms): [50] 0.00 - [95] 0.01 -
[99] 0.01 [100] 0.05

*Scenario 2*: the writer open a TCP socket and send the insert statement
string to a simple reader running on the same host. The reader then append
that insert statement string to a file on disk, mimicking a commit log of
some sort.

Results: sample size: 200002, percentiles (ms): [50] 0.01 - [95] 0.02 -
[99] 0.03 [100] 63.33

*Scenario 3*: is identical to scenario 2, but the reader is ran inside a
Docker container, to measure if there is any overhead from running in the
container.

Results: sample size: 200002, percentiles (ms): [50] 0.01 - [95] 0.01 -
[99] 0.01 [100] 4.45

* Scenario 4*: the writer asynchronously executes the insert statements
against a single-node C* cluster.

Results: sample size: 200002, percentiles (ms): [50] 0.07 - [95] 0.15 -
[99] 0.56 [100] 534.09

*Scenario 5*: the writer synchronously executes the insert statements
against a single-node C* cluster.

Results: sample size: 200002, percentiles (ms): [50] 1.40 - [95] 1.46 -
[99] 1.54 [100] 41.75

*Scenario 6*: the writer asynchronously executes the insert statements
against a four-nodes C* cluster.

Results: sample size: 200002, percentiles (ms): [50] 0.09 - [95] 0.14 -
[99] 0.16 [100] 838.83

*Scenario 7*: the writer synchronously executes the insert statements
against a four-nodes C* cluster.

Results: sample size: 200002, percentiles (ms): [50] 1.73 - [95] 1.89 -
[99] 2.15 [100] 50.94

Looking at scenario 3 & 5, a synchronous write statement to C* is about
150x slower than appending to a flat file. Now I understand write to a DB
is more involved than appending to a file, but I'm surprised by the
magnitude of the difference. I thought all C* did for writes with
consistency level of 1 was to append the write to its commit log and
return, then distribute the write across the cluster in an eventual
consistency manner. More than 1 ms per write is less than a 1000 writes per
second, far from big data velocity.

What am I doing wrong? Are writes supposed to be batched before inserted?
Instead of appending rows to the table, would it be more efficient to
append columns to the rows? Why writes are so slow?

Thanks for your time,
Alex

Re: Slow write speeds

["Laing, Michael" <mi...@nytimes.com>]

To add to what Jonathan and Jack have said...

To get high levels of performance with the python driver you should:

   - prepare your statements once (recent drivers default to Token Aware -
   and will correctly apply it if the statement is prepared).
   - execute asynchronously (up to ~150 futures - tho my [old] benchmarks
   showed smaller numbers worked fine.)
   - use multi-processing (performance leveled off in my [old] benchmark
   when each process consumed ~50% of a CPU)
   - Watch for network bottlenecks.

ml

On Thu, Dec 31, 2015 at 12:30 PM, Jack Krupansky <ja...@gmail.com>
wrote:

> Make sure the driver is configured for token aware routing, otherwise the
> coordinator node may have to redirect your write, adding a network hop.
>
> To be absolutely clear, Cassandra uses the distributed, parallel model for
> Big Data - lots of multi-threaded clients with lots of nodes. Clusters with
> less than six or eight nodes and using a single, single-threaded client are
> not a representative usage of Cassandra. Replication is presumed as well.
> Anything less than RF=3 is simply not a representative or recommended usage
> of Cassandra. Similarly, writes at less than QUORUM are neither
> representative nor recommended.
>
> CL=ONE has to update the memtable as well, not just the commit log.
> Flushing to sstables occurs once the memtables reach some threshold
> size.See:
>
> http://docs.datastax.com/en/cassandra/2.0/cassandra/dml/dml_config_consistency_c.html
>
>
> -- Jack Krupansky
>
> On Thu, Dec 31, 2015 at 11:13 AM, Jonathan Haddad <jo...@jonhaddad.com>
> wrote:
>
>> The limitation is on the driver side. Try looking at
>> execute_concurrent_with_args in the cassandra.concurrent module to get
>> parallel writes with prepared statements.
>>
>> https://datastax.github.io/python-driver/api/cassandra/concurrent.html
>>
>> On Wed, Dec 30, 2015 at 11:34 PM Alexandre Beaulne <
>> alexandre.beaulne@gmail.com> wrote:
>>
>>> Hi everyone,
>>>
>>> First and foremost thanks to everyone involved with making C* available
>>> to the world, it is a great technology to have access to.
>>>
>>> I'm experimenting with C* for one of our projects and I cannot reproduce
>>> the write speeds C* is lauded for. I would appreciate some guidance as to
>>> what I'm doing wrong.
>>>
>>> *Setup*: I have one, single-threaded, python client (using Datastax's
>>> python driver), writing (no reads) to a C* cluster. All C* nodes are
>>> launched by running the official Docker container. There's a single
>>> keyspace with replication factor of 1 and client is set to consistency
>>> level LOCAL ONE. In that keyspace there is a single table with ~40 columns
>>> of mixed types. Two columns are set as primary key and two more as
>>> clustering columns. The primary key is close to uniformly distributed in
>>> the dataset. The writer is in a tight-loop, building CQL 3 insert
>>> statements one by one and executing them against the C* cluster.
>>>
>>> *Specs*: Cassandra v3.0.1, python-driver v3.0.0, host is CentOS 7 with
>>> 40 cores @ 3GHz and 66Gb of RAM.
>>>
>>> In the course of my experimentation I came up with 7 scenarios trying to
>>> isolate the performance bottleneck:
>>>
>>> *Scenario 1*: the writer simply build the insert statement strings
>>> without doing anything with them.
>>>
>>> Results: sample size: 200002, percentiles (ms): [50] 0.00 - [95] 0.01 -
>>> [99] 0.01 [100] 0.05
>>>
>>> *Scenario 2*: the writer open a TCP socket and send the insert
>>> statement string to a simple reader running on the same host. The reader
>>> then append that insert statement string to a file on disk, mimicking a
>>> commit log of some sort.
>>>
>>> Results: sample size: 200002, percentiles (ms): [50] 0.01 - [95] 0.02 -
>>> [99] 0.03 [100] 63.33
>>>
>>> *Scenario 3*: is identical to scenario 2, but the reader is ran inside
>>> a Docker container, to measure if there is any overhead from running in the
>>> container.
>>>
>>> Results: sample size: 200002, percentiles (ms): [50] 0.01 - [95] 0.01 -
>>> [99] 0.01 [100] 4.45
>>>
>>> * Scenario 4*: the writer asynchronously executes the insert statements
>>> against a single-node C* cluster.
>>>
>>> Results: sample size: 200002, percentiles (ms): [50] 0.07 - [95] 0.15 -
>>> [99] 0.56 [100] 534.09
>>>
>>> *Scenario 5*: the writer synchronously executes the insert statements
>>> against a single-node C* cluster.
>>>
>>> Results: sample size: 200002, percentiles (ms): [50] 1.40 - [95] 1.46 -
>>> [99] 1.54 [100] 41.75
>>>
>>> *Scenario 6*: the writer asynchronously executes the insert statements
>>> against a four-nodes C* cluster.
>>>
>>> Results: sample size: 200002, percentiles (ms): [50] 0.09 - [95] 0.14 -
>>> [99] 0.16 [100] 838.83
>>>
>>> *Scenario 7*: the writer synchronously executes the insert statements
>>> against a four-nodes C* cluster.
>>>
>>> Results: sample size: 200002, percentiles (ms): [50] 1.73 - [95] 1.89 -
>>> [99] 2.15 [100] 50.94
>>>
>>> Looking at scenario 3 & 5, a synchronous write statement to C* is about
>>> 150x slower than appending to a flat file. Now I understand write to a DB
>>> is more involved than appending to a file, but I'm surprised by the
>>> magnitude of the difference. I thought all C* did for writes with
>>> consistency level of 1 was to append the write to its commit log and
>>> return, then distribute the write across the cluster in an eventual
>>> consistency manner. More than 1 ms per write is less than a 1000 writes per
>>> second, far from big data velocity.
>>>
>>> What am I doing wrong? Are writes supposed to be batched before
>>> inserted? Instead of appending rows to the table, would it be more
>>> efficient to append columns to the rows? Why writes are so slow?
>>>
>>> Thanks for your time,
>>> Alex
>>>
>>
>

Re: Slow write speeds

[Jack Krupansky <ja...@gmail.com>]

Make sure the driver is configured for token aware routing, otherwise the
coordinator node may have to redirect your write, adding a network hop.

To be absolutely clear, Cassandra uses the distributed, parallel model for
Big Data - lots of multi-threaded clients with lots of nodes. Clusters with
less than six or eight nodes and using a single, single-threaded client are
not a representative usage of Cassandra. Replication is presumed as well.
Anything less than RF=3 is simply not a representative or recommended usage
of Cassandra. Similarly, writes at less than QUORUM are neither
representative nor recommended.

CL=ONE has to update the memtable as well, not just the commit log.
Flushing to sstables occurs once the memtables reach some threshold
size.See:
http://docs.datastax.com/en/cassandra/2.0/cassandra/dml/dml_config_consistency_c.html


-- Jack Krupansky

On Thu, Dec 31, 2015 at 11:13 AM, Jonathan Haddad <jo...@jonhaddad.com> wrote:

> The limitation is on the driver side. Try looking at
> execute_concurrent_with_args in the cassandra.concurrent module to get
> parallel writes with prepared statements.
>
> https://datastax.github.io/python-driver/api/cassandra/concurrent.html
>
> On Wed, Dec 30, 2015 at 11:34 PM Alexandre Beaulne <
> alexandre.beaulne@gmail.com> wrote:
>
>> Hi everyone,
>>
>> First and foremost thanks to everyone involved with making C* available
>> to the world, it is a great technology to have access to.
>>
>> I'm experimenting with C* for one of our projects and I cannot reproduce
>> the write speeds C* is lauded for. I would appreciate some guidance as to
>> what I'm doing wrong.
>>
>> *Setup*: I have one, single-threaded, python client (using Datastax's
>> python driver), writing (no reads) to a C* cluster. All C* nodes are
>> launched by running the official Docker container. There's a single
>> keyspace with replication factor of 1 and client is set to consistency
>> level LOCAL ONE. In that keyspace there is a single table with ~40 columns
>> of mixed types. Two columns are set as primary key and two more as
>> clustering columns. The primary key is close to uniformly distributed in
>> the dataset. The writer is in a tight-loop, building CQL 3 insert
>> statements one by one and executing them against the C* cluster.
>>
>> *Specs*: Cassandra v3.0.1, python-driver v3.0.0, host is CentOS 7 with
>> 40 cores @ 3GHz and 66Gb of RAM.
>>
>> In the course of my experimentation I came up with 7 scenarios trying to
>> isolate the performance bottleneck:
>>
>> *Scenario 1*: the writer simply build the insert statement strings
>> without doing anything with them.
>>
>> Results: sample size: 200002, percentiles (ms): [50] 0.00 - [95] 0.01 -
>> [99] 0.01 [100] 0.05
>>
>> *Scenario 2*: the writer open a TCP socket and send the insert statement
>> string to a simple reader running on the same host. The reader then append
>> that insert statement string to a file on disk, mimicking a commit log of
>> some sort.
>>
>> Results: sample size: 200002, percentiles (ms): [50] 0.01 - [95] 0.02 -
>> [99] 0.03 [100] 63.33
>>
>> *Scenario 3*: is identical to scenario 2, but the reader is ran inside a
>> Docker container, to measure if there is any overhead from running in the
>> container.
>>
>> Results: sample size: 200002, percentiles (ms): [50] 0.01 - [95] 0.01 -
>> [99] 0.01 [100] 4.45
>>
>> * Scenario 4*: the writer asynchronously executes the insert statements
>> against a single-node C* cluster.
>>
>> Results: sample size: 200002, percentiles (ms): [50] 0.07 - [95] 0.15 -
>> [99] 0.56 [100] 534.09
>>
>> *Scenario 5*: the writer synchronously executes the insert statements
>> against a single-node C* cluster.
>>
>> Results: sample size: 200002, percentiles (ms): [50] 1.40 - [95] 1.46 -
>> [99] 1.54 [100] 41.75
>>
>> *Scenario 6*: the writer asynchronously executes the insert statements
>> against a four-nodes C* cluster.
>>
>> Results: sample size: 200002, percentiles (ms): [50] 0.09 - [95] 0.14 -
>> [99] 0.16 [100] 838.83
>>
>> *Scenario 7*: the writer synchronously executes the insert statements
>> against a four-nodes C* cluster.
>>
>> Results: sample size: 200002, percentiles (ms): [50] 1.73 - [95] 1.89 -
>> [99] 2.15 [100] 50.94
>>
>> Looking at scenario 3 & 5, a synchronous write statement to C* is about
>> 150x slower than appending to a flat file. Now I understand write to a DB
>> is more involved than appending to a file, but I'm surprised by the
>> magnitude of the difference. I thought all C* did for writes with
>> consistency level of 1 was to append the write to its commit log and
>> return, then distribute the write across the cluster in an eventual
>> consistency manner. More than 1 ms per write is less than a 1000 writes per
>> second, far from big data velocity.
>>
>> What am I doing wrong? Are writes supposed to be batched before inserted?
>> Instead of appending rows to the table, would it be more efficient to
>> append columns to the rows? Why writes are so slow?
>>
>> Thanks for your time,
>> Alex
>>
>

Re: Slow write speeds

[Jonathan Haddad <jo...@jonhaddad.com>]

The limitation is on the driver side. Try looking at
execute_concurrent_with_args in the cassandra.concurrent module to get
parallel writes with prepared statements.

https://datastax.github.io/python-driver/api/cassandra/concurrent.html
On Wed, Dec 30, 2015 at 11:34 PM Alexandre Beaulne <
alexandre.beaulne@gmail.com> wrote:

> Hi everyone,
>
> First and foremost thanks to everyone involved with making C* available to
> the world, it is a great technology to have access to.
>
> I'm experimenting with C* for one of our projects and I cannot reproduce
> the write speeds C* is lauded for. I would appreciate some guidance as to
> what I'm doing wrong.
>
> *Setup*: I have one, single-threaded, python client (using Datastax's
> python driver), writing (no reads) to a C* cluster. All C* nodes are
> launched by running the official Docker container. There's a single
> keyspace with replication factor of 1 and client is set to consistency
> level LOCAL ONE. In that keyspace there is a single table with ~40 columns
> of mixed types. Two columns are set as primary key and two more as
> clustering columns. The primary key is close to uniformly distributed in
> the dataset. The writer is in a tight-loop, building CQL 3 insert
> statements one by one and executing them against the C* cluster.
>
> *Specs*: Cassandra v3.0.1, python-driver v3.0.0, host is CentOS 7 with 40
> cores @ 3GHz and 66Gb of RAM.
>
> In the course of my experimentation I came up with 7 scenarios trying to
> isolate the performance bottleneck:
>
> *Scenario 1*: the writer simply build the insert statement strings
> without doing anything with them.
>
> Results: sample size: 200002, percentiles (ms): [50] 0.00 - [95] 0.01 -
> [99] 0.01 [100] 0.05
>
> *Scenario 2*: the writer open a TCP socket and send the insert statement
> string to a simple reader running on the same host. The reader then append
> that insert statement string to a file on disk, mimicking a commit log of
> some sort.
>
> Results: sample size: 200002, percentiles (ms): [50] 0.01 - [95] 0.02 -
> [99] 0.03 [100] 63.33
>
> *Scenario 3*: is identical to scenario 2, but the reader is ran inside a
> Docker container, to measure if there is any overhead from running in the
> container.
>
> Results: sample size: 200002, percentiles (ms): [50] 0.01 - [95] 0.01 -
> [99] 0.01 [100] 4.45
>
> * Scenario 4*: the writer asynchronously executes the insert statements
> against a single-node C* cluster.
>
> Results: sample size: 200002, percentiles (ms): [50] 0.07 - [95] 0.15 -
> [99] 0.56 [100] 534.09
>
> *Scenario 5*: the writer synchronously executes the insert statements
> against a single-node C* cluster.
>
> Results: sample size: 200002, percentiles (ms): [50] 1.40 - [95] 1.46 -
> [99] 1.54 [100] 41.75
>
> *Scenario 6*: the writer asynchronously executes the insert statements
> against a four-nodes C* cluster.
>
> Results: sample size: 200002, percentiles (ms): [50] 0.09 - [95] 0.14 -
> [99] 0.16 [100] 838.83
>
> *Scenario 7*: the writer synchronously executes the insert statements
> against a four-nodes C* cluster.
>
> Results: sample size: 200002, percentiles (ms): [50] 1.73 - [95] 1.89 -
> [99] 2.15 [100] 50.94
>
> Looking at scenario 3 & 5, a synchronous write statement to C* is about
> 150x slower than appending to a flat file. Now I understand write to a DB
> is more involved than appending to a file, but I'm surprised by the
> magnitude of the difference. I thought all C* did for writes with
> consistency level of 1 was to append the write to its commit log and
> return, then distribute the write across the cluster in an eventual
> consistency manner. More than 1 ms per write is less than a 1000 writes per
> second, far from big data velocity.
>
> What am I doing wrong? Are writes supposed to be batched before inserted?
> Instead of appending rows to the table, would it be more efficient to
> append columns to the rows? Why writes are so slow?
>
> Thanks for your time,
> Alex
>