You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Oleg Tsernetsov <sp...@hot.ee> on 2011/11/09 21:46:24 UTC
: Cassandra reads under write-only load, read degradation after massive writes
When monitoring JMX metrics of cassandra 0.8.7 loaded by write-only
test
I observe significant read activity on column family where I write to.
It seems strange to me, but I expected no read activity on write-only
load. The read activity is caused by writes, as when I stop the write
test, reads activity disappears. The test performs parallel column
writes to a single row, writing the values of fixed column set over and
over again. Furthermore, the second problem is that parallel massive
reads of such row degrade over time (even without parallel write load)
and cassandra starts burning 100% of CPU with read latency degrading
x20
times comparing with exactly the same row created from scratch. The
test
setup is 3 cassandra nodes, read/write consistency = Quorum. Row has 10
and above columns (tested with 10, 100, 1000, 10000 cols), the higher
is
the number of columns, the worse is observed degradation. Column family
has 2 indexed columns that are written with exactly the same values on
each and
every write. Row key, column name and column value are all Utf8Type.
Column family compaction on all the nodes does not help, and the row
remains "degraded". Read here means one of: read all the
the columns with slice query without bounds/with bounds; executing
column count query for a row with bounds/without bounds. I use Hector
as
cassandra client. I would be thankful if anyone could explain the read
activity on write load and give any hints on row read degradation after
massive write load on that row.
Regards,
Oleg
Re: a very simple indexing question (strange thing seen in CLI)
Posted by aaron morton <aa...@thelastpickle.com>.
> First off, why do I see (01)? I have a similar CF where I just see "1".
The CF uses BytesType as the comparator, which displays values as Hex. 01 is the hex representation of 1.
> Before inserting the data, I did "assume" to ascii
> on the keys, comparator and validator.
This is a feature of cassandra-cli and does not change the server side schema.
> What is happening? Sorry for the admittedly trivial question, obviously I'm stuck with something quite simple
> which I managed to do with zero effort in the past.
This works for me:
[default@dev] get files where '1'='1460103677';
0 Row Returned.
This fails.
[default@dev] assume files comparator as ascii;
Assumption for column family 'files' added successfully.
[default@dev] get files where '01'='1460103677';
No indexed columns present in index clause with operator EQ
[default@dev]
Do you still have the assume present ?
Cheers
-----------------
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com
On 8/04/2012, at 12:48 PM, Maxim Potekhin wrote:
> Greetings,
> Cassandra 0.8.8 is used.
>
> I'm trying to create an additional CF which is trivial in all respects. Just ascii columns and a few indexes.
>
> This is how I add an index:
> update column family files with column_metadata = [{column_name : '1', validation_class : AsciiType, index_type : 0, index_name : 'pandaid'}];
>
> When I do "show keyspaces", I see this:
>
> ColumnFamily: files
> Key Validation Class: org.apache.cassandra.db.marshal.BytesType
> Default column value validator: org.apache.cassandra.db.marshal.BytesType
> Columns sorted by: org.apache.cassandra.db.marshal.BytesType
> Row cache size / save period in seconds: 0.0/0
> Row Cache Provider: org.apache.cassandra.cache.ConcurrentLinkedHashCacheProvider
> Key cache size / save period in seconds: 200000.0/14400
> Memtable thresholds: 2.2828125/1440/487 (millions of ops/minutes/MB)
> GC grace seconds: 864000
> Compaction min/max thresholds: 4/32
> Read repair chance: 1.0
> Replicate on write: true
> Built indexes: [files.pandaid]
> Column Metadata:
> Column Name: (01)
> Validation Class: org.apache.cassandra.db.marshal.AsciiType
> Index Name: pandaid
> Index Type: KEYS
>
> First off, why do I see (01)? I have a similar CF where I just see "1". Before inserting the data, I did "assume" to ascii
> on the keys, comparator and validator. The index has been built. When I try to access the data via the index, I get this:
> [default@PANDA] get files where '1'='1460103677';
> InvalidRequestException(why:No indexed columns present in index clause with operator EQ)
>
>
> What is happening? Sorry for the admittedly trivial question, obviously I'm stuck with something quite simple
> which I managed to do with zero effort in the past.
>
> Maxim
>
>
>
a very simple indexing question (strange thing seen in CLI)
Posted by Maxim Potekhin <po...@bnl.gov>.
Greetings,
Cassandra 0.8.8 is used.
I'm trying to create an additional CF which is trivial in all respects.
Just ascii columns and a few indexes.
This is how I add an index:
update column family files with column_metadata = [{column_name : '1',
validation_class : AsciiType, index_type : 0, index_name : 'pandaid'}];
When I do "show keyspaces", I see this:
ColumnFamily: files
Key Validation Class: org.apache.cassandra.db.marshal.BytesType
Default column value validator:
org.apache.cassandra.db.marshal.BytesType
Columns sorted by: org.apache.cassandra.db.marshal.BytesType
Row cache size / save period in seconds: 0.0/0
Row Cache Provider:
org.apache.cassandra.cache.ConcurrentLinkedHashCacheProvider
Key cache size / save period in seconds: 200000.0/14400
Memtable thresholds: 2.2828125/1440/487 (millions of ops/minutes/MB)
GC grace seconds: 864000
Compaction min/max thresholds: 4/32
Read repair chance: 1.0
Replicate on write: true
Built indexes: [files.pandaid]
Column Metadata:
Column Name: (01)
Validation Class: org.apache.cassandra.db.marshal.AsciiType
Index Name: pandaid
Index Type: KEYS
First off, why do I see (01)? I have a similar CF where I just see "1".
Before inserting the data, I did "assume" to ascii
on the keys, comparator and validator. The index has been built. When I
try to access the data via the index, I get this:
[default@PANDA] get files where '1'='1460103677';
InvalidRequestException(why:No indexed columns present in index clause
with operator EQ)
What is happening? Sorry for the admittedly trivial question, obviously
I'm stuck with something quite simple
which I managed to do with zero effort in the past.
Maxim
Re: : Cassandra reads under write-only load, read degradation after
massive writes
Posted by Jeremiah Jordan <je...@morningstar.com>.
Indexed columns cause read before write so that the index can be updated
if the column already exists.
On 11/09/2011 02:46 PM, Oleg Tsernetsov wrote:
> When monitoring JMX metrics of cassandra 0.8.7 loaded by write-only
> test I observe significant read activity on column family where I
> write to. It seems strange to me, but I expected no read activity on
> write-only load. The read activity is caused by writes, as when I stop
> the write test, reads activity disappears. The test performs parallel
> column writes to a single row, writing the values of fixed column set
> over and over again. Furthermore, the second problem is that parallel
> massive reads of such row degrade over time (even without parallel
> write load) and cassandra starts burning 100% of CPU with read latency
> degrading x20 times comparing with exactly the same row created from
> scratch. The test setup is 3 cassandra nodes, read/write consistency =
> Quorum. Row has 10 and above columns (tested with 10, 100, 1000, 10000
> cols), the higher is the number of columns, the worse is observed
> degradation. Column family has 2 indexed columns that are written with
> exactly the same values on each and every write. Row key, column name
> and column value are all Utf8Type. Column family compaction on all the
> nodes does not help, and the row remains "degraded". Read here means
> one of: read all the the columns with slice query without bounds/with
> bounds; executing column count query for a row with bounds/without
> bounds. I use Hector as cassandra client. I would be thankful if
> anyone could explain the read activity on write load and give any
> hints on row read degradation after massive write load on that row.
>
> Regards,
> Oleg