You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Ivan Ganza (JIRA)" <ji...@apache.org> on 2012/06/14 16:06:42 UTC
[jira] [Created] (CASSANDRA-4340) Cassandra upgrade to 1.1.1
resulted in slow query issue
Ivan Ganza created CASSANDRA-4340:
-------------------------------------
Summary: Cassandra upgrade to 1.1.1 resulted in slow query issue
Key: CASSANDRA-4340
URL: https://issues.apache.org/jira/browse/CASSANDRA-4340
Project: Cassandra
Issue Type: Bug
Affects Versions: 1.1.1
Environment: Ubuntu Linux, Java 7, Hector 1.0-1
Reporter: Ivan Ganza
We have recently introduced Cassandra at the Globe and Mail here in Toronto, Canada. We are processing and storing the North American stock-market feed. We have found it to work very quickly and things have been looking very good.
Recently we upgraded to version 1.1.1 and then we have noticed some issues occurring.
I will try to describe it for you here. Basically one operation that we very often perform and is very critical is the ability to 'get the latest quote'. This would return to you the latest Quote adjusted against exchange delay rules. With Cassandra version 1.0.3 we could get a Quote in around 2ms. After update we are looking at time of at least 2-3 seconds.
The way we query the quote is using a REVERSED SuperSliceQuery with start=now, end=00:00:00.000 (beginning of day) LIMITED to 1.
Our investigation leads us to suspect that, since upgrade, Cassandra seems to be reading the sstable from disk even when we request a small range of day only 5 seconds back. If you look at the output below you can see that the query does NOT get slower as the lookback increases from 5 sec, 60 sec, 15 min, 60 min, and 24 hours.
We also noticed that the query was very fast for the first five minutes of trading, apparently until the first sstable was flushed to disk. After that we go into query times of 1-2 seconds or so.
Query time[lookback=5]:[1711ms]
Query time[lookback=60]:[1592ms]
Query time[lookback=900]:[1520ms]
Query time[lookback=3600]:[1294ms]
Query time[lookback=86400]:[1391ms]
We would really appreciate input or help on this.
Cassandra version: 1.1.1
Hector version: 1.0-1
---
public void testCassandraIssue() {
try {
int[] seconds = new int[]{ 5, 60, 60 * 15, 60 * 60, 60 * 60 * 24};
for(int sec : seconds) {
DateTime start = new DateTime();
SuperSliceQuery<String, String, String, String> superSliceQuery = HFactory.createSuperSliceQuery(keyspaceOperator, StringSerializer.get(), StringSerializer.get(), StringSerializer.get(), StringSerializer.get());
superSliceQuery.setKey("101390" + "." + testFormatter.print(start));
superSliceQuery.setColumnFamily("Quotes");
superSliceQuery.setRange(superKeyFormatter.print(start),
superKeyFormatter.print(start.minusSeconds(sec)),
true,
1);
long theStart = System.currentTimeMillis();
QueryResult<SuperSlice<String, String, String>> result = superSliceQuery.execute();
long end = System.currentTimeMillis();
System.out.println("Query time[lookback=" + sec + "]:[" + (end - theStart) + "ms]");
}
} catch(Exception e) {
e.printStackTrace();
fail(e.getMessage());
}
}
---
create column family Quotes
with column_type = Super
and comparator = BytesType
and subcomparator = BytesType
and keys_cached = 7000
and rows_cached = 0
and row_cache_save_period = 0
and key_cache_save_period = 3600
and memtable_throughput = 255
and memtable_operations = 0.29
AND compression_options={sstable_compression:SnappyCompressor, chunk_length_kb:64};
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-4340) Cassandra upgrade to 1.1.1
resulted in slow query issue
Posted by "Ivan Ganza (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ivan Ganza updated CASSANDRA-4340:
----------------------------------
Comment: was deleted
(was: public void testCassandraIssue() {
try {
int[] seconds = new int[]{ 5, 60, 60 * 15, 60 * 60, 60 * 60 * 24};
for(int sec : seconds) {
DateTime start = new DateTime();
SuperSliceQuery<String, String, String, String> superSliceQuery = HFactory.createSuperSliceQuery(keyspaceOperator, StringSerializer.get(), StringSerializer.get(), StringSerializer.get(), StringSerializer.get());
superSliceQuery.setKey("101390" + "." + testFormatter.print(start));
superSliceQuery.setColumnFamily("Quotes");
superSliceQuery.setRange(superKeyFormatter.print(start),
superKeyFormatter.print(start.minusSeconds(sec)),
true,
1);
long theStart = System.currentTimeMillis();
QueryResult<SuperSlice<String, String, String>> result = superSliceQuery.execute();
long end = System.currentTimeMillis();
System.out.println("Query time[lookback=" + sec + "]:[" + (end - theStart) + "ms]");
}
} catch(Exception e) {
e.printStackTrace();
fail(e.getMessage());
}
})
> Cassandra upgrade to 1.1.1 resulted in slow query issue
> -------------------------------------------------------
>
> Key: CASSANDRA-4340
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4340
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 1.1.1
> Environment: Ubuntu Linux, Java 7, Hector 1.0-1
> Reporter: Ivan Ganza
>
> We have recently introduced Cassandra at the Globe and Mail here in Toronto, Canada. We are processing and storing the North American stock-market feed. We have found it to work very quickly and things have been looking very good.
> Recently we upgraded to version 1.1.1 and then we have noticed some issues occurring.
> I will try to describe it for you here. Basically one operation that we very often perform and is very critical is the ability to 'get the latest quote'. This would return to you the latest Quote adjusted against exchange delay rules. With Cassandra version 1.0.3 we could get a Quote in around 2ms. After update we are looking at time of at least 2-3 seconds.
> The way we query the quote is using a REVERSED SuperSliceQuery with start=now, end=00:00:00.000 (beginning of day) LIMITED to 1.
> Our investigation leads us to suspect that, since upgrade, Cassandra seems to be reading the sstable from disk even when we request a small range of day only 5 seconds back. If you look at the output below you can see that the query does NOT get slower as the lookback increases from 5 sec, 60 sec, 15 min, 60 min, and 24 hours.
> We also noticed that the query was very fast for the first five minutes of trading, apparently until the first sstable was flushed to disk. After that we go into query times of 1-2 seconds or so.
> Query time[lookback=5]:[1711ms]
> Query time[lookback=60]:[1592ms]
> Query time[lookback=900]:[1520ms]
> Query time[lookback=3600]:[1294ms]
> Query time[lookback=86400]:[1391ms]
> We would really appreciate input or help on this.
> Cassandra version: 1.1.1
> Hector version: 1.0-1
> ---
> public void testCassandraIssue() {
> try {
> int[] seconds = new int[]{ 5, 60, 60 * 15, 60 * 60, 60 * 60 * 24};
> for(int sec : seconds) {
> DateTime start = new DateTime();
> SuperSliceQuery<String, String, String, String> superSliceQuery = HFactory.createSuperSliceQuery(keyspaceOperator, StringSerializer.get(), StringSerializer.get(), StringSerializer.get(), StringSerializer.get());
> superSliceQuery.setKey("101390" + "." + testFormatter.print(start));
> superSliceQuery.setColumnFamily("Quotes");
> superSliceQuery.setRange(superKeyFormatter.print(start),
> superKeyFormatter.print(start.minusSeconds(sec)),
> true,
> 1);
> long theStart = System.currentTimeMillis();
> QueryResult<SuperSlice<String, String, String>> result = superSliceQuery.execute();
> long end = System.currentTimeMillis();
> System.out.println("Query time[lookback=" + sec + "]:[" + (end - theStart) + "ms]");
> }
> } catch(Exception e) {
> e.printStackTrace();
> fail(e.getMessage());
> }
> }
> ---
> create column family Quotes
> with column_type = Super
> and comparator = BytesType
> and subcomparator = BytesType
> and keys_cached = 7000
> and rows_cached = 0
> and row_cache_save_period = 0
> and key_cache_save_period = 3600
> and memtable_throughput = 255
> and memtable_operations = 0.29
> AND compression_options={sstable_compression:SnappyCompressor, chunk_length_kb:64};
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Comment Edited] (CASSANDRA-4340) Cassandra upgrade to 1.1.1
resulted in slow query issue
Posted by "Ivan Ganza (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13295042#comment-13295042 ]
Ivan Ganza edited comment on CASSANDRA-4340 at 6/14/12 2:08 PM:
----------------------------------------------------------------
public void testCassandraIssue() {
try {
int[] seconds = new int[]{ 5, 60, 60 * 15, 60 * 60, 60 * 60 * 24};
for(int sec : seconds) {
DateTime start = new DateTime();
SuperSliceQuery<String, String, String, String> superSliceQuery = HFactory.createSuperSliceQuery(keyspaceOperator, StringSerializer.get(), StringSerializer.get(), StringSerializer.get(), StringSerializer.get());
superSliceQuery.setKey("101390" + "." + testFormatter.print(start));
superSliceQuery.setColumnFamily("Quotes");
superSliceQuery.setRange(superKeyFormatter.print(start),
superKeyFormatter.print(start.minusSeconds(sec)),
true,
1);
long theStart = System.currentTimeMillis();
QueryResult<SuperSlice<String, String, String>> result = superSliceQuery.execute();
long end = System.currentTimeMillis();
System.out.println("Query time[lookback=" + sec + "]:[" + (end - theStart) + "ms]");
}
} catch(Exception e) {
e.printStackTrace();
fail(e.getMessage());
}
}
was (Author: iganza@globeandmail.com):
public void testCassandraIssue() {
try {
int[] seconds = new int[]{ 5, 60, 60 * 15, 60 * 60, 60 * 60 * 24};
for(int sec : seconds) {
DateTime start = new DateTime();
SuperSliceQuery<String, String, String, String> superSliceQuery = HFactory.createSuperSliceQuery(keyspaceOperator, StringSerializer.get(), StringSerializer.get(), StringSerializer.get(), StringSerializer.get());
superSliceQuery.setKey("101390" + "." + testFormatter.print(start));
superSliceQuery.setColumnFamily("Quotes");
superSliceQuery.setRange(superKeyFormatter.print(start),
superKeyFormatter.print(start.minusSeconds(sec)),
true,
1);
long theStart = System.currentTimeMillis();
QueryResult<SuperSlice<String, String, String>> result = superSliceQuery.execute();
long end = System.currentTimeMillis();
System.out.println("Query time[lookback=" + sec + "]:[" + (end - theStart) + "ms]");
}
} catch(Exception e) {
e.printStackTrace();
fail(e.getMessage());
}
}
> Cassandra upgrade to 1.1.1 resulted in slow query issue
> -------------------------------------------------------
>
> Key: CASSANDRA-4340
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4340
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 1.1.1
> Environment: Ubuntu Linux, Java 7, Hector 1.0-1
> Reporter: Ivan Ganza
>
> We have recently introduced Cassandra at the Globe and Mail here in Toronto, Canada. We are processing and storing the North American stock-market feed. We have found it to work very quickly and things have been looking very good.
> Recently we upgraded to version 1.1.1 and then we have noticed some issues occurring.
> I will try to describe it for you here. Basically one operation that we very often perform and is very critical is the ability to 'get the latest quote'. This would return to you the latest Quote adjusted against exchange delay rules. With Cassandra version 1.0.3 we could get a Quote in around 2ms. After update we are looking at time of at least 2-3 seconds.
> The way we query the quote is using a REVERSED SuperSliceQuery with start=now, end=00:00:00.000 (beginning of day) LIMITED to 1.
> Our investigation leads us to suspect that, since upgrade, Cassandra seems to be reading the sstable from disk even when we request a small range of day only 5 seconds back. If you look at the output below you can see that the query does NOT get slower as the lookback increases from 5 sec, 60 sec, 15 min, 60 min, and 24 hours.
> We also noticed that the query was very fast for the first five minutes of trading, apparently until the first sstable was flushed to disk. After that we go into query times of 1-2 seconds or so.
> Query time[lookback=5]:[1711ms]
> Query time[lookback=60]:[1592ms]
> Query time[lookback=900]:[1520ms]
> Query time[lookback=3600]:[1294ms]
> Query time[lookback=86400]:[1391ms]
> We would really appreciate input or help on this.
> Cassandra version: 1.1.1
> Hector version: 1.0-1
> ---
> public void testCassandraIssue() {
> try {
> int[] seconds = new int[]{ 5, 60, 60 * 15, 60 * 60, 60 * 60 * 24};
> for(int sec : seconds) {
> DateTime start = new DateTime();
> SuperSliceQuery<String, String, String, String> superSliceQuery = HFactory.createSuperSliceQuery(keyspaceOperator, StringSerializer.get(), StringSerializer.get(), StringSerializer.get(), StringSerializer.get());
> superSliceQuery.setKey("101390" + "." + testFormatter.print(start));
> superSliceQuery.setColumnFamily("Quotes");
> superSliceQuery.setRange(superKeyFormatter.print(start),
> superKeyFormatter.print(start.minusSeconds(sec)),
> true,
> 1);
> long theStart = System.currentTimeMillis();
> QueryResult<SuperSlice<String, String, String>> result = superSliceQuery.execute();
> long end = System.currentTimeMillis();
> System.out.println("Query time[lookback=" + sec + "]:[" + (end - theStart) + "ms]");
> }
> } catch(Exception e) {
> e.printStackTrace();
> fail(e.getMessage());
> }
> }
> ---
> create column family Quotes
> with column_type = Super
> and comparator = BytesType
> and subcomparator = BytesType
> and keys_cached = 7000
> and rows_cached = 0
> and row_cache_save_period = 0
> and key_cache_save_period = 3600
> and memtable_throughput = 255
> and memtable_operations = 0.29
> AND compression_options={sstable_compression:SnappyCompressor, chunk_length_kb:64};
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4340) Cassandra upgrade to 1.1.1
resulted in slow query issue
Posted by "Ivan Ganza (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13295042#comment-13295042 ]
Ivan Ganza commented on CASSANDRA-4340:
---------------------------------------
public void testCassandraIssue() {
try {
int[] seconds = new int[]{ 5, 60, 60 * 15, 60 * 60, 60 * 60 * 24};
for(int sec : seconds) {
DateTime start = new DateTime();
SuperSliceQuery<String, String, String, String> superSliceQuery = HFactory.createSuperSliceQuery(keyspaceOperator, StringSerializer.get(), StringSerializer.get(), StringSerializer.get(), StringSerializer.get());
superSliceQuery.setKey("101390" + "." + testFormatter.print(start));
superSliceQuery.setColumnFamily("Quotes");
superSliceQuery.setRange(superKeyFormatter.print(start),
superKeyFormatter.print(start.minusSeconds(sec)),
true,
1);
long theStart = System.currentTimeMillis();
QueryResult<SuperSlice<String, String, String>> result = superSliceQuery.execute();
long end = System.currentTimeMillis();
System.out.println("Query time[lookback=" + sec + "]:[" + (end - theStart) + "ms]");
}
} catch(Exception e) {
e.printStackTrace();
fail(e.getMessage());
}
}
> Cassandra upgrade to 1.1.1 resulted in slow query issue
> -------------------------------------------------------
>
> Key: CASSANDRA-4340
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4340
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 1.1.1
> Environment: Ubuntu Linux, Java 7, Hector 1.0-1
> Reporter: Ivan Ganza
>
> We have recently introduced Cassandra at the Globe and Mail here in Toronto, Canada. We are processing and storing the North American stock-market feed. We have found it to work very quickly and things have been looking very good.
> Recently we upgraded to version 1.1.1 and then we have noticed some issues occurring.
> I will try to describe it for you here. Basically one operation that we very often perform and is very critical is the ability to 'get the latest quote'. This would return to you the latest Quote adjusted against exchange delay rules. With Cassandra version 1.0.3 we could get a Quote in around 2ms. After update we are looking at time of at least 2-3 seconds.
> The way we query the quote is using a REVERSED SuperSliceQuery with start=now, end=00:00:00.000 (beginning of day) LIMITED to 1.
> Our investigation leads us to suspect that, since upgrade, Cassandra seems to be reading the sstable from disk even when we request a small range of day only 5 seconds back. If you look at the output below you can see that the query does NOT get slower as the lookback increases from 5 sec, 60 sec, 15 min, 60 min, and 24 hours.
> We also noticed that the query was very fast for the first five minutes of trading, apparently until the first sstable was flushed to disk. After that we go into query times of 1-2 seconds or so.
> Query time[lookback=5]:[1711ms]
> Query time[lookback=60]:[1592ms]
> Query time[lookback=900]:[1520ms]
> Query time[lookback=3600]:[1294ms]
> Query time[lookback=86400]:[1391ms]
> We would really appreciate input or help on this.
> Cassandra version: 1.1.1
> Hector version: 1.0-1
> ---
> public void testCassandraIssue() {
> try {
> int[] seconds = new int[]{ 5, 60, 60 * 15, 60 * 60, 60 * 60 * 24};
> for(int sec : seconds) {
> DateTime start = new DateTime();
> SuperSliceQuery<String, String, String, String> superSliceQuery = HFactory.createSuperSliceQuery(keyspaceOperator, StringSerializer.get(), StringSerializer.get(), StringSerializer.get(), StringSerializer.get());
> superSliceQuery.setKey("101390" + "." + testFormatter.print(start));
> superSliceQuery.setColumnFamily("Quotes");
> superSliceQuery.setRange(superKeyFormatter.print(start),
> superKeyFormatter.print(start.minusSeconds(sec)),
> true,
> 1);
> long theStart = System.currentTimeMillis();
> QueryResult<SuperSlice<String, String, String>> result = superSliceQuery.execute();
> long end = System.currentTimeMillis();
> System.out.println("Query time[lookback=" + sec + "]:[" + (end - theStart) + "ms]");
> }
> } catch(Exception e) {
> e.printStackTrace();
> fail(e.getMessage());
> }
> }
> ---
> create column family Quotes
> with column_type = Super
> and comparator = BytesType
> and subcomparator = BytesType
> and keys_cached = 7000
> and rows_cached = 0
> and row_cache_save_period = 0
> and key_cache_save_period = 3600
> and memtable_throughput = 255
> and memtable_operations = 0.29
> AND compression_options={sstable_compression:SnappyCompressor, chunk_length_kb:64};
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4340) Cassandra upgrade to 1.1.1
resulted in slow query issue
Posted by "Ivan Ganza (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13403930#comment-13403930 ]
Ivan Ganza commented on CASSANDRA-4340:
---------------------------------------
I dropped and tried to rebuild the CF. At that point we noticed some schema options has changed from earlier (we had originally defined this schema a year ago).
After doing this speed is (so far) very good -- and some sstables have been saved -- so this is looking pretty good. However we have to wait and see for the entire day of trading as the trades build up.
New CF def looks like this:
create column family Quotes
with column_type = Super
and comparator = BytesType
and subcomparator = BytesType
and caching = all
AND compression_options={sstable_compression:SnappyCompressor, chunk_length_kb:64};
> Cassandra upgrade to 1.1.1 resulted in slow query issue
> -------------------------------------------------------
>
> Key: CASSANDRA-4340
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4340
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.1
> Environment: Ubuntu Linux, Java 7, Hector 1.0-1
> Reporter: Ivan Ganza
> Assignee: Pavel Yaskevich
> Fix For: 1.1.2
>
> Attachments: CassandraIssue.java
>
>
> We have recently introduced Cassandra at the Globe and Mail here in Toronto, Canada. We are processing and storing the North American stock-market feed. We have found it to work very quickly and things have been looking very good.
> Recently we upgraded to version 1.1.1 and then we have noticed some issues occurring.
> I will try to describe it for you here. Basically one operation that we very often perform and is very critical is the ability to 'get the latest quote'. This would return to you the latest Quote adjusted against exchange delay rules. With Cassandra version 1.0.3 we could get a Quote in around 2ms. After update we are looking at time of at least 2-3 seconds.
> The way we query the quote is using a REVERSED SuperSliceQuery with start=now, end=00:00:00.000 (beginning of day) LIMITED to 1.
> Our investigation leads us to suspect that, since upgrade, Cassandra seems to be reading the sstable from disk even when we request a small range of day only 5 seconds back. If you look at the output below you can see that the query does NOT get slower as the lookback increases from 5 sec, 60 sec, 15 min, 60 min, and 24 hours.
> We also noticed that the query was very fast for the first five minutes of trading, apparently until the first sstable was flushed to disk. After that we go into query times of 1-2 seconds or so.
> Query time[lookback=5]:[1711ms]
> Query time[lookback=60]:[1592ms]
> Query time[lookback=900]:[1520ms]
> Query time[lookback=3600]:[1294ms]
> Query time[lookback=86400]:[1391ms]
> We would really appreciate input or help on this.
> Cassandra version: 1.1.1
> Hector version: 1.0-1
> ---
> public void testCassandraIssue() {
> try {
> int[] seconds = new int[]{ 5, 60, 60 * 15, 60 * 60, 60 * 60 * 24};
> for(int sec : seconds) {
> DateTime start = new DateTime();
> SuperSliceQuery<String, String, String, String> superSliceQuery = HFactory.createSuperSliceQuery(keyspaceOperator, StringSerializer.get(), StringSerializer.get(), StringSerializer.get(), StringSerializer.get());
> superSliceQuery.setKey("101390" + "." + testFormatter.print(start));
> superSliceQuery.setColumnFamily("Quotes");
> superSliceQuery.setRange(superKeyFormatter.print(start),
> superKeyFormatter.print(start.minusSeconds(sec)),
> true,
> 1);
> long theStart = System.currentTimeMillis();
> QueryResult<SuperSlice<String, String, String>> result = superSliceQuery.execute();
> long end = System.currentTimeMillis();
> System.out.println("Query time[lookback=" + sec + "]:[" + (end - theStart) + "ms]");
> }
> } catch(Exception e) {
> e.printStackTrace();
> fail(e.getMessage());
> }
> }
> ---
> create column family Quotes
> with column_type = Super
> and comparator = BytesType
> and subcomparator = BytesType
> and keys_cached = 7000
> and rows_cached = 0
> and row_cache_save_period = 0
> and key_cache_save_period = 3600
> and memtable_throughput = 255
> and memtable_operations = 0.29
> AND compression_options={sstable_compression:SnappyCompressor, chunk_length_kb:64};
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4340) Cassandra upgrade to 1.1.1
resulted in slow query issue
Posted by "Ivan Ganza (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13403131#comment-13403131 ]
Ivan Ganza commented on CASSANDRA-4340:
---------------------------------------
We didn't use row caching on the Quotes CF prior to the upgrade and the response times were acceptable. We did try enabling row caching on the Quotes CF after the upgrade. Because Quotes has wide rows, the SerializingCacheProvider did not help much and resulted in instability. The ConcurrentLinkedHashmap provider does help, but the cache misses are still in the 2 second range.
Normally, we query recent data in the Quotes CF. In fact, it is possible to truncate this CF daily because the older data in it is not useful. On past versions of Cassandra, the data for our column slices appears to come from memtables almost exclusively. On 1.1.1, we get great performance right up to the point in which the first sstable is written to disk. Then query performance (on cache misses if caching is enabled) degrades gradually throughout the day.
We suspected an issue with the bloom filters, so we tried running nodetool upgradesstables, but it did not help query performance.
> Cassandra upgrade to 1.1.1 resulted in slow query issue
> -------------------------------------------------------
>
> Key: CASSANDRA-4340
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4340
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.1
> Environment: Ubuntu Linux, Java 7, Hector 1.0-1
> Reporter: Ivan Ganza
> Assignee: Pavel Yaskevich
> Fix For: 1.1.2
>
> Attachments: CassandraIssue.java
>
>
> We have recently introduced Cassandra at the Globe and Mail here in Toronto, Canada. We are processing and storing the North American stock-market feed. We have found it to work very quickly and things have been looking very good.
> Recently we upgraded to version 1.1.1 and then we have noticed some issues occurring.
> I will try to describe it for you here. Basically one operation that we very often perform and is very critical is the ability to 'get the latest quote'. This would return to you the latest Quote adjusted against exchange delay rules. With Cassandra version 1.0.3 we could get a Quote in around 2ms. After update we are looking at time of at least 2-3 seconds.
> The way we query the quote is using a REVERSED SuperSliceQuery with start=now, end=00:00:00.000 (beginning of day) LIMITED to 1.
> Our investigation leads us to suspect that, since upgrade, Cassandra seems to be reading the sstable from disk even when we request a small range of day only 5 seconds back. If you look at the output below you can see that the query does NOT get slower as the lookback increases from 5 sec, 60 sec, 15 min, 60 min, and 24 hours.
> We also noticed that the query was very fast for the first five minutes of trading, apparently until the first sstable was flushed to disk. After that we go into query times of 1-2 seconds or so.
> Query time[lookback=5]:[1711ms]
> Query time[lookback=60]:[1592ms]
> Query time[lookback=900]:[1520ms]
> Query time[lookback=3600]:[1294ms]
> Query time[lookback=86400]:[1391ms]
> We would really appreciate input or help on this.
> Cassandra version: 1.1.1
> Hector version: 1.0-1
> ---
> public void testCassandraIssue() {
> try {
> int[] seconds = new int[]{ 5, 60, 60 * 15, 60 * 60, 60 * 60 * 24};
> for(int sec : seconds) {
> DateTime start = new DateTime();
> SuperSliceQuery<String, String, String, String> superSliceQuery = HFactory.createSuperSliceQuery(keyspaceOperator, StringSerializer.get(), StringSerializer.get(), StringSerializer.get(), StringSerializer.get());
> superSliceQuery.setKey("101390" + "." + testFormatter.print(start));
> superSliceQuery.setColumnFamily("Quotes");
> superSliceQuery.setRange(superKeyFormatter.print(start),
> superKeyFormatter.print(start.minusSeconds(sec)),
> true,
> 1);
> long theStart = System.currentTimeMillis();
> QueryResult<SuperSlice<String, String, String>> result = superSliceQuery.execute();
> long end = System.currentTimeMillis();
> System.out.println("Query time[lookback=" + sec + "]:[" + (end - theStart) + "ms]");
> }
> } catch(Exception e) {
> e.printStackTrace();
> fail(e.getMessage());
> }
> }
> ---
> create column family Quotes
> with column_type = Super
> and comparator = BytesType
> and subcomparator = BytesType
> and keys_cached = 7000
> and rows_cached = 0
> and row_cache_save_period = 0
> and key_cache_save_period = 3600
> and memtable_throughput = 255
> and memtable_operations = 0.29
> AND compression_options={sstable_compression:SnappyCompressor, chunk_length_kb:64};
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4340) Cassandra upgrade to 1.1.1
resulted in slow query issue
Posted by "Pavel Yaskevich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13403097#comment-13403097 ]
Pavel Yaskevich commented on CASSANDRA-4340:
--------------------------------------------
In the Cassandra starting from version 1.1.x key/row caches _per-CF_ was replaced with _global_ key/row caches configurable via conf/cassandra.yaml, did you try to tune up {key,row}_cache_size_in_mb properties after upgrade?
> Cassandra upgrade to 1.1.1 resulted in slow query issue
> -------------------------------------------------------
>
> Key: CASSANDRA-4340
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4340
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.1
> Environment: Ubuntu Linux, Java 7, Hector 1.0-1
> Reporter: Ivan Ganza
> Assignee: Pavel Yaskevich
> Fix For: 1.1.2
>
> Attachments: CassandraIssue.java
>
>
> We have recently introduced Cassandra at the Globe and Mail here in Toronto, Canada. We are processing and storing the North American stock-market feed. We have found it to work very quickly and things have been looking very good.
> Recently we upgraded to version 1.1.1 and then we have noticed some issues occurring.
> I will try to describe it for you here. Basically one operation that we very often perform and is very critical is the ability to 'get the latest quote'. This would return to you the latest Quote adjusted against exchange delay rules. With Cassandra version 1.0.3 we could get a Quote in around 2ms. After update we are looking at time of at least 2-3 seconds.
> The way we query the quote is using a REVERSED SuperSliceQuery with start=now, end=00:00:00.000 (beginning of day) LIMITED to 1.
> Our investigation leads us to suspect that, since upgrade, Cassandra seems to be reading the sstable from disk even when we request a small range of day only 5 seconds back. If you look at the output below you can see that the query does NOT get slower as the lookback increases from 5 sec, 60 sec, 15 min, 60 min, and 24 hours.
> We also noticed that the query was very fast for the first five minutes of trading, apparently until the first sstable was flushed to disk. After that we go into query times of 1-2 seconds or so.
> Query time[lookback=5]:[1711ms]
> Query time[lookback=60]:[1592ms]
> Query time[lookback=900]:[1520ms]
> Query time[lookback=3600]:[1294ms]
> Query time[lookback=86400]:[1391ms]
> We would really appreciate input or help on this.
> Cassandra version: 1.1.1
> Hector version: 1.0-1
> ---
> public void testCassandraIssue() {
> try {
> int[] seconds = new int[]{ 5, 60, 60 * 15, 60 * 60, 60 * 60 * 24};
> for(int sec : seconds) {
> DateTime start = new DateTime();
> SuperSliceQuery<String, String, String, String> superSliceQuery = HFactory.createSuperSliceQuery(keyspaceOperator, StringSerializer.get(), StringSerializer.get(), StringSerializer.get(), StringSerializer.get());
> superSliceQuery.setKey("101390" + "." + testFormatter.print(start));
> superSliceQuery.setColumnFamily("Quotes");
> superSliceQuery.setRange(superKeyFormatter.print(start),
> superKeyFormatter.print(start.minusSeconds(sec)),
> true,
> 1);
> long theStart = System.currentTimeMillis();
> QueryResult<SuperSlice<String, String, String>> result = superSliceQuery.execute();
> long end = System.currentTimeMillis();
> System.out.println("Query time[lookback=" + sec + "]:[" + (end - theStart) + "ms]");
> }
> } catch(Exception e) {
> e.printStackTrace();
> fail(e.getMessage());
> }
> }
> ---
> create column family Quotes
> with column_type = Super
> and comparator = BytesType
> and subcomparator = BytesType
> and keys_cached = 7000
> and rows_cached = 0
> and row_cache_save_period = 0
> and key_cache_save_period = 3600
> and memtable_throughput = 255
> and memtable_operations = 0.29
> AND compression_options={sstable_compression:SnappyCompressor, chunk_length_kb:64};
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-4340) Cassandra upgrade to 1.1.1
resulted in slow query issue
Posted by "Ivan Ganza (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ivan Ganza updated CASSANDRA-4340:
----------------------------------
Comment: was deleted
(was: public void testCassandraIssue() {
try {
int[] seconds = new int[]{ 5, 60, 60 * 15, 60 * 60, 60 * 60 * 24};
for(int sec : seconds) {
DateTime start = new DateTime();
SuperSliceQuery<String, String, String, String> superSliceQuery = HFactory.createSuperSliceQuery(keyspaceOperator, StringSerializer.get(), StringSerializer.get(), StringSerializer.get(), StringSerializer.get());
superSliceQuery.setKey("101390" + "." + testFormatter.print(start));
superSliceQuery.setColumnFamily("Quotes");
superSliceQuery.setRange(superKeyFormatter.print(start),
superKeyFormatter.print(start.minusSeconds(sec)),
true,
1);
long theStart = System.currentTimeMillis();
QueryResult<SuperSlice<String, String, String>> result = superSliceQuery.execute();
long end = System.currentTimeMillis();
System.out.println("Query time[lookback=" + sec + "]:[" + (end - theStart) + "ms]");
}
} catch(Exception e) {
e.printStackTrace();
fail(e.getMessage());
}
}
)
> Cassandra upgrade to 1.1.1 resulted in slow query issue
> -------------------------------------------------------
>
> Key: CASSANDRA-4340
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4340
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 1.1.1
> Environment: Ubuntu Linux, Java 7, Hector 1.0-1
> Reporter: Ivan Ganza
>
> We have recently introduced Cassandra at the Globe and Mail here in Toronto, Canada. We are processing and storing the North American stock-market feed. We have found it to work very quickly and things have been looking very good.
> Recently we upgraded to version 1.1.1 and then we have noticed some issues occurring.
> I will try to describe it for you here. Basically one operation that we very often perform and is very critical is the ability to 'get the latest quote'. This would return to you the latest Quote adjusted against exchange delay rules. With Cassandra version 1.0.3 we could get a Quote in around 2ms. After update we are looking at time of at least 2-3 seconds.
> The way we query the quote is using a REVERSED SuperSliceQuery with start=now, end=00:00:00.000 (beginning of day) LIMITED to 1.
> Our investigation leads us to suspect that, since upgrade, Cassandra seems to be reading the sstable from disk even when we request a small range of day only 5 seconds back. If you look at the output below you can see that the query does NOT get slower as the lookback increases from 5 sec, 60 sec, 15 min, 60 min, and 24 hours.
> We also noticed that the query was very fast for the first five minutes of trading, apparently until the first sstable was flushed to disk. After that we go into query times of 1-2 seconds or so.
> Query time[lookback=5]:[1711ms]
> Query time[lookback=60]:[1592ms]
> Query time[lookback=900]:[1520ms]
> Query time[lookback=3600]:[1294ms]
> Query time[lookback=86400]:[1391ms]
> We would really appreciate input or help on this.
> Cassandra version: 1.1.1
> Hector version: 1.0-1
> ---
> public void testCassandraIssue() {
> try {
> int[] seconds = new int[]{ 5, 60, 60 * 15, 60 * 60, 60 * 60 * 24};
> for(int sec : seconds) {
> DateTime start = new DateTime();
> SuperSliceQuery<String, String, String, String> superSliceQuery = HFactory.createSuperSliceQuery(keyspaceOperator, StringSerializer.get(), StringSerializer.get(), StringSerializer.get(), StringSerializer.get());
> superSliceQuery.setKey("101390" + "." + testFormatter.print(start));
> superSliceQuery.setColumnFamily("Quotes");
> superSliceQuery.setRange(superKeyFormatter.print(start),
> superKeyFormatter.print(start.minusSeconds(sec)),
> true,
> 1);
> long theStart = System.currentTimeMillis();
> QueryResult<SuperSlice<String, String, String>> result = superSliceQuery.execute();
> long end = System.currentTimeMillis();
> System.out.println("Query time[lookback=" + sec + "]:[" + (end - theStart) + "ms]");
> }
> } catch(Exception e) {
> e.printStackTrace();
> fail(e.getMessage());
> }
> }
> ---
> create column family Quotes
> with column_type = Super
> and comparator = BytesType
> and subcomparator = BytesType
> and keys_cached = 7000
> and rows_cached = 0
> and row_cache_save_period = 0
> and key_cache_save_period = 3600
> and memtable_throughput = 255
> and memtable_operations = 0.29
> AND compression_options={sstable_compression:SnappyCompressor, chunk_length_kb:64};
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4340) Cassandra upgrade to 1.1.1
resulted in slow query issue
Posted by "Pavel Yaskevich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13403431#comment-13403431 ]
Pavel Yaskevich commented on CASSANDRA-4340:
--------------------------------------------
Also can you try to repo this effect on the single separate machine running 1.1.1 and query some toy dataset or maybe even a single super row?
> Cassandra upgrade to 1.1.1 resulted in slow query issue
> -------------------------------------------------------
>
> Key: CASSANDRA-4340
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4340
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.1
> Environment: Ubuntu Linux, Java 7, Hector 1.0-1
> Reporter: Ivan Ganza
> Assignee: Pavel Yaskevich
> Fix For: 1.1.2
>
> Attachments: CassandraIssue.java
>
>
> We have recently introduced Cassandra at the Globe and Mail here in Toronto, Canada. We are processing and storing the North American stock-market feed. We have found it to work very quickly and things have been looking very good.
> Recently we upgraded to version 1.1.1 and then we have noticed some issues occurring.
> I will try to describe it for you here. Basically one operation that we very often perform and is very critical is the ability to 'get the latest quote'. This would return to you the latest Quote adjusted against exchange delay rules. With Cassandra version 1.0.3 we could get a Quote in around 2ms. After update we are looking at time of at least 2-3 seconds.
> The way we query the quote is using a REVERSED SuperSliceQuery with start=now, end=00:00:00.000 (beginning of day) LIMITED to 1.
> Our investigation leads us to suspect that, since upgrade, Cassandra seems to be reading the sstable from disk even when we request a small range of day only 5 seconds back. If you look at the output below you can see that the query does NOT get slower as the lookback increases from 5 sec, 60 sec, 15 min, 60 min, and 24 hours.
> We also noticed that the query was very fast for the first five minutes of trading, apparently until the first sstable was flushed to disk. After that we go into query times of 1-2 seconds or so.
> Query time[lookback=5]:[1711ms]
> Query time[lookback=60]:[1592ms]
> Query time[lookback=900]:[1520ms]
> Query time[lookback=3600]:[1294ms]
> Query time[lookback=86400]:[1391ms]
> We would really appreciate input or help on this.
> Cassandra version: 1.1.1
> Hector version: 1.0-1
> ---
> public void testCassandraIssue() {
> try {
> int[] seconds = new int[]{ 5, 60, 60 * 15, 60 * 60, 60 * 60 * 24};
> for(int sec : seconds) {
> DateTime start = new DateTime();
> SuperSliceQuery<String, String, String, String> superSliceQuery = HFactory.createSuperSliceQuery(keyspaceOperator, StringSerializer.get(), StringSerializer.get(), StringSerializer.get(), StringSerializer.get());
> superSliceQuery.setKey("101390" + "." + testFormatter.print(start));
> superSliceQuery.setColumnFamily("Quotes");
> superSliceQuery.setRange(superKeyFormatter.print(start),
> superKeyFormatter.print(start.minusSeconds(sec)),
> true,
> 1);
> long theStart = System.currentTimeMillis();
> QueryResult<SuperSlice<String, String, String>> result = superSliceQuery.execute();
> long end = System.currentTimeMillis();
> System.out.println("Query time[lookback=" + sec + "]:[" + (end - theStart) + "ms]");
> }
> } catch(Exception e) {
> e.printStackTrace();
> fail(e.getMessage());
> }
> }
> ---
> create column family Quotes
> with column_type = Super
> and comparator = BytesType
> and subcomparator = BytesType
> and keys_cached = 7000
> and rows_cached = 0
> and row_cache_save_period = 0
> and key_cache_save_period = 3600
> and memtable_throughput = 255
> and memtable_operations = 0.29
> AND compression_options={sstable_compression:SnappyCompressor, chunk_length_kb:64};
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Comment Edited] (CASSANDRA-4340) Cassandra upgrade to 1.1.1
resulted in slow query issue
Posted by "Pavel Yaskevich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13403097#comment-13403097 ]
Pavel Yaskevich edited comment on CASSANDRA-4340 at 6/28/12 1:54 PM:
---------------------------------------------------------------------
In the Cassandra starting from version 1.1.x key/row caches _per-CF_ was replaced with _global_ key/row caches configurable via conf/cassandra.yaml, did you try to tune up {key,row}_cache_size_in_mb properties after upgrade?
Edit: also configuration of the cache could be done via JMX using o.a.c.service.CacheServiceMBean
was (Author: xedin):
In the Cassandra starting from version 1.1.x key/row caches _per-CF_ was replaced with _global_ key/row caches configurable via conf/cassandra.yaml, did you try to tune up {key,row}_cache_size_in_mb properties after upgrade?
> Cassandra upgrade to 1.1.1 resulted in slow query issue
> -------------------------------------------------------
>
> Key: CASSANDRA-4340
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4340
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.1
> Environment: Ubuntu Linux, Java 7, Hector 1.0-1
> Reporter: Ivan Ganza
> Assignee: Pavel Yaskevich
> Fix For: 1.1.2
>
> Attachments: CassandraIssue.java
>
>
> We have recently introduced Cassandra at the Globe and Mail here in Toronto, Canada. We are processing and storing the North American stock-market feed. We have found it to work very quickly and things have been looking very good.
> Recently we upgraded to version 1.1.1 and then we have noticed some issues occurring.
> I will try to describe it for you here. Basically one operation that we very often perform and is very critical is the ability to 'get the latest quote'. This would return to you the latest Quote adjusted against exchange delay rules. With Cassandra version 1.0.3 we could get a Quote in around 2ms. After update we are looking at time of at least 2-3 seconds.
> The way we query the quote is using a REVERSED SuperSliceQuery with start=now, end=00:00:00.000 (beginning of day) LIMITED to 1.
> Our investigation leads us to suspect that, since upgrade, Cassandra seems to be reading the sstable from disk even when we request a small range of day only 5 seconds back. If you look at the output below you can see that the query does NOT get slower as the lookback increases from 5 sec, 60 sec, 15 min, 60 min, and 24 hours.
> We also noticed that the query was very fast for the first five minutes of trading, apparently until the first sstable was flushed to disk. After that we go into query times of 1-2 seconds or so.
> Query time[lookback=5]:[1711ms]
> Query time[lookback=60]:[1592ms]
> Query time[lookback=900]:[1520ms]
> Query time[lookback=3600]:[1294ms]
> Query time[lookback=86400]:[1391ms]
> We would really appreciate input or help on this.
> Cassandra version: 1.1.1
> Hector version: 1.0-1
> ---
> public void testCassandraIssue() {
> try {
> int[] seconds = new int[]{ 5, 60, 60 * 15, 60 * 60, 60 * 60 * 24};
> for(int sec : seconds) {
> DateTime start = new DateTime();
> SuperSliceQuery<String, String, String, String> superSliceQuery = HFactory.createSuperSliceQuery(keyspaceOperator, StringSerializer.get(), StringSerializer.get(), StringSerializer.get(), StringSerializer.get());
> superSliceQuery.setKey("101390" + "." + testFormatter.print(start));
> superSliceQuery.setColumnFamily("Quotes");
> superSliceQuery.setRange(superKeyFormatter.print(start),
> superKeyFormatter.print(start.minusSeconds(sec)),
> true,
> 1);
> long theStart = System.currentTimeMillis();
> QueryResult<SuperSlice<String, String, String>> result = superSliceQuery.execute();
> long end = System.currentTimeMillis();
> System.out.println("Query time[lookback=" + sec + "]:[" + (end - theStart) + "ms]");
> }
> } catch(Exception e) {
> e.printStackTrace();
> fail(e.getMessage());
> }
> }
> ---
> create column family Quotes
> with column_type = Super
> and comparator = BytesType
> and subcomparator = BytesType
> and keys_cached = 7000
> and rows_cached = 0
> and row_cache_save_period = 0
> and key_cache_save_period = 3600
> and memtable_throughput = 255
> and memtable_operations = 0.29
> AND compression_options={sstable_compression:SnappyCompressor, chunk_length_kb:64};
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4340) Cassandra upgrade to 1.1.1
resulted in slow query issue
Posted by "Ivan Ganza (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13403285#comment-13403285 ]
Ivan Ganza commented on CASSANDRA-4340:
---------------------------------------
I was trying to duplicate our problem by fetching the same quote rapidly in succession and got the following error and stock trace. Could you add it to our Jira ticket? I will keep looking at this. Row caching was turned off.
java.lang.OutOfMemoryError: Java heap space
at org.apache.cassandra.io.compress.CompressedRandomAccessReader.<init>(CompressedRandomAccessReader.java:66)
at org.apache.cassandra.io.compress.CompressedRandomAccessReader.open(CompressedRandomAccessReader.java:46)
at org.apache.cassandra.io.compress.CompressedRandomAccessReader.open(CompressedRandomAccessReader.java:41)
at org.apache.cassandra.io.util.CompressedSegmentedFile.getSegment(CompressedSegmentedFile.java:63)
at org.apache.cassandra.io.sstable.SSTableReader.getFileDataInput(SSTableReader.java:871)
at org.apache.cassandra.db.columniterator.SSTableSliceIterator.<init>(SSTableSliceIterator.java:48)
at org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:66)
at org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:78)
at org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:256)
at org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:63)
at org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1321)
at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1183)
at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1118)
at org.apache.cassandra.db.Table.getRow(Table.java:374)
at org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:69)
at org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:816)
at org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1250)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
> Cassandra upgrade to 1.1.1 resulted in slow query issue
> -------------------------------------------------------
>
> Key: CASSANDRA-4340
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4340
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.1
> Environment: Ubuntu Linux, Java 7, Hector 1.0-1
> Reporter: Ivan Ganza
> Assignee: Pavel Yaskevich
> Fix For: 1.1.2
>
> Attachments: CassandraIssue.java
>
>
> We have recently introduced Cassandra at the Globe and Mail here in Toronto, Canada. We are processing and storing the North American stock-market feed. We have found it to work very quickly and things have been looking very good.
> Recently we upgraded to version 1.1.1 and then we have noticed some issues occurring.
> I will try to describe it for you here. Basically one operation that we very often perform and is very critical is the ability to 'get the latest quote'. This would return to you the latest Quote adjusted against exchange delay rules. With Cassandra version 1.0.3 we could get a Quote in around 2ms. After update we are looking at time of at least 2-3 seconds.
> The way we query the quote is using a REVERSED SuperSliceQuery with start=now, end=00:00:00.000 (beginning of day) LIMITED to 1.
> Our investigation leads us to suspect that, since upgrade, Cassandra seems to be reading the sstable from disk even when we request a small range of day only 5 seconds back. If you look at the output below you can see that the query does NOT get slower as the lookback increases from 5 sec, 60 sec, 15 min, 60 min, and 24 hours.
> We also noticed that the query was very fast for the first five minutes of trading, apparently until the first sstable was flushed to disk. After that we go into query times of 1-2 seconds or so.
> Query time[lookback=5]:[1711ms]
> Query time[lookback=60]:[1592ms]
> Query time[lookback=900]:[1520ms]
> Query time[lookback=3600]:[1294ms]
> Query time[lookback=86400]:[1391ms]
> We would really appreciate input or help on this.
> Cassandra version: 1.1.1
> Hector version: 1.0-1
> ---
> public void testCassandraIssue() {
> try {
> int[] seconds = new int[]{ 5, 60, 60 * 15, 60 * 60, 60 * 60 * 24};
> for(int sec : seconds) {
> DateTime start = new DateTime();
> SuperSliceQuery<String, String, String, String> superSliceQuery = HFactory.createSuperSliceQuery(keyspaceOperator, StringSerializer.get(), StringSerializer.get(), StringSerializer.get(), StringSerializer.get());
> superSliceQuery.setKey("101390" + "." + testFormatter.print(start));
> superSliceQuery.setColumnFamily("Quotes");
> superSliceQuery.setRange(superKeyFormatter.print(start),
> superKeyFormatter.print(start.minusSeconds(sec)),
> true,
> 1);
> long theStart = System.currentTimeMillis();
> QueryResult<SuperSlice<String, String, String>> result = superSliceQuery.execute();
> long end = System.currentTimeMillis();
> System.out.println("Query time[lookback=" + sec + "]:[" + (end - theStart) + "ms]");
> }
> } catch(Exception e) {
> e.printStackTrace();
> fail(e.getMessage());
> }
> }
> ---
> create column family Quotes
> with column_type = Super
> and comparator = BytesType
> and subcomparator = BytesType
> and keys_cached = 7000
> and rows_cached = 0
> and row_cache_save_period = 0
> and key_cache_save_period = 3600
> and memtable_throughput = 255
> and memtable_operations = 0.29
> AND compression_options={sstable_compression:SnappyCompressor, chunk_length_kb:64};
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4340) Cassandra upgrade to 1.1.1
resulted in slow query issue
Posted by "Ivan Ganza (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13295043#comment-13295043 ]
Ivan Ganza commented on CASSANDRA-4340:
---------------------------------------
public void testCassandraIssue() {
try {
int[] seconds = new int[]{ 5, 60, 60 * 15, 60 * 60, 60 * 60 * 24};
for(int sec : seconds) {
DateTime start = new DateTime();
SuperSliceQuery<String, String, String, String> superSliceQuery = HFactory.createSuperSliceQuery(keyspaceOperator, StringSerializer.get(), StringSerializer.get(), StringSerializer.get(), StringSerializer.get());
superSliceQuery.setKey("101390" + "." + testFormatter.print(start));
superSliceQuery.setColumnFamily("Quotes");
superSliceQuery.setRange(superKeyFormatter.print(start),
superKeyFormatter.print(start.minusSeconds(sec)),
true,
1);
long theStart = System.currentTimeMillis();
QueryResult<SuperSlice<String, String, String>> result = superSliceQuery.execute();
long end = System.currentTimeMillis();
System.out.println("Query time[lookback=" + sec + "]:[" + (end - theStart) + "ms]");
}
} catch(Exception e) {
e.printStackTrace();
fail(e.getMessage());
}
}
> Cassandra upgrade to 1.1.1 resulted in slow query issue
> -------------------------------------------------------
>
> Key: CASSANDRA-4340
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4340
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 1.1.1
> Environment: Ubuntu Linux, Java 7, Hector 1.0-1
> Reporter: Ivan Ganza
>
> We have recently introduced Cassandra at the Globe and Mail here in Toronto, Canada. We are processing and storing the North American stock-market feed. We have found it to work very quickly and things have been looking very good.
> Recently we upgraded to version 1.1.1 and then we have noticed some issues occurring.
> I will try to describe it for you here. Basically one operation that we very often perform and is very critical is the ability to 'get the latest quote'. This would return to you the latest Quote adjusted against exchange delay rules. With Cassandra version 1.0.3 we could get a Quote in around 2ms. After update we are looking at time of at least 2-3 seconds.
> The way we query the quote is using a REVERSED SuperSliceQuery with start=now, end=00:00:00.000 (beginning of day) LIMITED to 1.
> Our investigation leads us to suspect that, since upgrade, Cassandra seems to be reading the sstable from disk even when we request a small range of day only 5 seconds back. If you look at the output below you can see that the query does NOT get slower as the lookback increases from 5 sec, 60 sec, 15 min, 60 min, and 24 hours.
> We also noticed that the query was very fast for the first five minutes of trading, apparently until the first sstable was flushed to disk. After that we go into query times of 1-2 seconds or so.
> Query time[lookback=5]:[1711ms]
> Query time[lookback=60]:[1592ms]
> Query time[lookback=900]:[1520ms]
> Query time[lookback=3600]:[1294ms]
> Query time[lookback=86400]:[1391ms]
> We would really appreciate input or help on this.
> Cassandra version: 1.1.1
> Hector version: 1.0-1
> ---
> public void testCassandraIssue() {
> try {
> int[] seconds = new int[]{ 5, 60, 60 * 15, 60 * 60, 60 * 60 * 24};
> for(int sec : seconds) {
> DateTime start = new DateTime();
> SuperSliceQuery<String, String, String, String> superSliceQuery = HFactory.createSuperSliceQuery(keyspaceOperator, StringSerializer.get(), StringSerializer.get(), StringSerializer.get(), StringSerializer.get());
> superSliceQuery.setKey("101390" + "." + testFormatter.print(start));
> superSliceQuery.setColumnFamily("Quotes");
> superSliceQuery.setRange(superKeyFormatter.print(start),
> superKeyFormatter.print(start.minusSeconds(sec)),
> true,
> 1);
> long theStart = System.currentTimeMillis();
> QueryResult<SuperSlice<String, String, String>> result = superSliceQuery.execute();
> long end = System.currentTimeMillis();
> System.out.println("Query time[lookback=" + sec + "]:[" + (end - theStart) + "ms]");
> }
> } catch(Exception e) {
> e.printStackTrace();
> fail(e.getMessage());
> }
> }
> ---
> create column family Quotes
> with column_type = Super
> and comparator = BytesType
> and subcomparator = BytesType
> and keys_cached = 7000
> and rows_cached = 0
> and row_cache_save_period = 0
> and key_cache_save_period = 3600
> and memtable_throughput = 255
> and memtable_operations = 0.29
> AND compression_options={sstable_compression:SnappyCompressor, chunk_length_kb:64};
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4340) Cassandra upgrade to 1.1.1
resulted in slow query issue
Posted by "Pavel Yaskevich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13403403#comment-13403403 ]
Pavel Yaskevich commented on CASSANDRA-4340:
--------------------------------------------
This rings the bell that you need to add more capacity to your ring as you are overloading this one with the data (or the rows are too big), did the load on the ring change as well after upgrade?
> Cassandra upgrade to 1.1.1 resulted in slow query issue
> -------------------------------------------------------
>
> Key: CASSANDRA-4340
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4340
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.1
> Environment: Ubuntu Linux, Java 7, Hector 1.0-1
> Reporter: Ivan Ganza
> Assignee: Pavel Yaskevich
> Fix For: 1.1.2
>
> Attachments: CassandraIssue.java
>
>
> We have recently introduced Cassandra at the Globe and Mail here in Toronto, Canada. We are processing and storing the North American stock-market feed. We have found it to work very quickly and things have been looking very good.
> Recently we upgraded to version 1.1.1 and then we have noticed some issues occurring.
> I will try to describe it for you here. Basically one operation that we very often perform and is very critical is the ability to 'get the latest quote'. This would return to you the latest Quote adjusted against exchange delay rules. With Cassandra version 1.0.3 we could get a Quote in around 2ms. After update we are looking at time of at least 2-3 seconds.
> The way we query the quote is using a REVERSED SuperSliceQuery with start=now, end=00:00:00.000 (beginning of day) LIMITED to 1.
> Our investigation leads us to suspect that, since upgrade, Cassandra seems to be reading the sstable from disk even when we request a small range of day only 5 seconds back. If you look at the output below you can see that the query does NOT get slower as the lookback increases from 5 sec, 60 sec, 15 min, 60 min, and 24 hours.
> We also noticed that the query was very fast for the first five minutes of trading, apparently until the first sstable was flushed to disk. After that we go into query times of 1-2 seconds or so.
> Query time[lookback=5]:[1711ms]
> Query time[lookback=60]:[1592ms]
> Query time[lookback=900]:[1520ms]
> Query time[lookback=3600]:[1294ms]
> Query time[lookback=86400]:[1391ms]
> We would really appreciate input or help on this.
> Cassandra version: 1.1.1
> Hector version: 1.0-1
> ---
> public void testCassandraIssue() {
> try {
> int[] seconds = new int[]{ 5, 60, 60 * 15, 60 * 60, 60 * 60 * 24};
> for(int sec : seconds) {
> DateTime start = new DateTime();
> SuperSliceQuery<String, String, String, String> superSliceQuery = HFactory.createSuperSliceQuery(keyspaceOperator, StringSerializer.get(), StringSerializer.get(), StringSerializer.get(), StringSerializer.get());
> superSliceQuery.setKey("101390" + "." + testFormatter.print(start));
> superSliceQuery.setColumnFamily("Quotes");
> superSliceQuery.setRange(superKeyFormatter.print(start),
> superKeyFormatter.print(start.minusSeconds(sec)),
> true,
> 1);
> long theStart = System.currentTimeMillis();
> QueryResult<SuperSlice<String, String, String>> result = superSliceQuery.execute();
> long end = System.currentTimeMillis();
> System.out.println("Query time[lookback=" + sec + "]:[" + (end - theStart) + "ms]");
> }
> } catch(Exception e) {
> e.printStackTrace();
> fail(e.getMessage());
> }
> }
> ---
> create column family Quotes
> with column_type = Super
> and comparator = BytesType
> and subcomparator = BytesType
> and keys_cached = 7000
> and rows_cached = 0
> and row_cache_save_period = 0
> and key_cache_save_period = 3600
> and memtable_throughput = 255
> and memtable_operations = 0.29
> AND compression_options={sstable_compression:SnappyCompressor, chunk_length_kb:64};
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4340) Cassandra upgrade to 1.1.1
resulted in slow query issue
Posted by "Pavel Yaskevich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13403202#comment-13403202 ]
Pavel Yaskevich commented on CASSANDRA-4340:
--------------------------------------------
I did check the "get_slice" path and didn't found any significant difference between 1.0 and 1.1.1. I see that 'create column family' CLI statement in description still uses memtable_* options, have you tried making your memtables bigger to accomodate more updates (as result that would produce less SSTables)? How about the key cache, what is a hit rate? What compaction activity are you seeing (also interesting if memtable flush rate is different from 1.0)? It would be great if you could attach to one of the nodes with profiler (say YourKit) and profile what method is the most contended...
> Cassandra upgrade to 1.1.1 resulted in slow query issue
> -------------------------------------------------------
>
> Key: CASSANDRA-4340
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4340
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.1
> Environment: Ubuntu Linux, Java 7, Hector 1.0-1
> Reporter: Ivan Ganza
> Assignee: Pavel Yaskevich
> Fix For: 1.1.2
>
> Attachments: CassandraIssue.java
>
>
> We have recently introduced Cassandra at the Globe and Mail here in Toronto, Canada. We are processing and storing the North American stock-market feed. We have found it to work very quickly and things have been looking very good.
> Recently we upgraded to version 1.1.1 and then we have noticed some issues occurring.
> I will try to describe it for you here. Basically one operation that we very often perform and is very critical is the ability to 'get the latest quote'. This would return to you the latest Quote adjusted against exchange delay rules. With Cassandra version 1.0.3 we could get a Quote in around 2ms. After update we are looking at time of at least 2-3 seconds.
> The way we query the quote is using a REVERSED SuperSliceQuery with start=now, end=00:00:00.000 (beginning of day) LIMITED to 1.
> Our investigation leads us to suspect that, since upgrade, Cassandra seems to be reading the sstable from disk even when we request a small range of day only 5 seconds back. If you look at the output below you can see that the query does NOT get slower as the lookback increases from 5 sec, 60 sec, 15 min, 60 min, and 24 hours.
> We also noticed that the query was very fast for the first five minutes of trading, apparently until the first sstable was flushed to disk. After that we go into query times of 1-2 seconds or so.
> Query time[lookback=5]:[1711ms]
> Query time[lookback=60]:[1592ms]
> Query time[lookback=900]:[1520ms]
> Query time[lookback=3600]:[1294ms]
> Query time[lookback=86400]:[1391ms]
> We would really appreciate input or help on this.
> Cassandra version: 1.1.1
> Hector version: 1.0-1
> ---
> public void testCassandraIssue() {
> try {
> int[] seconds = new int[]{ 5, 60, 60 * 15, 60 * 60, 60 * 60 * 24};
> for(int sec : seconds) {
> DateTime start = new DateTime();
> SuperSliceQuery<String, String, String, String> superSliceQuery = HFactory.createSuperSliceQuery(keyspaceOperator, StringSerializer.get(), StringSerializer.get(), StringSerializer.get(), StringSerializer.get());
> superSliceQuery.setKey("101390" + "." + testFormatter.print(start));
> superSliceQuery.setColumnFamily("Quotes");
> superSliceQuery.setRange(superKeyFormatter.print(start),
> superKeyFormatter.print(start.minusSeconds(sec)),
> true,
> 1);
> long theStart = System.currentTimeMillis();
> QueryResult<SuperSlice<String, String, String>> result = superSliceQuery.execute();
> long end = System.currentTimeMillis();
> System.out.println("Query time[lookback=" + sec + "]:[" + (end - theStart) + "ms]");
> }
> } catch(Exception e) {
> e.printStackTrace();
> fail(e.getMessage());
> }
> }
> ---
> create column family Quotes
> with column_type = Super
> and comparator = BytesType
> and subcomparator = BytesType
> and keys_cached = 7000
> and rows_cached = 0
> and row_cache_save_period = 0
> and key_cache_save_period = 3600
> and memtable_throughput = 255
> and memtable_operations = 0.29
> AND compression_options={sstable_compression:SnappyCompressor, chunk_length_kb:64};
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Comment Edited] (CASSANDRA-4340) Cassandra upgrade to 1.1.1
resulted in slow query issue
Posted by "Ivan Ganza (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13295042#comment-13295042 ]
Ivan Ganza edited comment on CASSANDRA-4340 at 6/14/12 2:07 PM:
----------------------------------------------------------------
public void testCassandraIssue() {
try {
int[] seconds = new int[]{ 5, 60, 60 * 15, 60 * 60, 60 * 60 * 24};
for(int sec : seconds) {
DateTime start = new DateTime();
SuperSliceQuery<String, String, String, String> superSliceQuery = HFactory.createSuperSliceQuery(keyspaceOperator, StringSerializer.get(), StringSerializer.get(), StringSerializer.get(), StringSerializer.get());
superSliceQuery.setKey("101390" + "." + testFormatter.print(start));
superSliceQuery.setColumnFamily("Quotes");
superSliceQuery.setRange(superKeyFormatter.print(start),
superKeyFormatter.print(start.minusSeconds(sec)),
true,
1);
long theStart = System.currentTimeMillis();
QueryResult<SuperSlice<String, String, String>> result = superSliceQuery.execute();
long end = System.currentTimeMillis();
System.out.println("Query time[lookback=" + sec + "]:[" + (end - theStart) + "ms]");
}
} catch(Exception e) {
e.printStackTrace();
fail(e.getMessage());
}
}
was (Author: iganza@globeandmail.com):
public void testCassandraIssue() {
try {
int[] seconds = new int[]{ 5, 60, 60 * 15, 60 * 60, 60 * 60 * 24};
for(int sec : seconds) {
DateTime start = new DateTime();
SuperSliceQuery<String, String, String, String> superSliceQuery = HFactory.createSuperSliceQuery(keyspaceOperator, StringSerializer.get(), StringSerializer.get(), StringSerializer.get(), StringSerializer.get());
superSliceQuery.setKey("101390" + "." + testFormatter.print(start));
superSliceQuery.setColumnFamily("Quotes");
superSliceQuery.setRange(superKeyFormatter.print(start),
superKeyFormatter.print(start.minusSeconds(sec)),
true,
1);
long theStart = System.currentTimeMillis();
QueryResult<SuperSlice<String, String, String>> result = superSliceQuery.execute();
long end = System.currentTimeMillis();
System.out.println("Query time[lookback=" + sec + "]:[" + (end - theStart) + "ms]");
}
} catch(Exception e) {
e.printStackTrace();
fail(e.getMessage());
}
}
> Cassandra upgrade to 1.1.1 resulted in slow query issue
> -------------------------------------------------------
>
> Key: CASSANDRA-4340
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4340
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 1.1.1
> Environment: Ubuntu Linux, Java 7, Hector 1.0-1
> Reporter: Ivan Ganza
>
> We have recently introduced Cassandra at the Globe and Mail here in Toronto, Canada. We are processing and storing the North American stock-market feed. We have found it to work very quickly and things have been looking very good.
> Recently we upgraded to version 1.1.1 and then we have noticed some issues occurring.
> I will try to describe it for you here. Basically one operation that we very often perform and is very critical is the ability to 'get the latest quote'. This would return to you the latest Quote adjusted against exchange delay rules. With Cassandra version 1.0.3 we could get a Quote in around 2ms. After update we are looking at time of at least 2-3 seconds.
> The way we query the quote is using a REVERSED SuperSliceQuery with start=now, end=00:00:00.000 (beginning of day) LIMITED to 1.
> Our investigation leads us to suspect that, since upgrade, Cassandra seems to be reading the sstable from disk even when we request a small range of day only 5 seconds back. If you look at the output below you can see that the query does NOT get slower as the lookback increases from 5 sec, 60 sec, 15 min, 60 min, and 24 hours.
> We also noticed that the query was very fast for the first five minutes of trading, apparently until the first sstable was flushed to disk. After that we go into query times of 1-2 seconds or so.
> Query time[lookback=5]:[1711ms]
> Query time[lookback=60]:[1592ms]
> Query time[lookback=900]:[1520ms]
> Query time[lookback=3600]:[1294ms]
> Query time[lookback=86400]:[1391ms]
> We would really appreciate input or help on this.
> Cassandra version: 1.1.1
> Hector version: 1.0-1
> ---
> public void testCassandraIssue() {
> try {
> int[] seconds = new int[]{ 5, 60, 60 * 15, 60 * 60, 60 * 60 * 24};
> for(int sec : seconds) {
> DateTime start = new DateTime();
> SuperSliceQuery<String, String, String, String> superSliceQuery = HFactory.createSuperSliceQuery(keyspaceOperator, StringSerializer.get(), StringSerializer.get(), StringSerializer.get(), StringSerializer.get());
> superSliceQuery.setKey("101390" + "." + testFormatter.print(start));
> superSliceQuery.setColumnFamily("Quotes");
> superSliceQuery.setRange(superKeyFormatter.print(start),
> superKeyFormatter.print(start.minusSeconds(sec)),
> true,
> 1);
> long theStart = System.currentTimeMillis();
> QueryResult<SuperSlice<String, String, String>> result = superSliceQuery.execute();
> long end = System.currentTimeMillis();
> System.out.println("Query time[lookback=" + sec + "]:[" + (end - theStart) + "ms]");
> }
> } catch(Exception e) {
> e.printStackTrace();
> fail(e.getMessage());
> }
> }
> ---
> create column family Quotes
> with column_type = Super
> and comparator = BytesType
> and subcomparator = BytesType
> and keys_cached = 7000
> and rows_cached = 0
> and row_cache_save_period = 0
> and key_cache_save_period = 3600
> and memtable_throughput = 255
> and memtable_operations = 0.29
> AND compression_options={sstable_compression:SnappyCompressor, chunk_length_kb:64};
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-4340) Cassandra upgrade to 1.1.1
resulted in slow query issue
Posted by "Ivan Ganza (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ivan Ganza updated CASSANDRA-4340:
----------------------------------
Description:
We have recently introduced Cassandra at the Globe and Mail here in Toronto, Canada. We are processing and storing the North American stock-market feed. We have found it to work very quickly and things have been looking very good.
Recently we upgraded to version 1.1.1 and then we have noticed some issues occurring.
I will try to describe it for you here. Basically one operation that we very often perform and is very critical is the ability to 'get the latest quote'. This would return to you the latest Quote adjusted against exchange delay rules. With Cassandra version 1.0.3 we could get a Quote in around 2ms. After update we are looking at time of at least 2-3 seconds.
The way we query the quote is using a REVERSED SuperSliceQuery with start=now, end=00:00:00.000 (beginning of day) LIMITED to 1.
Our investigation leads us to suspect that, since upgrade, Cassandra seems to be reading the sstable from disk even when we request a small range of day only 5 seconds back. If you look at the output below you can see that the query does NOT get slower as the lookback increases from 5 sec, 60 sec, 15 min, 60 min, and 24 hours.
We also noticed that the query was very fast for the first five minutes of trading, apparently until the first sstable was flushed to disk. After that we go into query times of 1-2 seconds or so.
Query time[lookback=5]:[1711ms]
Query time[lookback=60]:[1592ms]
Query time[lookback=900]:[1520ms]
Query time[lookback=3600]:[1294ms]
Query time[lookback=86400]:[1391ms]
We would really appreciate input or help on this.
Cassandra version: 1.1.1
Hector version: 1.0-1
---
public void testCassandraIssue() {
try {
int[] seconds = new int[]{ 5, 60, 60 * 15, 60 * 60, 60 * 60 * 24};
for(int sec : seconds) {
DateTime start = new DateTime();
SuperSliceQuery<String, String, String, String> superSliceQuery = HFactory.createSuperSliceQuery(keyspaceOperator, StringSerializer.get(), StringSerializer.get(), StringSerializer.get(), StringSerializer.get());
superSliceQuery.setKey("101390" + "." + testFormatter.print(start));
superSliceQuery.setColumnFamily("Quotes");
superSliceQuery.setRange(superKeyFormatter.print(start),
superKeyFormatter.print(start.minusSeconds(sec)),
true,
1);
long theStart = System.currentTimeMillis();
QueryResult<SuperSlice<String, String, String>> result = superSliceQuery.execute();
long end = System.currentTimeMillis();
System.out.println("Query time[lookback=" + sec + "]:[" + (end - theStart) + "ms]");
}
} catch(Exception e) {
e.printStackTrace();
fail(e.getMessage());
}
}
---
create column family Quotes
with column_type = Super
and comparator = BytesType
and subcomparator = BytesType
and keys_cached = 7000
and rows_cached = 0
and row_cache_save_period = 0
and key_cache_save_period = 3600
and memtable_throughput = 255
and memtable_operations = 0.29
AND compression_options={sstable_compression:SnappyCompressor, chunk_length_kb:64};
was:
We have recently introduced Cassandra at the Globe and Mail here in Toronto, Canada. We are processing and storing the North American stock-market feed. We have found it to work very quickly and things have been looking very good.
Recently we upgraded to version 1.1.1 and then we have noticed some issues occurring.
I will try to describe it for you here. Basically one operation that we very often perform and is very critical is the ability to 'get the latest quote'. This would return to you the latest Quote adjusted against exchange delay rules. With Cassandra version 1.0.3 we could get a Quote in around 2ms. After update we are looking at time of at least 2-3 seconds.
The way we query the quote is using a REVERSED SuperSliceQuery with start=now, end=00:00:00.000 (beginning of day) LIMITED to 1.
Our investigation leads us to suspect that, since upgrade, Cassandra seems to be reading the sstable from disk even when we request a small range of day only 5 seconds back. If you look at the output below you can see that the query does NOT get slower as the lookback increases from 5 sec, 60 sec, 15 min, 60 min, and 24 hours.
We also noticed that the query was very fast for the first five minutes of trading, apparently until the first sstable was flushed to disk. After that we go into query times of 1-2 seconds or so.
Query time[lookback=5]:[1711ms]
Query time[lookback=60]:[1592ms]
Query time[lookback=900]:[1520ms]
Query time[lookback=3600]:[1294ms]
Query time[lookback=86400]:[1391ms]
We would really appreciate input or help on this.
Cassandra version: 1.1.1
Hector version: 1.0-1
---
<PRE>
public void testCassandraIssue() {
try {
int[] seconds = new int[]{ 5, 60, 60 * 15, 60 * 60, 60 * 60 * 24};
for(int sec : seconds) {
DateTime start = new DateTime();
SuperSliceQuery<String, String, String, String> superSliceQuery = HFactory.createSuperSliceQuery(keyspaceOperator, StringSerializer.get(), StringSerializer.get(), StringSerializer.get(), StringSerializer.get());
superSliceQuery.setKey("101390" + "." + testFormatter.print(start));
superSliceQuery.setColumnFamily("Quotes");
superSliceQuery.setRange(superKeyFormatter.print(start),
superKeyFormatter.print(start.minusSeconds(sec)),
true,
1);
long theStart = System.currentTimeMillis();
QueryResult<SuperSlice<String, String, String>> result = superSliceQuery.execute();
long end = System.currentTimeMillis();
System.out.println("Query time[lookback=" + sec + "]:[" + (end - theStart) + "ms]");
}
} catch(Exception e) {
e.printStackTrace();
fail(e.getMessage());
}
}
</PRE>
---
create column family Quotes
with column_type = Super
and comparator = BytesType
and subcomparator = BytesType
and keys_cached = 7000
and rows_cached = 0
and row_cache_save_period = 0
and key_cache_save_period = 3600
and memtable_throughput = 255
and memtable_operations = 0.29
AND compression_options={sstable_compression:SnappyCompressor, chunk_length_kb:64};
> Cassandra upgrade to 1.1.1 resulted in slow query issue
> -------------------------------------------------------
>
> Key: CASSANDRA-4340
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4340
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 1.1.1
> Environment: Ubuntu Linux, Java 7, Hector 1.0-1
> Reporter: Ivan Ganza
>
> We have recently introduced Cassandra at the Globe and Mail here in Toronto, Canada. We are processing and storing the North American stock-market feed. We have found it to work very quickly and things have been looking very good.
> Recently we upgraded to version 1.1.1 and then we have noticed some issues occurring.
> I will try to describe it for you here. Basically one operation that we very often perform and is very critical is the ability to 'get the latest quote'. This would return to you the latest Quote adjusted against exchange delay rules. With Cassandra version 1.0.3 we could get a Quote in around 2ms. After update we are looking at time of at least 2-3 seconds.
> The way we query the quote is using a REVERSED SuperSliceQuery with start=now, end=00:00:00.000 (beginning of day) LIMITED to 1.
> Our investigation leads us to suspect that, since upgrade, Cassandra seems to be reading the sstable from disk even when we request a small range of day only 5 seconds back. If you look at the output below you can see that the query does NOT get slower as the lookback increases from 5 sec, 60 sec, 15 min, 60 min, and 24 hours.
> We also noticed that the query was very fast for the first five minutes of trading, apparently until the first sstable was flushed to disk. After that we go into query times of 1-2 seconds or so.
> Query time[lookback=5]:[1711ms]
> Query time[lookback=60]:[1592ms]
> Query time[lookback=900]:[1520ms]
> Query time[lookback=3600]:[1294ms]
> Query time[lookback=86400]:[1391ms]
> We would really appreciate input or help on this.
> Cassandra version: 1.1.1
> Hector version: 1.0-1
> ---
> public void testCassandraIssue() {
> try {
> int[] seconds = new int[]{ 5, 60, 60 * 15, 60 * 60, 60 * 60 * 24};
> for(int sec : seconds) {
> DateTime start = new DateTime();
> SuperSliceQuery<String, String, String, String> superSliceQuery = HFactory.createSuperSliceQuery(keyspaceOperator, StringSerializer.get(), StringSerializer.get(), StringSerializer.get(), StringSerializer.get());
> superSliceQuery.setKey("101390" + "." + testFormatter.print(start));
> superSliceQuery.setColumnFamily("Quotes");
> superSliceQuery.setRange(superKeyFormatter.print(start),
> superKeyFormatter.print(start.minusSeconds(sec)),
> true,
> 1);
> long theStart = System.currentTimeMillis();
> QueryResult<SuperSlice<String, String, String>> result = superSliceQuery.execute();
> long end = System.currentTimeMillis();
> System.out.println("Query time[lookback=" + sec + "]:[" + (end - theStart) + "ms]");
> }
> } catch(Exception e) {
> e.printStackTrace();
> fail(e.getMessage());
> }
> }
> ---
> create column family Quotes
> with column_type = Super
> and comparator = BytesType
> and subcomparator = BytesType
> and keys_cached = 7000
> and rows_cached = 0
> and row_cache_save_period = 0
> and key_cache_save_period = 3600
> and memtable_throughput = 255
> and memtable_operations = 0.29
> AND compression_options={sstable_compression:SnappyCompressor, chunk_length_kb:64};
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (CASSANDRA-4340) Cassandra upgrade to 1.1.1
resulted in slow query issue
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis reassigned CASSANDRA-4340:
-----------------------------------------
Assignee: Pavel Yaskevich (was: Yuki Morishita)
> Cassandra upgrade to 1.1.1 resulted in slow query issue
> -------------------------------------------------------
>
> Key: CASSANDRA-4340
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4340
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.1
> Environment: Ubuntu Linux, Java 7, Hector 1.0-1
> Reporter: Ivan Ganza
> Assignee: Pavel Yaskevich
> Fix For: 1.1.2
>
> Attachments: CassandraIssue.java
>
>
> We have recently introduced Cassandra at the Globe and Mail here in Toronto, Canada. We are processing and storing the North American stock-market feed. We have found it to work very quickly and things have been looking very good.
> Recently we upgraded to version 1.1.1 and then we have noticed some issues occurring.
> I will try to describe it for you here. Basically one operation that we very often perform and is very critical is the ability to 'get the latest quote'. This would return to you the latest Quote adjusted against exchange delay rules. With Cassandra version 1.0.3 we could get a Quote in around 2ms. After update we are looking at time of at least 2-3 seconds.
> The way we query the quote is using a REVERSED SuperSliceQuery with start=now, end=00:00:00.000 (beginning of day) LIMITED to 1.
> Our investigation leads us to suspect that, since upgrade, Cassandra seems to be reading the sstable from disk even when we request a small range of day only 5 seconds back. If you look at the output below you can see that the query does NOT get slower as the lookback increases from 5 sec, 60 sec, 15 min, 60 min, and 24 hours.
> We also noticed that the query was very fast for the first five minutes of trading, apparently until the first sstable was flushed to disk. After that we go into query times of 1-2 seconds or so.
> Query time[lookback=5]:[1711ms]
> Query time[lookback=60]:[1592ms]
> Query time[lookback=900]:[1520ms]
> Query time[lookback=3600]:[1294ms]
> Query time[lookback=86400]:[1391ms]
> We would really appreciate input or help on this.
> Cassandra version: 1.1.1
> Hector version: 1.0-1
> ---
> public void testCassandraIssue() {
> try {
> int[] seconds = new int[]{ 5, 60, 60 * 15, 60 * 60, 60 * 60 * 24};
> for(int sec : seconds) {
> DateTime start = new DateTime();
> SuperSliceQuery<String, String, String, String> superSliceQuery = HFactory.createSuperSliceQuery(keyspaceOperator, StringSerializer.get(), StringSerializer.get(), StringSerializer.get(), StringSerializer.get());
> superSliceQuery.setKey("101390" + "." + testFormatter.print(start));
> superSliceQuery.setColumnFamily("Quotes");
> superSliceQuery.setRange(superKeyFormatter.print(start),
> superKeyFormatter.print(start.minusSeconds(sec)),
> true,
> 1);
> long theStart = System.currentTimeMillis();
> QueryResult<SuperSlice<String, String, String>> result = superSliceQuery.execute();
> long end = System.currentTimeMillis();
> System.out.println("Query time[lookback=" + sec + "]:[" + (end - theStart) + "ms]");
> }
> } catch(Exception e) {
> e.printStackTrace();
> fail(e.getMessage());
> }
> }
> ---
> create column family Quotes
> with column_type = Super
> and comparator = BytesType
> and subcomparator = BytesType
> and keys_cached = 7000
> and rows_cached = 0
> and row_cache_save_period = 0
> and key_cache_save_period = 3600
> and memtable_throughput = 255
> and memtable_operations = 0.29
> AND compression_options={sstable_compression:SnappyCompressor, chunk_length_kb:64};
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4340) Cassandra upgrade to 1.1.1
resulted in slow query issue
Posted by "Ivan Ganza (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13406486#comment-13406486 ]
Ivan Ganza commented on CASSANDRA-4340:
---------------------------------------
So far so good. We can probably close this issue for now. We will re-open if issue re-asserts itself. However for now seems good.
Thanks.
> Cassandra upgrade to 1.1.1 resulted in slow query issue
> -------------------------------------------------------
>
> Key: CASSANDRA-4340
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4340
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.1
> Environment: Ubuntu Linux, Java 7, Hector 1.0-1
> Reporter: Ivan Ganza
> Assignee: Pavel Yaskevich
> Fix For: 1.1.3
>
> Attachments: CassandraIssue.java
>
>
> We have recently introduced Cassandra at the Globe and Mail here in Toronto, Canada. We are processing and storing the North American stock-market feed. We have found it to work very quickly and things have been looking very good.
> Recently we upgraded to version 1.1.1 and then we have noticed some issues occurring.
> I will try to describe it for you here. Basically one operation that we very often perform and is very critical is the ability to 'get the latest quote'. This would return to you the latest Quote adjusted against exchange delay rules. With Cassandra version 1.0.3 we could get a Quote in around 2ms. After update we are looking at time of at least 2-3 seconds.
> The way we query the quote is using a REVERSED SuperSliceQuery with start=now, end=00:00:00.000 (beginning of day) LIMITED to 1.
> Our investigation leads us to suspect that, since upgrade, Cassandra seems to be reading the sstable from disk even when we request a small range of day only 5 seconds back. If you look at the output below you can see that the query does NOT get slower as the lookback increases from 5 sec, 60 sec, 15 min, 60 min, and 24 hours.
> We also noticed that the query was very fast for the first five minutes of trading, apparently until the first sstable was flushed to disk. After that we go into query times of 1-2 seconds or so.
> Query time[lookback=5]:[1711ms]
> Query time[lookback=60]:[1592ms]
> Query time[lookback=900]:[1520ms]
> Query time[lookback=3600]:[1294ms]
> Query time[lookback=86400]:[1391ms]
> We would really appreciate input or help on this.
> Cassandra version: 1.1.1
> Hector version: 1.0-1
> ---
> public void testCassandraIssue() {
> try {
> int[] seconds = new int[]{ 5, 60, 60 * 15, 60 * 60, 60 * 60 * 24};
> for(int sec : seconds) {
> DateTime start = new DateTime();
> SuperSliceQuery<String, String, String, String> superSliceQuery = HFactory.createSuperSliceQuery(keyspaceOperator, StringSerializer.get(), StringSerializer.get(), StringSerializer.get(), StringSerializer.get());
> superSliceQuery.setKey("101390" + "." + testFormatter.print(start));
> superSliceQuery.setColumnFamily("Quotes");
> superSliceQuery.setRange(superKeyFormatter.print(start),
> superKeyFormatter.print(start.minusSeconds(sec)),
> true,
> 1);
> long theStart = System.currentTimeMillis();
> QueryResult<SuperSlice<String, String, String>> result = superSliceQuery.execute();
> long end = System.currentTimeMillis();
> System.out.println("Query time[lookback=" + sec + "]:[" + (end - theStart) + "ms]");
> }
> } catch(Exception e) {
> e.printStackTrace();
> fail(e.getMessage());
> }
> }
> ---
> create column family Quotes
> with column_type = Super
> and comparator = BytesType
> and subcomparator = BytesType
> and keys_cached = 7000
> and rows_cached = 0
> and row_cache_save_period = 0
> and key_cache_save_period = 3600
> and memtable_throughput = 255
> and memtable_operations = 0.29
> AND compression_options={sstable_compression:SnappyCompressor, chunk_length_kb:64};
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-4340) Cassandra upgrade to 1.1.1
resulted in slow query issue
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-4340:
--------------------------------------
Component/s: Core
Fix Version/s: 1.1.2
> Cassandra upgrade to 1.1.1 resulted in slow query issue
> -------------------------------------------------------
>
> Key: CASSANDRA-4340
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4340
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.1
> Environment: Ubuntu Linux, Java 7, Hector 1.0-1
> Reporter: Ivan Ganza
> Assignee: Yuki Morishita
> Fix For: 1.1.2
>
> Attachments: CassandraIssue.java
>
>
> We have recently introduced Cassandra at the Globe and Mail here in Toronto, Canada. We are processing and storing the North American stock-market feed. We have found it to work very quickly and things have been looking very good.
> Recently we upgraded to version 1.1.1 and then we have noticed some issues occurring.
> I will try to describe it for you here. Basically one operation that we very often perform and is very critical is the ability to 'get the latest quote'. This would return to you the latest Quote adjusted against exchange delay rules. With Cassandra version 1.0.3 we could get a Quote in around 2ms. After update we are looking at time of at least 2-3 seconds.
> The way we query the quote is using a REVERSED SuperSliceQuery with start=now, end=00:00:00.000 (beginning of day) LIMITED to 1.
> Our investigation leads us to suspect that, since upgrade, Cassandra seems to be reading the sstable from disk even when we request a small range of day only 5 seconds back. If you look at the output below you can see that the query does NOT get slower as the lookback increases from 5 sec, 60 sec, 15 min, 60 min, and 24 hours.
> We also noticed that the query was very fast for the first five minutes of trading, apparently until the first sstable was flushed to disk. After that we go into query times of 1-2 seconds or so.
> Query time[lookback=5]:[1711ms]
> Query time[lookback=60]:[1592ms]
> Query time[lookback=900]:[1520ms]
> Query time[lookback=3600]:[1294ms]
> Query time[lookback=86400]:[1391ms]
> We would really appreciate input or help on this.
> Cassandra version: 1.1.1
> Hector version: 1.0-1
> ---
> public void testCassandraIssue() {
> try {
> int[] seconds = new int[]{ 5, 60, 60 * 15, 60 * 60, 60 * 60 * 24};
> for(int sec : seconds) {
> DateTime start = new DateTime();
> SuperSliceQuery<String, String, String, String> superSliceQuery = HFactory.createSuperSliceQuery(keyspaceOperator, StringSerializer.get(), StringSerializer.get(), StringSerializer.get(), StringSerializer.get());
> superSliceQuery.setKey("101390" + "." + testFormatter.print(start));
> superSliceQuery.setColumnFamily("Quotes");
> superSliceQuery.setRange(superKeyFormatter.print(start),
> superKeyFormatter.print(start.minusSeconds(sec)),
> true,
> 1);
> long theStart = System.currentTimeMillis();
> QueryResult<SuperSlice<String, String, String>> result = superSliceQuery.execute();
> long end = System.currentTimeMillis();
> System.out.println("Query time[lookback=" + sec + "]:[" + (end - theStart) + "ms]");
> }
> } catch(Exception e) {
> e.printStackTrace();
> fail(e.getMessage());
> }
> }
> ---
> create column family Quotes
> with column_type = Super
> and comparator = BytesType
> and subcomparator = BytesType
> and keys_cached = 7000
> and rows_cached = 0
> and row_cache_save_period = 0
> and key_cache_save_period = 3600
> and memtable_throughput = 255
> and memtable_operations = 0.29
> AND compression_options={sstable_compression:SnappyCompressor, chunk_length_kb:64};
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (CASSANDRA-4340) Cassandra upgrade to 1.1.1
resulted in slow query issue
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis reassigned CASSANDRA-4340:
-----------------------------------------
Assignee: Yuki Morishita
> Cassandra upgrade to 1.1.1 resulted in slow query issue
> -------------------------------------------------------
>
> Key: CASSANDRA-4340
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4340
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.1
> Environment: Ubuntu Linux, Java 7, Hector 1.0-1
> Reporter: Ivan Ganza
> Assignee: Yuki Morishita
> Fix For: 1.1.2
>
> Attachments: CassandraIssue.java
>
>
> We have recently introduced Cassandra at the Globe and Mail here in Toronto, Canada. We are processing and storing the North American stock-market feed. We have found it to work very quickly and things have been looking very good.
> Recently we upgraded to version 1.1.1 and then we have noticed some issues occurring.
> I will try to describe it for you here. Basically one operation that we very often perform and is very critical is the ability to 'get the latest quote'. This would return to you the latest Quote adjusted against exchange delay rules. With Cassandra version 1.0.3 we could get a Quote in around 2ms. After update we are looking at time of at least 2-3 seconds.
> The way we query the quote is using a REVERSED SuperSliceQuery with start=now, end=00:00:00.000 (beginning of day) LIMITED to 1.
> Our investigation leads us to suspect that, since upgrade, Cassandra seems to be reading the sstable from disk even when we request a small range of day only 5 seconds back. If you look at the output below you can see that the query does NOT get slower as the lookback increases from 5 sec, 60 sec, 15 min, 60 min, and 24 hours.
> We also noticed that the query was very fast for the first five minutes of trading, apparently until the first sstable was flushed to disk. After that we go into query times of 1-2 seconds or so.
> Query time[lookback=5]:[1711ms]
> Query time[lookback=60]:[1592ms]
> Query time[lookback=900]:[1520ms]
> Query time[lookback=3600]:[1294ms]
> Query time[lookback=86400]:[1391ms]
> We would really appreciate input or help on this.
> Cassandra version: 1.1.1
> Hector version: 1.0-1
> ---
> public void testCassandraIssue() {
> try {
> int[] seconds = new int[]{ 5, 60, 60 * 15, 60 * 60, 60 * 60 * 24};
> for(int sec : seconds) {
> DateTime start = new DateTime();
> SuperSliceQuery<String, String, String, String> superSliceQuery = HFactory.createSuperSliceQuery(keyspaceOperator, StringSerializer.get(), StringSerializer.get(), StringSerializer.get(), StringSerializer.get());
> superSliceQuery.setKey("101390" + "." + testFormatter.print(start));
> superSliceQuery.setColumnFamily("Quotes");
> superSliceQuery.setRange(superKeyFormatter.print(start),
> superKeyFormatter.print(start.minusSeconds(sec)),
> true,
> 1);
> long theStart = System.currentTimeMillis();
> QueryResult<SuperSlice<String, String, String>> result = superSliceQuery.execute();
> long end = System.currentTimeMillis();
> System.out.println("Query time[lookback=" + sec + "]:[" + (end - theStart) + "ms]");
> }
> } catch(Exception e) {
> e.printStackTrace();
> fail(e.getMessage());
> }
> }
> ---
> create column family Quotes
> with column_type = Super
> and comparator = BytesType
> and subcomparator = BytesType
> and keys_cached = 7000
> and rows_cached = 0
> and row_cache_save_period = 0
> and key_cache_save_period = 3600
> and memtable_throughput = 255
> and memtable_operations = 0.29
> AND compression_options={sstable_compression:SnappyCompressor, chunk_length_kb:64};
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-4340) Cassandra upgrade to 1.1.1
resulted in slow query issue
Posted by "Ivan Ganza (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ivan Ganza updated CASSANDRA-4340:
----------------------------------
Description:
We have recently introduced Cassandra at the Globe and Mail here in Toronto, Canada. We are processing and storing the North American stock-market feed. We have found it to work very quickly and things have been looking very good.
Recently we upgraded to version 1.1.1 and then we have noticed some issues occurring.
I will try to describe it for you here. Basically one operation that we very often perform and is very critical is the ability to 'get the latest quote'. This would return to you the latest Quote adjusted against exchange delay rules. With Cassandra version 1.0.3 we could get a Quote in around 2ms. After update we are looking at time of at least 2-3 seconds.
The way we query the quote is using a REVERSED SuperSliceQuery with start=now, end=00:00:00.000 (beginning of day) LIMITED to 1.
Our investigation leads us to suspect that, since upgrade, Cassandra seems to be reading the sstable from disk even when we request a small range of day only 5 seconds back. If you look at the output below you can see that the query does NOT get slower as the lookback increases from 5 sec, 60 sec, 15 min, 60 min, and 24 hours.
We also noticed that the query was very fast for the first five minutes of trading, apparently until the first sstable was flushed to disk. After that we go into query times of 1-2 seconds or so.
Query time[lookback=5]:[1711ms]
Query time[lookback=60]:[1592ms]
Query time[lookback=900]:[1520ms]
Query time[lookback=3600]:[1294ms]
Query time[lookback=86400]:[1391ms]
We would really appreciate input or help on this.
Cassandra version: 1.1.1
Hector version: 1.0-1
---
<PRE>
public void testCassandraIssue() {
try {
int[] seconds = new int[]{ 5, 60, 60 * 15, 60 * 60, 60 * 60 * 24};
for(int sec : seconds) {
DateTime start = new DateTime();
SuperSliceQuery<String, String, String, String> superSliceQuery = HFactory.createSuperSliceQuery(keyspaceOperator, StringSerializer.get(), StringSerializer.get(), StringSerializer.get(), StringSerializer.get());
superSliceQuery.setKey("101390" + "." + testFormatter.print(start));
superSliceQuery.setColumnFamily("Quotes");
superSliceQuery.setRange(superKeyFormatter.print(start),
superKeyFormatter.print(start.minusSeconds(sec)),
true,
1);
long theStart = System.currentTimeMillis();
QueryResult<SuperSlice<String, String, String>> result = superSliceQuery.execute();
long end = System.currentTimeMillis();
System.out.println("Query time[lookback=" + sec + "]:[" + (end - theStart) + "ms]");
}
} catch(Exception e) {
e.printStackTrace();
fail(e.getMessage());
}
}
</PRE>
---
create column family Quotes
with column_type = Super
and comparator = BytesType
and subcomparator = BytesType
and keys_cached = 7000
and rows_cached = 0
and row_cache_save_period = 0
and key_cache_save_period = 3600
and memtable_throughput = 255
and memtable_operations = 0.29
AND compression_options={sstable_compression:SnappyCompressor, chunk_length_kb:64};
was:
We have recently introduced Cassandra at the Globe and Mail here in Toronto, Canada. We are processing and storing the North American stock-market feed. We have found it to work very quickly and things have been looking very good.
Recently we upgraded to version 1.1.1 and then we have noticed some issues occurring.
I will try to describe it for you here. Basically one operation that we very often perform and is very critical is the ability to 'get the latest quote'. This would return to you the latest Quote adjusted against exchange delay rules. With Cassandra version 1.0.3 we could get a Quote in around 2ms. After update we are looking at time of at least 2-3 seconds.
The way we query the quote is using a REVERSED SuperSliceQuery with start=now, end=00:00:00.000 (beginning of day) LIMITED to 1.
Our investigation leads us to suspect that, since upgrade, Cassandra seems to be reading the sstable from disk even when we request a small range of day only 5 seconds back. If you look at the output below you can see that the query does NOT get slower as the lookback increases from 5 sec, 60 sec, 15 min, 60 min, and 24 hours.
We also noticed that the query was very fast for the first five minutes of trading, apparently until the first sstable was flushed to disk. After that we go into query times of 1-2 seconds or so.
Query time[lookback=5]:[1711ms]
Query time[lookback=60]:[1592ms]
Query time[lookback=900]:[1520ms]
Query time[lookback=3600]:[1294ms]
Query time[lookback=86400]:[1391ms]
We would really appreciate input or help on this.
Cassandra version: 1.1.1
Hector version: 1.0-1
---
public void testCassandraIssue() {
try {
int[] seconds = new int[]{ 5, 60, 60 * 15, 60 * 60, 60 * 60 * 24};
for(int sec : seconds) {
DateTime start = new DateTime();
SuperSliceQuery<String, String, String, String> superSliceQuery = HFactory.createSuperSliceQuery(keyspaceOperator, StringSerializer.get(), StringSerializer.get(), StringSerializer.get(), StringSerializer.get());
superSliceQuery.setKey("101390" + "." + testFormatter.print(start));
superSliceQuery.setColumnFamily("Quotes");
superSliceQuery.setRange(superKeyFormatter.print(start),
superKeyFormatter.print(start.minusSeconds(sec)),
true,
1);
long theStart = System.currentTimeMillis();
QueryResult<SuperSlice<String, String, String>> result = superSliceQuery.execute();
long end = System.currentTimeMillis();
System.out.println("Query time[lookback=" + sec + "]:[" + (end - theStart) + "ms]");
}
} catch(Exception e) {
e.printStackTrace();
fail(e.getMessage());
}
}
---
create column family Quotes
with column_type = Super
and comparator = BytesType
and subcomparator = BytesType
and keys_cached = 7000
and rows_cached = 0
and row_cache_save_period = 0
and key_cache_save_period = 3600
and memtable_throughput = 255
and memtable_operations = 0.29
AND compression_options={sstable_compression:SnappyCompressor, chunk_length_kb:64};
> Cassandra upgrade to 1.1.1 resulted in slow query issue
> -------------------------------------------------------
>
> Key: CASSANDRA-4340
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4340
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 1.1.1
> Environment: Ubuntu Linux, Java 7, Hector 1.0-1
> Reporter: Ivan Ganza
>
> We have recently introduced Cassandra at the Globe and Mail here in Toronto, Canada. We are processing and storing the North American stock-market feed. We have found it to work very quickly and things have been looking very good.
> Recently we upgraded to version 1.1.1 and then we have noticed some issues occurring.
> I will try to describe it for you here. Basically one operation that we very often perform and is very critical is the ability to 'get the latest quote'. This would return to you the latest Quote adjusted against exchange delay rules. With Cassandra version 1.0.3 we could get a Quote in around 2ms. After update we are looking at time of at least 2-3 seconds.
> The way we query the quote is using a REVERSED SuperSliceQuery with start=now, end=00:00:00.000 (beginning of day) LIMITED to 1.
> Our investigation leads us to suspect that, since upgrade, Cassandra seems to be reading the sstable from disk even when we request a small range of day only 5 seconds back. If you look at the output below you can see that the query does NOT get slower as the lookback increases from 5 sec, 60 sec, 15 min, 60 min, and 24 hours.
> We also noticed that the query was very fast for the first five minutes of trading, apparently until the first sstable was flushed to disk. After that we go into query times of 1-2 seconds or so.
> Query time[lookback=5]:[1711ms]
> Query time[lookback=60]:[1592ms]
> Query time[lookback=900]:[1520ms]
> Query time[lookback=3600]:[1294ms]
> Query time[lookback=86400]:[1391ms]
> We would really appreciate input or help on this.
> Cassandra version: 1.1.1
> Hector version: 1.0-1
> ---
> <PRE>
> public void testCassandraIssue() {
> try {
> int[] seconds = new int[]{ 5, 60, 60 * 15, 60 * 60, 60 * 60 * 24};
> for(int sec : seconds) {
> DateTime start = new DateTime();
> SuperSliceQuery<String, String, String, String> superSliceQuery = HFactory.createSuperSliceQuery(keyspaceOperator, StringSerializer.get(), StringSerializer.get(), StringSerializer.get(), StringSerializer.get());
> superSliceQuery.setKey("101390" + "." + testFormatter.print(start));
> superSliceQuery.setColumnFamily("Quotes");
> superSliceQuery.setRange(superKeyFormatter.print(start),
> superKeyFormatter.print(start.minusSeconds(sec)),
> true,
> 1);
> long theStart = System.currentTimeMillis();
> QueryResult<SuperSlice<String, String, String>> result = superSliceQuery.execute();
> long end = System.currentTimeMillis();
> System.out.println("Query time[lookback=" + sec + "]:[" + (end - theStart) + "ms]");
> }
> } catch(Exception e) {
> e.printStackTrace();
> fail(e.getMessage());
> }
> }
> </PRE>
> ---
> create column family Quotes
> with column_type = Super
> and comparator = BytesType
> and subcomparator = BytesType
> and keys_cached = 7000
> and rows_cached = 0
> and row_cache_save_period = 0
> and key_cache_save_period = 3600
> and memtable_throughput = 255
> and memtable_operations = 0.29
> AND compression_options={sstable_compression:SnappyCompressor, chunk_length_kb:64};
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4340) Cassandra upgrade to 1.1.1
resulted in slow query issue
Posted by "Ivan Ganza (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13403919#comment-13403919 ]
Ivan Ganza commented on CASSANDRA-4340:
---------------------------------------
Wanted to let you know; load did not change and was roughly the same before and after update.
> Cassandra upgrade to 1.1.1 resulted in slow query issue
> -------------------------------------------------------
>
> Key: CASSANDRA-4340
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4340
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.1
> Environment: Ubuntu Linux, Java 7, Hector 1.0-1
> Reporter: Ivan Ganza
> Assignee: Pavel Yaskevich
> Fix For: 1.1.2
>
> Attachments: CassandraIssue.java
>
>
> We have recently introduced Cassandra at the Globe and Mail here in Toronto, Canada. We are processing and storing the North American stock-market feed. We have found it to work very quickly and things have been looking very good.
> Recently we upgraded to version 1.1.1 and then we have noticed some issues occurring.
> I will try to describe it for you here. Basically one operation that we very often perform and is very critical is the ability to 'get the latest quote'. This would return to you the latest Quote adjusted against exchange delay rules. With Cassandra version 1.0.3 we could get a Quote in around 2ms. After update we are looking at time of at least 2-3 seconds.
> The way we query the quote is using a REVERSED SuperSliceQuery with start=now, end=00:00:00.000 (beginning of day) LIMITED to 1.
> Our investigation leads us to suspect that, since upgrade, Cassandra seems to be reading the sstable from disk even when we request a small range of day only 5 seconds back. If you look at the output below you can see that the query does NOT get slower as the lookback increases from 5 sec, 60 sec, 15 min, 60 min, and 24 hours.
> We also noticed that the query was very fast for the first five minutes of trading, apparently until the first sstable was flushed to disk. After that we go into query times of 1-2 seconds or so.
> Query time[lookback=5]:[1711ms]
> Query time[lookback=60]:[1592ms]
> Query time[lookback=900]:[1520ms]
> Query time[lookback=3600]:[1294ms]
> Query time[lookback=86400]:[1391ms]
> We would really appreciate input or help on this.
> Cassandra version: 1.1.1
> Hector version: 1.0-1
> ---
> public void testCassandraIssue() {
> try {
> int[] seconds = new int[]{ 5, 60, 60 * 15, 60 * 60, 60 * 60 * 24};
> for(int sec : seconds) {
> DateTime start = new DateTime();
> SuperSliceQuery<String, String, String, String> superSliceQuery = HFactory.createSuperSliceQuery(keyspaceOperator, StringSerializer.get(), StringSerializer.get(), StringSerializer.get(), StringSerializer.get());
> superSliceQuery.setKey("101390" + "." + testFormatter.print(start));
> superSliceQuery.setColumnFamily("Quotes");
> superSliceQuery.setRange(superKeyFormatter.print(start),
> superKeyFormatter.print(start.minusSeconds(sec)),
> true,
> 1);
> long theStart = System.currentTimeMillis();
> QueryResult<SuperSlice<String, String, String>> result = superSliceQuery.execute();
> long end = System.currentTimeMillis();
> System.out.println("Query time[lookback=" + sec + "]:[" + (end - theStart) + "ms]");
> }
> } catch(Exception e) {
> e.printStackTrace();
> fail(e.getMessage());
> }
> }
> ---
> create column family Quotes
> with column_type = Super
> and comparator = BytesType
> and subcomparator = BytesType
> and keys_cached = 7000
> and rows_cached = 0
> and row_cache_save_period = 0
> and key_cache_save_period = 3600
> and memtable_throughput = 255
> and memtable_operations = 0.29
> AND compression_options={sstable_compression:SnappyCompressor, chunk_length_kb:64};
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (CASSANDRA-4340) Cassandra upgrade to 1.1.1
resulted in slow query issue
Posted by "Pavel Yaskevich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Pavel Yaskevich resolved CASSANDRA-4340.
----------------------------------------
Resolution: Not A Problem
> Cassandra upgrade to 1.1.1 resulted in slow query issue
> -------------------------------------------------------
>
> Key: CASSANDRA-4340
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4340
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.1
> Environment: Ubuntu Linux, Java 7, Hector 1.0-1
> Reporter: Ivan Ganza
> Assignee: Pavel Yaskevich
> Fix For: 1.1.3
>
> Attachments: CassandraIssue.java
>
>
> We have recently introduced Cassandra at the Globe and Mail here in Toronto, Canada. We are processing and storing the North American stock-market feed. We have found it to work very quickly and things have been looking very good.
> Recently we upgraded to version 1.1.1 and then we have noticed some issues occurring.
> I will try to describe it for you here. Basically one operation that we very often perform and is very critical is the ability to 'get the latest quote'. This would return to you the latest Quote adjusted against exchange delay rules. With Cassandra version 1.0.3 we could get a Quote in around 2ms. After update we are looking at time of at least 2-3 seconds.
> The way we query the quote is using a REVERSED SuperSliceQuery with start=now, end=00:00:00.000 (beginning of day) LIMITED to 1.
> Our investigation leads us to suspect that, since upgrade, Cassandra seems to be reading the sstable from disk even when we request a small range of day only 5 seconds back. If you look at the output below you can see that the query does NOT get slower as the lookback increases from 5 sec, 60 sec, 15 min, 60 min, and 24 hours.
> We also noticed that the query was very fast for the first five minutes of trading, apparently until the first sstable was flushed to disk. After that we go into query times of 1-2 seconds or so.
> Query time[lookback=5]:[1711ms]
> Query time[lookback=60]:[1592ms]
> Query time[lookback=900]:[1520ms]
> Query time[lookback=3600]:[1294ms]
> Query time[lookback=86400]:[1391ms]
> We would really appreciate input or help on this.
> Cassandra version: 1.1.1
> Hector version: 1.0-1
> ---
> public void testCassandraIssue() {
> try {
> int[] seconds = new int[]{ 5, 60, 60 * 15, 60 * 60, 60 * 60 * 24};
> for(int sec : seconds) {
> DateTime start = new DateTime();
> SuperSliceQuery<String, String, String, String> superSliceQuery = HFactory.createSuperSliceQuery(keyspaceOperator, StringSerializer.get(), StringSerializer.get(), StringSerializer.get(), StringSerializer.get());
> superSliceQuery.setKey("101390" + "." + testFormatter.print(start));
> superSliceQuery.setColumnFamily("Quotes");
> superSliceQuery.setRange(superKeyFormatter.print(start),
> superKeyFormatter.print(start.minusSeconds(sec)),
> true,
> 1);
> long theStart = System.currentTimeMillis();
> QueryResult<SuperSlice<String, String, String>> result = superSliceQuery.execute();
> long end = System.currentTimeMillis();
> System.out.println("Query time[lookback=" + sec + "]:[" + (end - theStart) + "ms]");
> }
> } catch(Exception e) {
> e.printStackTrace();
> fail(e.getMessage());
> }
> }
> ---
> create column family Quotes
> with column_type = Super
> and comparator = BytesType
> and subcomparator = BytesType
> and keys_cached = 7000
> and rows_cached = 0
> and row_cache_save_period = 0
> and key_cache_save_period = 3600
> and memtable_throughput = 255
> and memtable_operations = 0.29
> AND compression_options={sstable_compression:SnappyCompressor, chunk_length_kb:64};
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4340) Cassandra upgrade to 1.1.1
resulted in slow query issue
Posted by "Pavel Yaskevich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13406206#comment-13406206 ]
Pavel Yaskevich commented on CASSANDRA-4340:
--------------------------------------------
Ivan, is everything alright after rebuild of the CF after a work day?
> Cassandra upgrade to 1.1.1 resulted in slow query issue
> -------------------------------------------------------
>
> Key: CASSANDRA-4340
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4340
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.1
> Environment: Ubuntu Linux, Java 7, Hector 1.0-1
> Reporter: Ivan Ganza
> Assignee: Pavel Yaskevich
> Fix For: 1.1.3
>
> Attachments: CassandraIssue.java
>
>
> We have recently introduced Cassandra at the Globe and Mail here in Toronto, Canada. We are processing and storing the North American stock-market feed. We have found it to work very quickly and things have been looking very good.
> Recently we upgraded to version 1.1.1 and then we have noticed some issues occurring.
> I will try to describe it for you here. Basically one operation that we very often perform and is very critical is the ability to 'get the latest quote'. This would return to you the latest Quote adjusted against exchange delay rules. With Cassandra version 1.0.3 we could get a Quote in around 2ms. After update we are looking at time of at least 2-3 seconds.
> The way we query the quote is using a REVERSED SuperSliceQuery with start=now, end=00:00:00.000 (beginning of day) LIMITED to 1.
> Our investigation leads us to suspect that, since upgrade, Cassandra seems to be reading the sstable from disk even when we request a small range of day only 5 seconds back. If you look at the output below you can see that the query does NOT get slower as the lookback increases from 5 sec, 60 sec, 15 min, 60 min, and 24 hours.
> We also noticed that the query was very fast for the first five minutes of trading, apparently until the first sstable was flushed to disk. After that we go into query times of 1-2 seconds or so.
> Query time[lookback=5]:[1711ms]
> Query time[lookback=60]:[1592ms]
> Query time[lookback=900]:[1520ms]
> Query time[lookback=3600]:[1294ms]
> Query time[lookback=86400]:[1391ms]
> We would really appreciate input or help on this.
> Cassandra version: 1.1.1
> Hector version: 1.0-1
> ---
> public void testCassandraIssue() {
> try {
> int[] seconds = new int[]{ 5, 60, 60 * 15, 60 * 60, 60 * 60 * 24};
> for(int sec : seconds) {
> DateTime start = new DateTime();
> SuperSliceQuery<String, String, String, String> superSliceQuery = HFactory.createSuperSliceQuery(keyspaceOperator, StringSerializer.get(), StringSerializer.get(), StringSerializer.get(), StringSerializer.get());
> superSliceQuery.setKey("101390" + "." + testFormatter.print(start));
> superSliceQuery.setColumnFamily("Quotes");
> superSliceQuery.setRange(superKeyFormatter.print(start),
> superKeyFormatter.print(start.minusSeconds(sec)),
> true,
> 1);
> long theStart = System.currentTimeMillis();
> QueryResult<SuperSlice<String, String, String>> result = superSliceQuery.execute();
> long end = System.currentTimeMillis();
> System.out.println("Query time[lookback=" + sec + "]:[" + (end - theStart) + "ms]");
> }
> } catch(Exception e) {
> e.printStackTrace();
> fail(e.getMessage());
> }
> }
> ---
> create column family Quotes
> with column_type = Super
> and comparator = BytesType
> and subcomparator = BytesType
> and keys_cached = 7000
> and rows_cached = 0
> and row_cache_save_period = 0
> and key_cache_save_period = 3600
> and memtable_throughput = 255
> and memtable_operations = 0.29
> AND compression_options={sstable_compression:SnappyCompressor, chunk_length_kb:64};
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-4340) Cassandra upgrade to 1.1.1
resulted in slow query issue
Posted by "Ivan Ganza (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ivan Ganza updated CASSANDRA-4340:
----------------------------------
Attachment: CassandraIssue.java
> Cassandra upgrade to 1.1.1 resulted in slow query issue
> -------------------------------------------------------
>
> Key: CASSANDRA-4340
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4340
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 1.1.1
> Environment: Ubuntu Linux, Java 7, Hector 1.0-1
> Reporter: Ivan Ganza
> Attachments: CassandraIssue.java
>
>
> We have recently introduced Cassandra at the Globe and Mail here in Toronto, Canada. We are processing and storing the North American stock-market feed. We have found it to work very quickly and things have been looking very good.
> Recently we upgraded to version 1.1.1 and then we have noticed some issues occurring.
> I will try to describe it for you here. Basically one operation that we very often perform and is very critical is the ability to 'get the latest quote'. This would return to you the latest Quote adjusted against exchange delay rules. With Cassandra version 1.0.3 we could get a Quote in around 2ms. After update we are looking at time of at least 2-3 seconds.
> The way we query the quote is using a REVERSED SuperSliceQuery with start=now, end=00:00:00.000 (beginning of day) LIMITED to 1.
> Our investigation leads us to suspect that, since upgrade, Cassandra seems to be reading the sstable from disk even when we request a small range of day only 5 seconds back. If you look at the output below you can see that the query does NOT get slower as the lookback increases from 5 sec, 60 sec, 15 min, 60 min, and 24 hours.
> We also noticed that the query was very fast for the first five minutes of trading, apparently until the first sstable was flushed to disk. After that we go into query times of 1-2 seconds or so.
> Query time[lookback=5]:[1711ms]
> Query time[lookback=60]:[1592ms]
> Query time[lookback=900]:[1520ms]
> Query time[lookback=3600]:[1294ms]
> Query time[lookback=86400]:[1391ms]
> We would really appreciate input or help on this.
> Cassandra version: 1.1.1
> Hector version: 1.0-1
> ---
> public void testCassandraIssue() {
> try {
> int[] seconds = new int[]{ 5, 60, 60 * 15, 60 * 60, 60 * 60 * 24};
> for(int sec : seconds) {
> DateTime start = new DateTime();
> SuperSliceQuery<String, String, String, String> superSliceQuery = HFactory.createSuperSliceQuery(keyspaceOperator, StringSerializer.get(), StringSerializer.get(), StringSerializer.get(), StringSerializer.get());
> superSliceQuery.setKey("101390" + "." + testFormatter.print(start));
> superSliceQuery.setColumnFamily("Quotes");
> superSliceQuery.setRange(superKeyFormatter.print(start),
> superKeyFormatter.print(start.minusSeconds(sec)),
> true,
> 1);
> long theStart = System.currentTimeMillis();
> QueryResult<SuperSlice<String, String, String>> result = superSliceQuery.execute();
> long end = System.currentTimeMillis();
> System.out.println("Query time[lookback=" + sec + "]:[" + (end - theStart) + "ms]");
> }
> } catch(Exception e) {
> e.printStackTrace();
> fail(e.getMessage());
> }
> }
> ---
> create column family Quotes
> with column_type = Super
> and comparator = BytesType
> and subcomparator = BytesType
> and keys_cached = 7000
> and rows_cached = 0
> and row_cache_save_period = 0
> and key_cache_save_period = 3600
> and memtable_throughput = 255
> and memtable_operations = 0.29
> AND compression_options={sstable_compression:SnappyCompressor, chunk_length_kb:64};
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira