You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Michael Shuler (JIRA)" <ji...@apache.org> on 2016/09/23 21:11:21 UTC
[jira] [Updated] (CASSANDRA-12435) Cassandra timeout when using
secondary index in cluster mode.
[ https://issues.apache.org/jira/browse/CASSANDRA-12435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Michael Shuler updated CASSANDRA-12435:
---------------------------------------
Fix Version/s: (was: 3.5)
3.x
> Cassandra timeout when using secondary index in cluster mode.
> -------------------------------------------------------------
>
> Key: CASSANDRA-12435
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12435
> Project: Cassandra
> Issue Type: Bug
> Environment: C* 3.5 | java-driver|AWS g2.2xlarge|ubuntu 14.04
> Reporter: Shijian Liu
> Fix For: 3.x
>
>
> I have a 3-node cluster with a keyspace with RF=3, then I created a table like below and inserted some data(about 10MB).
> CREATE TABLE vehicle_fuzzy_plate (
> day int,
> hour int,
> repo int,
> sensor_id int,
> ts timestamp,
> id timeuuid,
> plate_text_pattern_70 text,
> plate_text_pattern_10 text,
> plate_text_pattern_11 text,
> plate_text_pattern_12 text,
> plate_text_pattern_13 text,
> plate_text_pattern_14 text,
> plate_text_pattern_15 text,
> plate_text_pattern_16 text,
> plate_text_pattern_20 text,
> plate_text_pattern_21 text,
> plate_text_pattern_22 text,
> plate_text_pattern_23 text,
> plate_text_pattern_24 text,
> plate_text_pattern_25 text,
> plate_text_pattern_30 text,
> plate_text_pattern_31 text,
> plate_text_pattern_32 text,
> plate_text_pattern_33 text,
> plate_text_pattern_34 text,
> plate_text_pattern_40 text,
> plate_text_pattern_41 text,
> plate_text_pattern_42 text,
> plate_text_pattern_43 text,
> plate_text_pattern_50 text,
> plate_text_pattern_51 text,
> plate_text_pattern_52 text,
> plate_text_pattern_60 text,
> plate_text_pattern_61 text,
> PRIMARY KEY (day, repo, sensor_id, ts, id)
> ) WITH CLUSTERING ORDER BY (repo ASC, sensor_id ASC, ts ASC, id ASC);
> CREATE INDEX plate_text_pattern_10_idx ON vehicle_fuzzy_plate (plate_text_pattern_10);
> CREATE INDEX plate_text_pattern_11_idx ON vehicle_fuzzy_plate (plate_text_pattern_11);
> CREATE INDEX plate_text_pattern_12_idx ON vehicle_fuzzy_plate (plate_text_pattern_12);
> CREATE INDEX plate_text_pattern_13_idx ON vehicle_fuzzy_plate (plate_text_pattern_13);
> CREATE INDEX plate_text_pattern_14_idx ON vehicle_fuzzy_plate (plate_text_pattern_14);
> CREATE INDEX plate_text_pattern_15_idx ON vehicle_fuzzy_plate (plate_text_pattern_15);
> CREATE INDEX plate_text_pattern_16_idx ON vehicle_fuzzy_plate (plate_text_pattern_16);
> CREATE INDEX plate_text_pattern_20_idx ON vehicle_fuzzy_plate (plate_text_pattern_20);
> CREATE INDEX plate_text_pattern_21_idx ON vehicle_fuzzy_plate (plate_text_pattern_21);
> CREATE INDEX plate_text_pattern_22_idx ON vehicle_fuzzy_plate (plate_text_pattern_22);
> CREATE INDEX plate_text_pattern_23_idx ON vehicle_fuzzy_plate (plate_text_pattern_23);
> CREATE INDEX plate_text_pattern_24_idx ON vehicle_fuzzy_plate (plate_text_pattern_24);
> CREATE INDEX plate_text_pattern_25_idx ON vehicle_fuzzy_plate (plate_text_pattern_25);
> CREATE INDEX plate_text_pattern_30_idx ON vehicle_fuzzy_plate (plate_text_pattern_30);
> CREATE INDEX plate_text_pattern_31_idx ON vehicle_fuzzy_plate (plate_text_pattern_31);
> CREATE INDEX plate_text_pattern_32_idx ON vehicle_fuzzy_plate (plate_text_pattern_32);
> CREATE INDEX plate_text_pattern_33_idx ON vehicle_fuzzy_plate (plate_text_pattern_33);
> CREATE INDEX plate_text_pattern_34_idx ON vehicle_fuzzy_plate (plate_text_pattern_34);
> CREATE INDEX plate_text_pattern_40_idx ON vehicle_fuzzy_plate (plate_text_pattern_40);
> CREATE INDEX plate_text_pattern_41_idx ON vehicle_fuzzy_plate (plate_text_pattern_41);
> CREATE INDEX plate_text_pattern_42_idx ON vehicle_fuzzy_plate (plate_text_pattern_42);
> CREATE INDEX plate_text_pattern_43_idx ON vehicle_fuzzy_plate (plate_text_pattern_43);
> CREATE INDEX plate_text_pattern_50_idx ON vehicle_fuzzy_plate (plate_text_pattern_50);
> CREATE INDEX plate_text_pattern_51_idx ON vehicle_fuzzy_plate (plate_text_pattern_51);
> CREATE INDEX plate_text_pattern_52_idx ON vehicle_fuzzy_plate (plate_text_pattern_52);
> CREATE INDEX plate_text_pattern_60_idx ON vehicle_fuzzy_plate (plate_text_pattern_60);
> CREATE INDEX plate_text_pattern_61_idx ON vehicle_fuzzy_plate (plate_text_pattern_61);
> CREATE INDEX plate_text_pattern_70_idx ON vehicle_fuzzy_plate (plate_text_pattern_70);
> After that, I send some queries through Java driver with Consistency=Quorum one by one, for example,
> SELECT id,sensor_id,ts FROM vehicle_fuzzy_plate WHERE day IN (16993,16994,16995,16996,16997,16998,16999,17000,17001,17002,17003,17004,17005,17006,17007,17008,17009,17010,17011,17012,17013,17014,17015,17016,17017,17018,17019,17020,17021,17022,17023) AND repo IN (234) AND sensor_id IN (123,16) AND ts>=1468226847000 AND ts<=1470818847000 AND plate_text_pattern_70 IN ('70贵JBP378') ALLOW FILTERING;
> The weird part is that some queries will response timeout (about 30%). I checked the debug.log and found these exception,
> DEBUG [SharedPool-Worker-1] 2016-08-11 14:09:00,885 ReadCallback.java:126 - Timed out; received 1 of 2 responses (including data)
> ERROR [MessagingService-Incoming-/192.168.2.122] 2016-08-11 14:09:05,954 CassandraDaemon.java:217 - Exception in thread Thread[MessagingService-Incoming-/192.168.2.122,5,main]
> java.lang.ArrayIndexOutOfBoundsException: 74
> at org.apache.cassandra.db.filter.AbstractClusteringIndexFilter$FilterSerializer.deserialize(AbstractClusteringIndexFilter.java:74) ~[apache-cassandra-3.5.jar:3.5]
> at org.apache.cassandra.db.SinglePartitionReadCommand$Deserializer.deserialize(SinglePartitionReadCommand.java:1045) ~[apache-cassandra-3.5.jar:3.5]
> at org.apache.cassandra.db.ReadCommand$Serializer.deserialize(ReadCommand.java:667) ~[apache-cassandra-3.5.jar:3.5]
> at org.apache.cassandra.db.ReadCommand$Serializer.deserialize(ReadCommand.java:595) ~[apache-cassandra-3.5.jar:3.5]
> at org.apache.cassandra.net.MessageIn.read(MessageIn.java:114) ~[apache-cassandra-3.5.jar:3.5]
> at org.apache.cassandra.net.IncomingTcpConnection.receiveMessage(IncomingTcpConnection.java:190) ~[apache-cassandra-3.5.jar:3.5]
> at org.apache.cassandra.net.IncomingTcpConnection.receiveMessages(IncomingTcpConnection.java:178) ~[apache-cassandra-3.5.jar:3.5]
> at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:92) ~[apache-cassandra-3.5.jar:3.5]
> I try to clean table and send the query, but timeout comes out occasionally too.
> However, queries will work fine if I don't use the secondary index, just like this,
> SELECT id,sensor_id,ts FROM vehicle_fuzzy_plate WHERE day IN (16993,16994,16995,16996,16997,16998,16999,17000,17001,17002,17003,17004,17005,17006,17007,17008,17009,17010,17011,17012,17013,17014,17015,17016,17017,17018,17019,17020,17021,17022,17023) AND repo IN (234) AND sensor_id IN (123,16) AND ts>=1468226847000 AND ts<=1470818847000;
> Everything works fine on a single node with RF=1 with more than 10GB data.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)