You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Jason Turner <Ja...@viewpoint.com> on 2015/10/22 19:23:30 UTC
Data Streamed successfully but not queryable
Apologies for the long post but it's a complicated story...
I am running Cassandra in 3 datacenters 1,2 & 3. DCs 1 and 2 are working fine but I am having trouble selecting any data at all in DC 3.
I've stripped DC3 down from 4 nodes to a single brand new node to make debugging logs/traces etc easier. It's an AWS t2.medium, Cassandra 2.1.10 running on Amazon Linux AMI 2015.09. Java 1.8.0.
I am using the CQLSSTableWriter/BulkLoader utility to stream up a very small amount of test data to an empty table. The Bulkloader output reports no errors and the System log looks fine - session completed, all bytes received. I can see the data.db files in the appropriate data directory flushed to disk, however I can't select any of the data via CQL. No exceptions thrown, no rows returned. It's like it's simply ignoring the data which I know exists.
My first thought was the timestamps on the data might somehow be set in the future but SSTable2json shows the relevant data exactly as I'd expect it with timestamps that match the upload times and should be readable. It's a fresh install with NTP running so I am ruling out any timedrift across the nodes.
It's worth noting that the CQLSSTableWriter/BulkLoader utility works fine streaming in data centers 1 & 2. Data streamed from DCs 1 & 2 into DC 3 is immediately selectable. Data streamed from DC3 into DCs 1&2 is also immediately selectable. Only data streamed from DC3 into the DC3 cluster is ignored by select statements.
As an example I've streamed the same test row into DC3 from DC1 and DC3. There are now be 2 rows/partitions in the table in DC3, here they are in json via SSTable2json -
Doesn't Show (Streamed from same datacenter):
{"key": "e3589ff7-f753-4ff3-9809-365322306825",
"cells": [["bdff0420-78d2-11e5-a0cc-0607dbae485d:","",1445528225152000],
["bdff0420-78d2-11e5-a0cc-0607dbae485d:c001","Value1",1445528225152000],
["bdff0420-78d2-11e5-a0cc-0607dbae485d:c002","Value2",1445528225152000],
["bdff0420-78d2-11e5-a0cc-0607dbae485d:c003","Value3",1445528225152000],
["bdff0420-78d2-11e5-a0cc-0607dbae485d:c004","",1445528225152000],
["bdff0420-78d2-11e5-a0cc-0607dbae485d:c005","",1445528225152000],
["bdff0420-78d2-11e5-a0cc-0607dbae485d:c006","",1445528225152000],
["bdff0420-78d2-11e5-a0cc-0607dbae485d:c007","",1445528225152000],
["bdff0420-78d2-11e5-a0cc-0607dbae485d:c008","Value4",1445528225152000],
["bdff0420-78d2-11e5-a0cc-0607dbae485d:c009","Value5",1445528225152000],
["bdff0420-78d2-11e5-a0cc-0607dbae485d:c010","",1445528225152000],
["bdff0420-78d2-11e5-a0cc-0607dbae485d:c011","",1445528225152000],
["bdff0420-78d2-11e5-a0cc-0607dbae485d:c012","Value6",1445528225152000],
["bdff0420-78d2-11e5-a0cc-0607dbae485d:c013","",1445528225152000],
["bdff0420-78d2-11e5-a0cc-0607dbae485d:c014","",1445528225152000],
["bdff0420-78d2-11e5-a0cc-0607dbae485d:c015","",1445528225152000],
["bdff0420-78d2-11e5-a0cc-0607dbae485d:c016","",1445528225152000],
["bdff0420-78d2-11e5-a0cc-0607dbae485d:c017","",1445528225152000],
["bdff0420-78d2-11e5-a0cc-0607dbae485d:c018","",1445528225152000],
["bdff0420-78d2-11e5-a0cc-0607dbae485d:c019","",1445528225152000],
["bdff0420-78d2-11e5-a0cc-0607dbae485d:c020","Value7",1445528225152000]]}
Shows Correctly (Streamed over VPN from different datacenter):
{"key": "1448b4de-7ebe-4d46-a2a4-365443970109",
"cells": [["ccaba8f0-78d4-11e5-ae6a-12a3a8f7b9be:","",1445529108746000],
["ccaba8f0-78d4-11e5-ae6a-12a3a8f7b9be:c001","Value1",1445529108746000],
["ccaba8f0-78d4-11e5-ae6a-12a3a8f7b9be:c002","Value2",1445529108746000],
["ccaba8f0-78d4-11e5-ae6a-12a3a8f7b9be:c003","Value3",1445529108746000],
["ccaba8f0-78d4-11e5-ae6a-12a3a8f7b9be:c004","",1445529108746000],
["ccaba8f0-78d4-11e5-ae6a-12a3a8f7b9be:c005","",1445529108746000],
["ccaba8f0-78d4-11e5-ae6a-12a3a8f7b9be:c006","",1445529108746000],
["ccaba8f0-78d4-11e5-ae6a-12a3a8f7b9be:c007","",1445529108746000],
["ccaba8f0-78d4-11e5-ae6a-12a3a8f7b9be:c008","Value4",1445529108746000],
["ccaba8f0-78d4-11e5-ae6a-12a3a8f7b9be:c009","Value5",1445529108746000],
["ccaba8f0-78d4-11e5-ae6a-12a3a8f7b9be:c010","",1445529108746000],
["ccaba8f0-78d4-11e5-ae6a-12a3a8f7b9be:c011","",1445529108746000],
["ccaba8f0-78d4-11e5-ae6a-12a3a8f7b9be:c012","Value6",1445529108746000],
["ccaba8f0-78d4-11e5-ae6a-12a3a8f7b9be:c013","",1445529108746000],
["ccaba8f0-78d4-11e5-ae6a-12a3a8f7b9be:c014","",1445529108746000],
["ccaba8f0-78d4-11e5-ae6a-12a3a8f7b9be:c015","",1445529108746000],
["ccaba8f0-78d4-11e5-ae6a-12a3a8f7b9be:c016","",1445529108746000],
["ccaba8f0-78d4-11e5-ae6a-12a3a8f7b9be:c017","",1445529108746000],
["ccaba8f0-78d4-11e5-ae6a-12a3a8f7b9be:c018","",1445529108746000],
["ccaba8f0-78d4-11e5-ae6a-12a3a8f7b9be:c019","",1445529108746000],
["ccaba8f0-78d4-11e5-ae6a-12a3a8f7b9be:c020","Value7",1445529108746000]]}
Here is the trace on the select statement (no where clause - should return both rows), it looks to me like it's finding both rows. However only one row (from DC1) is being returned by the query.
Tracing session: eaafc370-78d6-11e5-9990-6f2f8a300360
activity | timestamp | source | source_elapsed
------------------------------------------------------------------------------------------------------------------------------+----------------------------+--------------+----------------
Execute CQL3 query | 2015-10-22 16:06:56.039000 | 10.129.1.112 | 0
Parsing select * from sheet1; [SharedPool-Worker-1] | 2015-10-22 16:06:56.044000 | 10.129.1.112 | 50
Preparing statement [SharedPool-Worker-1] | 2015-10-22 16:06:56.044000 | 10.129.1.112 | 93
Computing ranges to query [SharedPool-Worker-1] | 2015-10-22 16:06:56.045000 | 10.129.1.112 | 320
Submitting range requests on 257 ranges with a concurrency of 234 (0.42772275 rows per range expected) [SharedPool-Worker-1] | 2015-10-22 16:06:56.045000 | 10.129.1.112 | 726
Executing seq scan across 3 sstables for [min(-9223372036854775808), min(-9223372036854775808)] [SharedPool-Worker-2] | 2015-10-22 16:06:56.045000 | 10.129.1.112 | 1423
Read 0 live and 0 tombstone cells [SharedPool-Worker-2] | 2015-10-22 16:06:56.045000 | 10.129.1.112 | 1779
Read 1 live and 0 tombstone cells [SharedPool-Worker-2] | 2015-10-22 16:06:56.045000 | 10.129.1.112 | 1850
Scanned 2 rows and matched 2 [SharedPool-Worker-2] | 2015-10-22 16:06:56.045000 | 10.129.1.112 | 1881
Submitted 1 concurrent range requests covering 257 ranges [SharedPool-Worker-1] | 2015-10-22 16:06:56.046000 | 10.129.1.112 | 5390
Request complete | 2015-10-22 16:06:56.045807 | 10.129.1.112 | 6807
Can anyone suggest why my data isn't being returned or where to continue digging?
Thank you!
Jason