You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by David Ni <zn...@126.com> on 2018/09/03 03:33:08 UTC
Re:RE: Re:RE: data not beening syncd in different nodes
Hi Kyrylo,
The fact is that whenever I query from node 1,it returns about 60 rows,from node 2,it returns about 140 rows,from node 3,it returns about 30 rows ,from node it returns about 140 rows,from node 5 it returns about 70 rows,from node 6 it returns about 140 rows,I tried so many times,the result from every single node is the same,but different from other nodes.
At 2018-08-31 18:08:01, "Kyrylo Lebediev" <Ky...@epam.com.INVALID> wrote:
TTL 60 seconds – small value (even smaller than compaction window). This means that even if all replicas are consistent, data is deleted really quickly so that results may differ even for 2 consecutive queries. How about this theory?
CL in your driver – depends on which CL is default for your particular driver.
Regards,
Kyrill
From: David Ni <zn...@126.com>
Sent: Friday, August 31, 2018 12:53 PM
To:user@cassandra.apache.org
Subject: Re:RE: data not beening syncd in different nodes
Hi Kyrylo
I have already tried consistency quorum and all,still the same result.
the java code write data to cassandra does not set CL,does this mean the default CL is one?
the tpstats out is like below ,there are some dropped mutations, but it doesn't grow during a very long time
Pool Name Active Pending Completed Blocked All time blocked
MutationStage 0 0 300637997 0 0
ViewMutationStage 0 0 0 0 0
ReadStage 0 0 4357929 0 0
RequestResponseStage 0 0 306954791 0 0
ReadRepairStage 0 0 472027 0 0
CounterMutationStage 0 0 0 0 0
MiscStage 0 0 0 0 0
CompactionExecutor 0 0 17976139 0 0
MemtableReclaimMemory 0 0 53018 0 0
PendingRangeCalculator 0 0 11 0 0
GossipStage 0 0 59889799 0 0
SecondaryIndexManagement 0 0 0 0 0
HintsDispatcher 0 0 7 0 0
MigrationStage 0 0 101 0 0
MemtablePostFlush 0 0 41470 0 0
PerDiskMemtableFlushWriter_0 0 0 52779 0 0
ValidationExecutor 0 0 80 0 0
Sampler 0 0 0 0 0
MemtableFlushWriter 0 0 40301 0 0
InternalResponseStage 0 0 70 0 0
AntiEntropyStage 0 0 352 0 0
CacheCleanupExecutor 0 0 0 0 0
Native-Transport-Requests 0 0 158242159 0 13412
Message type Dropped
READ 0
RANGE_SLICE 0
_TRACE 1
HINT 0
MUTATION 34
COUNTER_MUTATION 0
BATCH_STORE 0
BATCH_REMOVE 0
REQUEST_RESPONSE 0
PAGED_RANGE 0
READ_REPAIR 0
and yes,we are inserting data with TTL=60 seconds
we have 200 vehicles and updating this table every 5 or 10 seconds;
At 2018-08-31 17:10:50, "Kyrylo Lebediev" <Ky...@epam.com.INVALID> wrote:
Looks like you’re querying the table at CL = ONE which is default for cqlsh.
If you run cqlsh on nodeX it doesn’t mean you retrieve data from this node. What this means is that nodeX will be coordinator, whereas actual data will be retrieved from any node, based on token range + dynamic snitch data (which, I assume, you use as it’s turned on by default).
Which CL you use when you write data?
Try querying using CL = QUORUM or ALL. What’s your result in this case?
If you run ‘nodetool tpstats’ across all the nodes, are there dropped mutations?
As you use TimeWindowCompactionStrategy, do you insert data with TTL?
These buckets seem to be too small for me: 'compaction_window_size': '2', 'compaction_window_unit': 'MINUTES'.
Do you have such a huge amount of writes so that such bucket size makes sense?
Regards,
Kyrill
From: David Ni <zn...@126.com>
Sent: Friday, August 31, 2018 11:39 AM
To:user@cassandra.apache.org
Subject: data not beening syncd in different nodes
Hi Experts,
I am using 3.9 cassandra in production environment,we have 6 nodes,the RF of keyspace is 3, I have a table which below definition:
CREATE TABLE nev_prod_tsp.heartbeat (
vin text PRIMARY KEY,
create_time timestamp,
pkg_sn text,
prot_ver text,
trace_time timestamp,
tsp_sn text
) WITH bloom_filter_fp_chance = 0.01
AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
AND comment = 'heartbeat'
AND compaction = {'class': 'org.apache.cassandra.db.compaction.TimeWindowCompactionStrategy', 'compaction_window_size': '2', 'compaction_window_unit': 'MINUTES', 'max_threshold': '32', 'min_threshold': '4'}
AND compression = {'chunk_length_in_kb': '64', 'class': 'org.apache.cassandra.io.compress.LZ4Compressor'}
AND crc_check_chance = 1.0
AND dclocal_read_repair_chance = 0.1
AND default_time_to_live = 60
AND gc_grace_seconds = 120
AND max_index_interval = 2048
AND memtable_flush_period_in_ms = 0
AND min_index_interval = 128
AND read_repair_chance = 0.2
AND speculative_retry = '99PERCENTILE';
I am updating this table's data every 5 or 10 seconds,
the data sample is as below:
cassandra@cqlsh> select * from nev_prod_tsp.heartbeat;
vin | create_time | pkg_sn | prot_ver | trace_time | tsp_sn
-------------------+---------------------------------+--------+----------+---------------------------------+--------------------------------------
LLXA2A408JA002371 | 2018-08-31 08:16:07.348000+0000 | 128209 | 2 | 2018-08-31 08:16:09.000000+0000 | ad4c2b13-d894-4804-9cf2-e07a3c5851bd
LLXA2A400JA002333 | 2018-08-31 08:16:06.386000+0000 | 142944 | 2 | 2018-08-31 08:16:04.000000+0000 | 6ba9655c-8542-4251-ba9b-93ae7420ecc7
LLXA2A402JA002351 | 2018-08-31 08:16:09.451000+0000 | 196040 | 2 | 2018-08-31 08:16:07.000000+0000 | 9b6a5d7d-4917-46bc-a247-8a0ff8c1e699
........
but we I select all data from this table,the results from different nodes are not the same:
node1: about 60 rows
node2:about 140 rows
node3:about 30 rows
node4:about 140 rows
node5:about 70 rows
node6:about 140 rows
I have tried compact and repair on all nodes,but didn't help.
Could any one give me some help?Thanks very much.