You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Vasileios Vlachos <va...@gmail.com> on 2015/01/18 00:32:28 UTC
Should one expect to see hints being stored/delivered occasionally?
Hello,
I thought hints are being stored on /node_A/ every time /node_B/ is
unavailable for whatever the reason. I also thought that these hints are
being delivered from /node_A/ to /node_B/ when /node_B/ is back and this
is true for a /period = max_hint_window_in_ms/. After that hints are
dropped and therefore never delivered to /node_B/.
Obviously I am wrong, because occasionally we get alerted from our
monitoring system that hints are being stored and delivered, which as
far as I know indicates a problem. Now, when that happens I cannot
correlate it with any network issues (all nodes are on the same LAN
anyway) or other problems. The output from /system.log/ looks like this:
INFO [CompactionExecutor:109085] 2015-01-17 15:35:13,536
CompactionTask.java (line 262) Compacted 2 sstables to
[/var/lib/cassandra/data/DataMining/quotebyquotereference/DataMining-quoteby
quotereference-ic-89765,]. 222,905,570 bytes to 222,881,859 (~99% of
original) in 91,850ms = 2.314172MB/s. 161,259 total rows, 161,253
unique. Row merge counts were {1:161247, 2:6, }
INFO [CompactionExecutor:109090] 2015-01-17 15:35:13,537
CompactionTask.java (line 105) Compacting
[SSTableReader(path='/var/lib/cassandra/data/DataMining/quotebyquotereference/DataMining-
quotebyquotereference-ic-89750-Data.db'),
SSTableReader(path='/var/lib/cassandra/data/DataMining/quotebyquotereference/DataMining-quotebyquotereference-ic-89765-Data.db')]
INFO [HintedHandoff:2] 2015-01-17 15:35:38,564
HintedHandOffManager.java (line 294) Started hinted handoff for host:
2ae2c679-8769-44da-a713-3bc21c670620 with IP: /10.3.5.3
INFO [HintedHandoff:1] 2015-01-17 15:35:38,564
HintedHandOffManager.java (line 294) Started hinted handoff for host:
0bb63124-6333-43fa-b1c8-3a8f6627b85a with IP: /10.3.5.2
INFO [HintedHandoff:1] 2015-01-17 15:35:38,967
HintedHandOffManager.java (line 326) Finished hinted handoff of 17 rows
to endpoint /10.3.5.2
INFO [HintedHandoff:1] 2015-01-17 15:35:38,968 ColumnFamilyStore.java
(line 633) Enqueuing flush of Memtable-hints@1779218028(614406/2848765
serialized/live bytes, 220 ops)
INFO [FlushWriter:9360] 2015-01-17 15:35:38,969 Memtable.java (line
398) Writing Memtable-hints@1779218028(614406/2848765 serialized/live
bytes, 220 ops)
INFO [FlushWriter:9360] 2015-01-17 15:35:39,192 Memtable.java (line
436) Completed flushing
/var/lib/cassandra/data/system/hints/system-hints-ic-89-Data.db (176861
bytes) for commitlog position ReplayPosition(segmentId=1418136927153,
position=20201767)
INFO [CompactionExecutor:109094] 2015-01-17 15:35:39,194
CompactionTask.java (line 105) Compacting
[SSTableReader(path='/var/lib/cassandra/data/system/hints/system-hints-ic-89-Data.db')]
INFO [CompactionExecutor:109094] 2015-01-17 15:35:39,485
CompactionTask.java (line 262) Compacted 1 sstables to
[/var/lib/cassandra/data/system/hints/system-hints-ic-90,]. 176,861
bytes to 177,355 (~100% of original) in 290ms = 0.583238MB/s. 4 total
rows, 3 unique. Row merge counts were {1:4, }
INFO [HintedHandoff:1] 2015-01-17 15:35:39,485
HintedHandOffManager.java (line 294) Started hinted handoff for host:
6b99058f-ba48-42b9-baa1-a878a74338cc with IP: /10.3.5.1
INFO [HintedHandoff:1] 2015-01-17 15:35:40,084
HintedHandOffManager.java (line 326) Finished hinted handoff of 22 rows
to endpoint /10.3.5.1
INFO [HintedHandoff:1] 2015-01-17 15:35:40,085 ColumnFamilyStore.java
(line 633) Enqueuing flush of Memtable-hints@1204004752(2356/10923
serialized/live bytes, 62 ops)
INFO [FlushWriter:9360] 2015-01-17 15:35:40,085 Memtable.java (line
398) Writing Memtable-hints@1204004752(2356/10923 serialized/live bytes,
62 ops)
Is there any other occasion that hints are stored and then being sent in
a cluster, other than network or other temporary or permanent failure?
Could it be that the client responsible for establishing a connection is
causing this? We use the Datastax C# driver for connecting to the
cluster and we run C* 1.2.18 on Ubuntu 12.04.
Many thanks,
Vasilis
Re: Should one expect to see hints being stored/delivered occasionally?
Posted by Vasileios Vlachos <va...@gmail.com>.
Thanks for your reply Rob, I am back to this after a while...
I am not sure if this is different in 1.2.18, but I remember from older
versions that GC pauses would only be logged in the /system.log/ if
their duration /was >= 200ms/. Also, when hints are detected, we cannot
correlate it with GC pauses. We are thinking of tweaking the GC logging
settings in the /cassandra-env/ file, but we are unsure as to which ones
are going to be heavy for the server and which ones are safer to modify.
Would you be able to advice on this?
The hints issue we seem to have, is not catastrophic in the sense that
it is not causing serious/obvious problems to the clients, but makes us
feel rather uncomfortable with the overall cluster health because, as
you said, is a warning sign that something is wrong. It doesn't happen
very often either, but I don't think this makes the situation any
better. Apart from increasing the GC logging, I don't see any other way
of debugging this further.
Thanks for your input,
Vasilis
On 20/01/15 22:53, Robert Coli wrote:
> On Sat, Jan 17, 2015 at 3:32 PM, Vasileios Vlachos
> <vasileiosvlachos@gmail.com <ma...@gmail.com>> wrote:
>
> Is there any other occasion that hints are stored and then being
> sent in a cluster, other than network or other temporary or
> permanent failure? Could it be that the client responsible for
> establishing a connection is causing this? We use the Datastax C#
> driver for connecting to the cluster and we run C* 1.2.18 on
> Ubuntu 12.04.
>
>
> Other than restarting nodes manually (which I consider a "temporary
> failure" for the purposes of this question), no. Seeing hints being
> stored and delivered outside of this context is a warning sign that
> something may be wrong with your cluster.
>
> Probably what is happening is that you have stop the world GCs long
> enough to trigger queueing of hints via timeouts during these GCs.
>
> =Rob
Re: Should one expect to see hints being stored/delivered occasionally?
Posted by Robert Coli <rc...@eventbrite.com>.
On Sat, Jan 17, 2015 at 3:32 PM, Vasileios Vlachos <
vasileiosvlachos@gmail.com> wrote:
> Is there any other occasion that hints are stored and then being sent in a
> cluster, other than network or other temporary or permanent failure? Could
> it be that the client responsible for establishing a connection is causing
> this? We use the Datastax C# driver for connecting to the cluster and we
> run C* 1.2.18 on Ubuntu 12.04.
>
Other than restarting nodes manually (which I consider a "temporary
failure" for the purposes of this question), no. Seeing hints being stored
and delivered outside of this context is a warning sign that something may
be wrong with your cluster.
Probably what is happening is that you have stop the world GCs long enough
to trigger queueing of hints via timeouts during these GCs.
=Rob