You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Michael Kjellman (JIRA)" <ji...@apache.org> on 2013/01/04 01:18:12 UTC
[jira] [Commented] (CASSANDRA-4446) nodetool drain sometimes
doesn't mark commitlog fully flushed
[ https://issues.apache.org/jira/browse/CASSANDRA-4446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13543446#comment-13543446 ]
Michael Kjellman commented on CASSANDRA-4446:
---------------------------------------------
did a nodetool drain before 1.1.7 -> 1.2.0. upon starting 1.2.0 every node in my cluster still replayed the commit logs and created mutations.
{code}
INFO 13:17:06,529 DRAINING: starting drain process
INFO 13:17:06,529 Stop listening to thrift clients
INFO 13:17:06,532 Announcing shutdown
INFO 13:17:07,536 Waiting for messaging service to quiesce
INFO 13:17:07,537 MessagingService shutting down server thread.
... normal startup stuff..
INFO 13:20:20,182 Replaying /ssd/commitlog/CommitLog-1355265349912.log
INFO 13:20:24,166 Finished reading /ssd/commitlog/CommitLog-1355265349912.log
INFO 13:20:24,166 Replaying /ssd/commitlog/CommitLog-1355265349914.log
INFO 13:20:26,700 Finished reading /ssd/commitlog/CommitLog-1355265349914.log
INFO 13:20:26,701 Replaying /ssd/commitlog/CommitLog-1355265349915.log
INFO 13:20:28,118 Finished reading /ssd/commitlog/CommitLog-1355265349915.log
... more replay lines ...
INFO 13:22:00,061 Log replay complete, 8052 replayed mutations
INFO 13:22:00,358 Possible old-format hints found. Truncating
INFO 13:22:00,370 Enqueuing flush of Memtable-local@1908923620(402/402 serialized/live bytes, 13 ops)
INFO 13:22:00,372 Writing Memtable-local@1908923620(402/402 serialized/live bytes, 13 ops)
INFO 13:22:00,494 Cassandra version: 1.2.0
INFO 13:22:00,495 Thrift API version: 19.35.0
INFO 13:22:00,495 CQL supported versions: 2.0.0,3.0.0 (default: 3.0.0)
INFO 13:22:00,534 Loading persisted ring state
INFO 13:22:00,537 Starting up server gossip
WARN 13:22:00,557 No host ID found, created dd3a40e2-fef1-4574-87b8-e2929fd80235 (Note: This should happen exactly once per node).
{code}
> nodetool drain sometimes doesn't mark commitlog fully flushed
> -------------------------------------------------------------
>
> Key: CASSANDRA-4446
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4446
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 1.0.10
> Environment: ubuntu 10.04 64bit
> Linux HOSTNAME 2.6.32-345-ec2 #48-Ubuntu SMP Wed May 2 19:29:55 UTC 2012 x86_64 GNU/Linux
> sun JVM
> cassandra 1.0.10 installed from apache deb
> Reporter: Robert Coli
> Attachments: cassandra.1.0.10.replaying.log.after.exception.during.drain.txt
>
>
> I recently wiped a customer's QA cluster. I drained each node and verified that they were drained. When I restarted the nodes, I saw the commitlog replay create a memtable and then flush it. I have attached a sanitized log snippet from a representative node at the time.
> It appears to show the following :
> 1) Drain begins
> 2) Drain triggers flush
> 3) Flush triggers compaction
> 4) StorageService logs DRAINED message
> 5) compaction thread excepts
> 6) on restart, same CF creates a memtable
> 7) and then flushes it [1]
> The columnfamily involved in the replay in 7) is the CF for which the compaction thread excepted in 5). This seems to suggest a timing issue whereby the exception in 5) prevents the flush in 3) from marking all the segments flushed, causing them to replay after restart.
> In case it might be relevant, I did an online change of compaction strategy from Leveled to SizeTiered during the uptime period preceding this drain.
> [1] Isn't commitlog replay not supposed to automatically trigger a flush in modern cassandra?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira