You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Fabien Rousseau (JIRA)" <ji...@apache.org> on 2012/10/09 18:52:02 UTC
[jira] [Created] (CASSANDRA-4782) Commitlog not replayed after
restart
Fabien Rousseau created CASSANDRA-4782:
------------------------------------------
Summary: Commitlog not replayed after restart
Key: CASSANDRA-4782
URL: https://issues.apache.org/jira/browse/CASSANDRA-4782
Project: Cassandra
Issue Type: Bug
Affects Versions: 1.1.5, 1.1.4, 1.1.3, 1.1.2, 1.1.1, 1.1.0
Reporter: Fabien Rousseau
It seems that there are two corner cases where commitlog is not replayed after a restart :
- After a reboot of a server + restart of cassandra (1.1.0 to 1.1.4)
- After doing an upgrade from cassandra 1.1.X to cassandra 1.1.5
This is due to the fact that the commitlog segment id should always be an incrementing number (see this condition : https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/db/commitlog/CommitLogReplayer.java#L247 )
But this assertion can be broken :
In the first case, it is generated by System.nanoTime() but it seems that System.nanoTime() is using the boot time as the base/reference (at least on java6 & linux), thus after a reboot, System.nanoTime() can return a lower number than before the reboot (and the javadoc says the reference is a relative point in time...)
In the second case, this was introduced by #4601 (which changes System.nanoTime() by System.currentTimeMillis() thus people starting with 1.1.5 are safe)
This could explain the following tickets : #4741 and #4481
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4782) Commitlog not replayed after
restart
Posted by "Omid Aladini (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13476948#comment-13476948 ]
Omid Aladini commented on CASSANDRA-4782:
-----------------------------------------
CASSANDRA-4446 does not happen to me any more (so far) when restarting 1.1.6 into 1.1.6. I could observe CASSANDRA-4446 on upgrade from 1.1.3 to 1.1.6 though.
> Commitlog not replayed after restart
> ------------------------------------
>
> Key: CASSANDRA-4782
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4782
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 1.1.0
> Reporter: Fabien Rousseau
> Assignee: Jonathan Ellis
> Priority: Critical
> Fix For: 1.1.6
>
> Attachments: 4782.txt
>
>
> It seems that there are two corner cases where commitlog is not replayed after a restart :
> - After a reboot of a server + restart of cassandra (1.1.0 to 1.1.4)
> - After doing an upgrade from cassandra 1.1.X to cassandra 1.1.5
> This is due to the fact that the commitlog segment id should always be an incrementing number (see this condition : https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/db/commitlog/CommitLogReplayer.java#L247 )
> But this assertion can be broken :
> In the first case, it is generated by System.nanoTime() but it seems that System.nanoTime() is using the boot time as the base/reference (at least on java6 & linux), thus after a reboot, System.nanoTime() can return a lower number than before the reboot (and the javadoc says the reference is a relative point in time...)
> In the second case, this was introduced by #4601 (which changes System.nanoTime() by System.currentTimeMillis() thus people starting with 1.1.5 are safe)
> This could explain the following tickets : #4741 and #4481
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4782) Commitlog not replayed after
restart
Posted by "Fabien Rousseau (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13473100#comment-13473100 ]
Fabien Rousseau commented on CASSANDRA-4782:
--------------------------------------------
To solve the first case, it's probably better, when possible, to upgrade to a newer version, then rewrite all SSTables (or at least all SSTables metadata)
For the second case, just rewrite all SSTables (or at least all SSTables metadata)
Each SSTable metadata contains the ReplayPosition (and the max is taken to know which commitlog to replay), thus if System.nanoTime() returned a number which is a timestamp in the future, previous commitlogs will be ignored),
thus a drain should prevent from losing data (because there is no commitlog to replay).
And because SSTables are immutables, then rewriting them completely seems a better option (rather than modifying previously written metadata)
Maybe the simplest to do is to have a new option to nodetool (or a new option to nodetool upgradesstables) which only changes the metadata using the following rule :
if the replayPosition.segment of the SSTable is in the future, then reset it to NONE otherwise, let it to its current value. (NONE is valid value if node was drained and restarted)
By using this rule :
- if a sstable was generated previously using a higher System.nanoTime() then it is reset
- if a sstable was generated previously using a lower System.nanoTime() OR System.currentTimeMillis() then it is left as is
- if a sstable is generated between the start & this rewriting process, then it is left as is
Thus the upgrade should be :
- drain node
- upgrade
- start
- run the process described above
What do you think ?
> Commitlog not replayed after restart
> ------------------------------------
>
> Key: CASSANDRA-4782
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4782
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 1.1.0, 1.1.1, 1.1.2, 1.1.3, 1.1.4, 1.1.5
> Reporter: Fabien Rousseau
>
> It seems that there are two corner cases where commitlog is not replayed after a restart :
> - After a reboot of a server + restart of cassandra (1.1.0 to 1.1.4)
> - After doing an upgrade from cassandra 1.1.X to cassandra 1.1.5
> This is due to the fact that the commitlog segment id should always be an incrementing number (see this condition : https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/db/commitlog/CommitLogReplayer.java#L247 )
> But this assertion can be broken :
> In the first case, it is generated by System.nanoTime() but it seems that System.nanoTime() is using the boot time as the base/reference (at least on java6 & linux), thus after a reboot, System.nanoTime() can return a lower number than before the reboot (and the javadoc says the reference is a relative point in time...)
> In the second case, this was introduced by #4601 (which changes System.nanoTime() by System.currentTimeMillis() thus people starting with 1.1.5 are safe)
> This could explain the following tickets : #4741 and #4481
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-4782) Commitlog not replayed after
restart
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-4782:
--------------------------------------
Attachment: 4782.txt
(patch revised to update CURRENT_VERSION as well)
> Commitlog not replayed after restart
> ------------------------------------
>
> Key: CASSANDRA-4782
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4782
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 1.1.0
> Reporter: Fabien Rousseau
> Assignee: Jonathan Ellis
> Priority: Critical
> Fix For: 1.1.6
>
> Attachments: 4782.txt
>
>
> It seems that there are two corner cases where commitlog is not replayed after a restart :
> - After a reboot of a server + restart of cassandra (1.1.0 to 1.1.4)
> - After doing an upgrade from cassandra 1.1.X to cassandra 1.1.5
> This is due to the fact that the commitlog segment id should always be an incrementing number (see this condition : https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/db/commitlog/CommitLogReplayer.java#L247 )
> But this assertion can be broken :
> In the first case, it is generated by System.nanoTime() but it seems that System.nanoTime() is using the boot time as the base/reference (at least on java6 & linux), thus after a reboot, System.nanoTime() can return a lower number than before the reboot (and the javadoc says the reference is a relative point in time...)
> In the second case, this was introduced by #4601 (which changes System.nanoTime() by System.currentTimeMillis() thus people starting with 1.1.5 are safe)
> This could explain the following tickets : #4741 and #4481
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4782) Commitlog not replayed after
restart
Posted by "Robert Coli (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13476266#comment-13476266 ]
Robert Coli commented on CASSANDRA-4782:
----------------------------------------
If drain is required between versions to avoid this issue then CASSANDRA-4446, where drain sometimes doesn't actually drain, seems to have become more significant.
> Commitlog not replayed after restart
> ------------------------------------
>
> Key: CASSANDRA-4782
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4782
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 1.1.0
> Reporter: Fabien Rousseau
> Assignee: Jonathan Ellis
> Priority: Critical
> Fix For: 1.1.6
>
> Attachments: 4782.txt
>
>
> It seems that there are two corner cases where commitlog is not replayed after a restart :
> - After a reboot of a server + restart of cassandra (1.1.0 to 1.1.4)
> - After doing an upgrade from cassandra 1.1.X to cassandra 1.1.5
> This is due to the fact that the commitlog segment id should always be an incrementing number (see this condition : https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/db/commitlog/CommitLogReplayer.java#L247 )
> But this assertion can be broken :
> In the first case, it is generated by System.nanoTime() but it seems that System.nanoTime() is using the boot time as the base/reference (at least on java6 & linux), thus after a reboot, System.nanoTime() can return a lower number than before the reboot (and the javadoc says the reference is a relative point in time...)
> In the second case, this was introduced by #4601 (which changes System.nanoTime() by System.currentTimeMillis() thus people starting with 1.1.5 are safe)
> This could explain the following tickets : #4741 and #4481
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4782) Commitlog not replayed after
restart
Posted by "Brandon Williams (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13472573#comment-13472573 ]
Brandon Williams commented on CASSANDRA-4782:
---------------------------------------------
Thanks for the summary. What are you proposing we do?
> Commitlog not replayed after restart
> ------------------------------------
>
> Key: CASSANDRA-4782
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4782
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 1.1.0, 1.1.1, 1.1.2, 1.1.3, 1.1.4, 1.1.5
> Reporter: Fabien Rousseau
>
> It seems that there are two corner cases where commitlog is not replayed after a restart :
> - After a reboot of a server + restart of cassandra (1.1.0 to 1.1.4)
> - After doing an upgrade from cassandra 1.1.X to cassandra 1.1.5
> This is due to the fact that the commitlog segment id should always be an incrementing number (see this condition : https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/db/commitlog/CommitLogReplayer.java#L247 )
> But this assertion can be broken :
> In the first case, it is generated by System.nanoTime() but it seems that System.nanoTime() is using the boot time as the base/reference (at least on java6 & linux), thus after a reboot, System.nanoTime() can return a lower number than before the reboot (and the javadoc says the reference is a relative point in time...)
> In the second case, this was introduced by #4601 (which changes System.nanoTime() by System.currentTimeMillis() thus people starting with 1.1.5 are safe)
> This could explain the following tickets : #4741 and #4481
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4782) Commitlog not replayed after
restart
Posted by "Fabien Rousseau (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13474067#comment-13474067 ]
Fabien Rousseau commented on CASSANDRA-4782:
--------------------------------------------
Thanks for the patch. This solution is more simple and elegant than the one I proposed.
I tested it and it worked like a charm.
Nevertheless, if there are counters CF, a drain is probably necessary to avoid replaying the full commitlog and avoid having overcounts. (I don't think it is a problem, just something to know before the upgrade...)
> Commitlog not replayed after restart
> ------------------------------------
>
> Key: CASSANDRA-4782
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4782
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 1.1.0
> Reporter: Fabien Rousseau
> Assignee: Jonathan Ellis
> Priority: Critical
> Fix For: 1.1.6
>
> Attachments: 4782.txt
>
>
> It seems that there are two corner cases where commitlog is not replayed after a restart :
> - After a reboot of a server + restart of cassandra (1.1.0 to 1.1.4)
> - After doing an upgrade from cassandra 1.1.X to cassandra 1.1.5
> This is due to the fact that the commitlog segment id should always be an incrementing number (see this condition : https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/db/commitlog/CommitLogReplayer.java#L247 )
> But this assertion can be broken :
> In the first case, it is generated by System.nanoTime() but it seems that System.nanoTime() is using the boot time as the base/reference (at least on java6 & linux), thus after a reboot, System.nanoTime() can return a lower number than before the reboot (and the javadoc says the reference is a relative point in time...)
> In the second case, this was introduced by #4601 (which changes System.nanoTime() by System.currentTimeMillis() thus people starting with 1.1.5 are safe)
> This could explain the following tickets : #4741 and #4481
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4782) Commitlog not replayed after
restart
Posted by "Fabien Rousseau (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13474098#comment-13474098 ]
Fabien Rousseau commented on CASSANDRA-4782:
--------------------------------------------
By the way, I just noticed that the commitlog files were not replayed in the order of their ids.
It seems that they are sorted by "last modification date" before being replayed, but this does not corresponds to their ids.
Moreover, "last modification date" is changed when a file is copied, so, this could also change the order of archived commitlogs.
I suppose the sort order of commit log files is for schemas ?
Maybe it's safer to sort them using the id in the file name ?
> Commitlog not replayed after restart
> ------------------------------------
>
> Key: CASSANDRA-4782
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4782
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 1.1.0
> Reporter: Fabien Rousseau
> Assignee: Jonathan Ellis
> Priority: Critical
> Fix For: 1.1.6
>
> Attachments: 4782.txt
>
>
> It seems that there are two corner cases where commitlog is not replayed after a restart :
> - After a reboot of a server + restart of cassandra (1.1.0 to 1.1.4)
> - After doing an upgrade from cassandra 1.1.X to cassandra 1.1.5
> This is due to the fact that the commitlog segment id should always be an incrementing number (see this condition : https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/db/commitlog/CommitLogReplayer.java#L247 )
> But this assertion can be broken :
> In the first case, it is generated by System.nanoTime() but it seems that System.nanoTime() is using the boot time as the base/reference (at least on java6 & linux), thus after a reboot, System.nanoTime() can return a lower number than before the reboot (and the javadoc says the reference is a relative point in time...)
> In the second case, this was introduced by #4601 (which changes System.nanoTime() by System.currentTimeMillis() thus people starting with 1.1.5 are safe)
> This could explain the following tickets : #4741 and #4481
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-4782) Commitlog not replayed after
restart
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-4782:
--------------------------------------
Attachment: (was: 4782.txt)
> Commitlog not replayed after restart
> ------------------------------------
>
> Key: CASSANDRA-4782
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4782
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 1.1.0
> Reporter: Fabien Rousseau
> Assignee: Jonathan Ellis
> Priority: Critical
> Fix For: 1.1.6
>
>
> It seems that there are two corner cases where commitlog is not replayed after a restart :
> - After a reboot of a server + restart of cassandra (1.1.0 to 1.1.4)
> - After doing an upgrade from cassandra 1.1.X to cassandra 1.1.5
> This is due to the fact that the commitlog segment id should always be an incrementing number (see this condition : https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/db/commitlog/CommitLogReplayer.java#L247 )
> But this assertion can be broken :
> In the first case, it is generated by System.nanoTime() but it seems that System.nanoTime() is using the boot time as the base/reference (at least on java6 & linux), thus after a reboot, System.nanoTime() can return a lower number than before the reboot (and the javadoc says the reference is a relative point in time...)
> In the second case, this was introduced by #4601 (which changes System.nanoTime() by System.currentTimeMillis() thus people starting with 1.1.5 are safe)
> This could explain the following tickets : #4741 and #4481
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4782) Commitlog not replayed after
restart
Posted by "Arya Goudarzi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13479127#comment-13479127 ]
Arya Goudarzi commented on CASSANDRA-4782:
------------------------------------------
+1. I think the issue I had with data loss after node restart, actually had to do with this bug. Thanks for the fix.
> Commitlog not replayed after restart
> ------------------------------------
>
> Key: CASSANDRA-4782
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4782
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 1.1.0
> Reporter: Fabien Rousseau
> Assignee: Jonathan Ellis
> Priority: Critical
> Fix For: 1.1.6
>
> Attachments: 4782.txt
>
>
> It seems that there are two corner cases where commitlog is not replayed after a restart :
> - After a reboot of a server + restart of cassandra (1.1.0 to 1.1.4)
> - After doing an upgrade from cassandra 1.1.X to cassandra 1.1.5
> This is due to the fact that the commitlog segment id should always be an incrementing number (see this condition : https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/db/commitlog/CommitLogReplayer.java#L247 )
> But this assertion can be broken :
> In the first case, it is generated by System.nanoTime() but it seems that System.nanoTime() is using the boot time as the base/reference (at least on java6 & linux), thus after a reboot, System.nanoTime() can return a lower number than before the reboot (and the javadoc says the reference is a relative point in time...)
> In the second case, this was introduced by #4601 (which changes System.nanoTime() by System.currentTimeMillis() thus people starting with 1.1.5 are safe)
> This could explain the following tickets : #4741 and #4481
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (CASSANDRA-4782) Commitlog not replayed after
restart
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis resolved CASSANDRA-4782.
---------------------------------------
Resolution: Fixed
Reviewer: frousseau
Committed with a warning in NEWS to drain.
Go ahead and open a separate ticket to fix sort order.
> Commitlog not replayed after restart
> ------------------------------------
>
> Key: CASSANDRA-4782
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4782
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 1.1.0
> Reporter: Fabien Rousseau
> Assignee: Jonathan Ellis
> Priority: Critical
> Fix For: 1.1.6
>
> Attachments: 4782.txt
>
>
> It seems that there are two corner cases where commitlog is not replayed after a restart :
> - After a reboot of a server + restart of cassandra (1.1.0 to 1.1.4)
> - After doing an upgrade from cassandra 1.1.X to cassandra 1.1.5
> This is due to the fact that the commitlog segment id should always be an incrementing number (see this condition : https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/db/commitlog/CommitLogReplayer.java#L247 )
> But this assertion can be broken :
> In the first case, it is generated by System.nanoTime() but it seems that System.nanoTime() is using the boot time as the base/reference (at least on java6 & linux), thus after a reboot, System.nanoTime() can return a lower number than before the reboot (and the javadoc says the reference is a relative point in time...)
> In the second case, this was introduced by #4601 (which changes System.nanoTime() by System.currentTimeMillis() thus people starting with 1.1.5 are safe)
> This could explain the following tickets : #4741 and #4481
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-4782) Commitlog not replayed after
restart
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-4782:
--------------------------------------
Attachment: 4782.txt
Thanks, Fabien. I admit that I didn't realize at first the implications of the millis fix on existing sstable metadata.
Patch attached that bumps the sstable version to hf as a marker that we know metadata with that version has sane replay positions. Replay positions from older metadata will be treated as NONE.
This will force a full replay the first restart on 1.1.6; afterwards, any newly flushed sstables will have the sane-replay-position marker and future restarts will not need to replay data unnecessarily.
What do you think?
> Commitlog not replayed after restart
> ------------------------------------
>
> Key: CASSANDRA-4782
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4782
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 1.1.0
> Reporter: Fabien Rousseau
> Assignee: Jonathan Ellis
> Priority: Critical
> Fix For: 1.1.6
>
> Attachments: 4782.txt
>
>
> It seems that there are two corner cases where commitlog is not replayed after a restart :
> - After a reboot of a server + restart of cassandra (1.1.0 to 1.1.4)
> - After doing an upgrade from cassandra 1.1.X to cassandra 1.1.5
> This is due to the fact that the commitlog segment id should always be an incrementing number (see this condition : https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/db/commitlog/CommitLogReplayer.java#L247 )
> But this assertion can be broken :
> In the first case, it is generated by System.nanoTime() but it seems that System.nanoTime() is using the boot time as the base/reference (at least on java6 & linux), thus after a reboot, System.nanoTime() can return a lower number than before the reboot (and the javadoc says the reference is a relative point in time...)
> In the second case, this was introduced by #4601 (which changes System.nanoTime() by System.currentTimeMillis() thus people starting with 1.1.5 are safe)
> This could explain the following tickets : #4741 and #4481
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-4782) Commitlog not replayed after
restart
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-4782:
--------------------------------------
Priority: Critical (was: Major)
Affects Version/s: (was: 1.1.5)
(was: 1.1.4)
(was: 1.1.3)
(was: 1.1.2)
(was: 1.1.1)
Fix Version/s: 1.1.6
Assignee: Jonathan Ellis
> Commitlog not replayed after restart
> ------------------------------------
>
> Key: CASSANDRA-4782
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4782
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 1.1.0
> Reporter: Fabien Rousseau
> Assignee: Jonathan Ellis
> Priority: Critical
> Fix For: 1.1.6
>
>
> It seems that there are two corner cases where commitlog is not replayed after a restart :
> - After a reboot of a server + restart of cassandra (1.1.0 to 1.1.4)
> - After doing an upgrade from cassandra 1.1.X to cassandra 1.1.5
> This is due to the fact that the commitlog segment id should always be an incrementing number (see this condition : https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/db/commitlog/CommitLogReplayer.java#L247 )
> But this assertion can be broken :
> In the first case, it is generated by System.nanoTime() but it seems that System.nanoTime() is using the boot time as the base/reference (at least on java6 & linux), thus after a reboot, System.nanoTime() can return a lower number than before the reboot (and the javadoc says the reference is a relative point in time...)
> In the second case, this was introduced by #4601 (which changes System.nanoTime() by System.currentTimeMillis() thus people starting with 1.1.5 are safe)
> This could explain the following tickets : #4741 and #4481
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira