[jira] [Created] (CASSANDRA-5535) Manifest file not fsynced

["Terry Cumaranatunge (JIRA)" <ji...@apache.org>]

Terry Cumaranatunge created CASSANDRA-5535:
----------------------------------------------

             Summary: Manifest file not fsynced
                 Key: CASSANDRA-5535
                 URL: https://issues.apache.org/jira/browse/CASSANDRA-5535
             Project: Cassandra
          Issue Type: Bug
    Affects Versions: 1.2.3
         Environment: RHEL 6.4
java -version
java version "1.6.0_31"
Java(TM) SE Runtime Environment (build 1.6.0_31-b04)
Java HotSpot(TM) 64-Bit Server VM (build 20.6-b01, mixed mode)

            Reporter: Terry Cumaranatunge
            Priority: Critical


We had several cases where the the manifest file would get corrupted when doing power reset tests or iLO resets to mimic power failure scenarios, ungraceful resets, kernel panics etc. It wasn't clear at the time where the problem was, but I think the data below shows that Cassandra is missing an fsync call to the manifest file prior to closing it. This particular stack trace from below is on Cassandra 1.2.4.
The trace was captured using strace options:

strace -f -p 2200 -e trace=open,close,write,fsync,fdatasync,rename
[pid 9710] open("/opt/mp/storage/persistent/cassandra/cassandra-lib/data/MSA/subinfo/subinfo-tmp.json", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 238
[pid 9710] write(238, "{\n \"generations\" : [ {\n \"gen"..., 3996) = 3996
[pid 9710] write(238, "14, 263161, 263484, 270816, 2593"..., 3996) = 3996
[pid 9710] write(238, "275136, 275137, 275138, 275139, "..., 1173) = 1173
[pid 9710] close(238) = 0
[pid 9710] rename("/opt/mp/storage/persistent/cassandra/cassandra-lib/data/MSA/subinfo/subinfo.json"
, "/opt/mp/storage/persistent/cassandra/cassandra-lib/data/MSA/subinfo/subinfo-old.json") = 0
[pid 9710] rename("/opt/mp/storage/persistent/cassandra/cassandra-lib/data/MSA/subinfo/subinfo-tmp.j
son", "/opt/mp/storage/persistent/cassandra/cassandra-lib/data/MSA/subinfo/subinfo.json") = 0


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira