You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@jena.apache.org by "Simon Helsen (JIRA)" <ji...@apache.org> on 2012/08/30 00:30:12 UTC
[jira] [Created] (JENA-308) Index corruption after killing process
Simon Helsen created JENA-308:
---------------------------------
Summary: Index corruption after killing process
Key: JENA-308
URL: https://issues.apache.org/jira/browse/JENA-308
Project: Apache Jena
Issue Type: Bug
Components: TDB
Affects Versions: TDB 0.9.1, TDB 0.9.4
Environment: Windows 2008/7 (64 bit) both on IBM and SUN JVM 6. Direct mode
Reporter: Simon Helsen
Priority: Critical
We are faced with a series of possible exceptions which may or may not occur after an active tdb store is killed. The most common exception has the form:
Exception in thread "Thread-3" com.hp.hpl.jena.tdb.transaction.TDBTransactionException: Feaild to read block checksum.
at com.hp.hpl.jena.tdb.transaction.Journal._read(Journal.java:238)
at com.hp.hpl.jena.tdb.transaction.Journal._readJournal(Journal.java:197)
at com.hp.hpl.jena.tdb.transaction.Journal.access$1(Journal.java:192)
at com.hp.hpl.jena.tdb.transaction.Journal$IteratorEntries.moveToNext(Journal.java:267)
at com.hp.hpl.jena.tdb.transaction.Journal$IteratorEntries.moveToNext(Journal.java:1)
at org.openjena.atlas.iterator.IteratorSlotted.hasNext(IteratorSlotted.java:67)
at com.hp.hpl.jena.tdb.transaction.JournalControl.scanForCommit(JournalControl.java:148)
at com.hp.hpl.jena.tdb.transaction.JournalControl.recoverFromJournal(JournalControl.java:127)
at com.hp.hpl.jena.tdb.StoreConnection._makeAndCache(StoreConnection.java:234)
at com.hp.hpl.jena.tdb.StoreConnection.make(StoreConnection.java:214)
at com.hp.hpl.jena.tdb.transaction.DatasetGraphTransaction.<init>(DatasetGraphTransaction.java:76)
at com.hp.hpl.jena.tdb.sys.TDBMaker._create(TDBMaker.java:57)
at com.hp.hpl.jena.tdb.sys.TDBMaker.createDatasetGraphTransaction(TDBMaker.java:45)
at com.hp.hpl.jena.tdb.TDBFactory._createDatasetGraph(TDBFactory.java:104)
at com.hp.hpl.jena.tdb.TDBFactory.createDatasetGraph(TDBFactory.java:73)
at com.hp.hpl.jena.tdb.TDBFactory.createDataset(TDBFactory.java:52)
Other exceptions are possible (see comments below). I have been able to produce a standalone java program/test case which simulates a process kill and is able to produce the exception after a non-deterministic time. It runs an iteration until it hits one of the possible problems. Sometimes, an exception hits after just 2 iterations, other times, it can take 40 runs or more before hitting a problem. Because I don't know what the cause is here, I have not been able to make the test case more specific
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (JENA-308) Index corruption after killing
process
Posted by "Simon Helsen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/JENA-308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13444488#comment-13444488 ]
Simon Helsen commented on JENA-308:
-----------------------------------
The exception as shown in the description is the most common and has been observed on both 0.9.1 and 0.9.4. However, the following exception very rarely occurs instead (also seen in both 0.9.1. and 0.9.4):
1346278257925 : Iteration 1
1346278257931 : create dataset
************* UNEXPECTED [1]
1346278257945 : read
1346278257947 : write
18:10:57,969 [Thread-2] WARN com.hp.hpl.jena.tdb.transaction.NodeTableTrans - Txn[4]/W journalStartOffset not zero: 1048/0x418
Different ids for https://localhost:9443/jazz/storage/net.jazz.ajax.theme0: allocated: expected [0000000000000665], got [0000000000000000]
>>>>>>>>>>
label = nodes
txn = Transaction: 4 : Mode=WRITE : State=PREPARING : C:\Temp\testIndexLocation\
offset = 1637
journalStartOffset = 1048
journal = nodes.dat-jrnl
com.hp.hpl.jena.tdb.TDBException: Different ids for https://localhost:9443/jazz/storage/net.jazz.ajax.theme0: allocated: expected [0000000000000665], got [0000000000000000]
at com.hp.hpl.jena.tdb.transaction.NodeTableTrans.inconsistent(NodeTableTrans.java:212)
at com.hp.hpl.jena.tdb.transaction.NodeTableTrans.append(NodeTableTrans.java:200)
at com.hp.hpl.jena.tdb.transaction.NodeTableTrans.writeNodeJournal(NodeTableTrans.java:306)
at com.hp.hpl.jena.tdb.transaction.NodeTableTrans.commitPrepare(NodeTableTrans.java:266)
at com.hp.hpl.jena.tdb.transaction.Transaction.prepare(Transaction.java:131)
at com.hp.hpl.jena.tdb.transaction.Transaction.commit(Transaction.java:112)
at com.hp.hpl.jena.tdb.transaction.DatasetGraphTxn.commit(DatasetGraphTxn.java:44)
at com.hp.hpl.jena.tdb.transaction.DatasetGraphTransaction._commit(DatasetGraphTransaction.java:148)
at com.hp.hpl.jena.sparql.core.DatasetGraphTrackActive.commit(DatasetGraphTrackActive.java:54)
at com.hp.hpl.jena.sparql.core.DatasetImpl.commit(DatasetImpl.java:141)
at com.ibm.jena.test.JournalRecoveryTest.storeOperation(JournalRecoveryTest.java:92)
at com.ibm.jena.test.JournalRecoveryTest.writeModel(JournalRecoveryTest.java:212)
at com.ibm.jena.test.JournalRecoveryTest$TestRunnable.run(JournalRecoveryTest.java:277)
at java.lang.Thread.run(Thread.java:662)
> Index corruption after killing process
> --------------------------------------
>
> Key: JENA-308
> URL: https://issues.apache.org/jira/browse/JENA-308
> Project: Apache Jena
> Issue Type: Bug
> Components: TDB
> Affects Versions: TDB 0.9.1, TDB 0.9.4
> Environment: Windows 2008/7 (64 bit) both on IBM and SUN JVM 6. Direct mode
> Reporter: Simon Helsen
> Priority: Critical
>
> We are faced with a series of possible exceptions which may or may not occur after an active tdb store is killed. The most common exception has the form:
> Exception in thread "Thread-3" com.hp.hpl.jena.tdb.transaction.TDBTransactionException: Feaild to read block checksum.
> at com.hp.hpl.jena.tdb.transaction.Journal._read(Journal.java:238)
> at com.hp.hpl.jena.tdb.transaction.Journal._readJournal(Journal.java:197)
> at com.hp.hpl.jena.tdb.transaction.Journal.access$1(Journal.java:192)
> at com.hp.hpl.jena.tdb.transaction.Journal$IteratorEntries.moveToNext(Journal.java:267)
> at com.hp.hpl.jena.tdb.transaction.Journal$IteratorEntries.moveToNext(Journal.java:1)
> at org.openjena.atlas.iterator.IteratorSlotted.hasNext(IteratorSlotted.java:67)
> at com.hp.hpl.jena.tdb.transaction.JournalControl.scanForCommit(JournalControl.java:148)
> at com.hp.hpl.jena.tdb.transaction.JournalControl.recoverFromJournal(JournalControl.java:127)
> at com.hp.hpl.jena.tdb.StoreConnection._makeAndCache(StoreConnection.java:234)
> at com.hp.hpl.jena.tdb.StoreConnection.make(StoreConnection.java:214)
> at com.hp.hpl.jena.tdb.transaction.DatasetGraphTransaction.<init>(DatasetGraphTransaction.java:76)
> at com.hp.hpl.jena.tdb.sys.TDBMaker._create(TDBMaker.java:57)
> at com.hp.hpl.jena.tdb.sys.TDBMaker.createDatasetGraphTransaction(TDBMaker.java:45)
> at com.hp.hpl.jena.tdb.TDBFactory._createDatasetGraph(TDBFactory.java:104)
> at com.hp.hpl.jena.tdb.TDBFactory.createDatasetGraph(TDBFactory.java:73)
> at com.hp.hpl.jena.tdb.TDBFactory.createDataset(TDBFactory.java:52)
> Other exceptions are possible (see comments below). I have been able to produce a standalone java program/test case which simulates a process kill and is able to produce the exception after a non-deterministic time. It runs an iteration until it hits one of the possible problems. Sometimes, an exception hits after just 2 iterations, other times, it can take 40 runs or more before hitting a problem. Because I don't know what the cause is here, I have not been able to make the test case more specific
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (JENA-308) Index corruption after killing
process
Posted by "Simon Helsen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/JENA-308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13444984#comment-13444984 ]
Simon Helsen commented on JENA-308:
-----------------------------------
According to http://mail-archives.apache.org/mod_mbox/jena-dev/201208.mbox/%3C503F2C80.1040303%40apache.org%3E, the exception com.hp.hpl.jena.tdb.base.block.BlockException: BlockAccessBase: Bounds exception: C:\Temp\testIndexLocation\GSPO.dat: (9,9) has been fixed in 0.9.4, which corresponds to my observations. However, the other 2 exceptions are still observable in 0.9.4
> Index corruption after killing process
> --------------------------------------
>
> Key: JENA-308
> URL: https://issues.apache.org/jira/browse/JENA-308
> Project: Apache Jena
> Issue Type: Bug
> Components: TDB
> Affects Versions: TDB 0.9.1, TDB 0.9.4
> Environment: Windows 2008/7 (64 bit) both on IBM and SUN JVM 6. Direct mode
> Reporter: Simon Helsen
> Priority: Critical
> Attachments: journalRecoveryTest.zip
>
>
> We are faced with a series of possible exceptions which may or may not occur after an active tdb store is killed. The most common exception has the form:
> Exception in thread "Thread-3" com.hp.hpl.jena.tdb.transaction.TDBTransactionException: Feaild to read block checksum.
> at com.hp.hpl.jena.tdb.transaction.Journal._read(Journal.java:238)
> at com.hp.hpl.jena.tdb.transaction.Journal._readJournal(Journal.java:197)
> at com.hp.hpl.jena.tdb.transaction.Journal.access$1(Journal.java:192)
> at com.hp.hpl.jena.tdb.transaction.Journal$IteratorEntries.moveToNext(Journal.java:267)
> at com.hp.hpl.jena.tdb.transaction.Journal$IteratorEntries.moveToNext(Journal.java:1)
> at org.openjena.atlas.iterator.IteratorSlotted.hasNext(IteratorSlotted.java:67)
> at com.hp.hpl.jena.tdb.transaction.JournalControl.scanForCommit(JournalControl.java:148)
> at com.hp.hpl.jena.tdb.transaction.JournalControl.recoverFromJournal(JournalControl.java:127)
> at com.hp.hpl.jena.tdb.StoreConnection._makeAndCache(StoreConnection.java:234)
> at com.hp.hpl.jena.tdb.StoreConnection.make(StoreConnection.java:214)
> at com.hp.hpl.jena.tdb.transaction.DatasetGraphTransaction.<init>(DatasetGraphTransaction.java:76)
> at com.hp.hpl.jena.tdb.sys.TDBMaker._create(TDBMaker.java:57)
> at com.hp.hpl.jena.tdb.sys.TDBMaker.createDatasetGraphTransaction(TDBMaker.java:45)
> at com.hp.hpl.jena.tdb.TDBFactory._createDatasetGraph(TDBFactory.java:104)
> at com.hp.hpl.jena.tdb.TDBFactory.createDatasetGraph(TDBFactory.java:73)
> at com.hp.hpl.jena.tdb.TDBFactory.createDataset(TDBFactory.java:52)
> Other exceptions are possible (see comments below). I have been able to produce a standalone java program/test case which simulates a process kill and is able to produce the exception after a non-deterministic time. It runs an iteration until it hits one of the possible problems. Sometimes, an exception hits after just 2 iterations, other times, it can take 40 runs or more before hitting a problem. Because I don't know what the cause is here, I have not been able to make the test case more specific
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (JENA-308) Index corruption after killing
process
Posted by "Andy Seaborne (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/JENA-308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13445307#comment-13445307 ]
Andy Seaborne commented on JENA-308:
------------------------------------
"BlockAccessBase: Bounds exception" -- already fixed.
"TDBTransactionException: Failed to read block checksum."
This is due to uncommited junk at the end of the main journal after a crash.
Fixed in svn.
"journalStartOffset not zero"
"TDBException: Different ids"
would be explained by failing to empty a file that was part-prepared. Fix applied - not tested but I haven't managed to reproduce it anyway even after a lot of runs.
Please open a new JIRA for this last point with a refined test case if the new TDB code still exhibits it.
This test code locks up after many iterations, 230 or so in the debugger, 728 outside (possibly dead threads causing some kind of resource exhaustion)
> Index corruption after killing process
> --------------------------------------
>
> Key: JENA-308
> URL: https://issues.apache.org/jira/browse/JENA-308
> Project: Apache Jena
> Issue Type: Bug
> Components: TDB
> Affects Versions: TDB 0.9.1, TDB 0.9.4
> Environment: Windows 2008/7 (64 bit) both on IBM and SUN JVM 6. Direct mode
> Reporter: Simon Helsen
> Priority: Critical
> Fix For: TDB 0.9.4
>
> Attachments: journalRecoveryTest.zip
>
>
> We are faced with a series of possible exceptions which may or may not occur after an active tdb store is killed. The most common exception has the form:
> Exception in thread "Thread-3" com.hp.hpl.jena.tdb.transaction.TDBTransactionException: Feaild to read block checksum.
> at com.hp.hpl.jena.tdb.transaction.Journal._read(Journal.java:238)
> at com.hp.hpl.jena.tdb.transaction.Journal._readJournal(Journal.java:197)
> at com.hp.hpl.jena.tdb.transaction.Journal.access$1(Journal.java:192)
> at com.hp.hpl.jena.tdb.transaction.Journal$IteratorEntries.moveToNext(Journal.java:267)
> at com.hp.hpl.jena.tdb.transaction.Journal$IteratorEntries.moveToNext(Journal.java:1)
> at org.openjena.atlas.iterator.IteratorSlotted.hasNext(IteratorSlotted.java:67)
> at com.hp.hpl.jena.tdb.transaction.JournalControl.scanForCommit(JournalControl.java:148)
> at com.hp.hpl.jena.tdb.transaction.JournalControl.recoverFromJournal(JournalControl.java:127)
> at com.hp.hpl.jena.tdb.StoreConnection._makeAndCache(StoreConnection.java:234)
> at com.hp.hpl.jena.tdb.StoreConnection.make(StoreConnection.java:214)
> at com.hp.hpl.jena.tdb.transaction.DatasetGraphTransaction.<init>(DatasetGraphTransaction.java:76)
> at com.hp.hpl.jena.tdb.sys.TDBMaker._create(TDBMaker.java:57)
> at com.hp.hpl.jena.tdb.sys.TDBMaker.createDatasetGraphTransaction(TDBMaker.java:45)
> at com.hp.hpl.jena.tdb.TDBFactory._createDatasetGraph(TDBFactory.java:104)
> at com.hp.hpl.jena.tdb.TDBFactory.createDatasetGraph(TDBFactory.java:73)
> at com.hp.hpl.jena.tdb.TDBFactory.createDataset(TDBFactory.java:52)
> Other exceptions are possible (see comments below). I have been able to produce a standalone java program/test case which simulates a process kill and is able to produce the exception after a non-deterministic time. It runs an iteration until it hits one of the possible problems. Sometimes, an exception hits after just 2 iterations, other times, it can take 40 runs or more before hitting a problem. Because I don't know what the cause is here, I have not been able to make the test case more specific
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (JENA-308) Index corruption after killing
process
Posted by "Simon Helsen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/JENA-308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13444493#comment-13444493 ]
Simon Helsen commented on JENA-308:
-----------------------------------
Finally, I should make a few comments about the actual test case. Because we need to kill ongoing (write) transactions and repeat the process often enough, I decided to simulate this by running the actual logic in a separate thread which I would kill after a certain time. To make sure that no cleanup activities take place, I use ThreadDeath to immediately leave the process.
Moreover, because I want to simulate a fresh startup, I have to call StoreConnection.reset();
I realize this is not a 100% perfect simulation, I think the relevant behavior is similar to what would happen on process kill. Either way, we have seen the exceptions with a real process kill
> Index corruption after killing process
> --------------------------------------
>
> Key: JENA-308
> URL: https://issues.apache.org/jira/browse/JENA-308
> Project: Apache Jena
> Issue Type: Bug
> Components: TDB
> Affects Versions: TDB 0.9.1, TDB 0.9.4
> Environment: Windows 2008/7 (64 bit) both on IBM and SUN JVM 6. Direct mode
> Reporter: Simon Helsen
> Priority: Critical
>
> We are faced with a series of possible exceptions which may or may not occur after an active tdb store is killed. The most common exception has the form:
> Exception in thread "Thread-3" com.hp.hpl.jena.tdb.transaction.TDBTransactionException: Feaild to read block checksum.
> at com.hp.hpl.jena.tdb.transaction.Journal._read(Journal.java:238)
> at com.hp.hpl.jena.tdb.transaction.Journal._readJournal(Journal.java:197)
> at com.hp.hpl.jena.tdb.transaction.Journal.access$1(Journal.java:192)
> at com.hp.hpl.jena.tdb.transaction.Journal$IteratorEntries.moveToNext(Journal.java:267)
> at com.hp.hpl.jena.tdb.transaction.Journal$IteratorEntries.moveToNext(Journal.java:1)
> at org.openjena.atlas.iterator.IteratorSlotted.hasNext(IteratorSlotted.java:67)
> at com.hp.hpl.jena.tdb.transaction.JournalControl.scanForCommit(JournalControl.java:148)
> at com.hp.hpl.jena.tdb.transaction.JournalControl.recoverFromJournal(JournalControl.java:127)
> at com.hp.hpl.jena.tdb.StoreConnection._makeAndCache(StoreConnection.java:234)
> at com.hp.hpl.jena.tdb.StoreConnection.make(StoreConnection.java:214)
> at com.hp.hpl.jena.tdb.transaction.DatasetGraphTransaction.<init>(DatasetGraphTransaction.java:76)
> at com.hp.hpl.jena.tdb.sys.TDBMaker._create(TDBMaker.java:57)
> at com.hp.hpl.jena.tdb.sys.TDBMaker.createDatasetGraphTransaction(TDBMaker.java:45)
> at com.hp.hpl.jena.tdb.TDBFactory._createDatasetGraph(TDBFactory.java:104)
> at com.hp.hpl.jena.tdb.TDBFactory.createDatasetGraph(TDBFactory.java:73)
> at com.hp.hpl.jena.tdb.TDBFactory.createDataset(TDBFactory.java:52)
> Other exceptions are possible (see comments below). I have been able to produce a standalone java program/test case which simulates a process kill and is able to produce the exception after a non-deterministic time. It runs an iteration until it hits one of the possible problems. Sometimes, an exception hits after just 2 iterations, other times, it can take 40 runs or more before hitting a problem. Because I don't know what the cause is here, I have not been able to make the test case more specific
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (JENA-308) Index corruption after killing
process
Posted by "Simon Helsen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/JENA-308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13446374#comment-13446374 ]
Simon Helsen commented on JENA-308:
-----------------------------------
looks good for now. I would need to see what happens in our system verification tests once we adopt a successor release of course
> Index corruption after killing process
> --------------------------------------
>
> Key: JENA-308
> URL: https://issues.apache.org/jira/browse/JENA-308
> Project: Apache Jena
> Issue Type: Bug
> Components: TDB
> Affects Versions: TDB 0.9.1, TDB 0.9.4
> Environment: Windows 2008/7 (64 bit) both on IBM and SUN JVM 6. Direct mode
> Reporter: Simon Helsen
> Assignee: Andy Seaborne
> Priority: Critical
> Fix For: TDB 0.9.4
>
> Attachments: journalRecoveryTest.zip
>
>
> We are faced with a series of possible exceptions which may or may not occur after an active tdb store is killed. The most common exception has the form:
> Exception in thread "Thread-3" com.hp.hpl.jena.tdb.transaction.TDBTransactionException: Feaild to read block checksum.
> at com.hp.hpl.jena.tdb.transaction.Journal._read(Journal.java:238)
> at com.hp.hpl.jena.tdb.transaction.Journal._readJournal(Journal.java:197)
> at com.hp.hpl.jena.tdb.transaction.Journal.access$1(Journal.java:192)
> at com.hp.hpl.jena.tdb.transaction.Journal$IteratorEntries.moveToNext(Journal.java:267)
> at com.hp.hpl.jena.tdb.transaction.Journal$IteratorEntries.moveToNext(Journal.java:1)
> at org.openjena.atlas.iterator.IteratorSlotted.hasNext(IteratorSlotted.java:67)
> at com.hp.hpl.jena.tdb.transaction.JournalControl.scanForCommit(JournalControl.java:148)
> at com.hp.hpl.jena.tdb.transaction.JournalControl.recoverFromJournal(JournalControl.java:127)
> at com.hp.hpl.jena.tdb.StoreConnection._makeAndCache(StoreConnection.java:234)
> at com.hp.hpl.jena.tdb.StoreConnection.make(StoreConnection.java:214)
> at com.hp.hpl.jena.tdb.transaction.DatasetGraphTransaction.<init>(DatasetGraphTransaction.java:76)
> at com.hp.hpl.jena.tdb.sys.TDBMaker._create(TDBMaker.java:57)
> at com.hp.hpl.jena.tdb.sys.TDBMaker.createDatasetGraphTransaction(TDBMaker.java:45)
> at com.hp.hpl.jena.tdb.TDBFactory._createDatasetGraph(TDBFactory.java:104)
> at com.hp.hpl.jena.tdb.TDBFactory.createDatasetGraph(TDBFactory.java:73)
> at com.hp.hpl.jena.tdb.TDBFactory.createDataset(TDBFactory.java:52)
> Other exceptions are possible (see comments below). I have been able to produce a standalone java program/test case which simulates a process kill and is able to produce the exception after a non-deterministic time. It runs an iteration until it hits one of the possible problems. Sometimes, an exception hits after just 2 iterations, other times, it can take 40 runs or more before hitting a problem. Because I don't know what the cause is here, I have not been able to make the test case more specific
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (JENA-308) Index corruption after killing process
Posted by "Simon Helsen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/JENA-308?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Simon Helsen updated JENA-308:
------------------------------
Attachment: journalRecoveryTest.zip
> Index corruption after killing process
> --------------------------------------
>
> Key: JENA-308
> URL: https://issues.apache.org/jira/browse/JENA-308
> Project: Apache Jena
> Issue Type: Bug
> Components: TDB
> Affects Versions: TDB 0.9.1, TDB 0.9.4
> Environment: Windows 2008/7 (64 bit) both on IBM and SUN JVM 6. Direct mode
> Reporter: Simon Helsen
> Priority: Critical
> Attachments: journalRecoveryTest.zip
>
>
> We are faced with a series of possible exceptions which may or may not occur after an active tdb store is killed. The most common exception has the form:
> Exception in thread "Thread-3" com.hp.hpl.jena.tdb.transaction.TDBTransactionException: Feaild to read block checksum.
> at com.hp.hpl.jena.tdb.transaction.Journal._read(Journal.java:238)
> at com.hp.hpl.jena.tdb.transaction.Journal._readJournal(Journal.java:197)
> at com.hp.hpl.jena.tdb.transaction.Journal.access$1(Journal.java:192)
> at com.hp.hpl.jena.tdb.transaction.Journal$IteratorEntries.moveToNext(Journal.java:267)
> at com.hp.hpl.jena.tdb.transaction.Journal$IteratorEntries.moveToNext(Journal.java:1)
> at org.openjena.atlas.iterator.IteratorSlotted.hasNext(IteratorSlotted.java:67)
> at com.hp.hpl.jena.tdb.transaction.JournalControl.scanForCommit(JournalControl.java:148)
> at com.hp.hpl.jena.tdb.transaction.JournalControl.recoverFromJournal(JournalControl.java:127)
> at com.hp.hpl.jena.tdb.StoreConnection._makeAndCache(StoreConnection.java:234)
> at com.hp.hpl.jena.tdb.StoreConnection.make(StoreConnection.java:214)
> at com.hp.hpl.jena.tdb.transaction.DatasetGraphTransaction.<init>(DatasetGraphTransaction.java:76)
> at com.hp.hpl.jena.tdb.sys.TDBMaker._create(TDBMaker.java:57)
> at com.hp.hpl.jena.tdb.sys.TDBMaker.createDatasetGraphTransaction(TDBMaker.java:45)
> at com.hp.hpl.jena.tdb.TDBFactory._createDatasetGraph(TDBFactory.java:104)
> at com.hp.hpl.jena.tdb.TDBFactory.createDatasetGraph(TDBFactory.java:73)
> at com.hp.hpl.jena.tdb.TDBFactory.createDataset(TDBFactory.java:52)
> Other exceptions are possible (see comments below). I have been able to produce a standalone java program/test case which simulates a process kill and is able to produce the exception after a non-deterministic time. It runs an iteration until it hits one of the possible problems. Sometimes, an exception hits after just 2 iterations, other times, it can take 40 runs or more before hitting a problem. Because I don't know what the cause is here, I have not been able to make the test case more specific
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Closed] (JENA-308) Index corruption after killing process
Posted by "Andy Seaborne (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/JENA-308?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Andy Seaborne closed JENA-308.
------------------------------
> Index corruption after killing process
> --------------------------------------
>
> Key: JENA-308
> URL: https://issues.apache.org/jira/browse/JENA-308
> Project: Apache Jena
> Issue Type: Bug
> Components: TDB
> Affects Versions: TDB 0.9.1, TDB 0.9.4
> Environment: Windows 2008/7 (64 bit) both on IBM and SUN JVM 6. Direct mode
> Reporter: Simon Helsen
> Assignee: Andy Seaborne
> Priority: Critical
> Fix For: TDB 0.9.4
>
> Attachments: journalRecoveryTest.zip
>
>
> We are faced with a series of possible exceptions which may or may not occur after an active tdb store is killed. The most common exception has the form:
> Exception in thread "Thread-3" com.hp.hpl.jena.tdb.transaction.TDBTransactionException: Feaild to read block checksum.
> at com.hp.hpl.jena.tdb.transaction.Journal._read(Journal.java:238)
> at com.hp.hpl.jena.tdb.transaction.Journal._readJournal(Journal.java:197)
> at com.hp.hpl.jena.tdb.transaction.Journal.access$1(Journal.java:192)
> at com.hp.hpl.jena.tdb.transaction.Journal$IteratorEntries.moveToNext(Journal.java:267)
> at com.hp.hpl.jena.tdb.transaction.Journal$IteratorEntries.moveToNext(Journal.java:1)
> at org.openjena.atlas.iterator.IteratorSlotted.hasNext(IteratorSlotted.java:67)
> at com.hp.hpl.jena.tdb.transaction.JournalControl.scanForCommit(JournalControl.java:148)
> at com.hp.hpl.jena.tdb.transaction.JournalControl.recoverFromJournal(JournalControl.java:127)
> at com.hp.hpl.jena.tdb.StoreConnection._makeAndCache(StoreConnection.java:234)
> at com.hp.hpl.jena.tdb.StoreConnection.make(StoreConnection.java:214)
> at com.hp.hpl.jena.tdb.transaction.DatasetGraphTransaction.<init>(DatasetGraphTransaction.java:76)
> at com.hp.hpl.jena.tdb.sys.TDBMaker._create(TDBMaker.java:57)
> at com.hp.hpl.jena.tdb.sys.TDBMaker.createDatasetGraphTransaction(TDBMaker.java:45)
> at com.hp.hpl.jena.tdb.TDBFactory._createDatasetGraph(TDBFactory.java:104)
> at com.hp.hpl.jena.tdb.TDBFactory.createDatasetGraph(TDBFactory.java:73)
> at com.hp.hpl.jena.tdb.TDBFactory.createDataset(TDBFactory.java:52)
> Other exceptions are possible (see comments below). I have been able to produce a standalone java program/test case which simulates a process kill and is able to produce the exception after a non-deterministic time. It runs an iteration until it hits one of the possible problems. Sometimes, an exception hits after just 2 iterations, other times, it can take 40 runs or more before hitting a problem. Because I don't know what the cause is here, I have not been able to make the test case more specific
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (JENA-308) Index corruption after killing
process
Posted by "Simon Helsen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/JENA-308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13445313#comment-13445313 ]
Simon Helsen commented on JENA-308:
-----------------------------------
Thanks Andy, I will certainly test it out. I rarely had to run it more than 40 iterations to hit any of the indicated exceptions, but yes, it is not ideal test code.
> Index corruption after killing process
> --------------------------------------
>
> Key: JENA-308
> URL: https://issues.apache.org/jira/browse/JENA-308
> Project: Apache Jena
> Issue Type: Bug
> Components: TDB
> Affects Versions: TDB 0.9.1, TDB 0.9.4
> Environment: Windows 2008/7 (64 bit) both on IBM and SUN JVM 6. Direct mode
> Reporter: Simon Helsen
> Assignee: Andy Seaborne
> Priority: Critical
> Fix For: TDB 0.9.4
>
> Attachments: journalRecoveryTest.zip
>
>
> We are faced with a series of possible exceptions which may or may not occur after an active tdb store is killed. The most common exception has the form:
> Exception in thread "Thread-3" com.hp.hpl.jena.tdb.transaction.TDBTransactionException: Feaild to read block checksum.
> at com.hp.hpl.jena.tdb.transaction.Journal._read(Journal.java:238)
> at com.hp.hpl.jena.tdb.transaction.Journal._readJournal(Journal.java:197)
> at com.hp.hpl.jena.tdb.transaction.Journal.access$1(Journal.java:192)
> at com.hp.hpl.jena.tdb.transaction.Journal$IteratorEntries.moveToNext(Journal.java:267)
> at com.hp.hpl.jena.tdb.transaction.Journal$IteratorEntries.moveToNext(Journal.java:1)
> at org.openjena.atlas.iterator.IteratorSlotted.hasNext(IteratorSlotted.java:67)
> at com.hp.hpl.jena.tdb.transaction.JournalControl.scanForCommit(JournalControl.java:148)
> at com.hp.hpl.jena.tdb.transaction.JournalControl.recoverFromJournal(JournalControl.java:127)
> at com.hp.hpl.jena.tdb.StoreConnection._makeAndCache(StoreConnection.java:234)
> at com.hp.hpl.jena.tdb.StoreConnection.make(StoreConnection.java:214)
> at com.hp.hpl.jena.tdb.transaction.DatasetGraphTransaction.<init>(DatasetGraphTransaction.java:76)
> at com.hp.hpl.jena.tdb.sys.TDBMaker._create(TDBMaker.java:57)
> at com.hp.hpl.jena.tdb.sys.TDBMaker.createDatasetGraphTransaction(TDBMaker.java:45)
> at com.hp.hpl.jena.tdb.TDBFactory._createDatasetGraph(TDBFactory.java:104)
> at com.hp.hpl.jena.tdb.TDBFactory.createDatasetGraph(TDBFactory.java:73)
> at com.hp.hpl.jena.tdb.TDBFactory.createDataset(TDBFactory.java:52)
> Other exceptions are possible (see comments below). I have been able to produce a standalone java program/test case which simulates a process kill and is able to produce the exception after a non-deterministic time. It runs an iteration until it hits one of the possible problems. Sometimes, an exception hits after just 2 iterations, other times, it can take 40 runs or more before hitting a problem. Because I don't know what the cause is here, I have not been able to make the test case more specific
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (JENA-308) Index corruption after killing process
Posted by "Andy Seaborne (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/JENA-308?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Andy Seaborne resolved JENA-308.
--------------------------------
Resolution: Fixed
Fix Version/s: TDB 0.9.4
Assignee: Andy Seaborne
> Index corruption after killing process
> --------------------------------------
>
> Key: JENA-308
> URL: https://issues.apache.org/jira/browse/JENA-308
> Project: Apache Jena
> Issue Type: Bug
> Components: TDB
> Affects Versions: TDB 0.9.1, TDB 0.9.4
> Environment: Windows 2008/7 (64 bit) both on IBM and SUN JVM 6. Direct mode
> Reporter: Simon Helsen
> Assignee: Andy Seaborne
> Priority: Critical
> Fix For: TDB 0.9.4
>
> Attachments: journalRecoveryTest.zip
>
>
> We are faced with a series of possible exceptions which may or may not occur after an active tdb store is killed. The most common exception has the form:
> Exception in thread "Thread-3" com.hp.hpl.jena.tdb.transaction.TDBTransactionException: Feaild to read block checksum.
> at com.hp.hpl.jena.tdb.transaction.Journal._read(Journal.java:238)
> at com.hp.hpl.jena.tdb.transaction.Journal._readJournal(Journal.java:197)
> at com.hp.hpl.jena.tdb.transaction.Journal.access$1(Journal.java:192)
> at com.hp.hpl.jena.tdb.transaction.Journal$IteratorEntries.moveToNext(Journal.java:267)
> at com.hp.hpl.jena.tdb.transaction.Journal$IteratorEntries.moveToNext(Journal.java:1)
> at org.openjena.atlas.iterator.IteratorSlotted.hasNext(IteratorSlotted.java:67)
> at com.hp.hpl.jena.tdb.transaction.JournalControl.scanForCommit(JournalControl.java:148)
> at com.hp.hpl.jena.tdb.transaction.JournalControl.recoverFromJournal(JournalControl.java:127)
> at com.hp.hpl.jena.tdb.StoreConnection._makeAndCache(StoreConnection.java:234)
> at com.hp.hpl.jena.tdb.StoreConnection.make(StoreConnection.java:214)
> at com.hp.hpl.jena.tdb.transaction.DatasetGraphTransaction.<init>(DatasetGraphTransaction.java:76)
> at com.hp.hpl.jena.tdb.sys.TDBMaker._create(TDBMaker.java:57)
> at com.hp.hpl.jena.tdb.sys.TDBMaker.createDatasetGraphTransaction(TDBMaker.java:45)
> at com.hp.hpl.jena.tdb.TDBFactory._createDatasetGraph(TDBFactory.java:104)
> at com.hp.hpl.jena.tdb.TDBFactory.createDatasetGraph(TDBFactory.java:73)
> at com.hp.hpl.jena.tdb.TDBFactory.createDataset(TDBFactory.java:52)
> Other exceptions are possible (see comments below). I have been able to produce a standalone java program/test case which simulates a process kill and is able to produce the exception after a non-deterministic time. It runs an iteration until it hits one of the possible problems. Sometimes, an exception hits after just 2 iterations, other times, it can take 40 runs or more before hitting a problem. Because I don't know what the cause is here, I have not been able to make the test case more specific
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (JENA-308) Index corruption after killing
process
Posted by "Simon Helsen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/JENA-308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13444489#comment-13444489 ]
Simon Helsen commented on JENA-308:
-----------------------------------
in 0.9.1 (not yet seen in 0.9.4), we also sometimes see:
1346278896399 : Iteration 10
1346278896400 : create dataset
Exception in thread "Thread-11" com.hp.hpl.jena.tdb.base.block.BlockException: BlockAccessBase: Bounds exception: C:\Temp\testIndexLocation\GSPO.dat: (9,9)
at com.hp.hpl.jena.tdb.base.file.BlockAccessBase.check(BlockAccessBase.java:107)
at com.hp.hpl.jena.tdb.base.file.BlockAccessBase.check(BlockAccessBase.java:114)
at com.hp.hpl.jena.tdb.base.file.BlockAccessDirect.write(BlockAccessDirect.java:85)
at com.hp.hpl.jena.tdb.base.file.BlockAccessDirect.overwrite(BlockAccessDirect.java:104)
at com.hp.hpl.jena.tdb.base.block.BlockMgrFileAccess.overwrite(BlockMgrFileAccess.java:102)
at com.hp.hpl.jena.tdb.base.block.BlockMgrWrapper.overwrite(BlockMgrWrapper.java:89)
at com.hp.hpl.jena.tdb.base.block.BlockMgrSync.overwrite(BlockMgrSync.java:90)
at com.hp.hpl.jena.tdb.base.block.BlockMgrCache.overwrite(BlockMgrCache.java:206)
at com.hp.hpl.jena.tdb.transaction.JournalControl.replay(JournalControl.java:305)
at com.hp.hpl.jena.tdb.transaction.JournalControl.recoverSegment(JournalControl.java:175)
at com.hp.hpl.jena.tdb.transaction.JournalControl.recoverFromJournal(JournalControl.java:128)
at com.hp.hpl.jena.tdb.StoreConnection._makeAndCache(StoreConnection.java:231)
at com.hp.hpl.jena.tdb.StoreConnection.make(StoreConnection.java:190)
at com.hp.hpl.jena.tdb.transaction.DatasetGraphTransaction.<init>(DatasetGraphTransaction.java:63)
at com.hp.hpl.jena.tdb.sys.TDBMakerTxn._create(TDBMakerTxn.java:49)
at com.hp.hpl.jena.tdb.sys.TDBMakerTxn.createDatasetGraph(TDBMakerTxn.java:37)
at com.hp.hpl.jena.tdb.TDBFactory._createDatasetGraph(TDBFactory.java:104)
at com.hp.hpl.jena.tdb.TDBFactory.createDatasetGraph(TDBFactory.java:73)
at com.hp.hpl.jena.tdb.TDBFactory.createDataset(TDBFactory.java:52)
at com.ibm.jena.test.JournalRecoveryTest$TestRunnable.run(JournalRecoveryTest.java:161)
at java.lang.Thread.run(Thread.java:662)
This was the original exception which triggered my investigation. It was observed by one of our clients after a windows 2008 server machine was rebooted killing the Java process. It is unclear if this issues has disappeared in 0.9.4
> Index corruption after killing process
> --------------------------------------
>
> Key: JENA-308
> URL: https://issues.apache.org/jira/browse/JENA-308
> Project: Apache Jena
> Issue Type: Bug
> Components: TDB
> Affects Versions: TDB 0.9.1, TDB 0.9.4
> Environment: Windows 2008/7 (64 bit) both on IBM and SUN JVM 6. Direct mode
> Reporter: Simon Helsen
> Priority: Critical
>
> We are faced with a series of possible exceptions which may or may not occur after an active tdb store is killed. The most common exception has the form:
> Exception in thread "Thread-3" com.hp.hpl.jena.tdb.transaction.TDBTransactionException: Feaild to read block checksum.
> at com.hp.hpl.jena.tdb.transaction.Journal._read(Journal.java:238)
> at com.hp.hpl.jena.tdb.transaction.Journal._readJournal(Journal.java:197)
> at com.hp.hpl.jena.tdb.transaction.Journal.access$1(Journal.java:192)
> at com.hp.hpl.jena.tdb.transaction.Journal$IteratorEntries.moveToNext(Journal.java:267)
> at com.hp.hpl.jena.tdb.transaction.Journal$IteratorEntries.moveToNext(Journal.java:1)
> at org.openjena.atlas.iterator.IteratorSlotted.hasNext(IteratorSlotted.java:67)
> at com.hp.hpl.jena.tdb.transaction.JournalControl.scanForCommit(JournalControl.java:148)
> at com.hp.hpl.jena.tdb.transaction.JournalControl.recoverFromJournal(JournalControl.java:127)
> at com.hp.hpl.jena.tdb.StoreConnection._makeAndCache(StoreConnection.java:234)
> at com.hp.hpl.jena.tdb.StoreConnection.make(StoreConnection.java:214)
> at com.hp.hpl.jena.tdb.transaction.DatasetGraphTransaction.<init>(DatasetGraphTransaction.java:76)
> at com.hp.hpl.jena.tdb.sys.TDBMaker._create(TDBMaker.java:57)
> at com.hp.hpl.jena.tdb.sys.TDBMaker.createDatasetGraphTransaction(TDBMaker.java:45)
> at com.hp.hpl.jena.tdb.TDBFactory._createDatasetGraph(TDBFactory.java:104)
> at com.hp.hpl.jena.tdb.TDBFactory.createDatasetGraph(TDBFactory.java:73)
> at com.hp.hpl.jena.tdb.TDBFactory.createDataset(TDBFactory.java:52)
> Other exceptions are possible (see comments below). I have been able to produce a standalone java program/test case which simulates a process kill and is able to produce the exception after a non-deterministic time. It runs an iteration until it hits one of the possible problems. Sometimes, an exception hits after just 2 iterations, other times, it can take 40 runs or more before hitting a problem. Because I don't know what the cause is here, I have not been able to make the test case more specific
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira