You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@asterixdb.apache.org by "Ildar Absalyamov (JIRA)" <ji...@apache.org> on 2017/12/04 05:48:01 UTC
[jira] [Commented] (ASTERIXDB-1708) Rollback failure at scale
[ https://issues.apache.org/jira/browse/ASTERIXDB-1708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16276320#comment-16276320 ]
Ildar Absalyamov commented on ASTERIXDB-1708:
---------------------------------------------
It is still an issue. Just encountered it on my 4-node cluster setup:
{code}
rg.apache.asterix.common.exceptions.ACIDException: java.io.IOException: Log file with id(1) was not found. Requested LSN: 3969793214
at org.apache.asterix.transaction.management.service.logging.LogReader.getLogFile(LogReader.java:293)
at org.apache.asterix.transaction.management.service.logging.LogReader.initializeScan(LogReader.java:76)
at org.apache.asterix.app.nc.RecoveryManager.rollbackTransaction(RecoveryManager.java:542)
at org.apache.asterix.transaction.management.service.transaction.TransactionManager.abortTransaction(TransactionManager.java:65)
at org.apache.asterix.transaction.management.service.transaction.TransactionManager.completedTransaction(TransactionManager.java:132)
at org.apache.asterix.runtime.job.listener.MultiTransactionJobletEventListenerFactory$1.jobletFinish(MultiTransactionJobletEventListenerFactory.java:62)
at org.apache.hyracks.control.nc.Joblet.performCleanup(Joblet.java:317)
at org.apache.hyracks.control.nc.Joblet.removeTask(Joblet.java:152)
at org.apache.hyracks.control.nc.work.NotifyTaskFailureWork.run(NotifyTaskFailureWork.java:63)
at org.apache.hyracks.control.common.work.WorkQueue$WorkerThread.run(WorkQueue.java:127)
Caused by: java.io.IOException: Log file with id(1) was not found. Requested LSN: 3969793214
at org.apache.asterix.transaction.management.service.logging.LogManager.getLogFile(LogManager.java:570)
at org.apache.asterix.transaction.management.service.logging.LogReader.getLogFile(LogReader.java:290)
... 9 more
Exception in thread "Worker:a1_node2" java.lang.Error: org.apache.asterix.common.exceptions.ACIDException: Could not complete rollback! System is in an inconsistent state
at org.apache.asterix.runtime.job.listener.MultiTransactionJobletEventListenerFactory$1.jobletFinish(MultiTransactionJobletEventListenerFactory.java:66)
at org.apache.hyracks.control.nc.Joblet.performCleanup(Joblet.java:317)
at org.apache.hyracks.control.nc.Joblet.removeTask(Joblet.java:152)
at org.apache.hyracks.control.nc.work.NotifyTaskFailureWork.run(NotifyTaskFailureWork.java:63)
at org.apache.hyracks.control.common.work.WorkQueue$WorkerThread.run(WorkQueue.java:127)
Caused by: org.apache.asterix.common.exceptions.ACIDException: Could not complete rollback! System is in an inconsistent state
at org.apache.asterix.transaction.management.service.transaction.TransactionManager.abortTransaction(TransactionManager.java:73)
at org.apache.asterix.transaction.management.service.transaction.TransactionManager.completedTransaction(TransactionManager.java:132)
at org.apache.asterix.runtime.job.listener.MultiTransactionJobletEventListenerFactory$1.jobletFinish(MultiTransactionJobletEventListenerFactory.java:62)
... 4 more
Caused by: org.apache.asterix.common.exceptions.ACIDException: java.io.IOException: Log file with id(1) was not found. Requested LSN: 3969793214
at org.apache.asterix.transaction.management.service.logging.LogReader.getLogFile(LogReader.java:293)
at org.apache.asterix.transaction.management.service.logging.LogReader.initializeScan(LogReader.java:76)
at org.apache.asterix.app.nc.RecoveryManager.rollbackTransaction(RecoveryManager.java:542)
at org.apache.asterix.transaction.management.service.transaction.TransactionManager.abortTransaction(TransactionManager.java:65)
... 6 more
Caused by: java.io.IOException: Log file with id(1) was not found. Requested LSN: 3969793214
at org.apache.asterix.transaction.management.service.logging.LogManager.getLogFile(LogManager.java:570)
at org.apache.asterix.transaction.management.service.logging.LogReader.getLogFile(LogReader.java:290)
... 9 more
{code}
> Rollback failure at scale
> -------------------------
>
> Key: ASTERIXDB-1708
> URL: https://issues.apache.org/jira/browse/ASTERIXDB-1708
> Project: Apache AsterixDB
> Issue Type: Bug
> Reporter: Ian Maxon
> Assignee: Ian Maxon
>
> Seems that transaction rollback can fail at certain points. This happened with the same file ID on a cluster of 5 nodes which is an interesting coincidence.
> org.apache.asterix.common.exceptions.ACIDException: java.io.IOException: Log file with id(37) was not found. Requested LSN: 80892085216
> at org.apache.asterix.transaction.management.service.logging.LogReader.getLogFile(LogReader.java:293)
> at org.apache.asterix.transaction.management.service.logging.LogReader.initializeScan(LogReader.java:76)
> at org.apache.asterix.transaction.management.service.recovery.RecoveryManager.rollbackTransaction(RecoveryManager.java:734)
> at org.apache.asterix.transaction.management.service.transaction.TransactionManager.abortTransaction(TransactionManager.java:64)
> at org.apache.asterix.transaction.management.service.transaction.TransactionManager.completedTransaction(TransactionManager.java:130)
> at org.apache.asterix.runtime.job.listener.JobEventListenerFactory$1.jobletFinish(JobEventListenerFactory.java:58)
> at org.apache.hyracks.control.nc.Joblet.performCleanup(Joblet.java:318)
> at org.apache.hyracks.control.nc.Joblet.cleanup(Joblet.java:310)
> at org.apache.hyracks.control.nc.work.CleanupJobletWork.run(CleanupJobletWork.java:67)
> at org.apache.hyracks.control.common.work.WorkQueue$WorkerThread.run(WorkQueue.java:127)
> Caused by: java.io.IOException: Log file with id(37) was not found. Requested LSN: 80892085216
> at org.apache.asterix.transaction.management.service.logging.LogManager.getLogFile(LogManager.java:544)
> at org.apache.asterix.transaction.management.service.logging.LogReader.getLogFile(LogReader.java:290)
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)