You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@asterixdb.apache.org by "Michael Blow (JIRA)" <ji...@apache.org> on 2016/06/23 14:37:16 UTC
[jira] [Resolved] (ASTERIXDB-1474) Cannot query a dataset with big
object after restart
[ https://issues.apache.org/jira/browse/ASTERIXDB-1474?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Michael Blow resolved ASTERIXDB-1474.
-------------------------------------
Resolution: Fixed
Change-Id: I2f47c551a053e193519a4337c338934f3e171afd
Reviewed-on: https://asterix-gerrit.ics.uci.edu/920
Tested-by: Jenkins <je...@fulliautomatix.ics.uci.edu>
Reviewed-by: abdullah alamoudi <ba...@gmail.com>
Project: http://git-wip-us.apache.org/repos/asf/asterixdb/repo
Commit: http://git-wip-us.apache.org/repos/asf/asterixdb/commit/171ff2ff
Tree: http://git-wip-us.apache.org/repos/asf/asterixdb/tree/171ff2ff
Diff: http://git-wip-us.apache.org/repos/asf/asterixdb/diff/171ff2ff
Branch: refs/heads/master
Commit: 171ff2ffedb9b6c8ffea8c5e0087277c0a9946c0
Parents: 809be1b
Author: Michael Blow <mb...@apache.org>
Authored: Mon Jun 13 17:43:39 2016 -0400
Committer: Michael Blow <mi...@couchbase.com>
Committed: Mon Jun 13 19:31:24 2016 -0700
> Cannot query a dataset with big object after restart
> ----------------------------------------------------
>
> Key: ASTERIXDB-1474
> URL: https://issues.apache.org/jira/browse/ASTERIXDB-1474
> Project: Apache AsterixDB
> Issue Type: Bug
> Components: Storage
> Reporter: Murtadha Hubail
> Assignee: Michael Blow
> Priority: Critical
>
> If you try to query a dataset which contains a big object record after an instance restart (or simply after the dataset gets deactivated and activated again), you get the following exception
> {noformat}
> attempt to unlock read lock, not locked by current thread [IllegalMonitorStateException]
> {noformat}
> The complete stacktrace is below
> {noformat}
> java.lang.ArrayIndexOutOfBoundsException: 1948279919
> at org.apache.hyracks.storage.am.lsm.btree.tuples.LSMBTreeTupleReference.isAntimatter(LSMBTreeTupleReference.java:88)
> at org.apache.hyracks.storage.am.lsm.btree.tuples.LSMBTreeTupleReference.resetByTupleOffset(LSMBTreeTupleReference.java:60)
> at org.apache.hyracks.storage.am.lsm.btree.tuples.LSMBTreeTupleReference.resetByTupleIndex(LSMBTreeTupleReference.java:75)
> at org.apache.hyracks.storage.am.btree.frames.OrderedSlotManager.findTupleIndex(OrderedSlotManager.java:44)
> at org.apache.hyracks.storage.am.btree.frames.BTreeNSMLeafFrame.findTupleIndex(BTreeNSMLeafFrame.java:388)
> at org.apache.hyracks.storage.am.btree.impls.BTreeRangeSearchCursor.getLowKeyIndex(BTreeRangeSearchCursor.java:219)
> at org.apache.hyracks.storage.am.btree.impls.BTreeRangeSearchCursor.open(BTreeRangeSearchCursor.java:291)
> at org.apache.hyracks.storage.am.btree.impls.BTree.performOp(BTree.java:794)
> at org.apache.hyracks.storage.am.btree.impls.BTree.performOp(BTree.java:679)
> at org.apache.hyracks.storage.am.btree.impls.BTree.search(BTree.java:206)
> at org.apache.hyracks.storage.am.btree.impls.BTree.access$6(BTree.java:190)
> at org.apache.hyracks.storage.am.btree.impls.BTree$BTreeAccessor.search(BTree.java:973)
> at org.apache.hyracks.storage.am.lsm.btree.impls.LSMBTreePointSearchCursor.hasNext(LSMBTreePointSearchCursor.java:74)
> at org.apache.hyracks.storage.am.lsm.btree.impls.LSMBTreeSearchCursor.hasNext(LSMBTreeSearchCursor.java:86)
> at org.apache.hyracks.storage.am.common.dataflow.IndexSearchOperatorNodePushable.writeSearchResults(IndexSearchOperatorNodePushable.java:149)
> at org.apache.hyracks.storage.am.common.dataflow.IndexSearchOperatorNodePushable.nextFrame(IndexSearchOperatorNodePushable.java:184)
> at org.apache.hyracks.dataflow.common.comm.io.AbstractFrameAppender.write(AbstractFrameAppender.java:93)
> at org.apache.hyracks.algebricks.runtime.operators.base.AbstractOneInputOneOutputOneFramePushRuntime.flushAndReset(AbstractOneInputOneOutputOneFramePushRuntime.java:63)
> at org.apache.hyracks.algebricks.runtime.operators.base.AbstractOneInputOneOutputOneFramePushRuntime.flushIfNotFailed(AbstractOneInputOneOutputOneFramePushRuntime.java:69)
> at org.apache.hyracks.algebricks.runtime.operators.base.AbstractOneInputOneOutputOneFramePushRuntime.close(AbstractOneInputOneOutputOneFramePushRuntime.java:55)
> at org.apache.hyracks.algebricks.runtime.operators.std.AssignRuntimeFactory$1.close(AssignRuntimeFactory.java:122)
> at org.apache.hyracks.algebricks.runtime.operators.std.EmptyTupleSourceRuntimeFactory$1.close(EmptyTupleSourceRuntimeFactory.java:60)
> at org.apache.hyracks.algebricks.runtime.operators.meta.AlgebricksMetaOperatorDescriptor$1.initialize(AlgebricksMetaOperatorDescriptor.java:116)
> at org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePushable.lambda$initialize$0(SuperActivityOperatorNodePushable.java:83)
> at org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePushable$1.call(SuperActivityOperatorNodePushable.java:205)
> at org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePushable$1.call(SuperActivityOperatorNodePushable.java:202)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> org.apache.hyracks.api.exceptions.HyracksDataException: java.util.concurrent.ExecutionException: org.apache.hyracks.api.exceptions.HyracksDataException: java.lang.NullPointerException
> at org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePushable.runInParallel(SuperActivityOperatorNodePushable.java:218)
> at org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePushable.initialize(SuperActivityOperatorNodePushable.java:83)
> at org.apache.hyracks.control.nc.Task.run(Task.java:263)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.util.concurrent.ExecutionException: org.apache.hyracks.api.exceptions.HyracksDataException: java.lang.NullPointerException
> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> at java.util.concurrent.FutureTask.get(FutureTask.java:192)
> at org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePushable.runInParallel(SuperActivityOperatorNodePushable.java:212)
> ... 5 more
> Caused by: org.apache.hyracks.api.exceptions.HyracksDataException: java.lang.NullPointerException
> at org.apache.hyracks.storage.am.common.dataflow.IndexSearchOperatorNodePushable.nextFrame(IndexSearchOperatorNodePushable.java:187)
> at org.apache.hyracks.dataflow.common.comm.io.AbstractFrameAppender.write(AbstractFrameAppender.java:93)
> at org.apache.hyracks.algebricks.runtime.operators.base.AbstractOneInputOneOutputOneFramePushRuntime.flushAndReset(AbstractOneInputOneOutputOneFramePushRuntime.java:63)
> at org.apache.hyracks.algebricks.runtime.operators.base.AbstractOneInputOneOutputOneFramePushRuntime.flushIfNotFailed(AbstractOneInputOneOutputOneFramePushRuntime.java:69)
> at org.apache.hyracks.algebricks.runtime.operators.base.AbstractOneInputOneOutputOneFramePushRuntime.close(AbstractOneInputOneOutputOneFramePushRuntime.java:55)
> at org.apache.hyracks.algebricks.runtime.operators.std.AssignRuntimeFactory$1.close(AssignRuntimeFactory.java:122)
> at org.apache.hyracks.algebricks.runtime.operators.std.EmptyTupleSourceRuntimeFactory$1.close(EmptyTupleSourceRuntimeFactory.java:60)
> at org.apache.hyracks.algebricks.runtime.operators.meta.AlgebricksMetaOperatorDescriptor$1.initialize(AlgebricksMetaOperatorDescriptor.java:116)
> at org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePushable.lambda$initialize$0(SuperActivityOperatorNodePushable.java:83)
> at org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePushable$1.call(SuperActivityOperatorNodePushable.java:205)
> at org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePushable$1.call(SuperActivityOperatorNodePushable.java:202)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> ... 3 more
> Caused by: java.lang.NullPointerException
> at org.apache.hyracks.storage.am.bloomfilter.impls.MurmurHash128Bit.hash3_x64_128(MurmurHash128Bit.java:49)
> at org.apache.hyracks.storage.am.bloomfilter.impls.BloomFilter.contains(BloomFilter.java:93)
> at org.apache.hyracks.storage.am.lsm.common.impls.BloomFilterAwareBTreePointSearchCursor.hasNext(BloomFilterAwareBTreePointSearchCursor.java:39)
> at org.apache.hyracks.storage.am.lsm.btree.impls.LSMBTreePointSearchCursor.hasNext(LSMBTreePointSearchCursor.java:75)
> at org.apache.hyracks.storage.am.lsm.btree.impls.LSMBTreeSearchCursor.hasNext(LSMBTreeSearchCursor.java:86)
> at org.apache.hyracks.storage.am.common.dataflow.IndexSearchOperatorNodePushable.writeSearchResults(IndexSearchOperatorNodePushable.java:149)
> at org.apache.hyracks.storage.am.common.dataflow.IndexSearchOperatorNodePushable.nextFrame(IndexSearchOperatorNodePushable.java:184)
> ... 14 more
> org.apache.hyracks.api.exceptions.HyracksDataException: java.util.concurrent.ExecutionException: org.apache.hyracks.api.exceptions.HyracksDataException: java.lang.IllegalMonitorStateException: attempt to unlock read lock, not locked by current thread
> at org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePushable.runInParallel(SuperActivityOperatorNodePushable.java:218)
> at org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePushable.initialize(SuperActivityOperatorNodePushable.java:83)
> at org.apache.hyracks.control.nc.Task.run(Task.java:263)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.util.concurrent.ExecutionException: org.apache.hyracks.api.exceptions.HyracksDataException: java.lang.IllegalMonitorStateException: attempt to unlock read lock, not locked by current thread
> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> at java.util.concurrent.FutureTask.get(FutureTask.java:192)
> at org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePushable.runInParallel(SuperActivityOperatorNodePushable.java:212)
> ... 5 more
> Caused by: org.apache.hyracks.api.exceptions.HyracksDataException: java.lang.IllegalMonitorStateException: attempt to unlock read lock, not locked by current thread
> at org.apache.hyracks.storage.am.common.dataflow.IndexSearchOperatorNodePushable.close(IndexSearchOperatorNodePushable.java:211)
> at org.apache.hyracks.algebricks.runtime.operators.base.AbstractOneInputOneOutputOneFramePushRuntime.close(AbstractOneInputOneOutputOneFramePushRuntime.java:57)
> at org.apache.hyracks.algebricks.runtime.operators.std.AssignRuntimeFactory$1.close(AssignRuntimeFactory.java:122)
> at org.apache.hyracks.algebricks.runtime.operators.std.EmptyTupleSourceRuntimeFactory$1.close(EmptyTupleSourceRuntimeFactory.java:60)
> at org.apache.hyracks.algebricks.runtime.operators.meta.AlgebricksMetaOperatorDescriptor$1.initialize(AlgebricksMetaOperatorDescriptor.java:116)
> at org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePushable.lambda$initialize$0(SuperActivityOperatorNodePushable.java:83)
> at org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePushable$1.call(SuperActivityOperatorNodePushable.java:205)
> at org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePushable$1.call(SuperActivityOperatorNodePushable.java:202)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> ... 3 more
> Caused by: java.lang.IllegalMonitorStateException: attempt to unlock read lock, not locked by current thread
> at java.util.concurrent.locks.ReentrantReadWriteLock$Sync.unmatchedUnlockException(ReentrantReadWriteLock.java:444)
> at java.util.concurrent.locks.ReentrantReadWriteLock$Sync.tryReleaseShared(ReentrantReadWriteLock.java:428)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer.releaseShared(AbstractQueuedSynchronizer.java:1341)
> at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.unlock(ReentrantReadWriteLock.java:881)
> at org.apache.hyracks.storage.common.buffercache.CachedPage.releaseReadLatch(CachedPage.java:121)
> at org.apache.hyracks.storage.am.btree.impls.BTreeRangeSearchCursor.close(BTreeRangeSearchCursor.java:91)
> at org.apache.hyracks.storage.am.lsm.btree.impls.LSMBTreePointSearchCursor.close(LSMBTreePointSearchCursor.java:211)
> at org.apache.hyracks.storage.am.lsm.btree.impls.LSMBTreeSearchCursor.close(LSMBTreeSearchCursor.java:97)
> at org.apache.hyracks.storage.am.common.dataflow.IndexSearchOperatorNodePushable.close(IndexSearchOperatorNodePushable.java:208)
> ... 11 more
> Jun 07, 2016 4:20:13 AM org.apache.hyracks.control.cc.work.JobCleanupWork run
> WARNING: Ignoring duplicate cleanup for JobRun with id: JID:0
> Jun 07, 2016 4:20:13 AM org.apache.hyracks.control.nc.Joblet close
> WARNING: Freeing leaked 327680 bytes
> Jun 07, 2016 4:20:13 AM org.apache.hyracks.control.nc.Joblet close
> WARNING: Freeing leaked 327680 bytes
> org.apache.hyracks.api.exceptions.HyracksException: Job failed on account of:
> org.apache.hyracks.api.exceptions.HyracksDataException: java.util.concurrent.ExecutionException: org.apache.hyracks.api.exceptions.HyracksDataException: java.lang.IllegalMonitorStateException: attempt to unlock read lock, not locked by current thread
> at org.apache.hyracks.control.cc.job.JobRun.waitForCompletion(JobRun.java:212)
> at org.apache.hyracks.control.cc.work.WaitForJobCompletionWork$1.run(WaitForJobCompletionWork.java:48)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.hyracks.api.exceptions.HyracksDataException: org.apache.hyracks.api.exceptions.HyracksDataException: java.util.concurrent.ExecutionException: org.apache.hyracks.api.exceptions.HyracksDataException: java.lang.IllegalMonitorStateException: attempt to unlock read lock, not locked by current thread
> at org.apache.hyracks.control.common.utils.ExceptionUtils.setNodeIds(ExceptionUtils.java:45)
> at org.apache.hyracks.control.nc.Task.run(Task.java:319)
> ... 3 more
> Caused by: org.apache.hyracks.api.exceptions.HyracksDataException: java.util.concurrent.ExecutionException: org.apache.hyracks.api.exceptions.HyracksDataException: java.lang.IllegalMonitorStateException: attempt to unlock read lock, not locked by current thread
> at org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePushable.runInParallel(SuperActivityOperatorNodePushable.java:218)
> at org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePushable.initialize(SuperActivityOperatorNodePushable.java:83)
> at org.apache.hyracks.control.nc.Task.run(Task.java:263)
> ... 3 more
> Caused by: java.util.concurrent.ExecutionException: org.apache.hyracks.api.exceptions.HyracksDataException: java.lang.IllegalMonitorStateException: attempt to unlock read lock, not locked by current thread
> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> at java.util.concurrent.FutureTask.get(FutureTask.java:192)
> at org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePushable.runInParallel(SuperActivityOperatorNodePushable.java:212)
> ... 5 more
> Caused by: org.apache.hyracks.api.exceptions.HyracksDataException: java.lang.IllegalMonitorStateException: attempt to unlock read lock, not locked by current thread
> at org.apache.hyracks.storage.am.common.dataflow.IndexSearchOperatorNodePushable.close(IndexSearchOperatorNodePushable.java:211)
> at org.apache.hyracks.algebricks.runtime.operators.base.AbstractOneInputOneOutputOneFramePushRuntime.close(AbstractOneInputOneOutputOneFramePushRuntime.java:57)
> at org.apache.hyracks.algebricks.runtime.operators.std.AssignRuntimeFactory$1.close(AssignRuntimeFactory.java:122)
> at org.apache.hyracks.algebricks.runtime.operators.std.EmptyTupleSourceRuntimeFactory$1.close(EmptyTupleSourceRuntimeFactory.java:60)
> at org.apache.hyracks.algebricks.runtime.operators.meta.AlgebricksMetaOperatorDescriptor$1.initialize(AlgebricksMetaOperatorDescriptor.java:116)
> at org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePushable.lambda$initialize$0(SuperActivityOperatorNodePushable.java:83)
> at org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePushable$1.call(SuperActivityOperatorNodePushable.java:205)
> at org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePushable$1.call(SuperActivityOperatorNodePushable.java:202)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> ... 3 more
> Caused by: java.lang.IllegalMonitorStateException: attempt to unlock read lock, not locked by current thread
> at java.util.concurrent.locks.ReentrantReadWriteLock$Sync.unmatchedUnlockException(ReentrantReadWriteLock.java:444)
> at java.util.concurrent.locks.ReentrantReadWriteLock$Sync.tryReleaseShared(ReentrantReadWriteLock.java:428)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer.releaseShared(AbstractQueuedSynchronizer.java:1341)
> at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.unlock(ReentrantReadWriteLock.java:881)
> at org.apache.hyracks.storage.common.buffercache.CachedPage.releaseReadLatch(CachedPage.java:121)
> at org.apache.hyracks.storage.am.btree.impls.BTreeRangeSearchCursor.close(BTreeRangeSearchCursor.java:91)
> at org.apache.hyracks.storage.am.lsm.btree.impls.LSMBTreePointSearchCursor.close(LSMBTreePointSearchCursor.java:211)
> at org.apache.hyracks.storage.am.lsm.btree.impls.LSMBTreeSearchCursor.close(LSMBTreeSearchCursor.java:97)
> at org.apache.hyracks.storage.am.common.dataflow.IndexSearchOperatorNodePushable.close(IndexSearchOperatorNodePushable.java:208)
> ... 11 more
> org.apache.hyracks.api.exceptions.HyracksException: Job failed on account of:
> org.apache.hyracks.api.exceptions.HyracksDataException: java.util.concurrent.ExecutionException: org.apache.hyracks.api.exceptions.HyracksDataException: java.lang.IllegalMonitorStateException: attempt to unlock read lock, not locked by current thread
> at org.apache.hyracks.control.cc.job.JobRun.waitForCompletion(JobRun.java:212)
> at org.apache.hyracks.control.cc.work.WaitForJobCompletionWork$1.run(WaitForJobCompletionWork.java:48)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.hyracks.api.exceptions.HyracksDataException: org.apache.hyracks.api.exceptions.HyracksDataException: java.util.concurrent.ExecutionException: org.apache.hyracks.api.exceptions.HyracksDataException: java.lang.IllegalMonitorStateException: attempt to unlock read lock, not locked by current thread
> at org.apache.hyracks.control.common.utils.ExceptionUtils.setNodeIds(ExceptionUtils.java:45)
> at org.apache.hyracks.control.nc.Task.run(Task.java:319)
> ... 3 more
> Caused by: org.apache.hyracks.api.exceptions.HyracksDataException: java.util.concurrent.ExecutionException: org.apache.hyracks.api.exceptions.HyracksDataException: java.lang.IllegalMonitorStateException: attempt to unlock read lock, not locked by current thread
> at org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePushable.runInParallel(SuperActivityOperatorNodePushable.java:218)
> at org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePushable.initialize(SuperActivityOperatorNodePushable.java:83)
> at org.apache.hyracks.control.nc.Task.run(Task.java:263)
> ... 3 more
> Caused by: java.util.concurrent.ExecutionException: org.apache.hyracks.api.exceptions.HyracksDataException: java.lang.IllegalMonitorStateException: attempt to unlock read lock, not locked by current thread
> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> at java.util.concurrent.FutureTask.get(FutureTask.java:192)
> at org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePushable.runInParallel(SuperActivityOperatorNodePushable.java:212)
> ... 5 more
> Caused by: org.apache.hyracks.api.exceptions.HyracksDataException: java.lang.IllegalMonitorStateException: attempt to unlock read lock, not locked by current thread
> at org.apache.hyracks.storage.am.common.dataflow.IndexSearchOperatorNodePushable.close(IndexSearchOperatorNodePushable.java:211)
> at org.apache.hyracks.algebricks.runtime.operators.base.AbstractOneInputOneOutputOneFramePushRuntime.close(AbstractOneInputOneOutputOneFramePushRuntime.java:57)
> at org.apache.hyracks.algebricks.runtime.operators.std.AssignRuntimeFactory$1.close(AssignRuntimeFactory.java:122)
> at org.apache.hyracks.algebricks.runtime.operators.std.EmptyTupleSourceRuntimeFactory$1.close(EmptyTupleSourceRuntimeFactory.java:60)
> at org.apache.hyracks.algebricks.runtime.operators.meta.AlgebricksMetaOperatorDescriptor$1.initialize(AlgebricksMetaOperatorDescriptor.java:116)
> at org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePushable.lambda$initialize$0(SuperActivityOperatorNodePushable.java:83)
> at org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePushable$1.call(SuperActivityOperatorNodePushable.java:205)
> at org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePushable$1.call(SuperActivityOperatorNodePushable.java:202)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> ... 3 more
> Caused by: java.lang.IllegalMonitorStateException: attempt to unlock read lock, not locked by current thread
> at java.util.concurrent.locks.ReentrantReadWriteLock$Sync.unmatchedUnlockException(ReentrantReadWriteLock.java:444)
> at java.util.concurrent.locks.ReentrantReadWriteLock$Sync.tryReleaseShared(ReentrantReadWriteLock.java:428)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer.releaseShared(AbstractQueuedSynchronizer.java:1341)
> at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.unlock(ReentrantReadWriteLock.java:881)
> at org.apache.hyracks.storage.common.buffercache.CachedPage.releaseReadLatch(CachedPage.java:121)
> at org.apache.hyracks.storage.am.btree.impls.BTreeRangeSearchCursor.close(BTreeRangeSearchCursor.java:91)
> at org.apache.hyracks.storage.am.lsm.btree.impls.LSMBTreePointSearchCursor.close(LSMBTreePointSearchCursor.java:211)
> at org.apache.hyracks.storage.am.lsm.btree.impls.LSMBTreeSearchCursor.close(LSMBTreeSearchCursor.java:97)
> at org.apache.hyracks.storage.am.common.dataflow.IndexSearchOperatorNodePushable.close(IndexSearchOperatorNodePushable.java:208)
> ... 11 more
> Jun 07, 2016 4:20:13 AM org.apache.asterix.api.http.servlet.APIServlet doPost
> SEVERE: Job failed on account of:
> org.apache.hyracks.api.exceptions.HyracksDataException: java.util.concurrent.ExecutionException: org.apache.hyracks.api.exceptions.HyracksDataException: java.lang.IllegalMonitorStateException: attempt to unlock read lock, not locked by current thread
> org.apache.hyracks.api.exceptions.HyracksException: Job failed on account of:
> org.apache.hyracks.api.exceptions.HyracksDataException: java.util.concurrent.ExecutionException: org.apache.hyracks.api.exceptions.HyracksDataException: java.lang.IllegalMonitorStateException: attempt to unlock read lock, not locked by current thread
> at org.apache.hyracks.control.cc.job.JobRun.waitForCompletion(JobRun.java:212)
> at org.apache.hyracks.control.cc.work.WaitForJobCompletionWork$1.run(WaitForJobCompletionWork.java:48)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.hyracks.api.exceptions.HyracksDataException: org.apache.hyracks.api.exceptions.HyracksDataException: java.util.concurrent.ExecutionException: org.apache.hyracks.api.exceptions.HyracksDataException: java.lang.IllegalMonitorStateException: attempt to unlock read lock, not locked by current thread
> at org.apache.hyracks.control.common.utils.ExceptionUtils.setNodeIds(ExceptionUtils.java:45)
> at org.apache.hyracks.control.nc.Task.run(Task.java:319)
> ... 3 more
> Caused by: org.apache.hyracks.api.exceptions.HyracksDataException: java.util.concurrent.ExecutionException: org.apache.hyracks.api.exceptions.HyracksDataException: java.lang.IllegalMonitorStateException: attempt to unlock read lock, not locked by current thread
> at org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePushable.runInParallel(SuperActivityOperatorNodePushable.java:218)
> at org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePushable.initialize(SuperActivityOperatorNodePushable.java:83)
> at org.apache.hyracks.control.nc.Task.run(Task.java:263)
> ... 3 more
> Caused by: java.util.concurrent.ExecutionException: org.apache.hyracks.api.exceptions.HyracksDataException: java.lang.IllegalMonitorStateException: attempt to unlock read lock, not locked by current thread
> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> at java.util.concurrent.FutureTask.get(FutureTask.java:192)
> at org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePushable.runInParallel(SuperActivityOperatorNodePushable.java:212)
> ... 5 more
> Caused by: org.apache.hyracks.api.exceptions.HyracksDataException: java.lang.IllegalMonitorStateException: attempt to unlock read lock, not locked by current thread
> at org.apache.hyracks.storage.am.common.dataflow.IndexSearchOperatorNodePushable.close(IndexSearchOperatorNodePushable.java:211)
> at org.apache.hyracks.algebricks.runtime.operators.base.AbstractOneInputOneOutputOneFramePushRuntime.close(AbstractOneInputOneOutputOneFramePushRuntime.java:57)
> at org.apache.hyracks.algebricks.runtime.operators.std.AssignRuntimeFactory$1.close(AssignRuntimeFactory.java:122)
> at org.apache.hyracks.algebricks.runtime.operators.std.EmptyTupleSourceRuntimeFactory$1.close(EmptyTupleSourceRuntimeFactory.java:60)
> at org.apache.hyracks.algebricks.runtime.operators.meta.AlgebricksMetaOperatorDescriptor$1.initialize(AlgebricksMetaOperatorDescriptor.java:116)
> at org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePushable.lambda$initialize$0(SuperActivityOperatorNodePushable.java:83)
> at org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePushable$1.call(SuperActivityOperatorNodePushable.java:205)
> at org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePushable$1.call(SuperActivityOperatorNodePushable.java:202)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> ... 3 more
> Caused by: java.lang.IllegalMonitorStateException: attempt to unlock read lock, not locked by current thread
> at java.util.concurrent.locks.ReentrantReadWriteLock$Sync.unmatchedUnlockException(ReentrantReadWriteLock.java:444)
> at java.util.concurrent.locks.ReentrantReadWriteLock$Sync.tryReleaseShared(ReentrantReadWriteLock.java:428)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer.releaseShared(AbstractQueuedSynchronizer.java:1341)
> at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.unlock(ReentrantReadWriteLock.java:881)
> at org.apache.hyracks.storage.common.buffercache.CachedPage.releaseReadLatch(CachedPage.java:121)
> at org.apache.hyracks.storage.am.btree.impls.BTreeRangeSearchCursor.close(BTreeRangeSearchCursor.java:91)
> at org.apache.hyracks.storage.am.lsm.btree.impls.LSMBTreePointSearchCursor.close(LSMBTreePointSearchCursor.java:211)
> at org.apache.hyracks.storage.am.lsm.btree.impls.LSMBTreeSearchCursor.close(LSMBTreeSearchCursor.java:97)
> at org.apache.hyracks.storage.am.common.dataflow.IndexSearchOperatorNodePushable.close(IndexSearchOperatorNodePushable.java:208)
> ... 11 more
> {noformat}
> To reproduce the issue execute the following:
> {noformat}
> drop dataverse testdv2 if exists;
> create dataverse testdv2;
> use dataverse testdv2;
> create type testtype as closed {
> id: int64,
> name: string,
> hobbies: {{string}}
> }
> create dataset testds(testtype) primary key id;
> use dataverse testdv2;
> load dataset testds
> using localfs
> (("path"="asterix_nc1://target/data/big-object/big_object_20M.adm"),("format"="adm")) pre-sorted;
> {noformat}
> After that, restart the instance and run the following query:
> {noformat}
> use dataverse testdv2;
> for $d in dataset("testds")
> where $d.id = 1
> return $d
> {noformat}
> I traced the problem and found out the issue is related to checking the first metadata page of the index. If the first page in the index contains supplement blocks, it will be cached in the buffer cache without its supplement blocks during the metadata page checking. Then, when the same page is requested again during a search, it will be returned from the buffer cache without its supplement blocks, causing all types of unexpected behaviors.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)