You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2022/06/04 20:47:30 UTC
[GitHub] [beam] damccorm opened a new issue, #20977: SamzaStoreStateInternalsTest is flaky
damccorm opened a new issue, #20977:
URL: https://github.com/apache/beam/issues/20977
Happened in precommits: https://ci-beam.apache.org/job/beam_PreCommit_Java_Commit/17290/testReport/junit/org.apache.beam.runners.samza.runtime/
```
Error Message
org.apache.beam.sdk.Pipeline$PipelineExecutionException: org.apache.samza.SamzaException:
Exception when restoring
Stacktrace
org.apache.beam.sdk.Pipeline$PipelineExecutionException: org.apache.samza.SamzaException:
Exception when restoring
at org.apache.beam.runners.samza.SamzaPipelineResult.getStateInfo(SamzaPipelineResult.java:129)
at
org.apache.beam.runners.samza.SamzaPipelineResult.waitUntilFinish(SamzaPipelineResult.java:92)
at
org.apache.beam.runners.samza.SamzaPipelineResult.waitUntilFinish(SamzaPipelineResult.java:108)
at
org.apache.beam.runners.samza.TestSamzaRunner.run(TestSamzaRunner.java:78)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:322)
at
org.apache.beam.sdk.testing.TestPipeline.run(TestPipeline.java:398)
at org.apache.beam.sdk.testing.TestPipeline.run(TestPipeline.java:334)
at
org.apache.beam.runners.samza.runtime.SamzaStoreStateInternalsTest.testSetStateIterator(SamzaStoreStateInternalsTest.java:192)
at
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at
org.apache.beam.sdk.testing.TestPipeline$1.evaluate(TestPipeline.java:322)
at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
at
org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
at
org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
at
org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
at
org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
at
org.junit.runners.ParentRunner.run(ParentRunner.java:413)
at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.runTestClass(JUnitTestClassExecutor.java:110)
at
org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:58)
at
org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:38)
at
org.gradle.api.internal.tasks.testing.junit.AbstractJUnitTestClassProcessor.processTestClass(AbstractJUnitTestClassProcessor.java:62)
at
org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:51)
at
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36)
at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
at
org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:33)
at
org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:94)
at
com.sun.proxy.$Proxy2.processTestClass(Unknown Source)
at org.gradle.api.internal.tasks.testing.worker.TestWorker.processTestClass(TestWorker.java:119)
at
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36)
at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
at
org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection$DispatchWrapper.dispatch(MessageHubBackedObjectConnection.java:182)
at
org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection$DispatchWrapper.dispatch(MessageHubBackedObjectConnection.java:164)
at
org.gradle.internal.remote.internal.hub.MessageHub$Handler.run(MessageHub.java:414)
at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:64)
at
org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:48)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at org.gradle.internal.concurrent.ThreadFactoryImpl$ManagedThreadRunnable.run(ThreadFactoryImpl.java:56)
at
java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.samza.SamzaException: Exception when restoring
at org.apache.samza.storage.ContainerStorageManager.restoreStores(ContainerStorageManager.java:695)
at
org.apache.samza.storage.ContainerStorageManager.start(ContainerStorageManager.java:653)
at org.apache.samza.container.SamzaContainer.startStores(SamzaContainer.scala:966)
at
org.apache.samza.container.SamzaContainer.run(SamzaContainer.scala:764)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at
java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
... 1 more
Caused
by: java.util.concurrent.ExecutionException: org.apache.samza.SamzaException: Error opening RocksDB
store beamStore at location /tmp/beam-samza-test/beamStore/Partition_0
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at
java.util.concurrent.FutureTask.get(FutureTask.java:192)
at org.apache.samza.storage.ContainerStorageManager.restoreStores(ContainerStorageManager.java:687)
...
8 more
Caused by: org.apache.samza.SamzaException: Error opening RocksDB store beamStore at location
/tmp/beam-samza-test/beamStore/Partition_0
at org.apache.samza.storage.kv.RocksDbKeyValueStore$.openDB(RocksDbKeyValueStore.scala:107)
at
org.apache.samza.storage.kv.RocksDbKeyValueStore.org$apache$samza$storage$kv$RocksDbKeyValueStore$$db$lzycompute(RocksDbKeyValueStore.scala:125)
at
org.apache.samza.storage.kv.RocksDbKeyValueStore.org$apache$samza$storage$kv$RocksDbKeyValueStore$$db(RocksDbKeyValueStore.scala:125)
at
org.apache.samza.storage.kv.RocksDbKeyValueStore$$anonfun$putAll$1.apply$mcJ$sp(RocksDbKeyValueStore.scala:195)
at
org.apache.samza.storage.kv.RocksDbKeyValueStore$$anonfun$putAll$1.apply(RocksDbKeyValueStore.scala:176)
at
org.apache.samza.storage.kv.RocksDbKeyValueStore$$anonfun$putAll$1.apply(RocksDbKeyValueStore.scala:176)
at
org.apache.samza.storage.kv.RocksDbKeyValueStore.org$apache$samza$storage$kv$RocksDbKeyValueStore$$ifOpen(RocksDbKeyValueStore.scala:275)
at
org.apache.samza.storage.kv.RocksDbKeyValueStore.putAll(RocksDbKeyValueStore.scala:176)
at org.apache.samza.storage.kv.SerializedKeyValueStore.putAll(SerializedKeyValueStore.scala:89)
at
org.apache.samza.storage.kv.CachedStore.org$apache$samza$storage$kv$CachedStore$$putAllDirtyEntries(CachedStore.scala:218)
at
org.apache.samza.storage.kv.CachedStore.flush(CachedStore.scala:198)
at org.apache.samza.storage.kv.NullSafeKeyValueStore.flush(NullSafeKeyValueStore.scala:85)
at
org.apache.samza.storage.kv.KeyValueStorageEngine$$anonfun$flush$1.apply$mcV$sp(KeyValueStorageEngine.scala:214)
at
org.apache.samza.storage.kv.KeyValueStorageEngine$$anonfun$flush$1.apply(KeyValueStorageEngine.scala:211)
at
org.apache.samza.storage.kv.KeyValueStorageEngine$$anonfun$flush$1.apply(KeyValueStorageEngine.scala:211)
at
org.apache.samza.util.TimerUtil$class.updateTimer(TimerUtil.scala:37)
at org.apache.samza.storage.kv.KeyValueStorageEngine.updateTimer(KeyValueStorageEngine.scala:39)
at
org.apache.samza.storage.kv.KeyValueStorageEngine.flush(KeyValueStorageEngine.scala:211)
at org.apache.samza.storage.kv.KeyValueStorageEngine.close(KeyValueStorageEngine.scala:235)
at
org.apache.samza.storage.kv.KeyValueStorageEngine.stop(KeyValueStorageEngine.scala:229)
at org.apache.samza.storage.TransactionalStateTaskRestoreManager.lambda$stopPersistentStores$0(TransactionalStateTaskRestoreManager.java:148)
at
java.util.HashMap.forEach(HashMap.java:1289)
at org.apache.samza.storage.TransactionalStateTaskRestoreManager.stopPersistentStores(TransactionalStateTaskRestoreManager.java:146)
at
org.apache.samza.storage.ContainerStorageManager$TaskRestoreCallable.call(ContainerStorageManager.java:952)
at
org.apache.samza.storage.ContainerStorageManager$TaskRestoreCallable.call(ContainerStorageManager.java:922)
...
4 more
Caused by: org.rocksdb.RocksDBException: lock hold by current process, acquire time 1618957171
acquiring thread 140263348766464: /tmp/beam-samza-test/beamStore/Partition_0/LOCK: No locks available
at
org.rocksdb.RocksDB.open(Native Method)
at org.rocksdb.RocksDB.open(RocksDB.java:251)
at org.apache.samza.storage.kv.RocksDbKeyValueStore$.openDB(RocksDbKeyValueStore.scala:75)
...
28 more
```
Imported from Jira [BEAM-12200](https://issues.apache.org/jira/browse/BEAM-12200). Original Jira may contain additional context.
Reported by: tvalentyn.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] kennknowles commented on issue #20977: SamzaStoreStateInternalsTest is flaky
Posted by GitBox <gi...@apache.org>.
kennknowles commented on issue #20977:
URL: https://github.com/apache/beam/issues/20977#issuecomment-1287471040
Didn't repro, and isn't disabled or flaking now.
```
(set +e
for i in `seq 1 1000` ; do
echo "RUN $i"
./gradlew --quiet :runners:samza:cleanTest
./gradlew --quiet :runners:samza:test \
--tests 'SamzaStoreStateInternalsTest'
done
)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] kennknowles closed issue #20977: SamzaStoreStateInternalsTest is flaky
Posted by GitBox <gi...@apache.org>.
kennknowles closed issue #20977: SamzaStoreStateInternalsTest is flaky
URL: https://github.com/apache/beam/issues/20977
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org