You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@accumulo.apache.org by GitBox <gi...@apache.org> on 2022/08/01 22:44:21 UTC
[GitHub] [accumulo] EdColeman opened a new issue, #2837: IT Test failures - waitForFateOperation and Fate Clean-up
EdColeman opened a new issue, #2837:
URL: https://github.com/apache/accumulo/issues/2837
**Describe the bug**
Main branch is failing tests - most common that I saw was the fate print and fate summary are failing because the transaction is not reliably being cleaned-up - the commands return `SUCCESS`when an empty result is expected,
I also kicked of a full set of ITs that have a number of failures that seem to stem from an exception in the waitForFateOperation. These errors also seemed to cause the recent Jenkins run to fail.
**Versions (OS, Maven, Java, and others, as appropriate):**
Accumulo 2.1 (main)
Two of the failed IT tests mvn output.
```
[ERROR] ConcurrentDeleteTableIT.testConcurrentDeleteTablesOps:105 » Execution java.lang.RuntimeException: org.apache.accumulo.core.client.AccumuloException: Internal error processing waitForFateOperation
[ERROR] ConcurrentDeleteTableIT.testConcurrentFateOpsWithDelete:210 » Execution java.lang.RuntimeException: org.apache.accumulo.core.client.AccumuloException: Internal error processing waitForFateOperation
```
One of the stack traces (not necessarily from above mvn output)
```
2022-08-01T21:32:41.0313545Z Caused by: org.apache.accumulo.core.client.AccumuloException: Internal error processing waitForFateOperation
2022-08-01T21:32:41.0314168Z at org.apache.accumulo.core.clientImpl.TableOperationsImpl.doFateOperation(TableOperationsImpl.java:407)
2022-08-01T21:32:41.0314840Z at org.apache.accumulo.core.clientImpl.TableOperationsImpl.doFateOperation(TableOperationsImpl.java:359)
2022-08-01T21:32:41.0315576Z at org.apache.accumulo.core.clientImpl.NamespaceOperationsImpl.doNamespaceFateOperation(NamespaceOperationsImpl.java:296)
2022-08-01T21:32:41.0316510Z at org.apache.accumulo.core.clientImpl.NamespaceOperationsImpl.create(NamespaceOperationsImpl.java:126)
2022-08-01T21:32:41.0317139Z at org.apache.accumulo.test.NamespacesIT.lambda$verifyNamespaceOperationsExceptions$83(NamespacesIT.java:1088)
2022-08-01T21:32:41.0317667Z at org.junit.jupiter.api.AssertThrows.assertThrows(AssertThrows.java:55)
2022-08-01T21:32:41.0317992Z ... 16 more
2022-08-01T21:32:41.0318355Z Caused by: org.apache.thrift.TApplicationException: Internal error processing waitForFateOperation
2022-08-01T21:32:41.0318835Z at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:79)
2022-08-01T21:32:41.0319345Z at org.apache.accumulo.core.manager.thrift.FateService$Client.recv_waitForFateOperation(FateService.java:161)
2022-08-01T21:32:41.0319925Z at org.apache.accumulo.core.manager.thrift.FateService$Client.waitForFateOperation(FateService.java:146)
2022-08-01T21:32:41.0320575Z at org.apache.accumulo.core.clientImpl.TableOperationsImpl.waitForFateOperation(TableOperationsImpl.java:306)
2022-08-01T21:32:41.0321262Z at org.apache.accumulo.core.clientImpl.TableOperationsImpl.doFateOperation(TableOperationsImpl.java:375)
2022-08-01T21:32:41.0321784Z ... 21 more
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] dlmarion closed issue #2837: IT failures - waitForFateOperation and Fate Clean-up
Posted by GitBox <gi...@apache.org>.
dlmarion closed issue #2837: IT failures - waitForFateOperation and Fate Clean-up
URL: https://github.com/apache/accumulo/issues/2837
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] dlmarion commented on issue #2837: IT failures - waitForFateOperation and Fate Clean-up
Posted by GitBox <gi...@apache.org>.
dlmarion commented on issue #2837:
URL: https://github.com/apache/accumulo/issues/2837#issuecomment-1202436991
I reverted the changes to ZooStore in [5b0ad9087fc205b4aeffa16902b0d722b28c0dcf ](https://github.com/apache/accumulo/commit/5b0ad9087fc205b4aeffa16902b0d722b28c0dcf) and ConcurrentDeleteTableIT passes locally for me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] milleruntime commented on issue #2837: IT failures - waitForFateOperation and Fate Clean-up
Posted by GitBox <gi...@apache.org>.
milleruntime commented on issue #2837:
URL: https://github.com/apache/accumulo/issues/2837#issuecomment-1202531400
I ran the failing ITs with the fix in #2839 and they all passed except FateConcurrencyIT
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] EdColeman commented on issue #2837: IT Test failures - waitForFateOperation and Fate Clean-up
Posted by GitBox <gi...@apache.org>.
EdColeman commented on issue #2837:
URL: https://github.com/apache/accumulo/issues/2837#issuecomment-1201830914
From running `ConcurrentDeleteTableIT` locally
```
[ERROR] org.apache.accumulo.test.functional.ConcurrentDeleteTableIT.testConcurrentDeleteTablesOps Time elapsed: 22.382 s <<< ERROR!
java.util.concurrent.ExecutionException: java.lang.RuntimeException: org.apache.accumulo.core.client.AccumuloException: Internal error processing waitForFateOperation
at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191)
at org.apache.accumulo.test.functional.ConcurrentDeleteTableIT.testConcurrentDeleteTablesOps(ConcurrentDeleteTableIT.java:105)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:725)
at org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60)
at org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:131)
at org.junit.jupiter.api.AssertTimeout.lambda$assertTimeoutPreemptively$4(AssertTimeout.java:138)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.lang.RuntimeException: org.apache.accumulo.core.client.AccumuloException: Internal error processing waitForFateOperation
at org.apache.accumulo.test.functional.ConcurrentDeleteTableIT.lambda$testConcurrentDeleteTablesOps$0(ConcurrentDeleteTableIT.java:97)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
... 4 more
Caused by: org.apache.accumulo.core.client.AccumuloException: Internal error processing waitForFateOperation
at org.apache.accumulo.core.clientImpl.TableOperationsImpl.doFateOperation(TableOperationsImpl.java:407)
at org.apache.accumulo.core.clientImpl.TableOperationsImpl.doFateOperation(TableOperationsImpl.java:359)
at org.apache.accumulo.core.clientImpl.TableOperationsImpl.doTableFateOperation(TableOperationsImpl.java:1700)
at org.apache.accumulo.core.clientImpl.TableOperationsImpl.delete(TableOperationsImpl.java:752)
at org.apache.accumulo.test.functional.ConcurrentDeleteTableIT.lambda$testConcurrentDeleteTablesOps$0(ConcurrentDeleteTableIT.java:93)
... 5 more
Caused by: org.apache.thrift.TApplicationException: Internal error processing waitForFateOperation
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:79)
at org.apache.accumulo.core.manager.thrift.FateService$Client.recv_waitForFateOperation(FateService.java:161)
at org.apache.accumulo.core.manager.thrift.FateService$Client.waitForFateOperation(FateService.java:146)
at org.apache.accumulo.core.clientImpl.TableOperationsImpl.waitForFateOperation(TableOperationsImpl.java:306)
at org.apache.accumulo.core.clientImpl.TableOperationsImpl.doFateOperation(TableOperationsImpl.java:375)
... 9 more
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] milleruntime commented on issue #2837: IT failures - waitForFateOperation and Fate Clean-up
Posted by GitBox <gi...@apache.org>.
milleruntime commented on issue #2837:
URL: https://github.com/apache/accumulo/issues/2837#issuecomment-1202505637
These failures are due to changes in #2829 but looks like Dave already found a fix.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] dlmarion commented on issue #2837: IT failures - waitForFateOperation and Fate Clean-up
Posted by GitBox <gi...@apache.org>.
dlmarion commented on issue #2837:
URL: https://github.com/apache/accumulo/issues/2837#issuecomment-1202464259
I think I see what's happening. ZooStore.getProperty is retrieving all properties using just the property name. But, ZooStore.setProperty is only setting String properties using just the property name. Other types of properties are still being set with the `prop_` prefix. Removing the other "prop_" prefix in ZooStore.setProperty when the object is not a String also allows the test to pass.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] EdColeman commented on issue #2837: IT Test failures - waitForFateOperation and Fate Clean-up
Posted by GitBox <gi...@apache.org>.
EdColeman commented on issue #2837:
URL: https://github.com/apache/accumulo/issues/2837#issuecomment-1201850947
From a manager log
```
2022-08-01T23:05:34,027 [thrift.ProcessFunction] ERROR: Internal error processing waitForFateOperation
java.lang.RuntimeException: null
at org.apache.accumulo.manager.FateServiceHandler.waitForFateOperation(FateServiceHandler.java:711) ~[accumulo-manager-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at jdk.internal.reflect.GeneratedMethodAccessor16.invoke(Unknown Source) ~[?:?]
at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
at org.apache.accumulo.core.trace.TraceUtil.lambda$wrapService$0(TraceUtil.java:206) ~[accumulo-core-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at com.sun.proxy.$Proxy28.waitForFateOperation(Unknown Source) ~[?:?]
at org.apache.accumulo.core.manager.thrift.FateService$Processor$waitForFateOperation.getResult(FateService.java:560) ~[accumulo-core-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at org.apache.accumulo.core.manager.thrift.FateService$Processor$waitForFateOperation.getResult(FateService.java:539) ~[accumulo-core-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:38) ~[libthrift-0.16.0.jar:0.16.0]
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:38) ~[libthrift-0.16.0.jar:0.16.0]
at org.apache.thrift.TMultiplexedProcessor.process(TMultiplexedProcessor.java:138) ~[libthrift-0.16.0.jar:0.16.0]
at org.apache.accumulo.server.rpc.TimedProcessor.process(TimedProcessor.java:54) ~[accumulo-server-base-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at org.apache.thrift.server.AbstractNonblockingServer$FrameBuffer.invoke(AbstractNonblockingServer.java:524) ~[libthrift-0.16.0.jar:0.16.0]
at org.apache.accumulo.server.rpc.CustomNonBlockingServer$CustomFrameBuffer.invoke(CustomNonBlockingServer.java:129) ~[accumulo-server-base-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at org.apache.thrift.server.Invocation.run(Invocation.java:18) ~[libthrift-0.16.0.jar:0.16.0]
at org.apache.accumulo.core.trace.TraceWrappedRunnable.run(TraceWrappedRunnable.java:52) ~[accumulo-core-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at org.apache.accumulo.core.trace.TraceWrappedRunnable.run(TraceWrappedRunnable.java:52) ~[accumulo-core-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at java.lang.Thread.run(Thread.java:834) ~[?:?]
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org