You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2021/06/09 11:40:48 UTC
[GitHub] [pulsar] lhotari opened a new issue #10872: Flaky-test: MetadataCacheTest.crossStoreUpdates
lhotari opened a new issue #10872:
URL: https://github.com/apache/pulsar/issues/10872
<!---
Instructions for reporting a flaky test using this issue template:
1. Replace [test class] in title and body with the test class name
2. Replace [test method] in title and body with the test method that failed. Multiple methods are flaky, remove the content that refers to the test method.
3. Replace "url here" with a url to an example failure. In the Github Actions workflow run logs, you can right click on the line number to copy a link to the line. Example of such url is https://github.com/apache/pulsar/pull/8892/checks?check_run_id=1531075794#step:9:377 . The logs are available for a limited amount of time (usually for a few weeks).
4. Replace "relevant parts of the exception stacktrace here" with the a few lines of the stack trace that shows at leat the exception message and the line of test code where the stacktrace occured.
5. Replace "full exception stacktrace here" with the full exception stacktrace from logs. This section will be hidded by default.
6. Remove all unused fields / content to unclutter the reported issue. Remove this comment too.
-->
MetadataCacheTest is flaky. The crossStoreUpdates test method fails sporadically.
[example failure](https://github.com/apache/pulsar/pull/10864/checks?check_run_id=2782726910#step:9:6862)
```
Error: Tests run: 26, Failures: 1, Errors: 0, Skipped: 23, Time elapsed: 1.557 s <<< FAILURE! - in org.apache.pulsar.metadata.MetadataCacheTest
Error: crossStoreUpdates(org.apache.pulsar.metadata.MetadataCacheTest) Time elapsed: 0.346 s <<< FAILURE!
java.util.concurrent.CompletionException: org.apache.pulsar.metadata.api.MetadataStoreException$AlreadyExistsException: org.apache.pulsar.metadata.api.MetadataStoreException$BadVersionException: org.apache.zookeeper.KeeperException$BadVersionException: KeeperErrorCode = BadVersion for /test-key1
at java.base/java.util.concurrent.CompletableFuture.reportJoin(CompletableFuture.java:412)
at java.base/java.util.concurrent.CompletableFuture.join(CompletableFuture.java:2044)
at org.apache.pulsar.metadata.MetadataCacheTest.multiStoreAddDelete(MetadataCacheTest.java:142)
at org.apache.pulsar.metadata.MetadataCacheTest.crossStoreUpdates(MetadataCacheTest.java:124)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:132)
at org.testng.internal.InvokeMethodRunnable.runOne(InvokeMethodRunnable.java:45)
at org.testng.internal.InvokeMethodRunnable.call(InvokeMethodRunnable.java:73)
at org.testng.internal.InvokeMethodRunnable.call(InvokeMethodRunnable.java:11)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: org.apache.pulsar.metadata.api.MetadataStoreException$AlreadyExistsException: org.apache.pulsar.metadata.api.MetadataStoreException$BadVersionException: org.apache.zookeeper.KeeperException$BadVersionException: KeeperErrorCode = BadVersion for /test-key1
at org.apache.pulsar.metadata.cache.impl.MetadataCacheImpl.lambda$create$12(MetadataCacheImpl.java:228)
at java.base/java.util.concurrent.CompletableFuture.uniExceptionally(CompletableFuture.java:986)
at java.base/java.util.concurrent.CompletableFuture$UniExceptionally.tryFire(CompletableFuture.java:970)
at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506)
at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2088)
at org.apache.pulsar.metadata.impl.ZKMetadataStore.lambda$storePut$15(ZKMetadataStore.java:226)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 more
Caused by: org.apache.pulsar.metadata.api.MetadataStoreException$BadVersionException: org.apache.zookeeper.KeeperException$BadVersionException: KeeperErrorCode = BadVersion for /test-key1
at org.apache.pulsar.metadata.impl.ZKMetadataStore.getException(ZKMetadataStore.java:308)
... 5 more
Caused by: org.apache.zookeeper.KeeperException$BadVersionException: KeeperErrorCode = BadVersion for /test-key1
at org.apache.zookeeper.KeeperException.create(KeeperException.java:122)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:54)
at org.apache.pulsar.metadata.impl.ZKMetadataStore.getException(ZKMetadataStore.java:304)
... 5 more
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lhotari commented on issue #10872: Flaky-test: MetadataCacheTest.crossStoreUpdates
Posted by GitBox <gi...@apache.org>.
lhotari commented on issue #10872:
URL: https://github.com/apache/pulsar/issues/10872#issuecomment-857625275
It looks like this flaky test was introduced by #10862 . @sursingh Would you mind taking a look ?
It might help to use Awaitility instead of Thread.sleep in this test.
example of using Awaitility (ignore any exceptions thrown using evaluation) from another test:
https://github.com/apache/pulsar/blob/da71ec2b2a6682ffbab60843756f638bedbd4460/pulsar-broker/src/test/java/org/apache/pulsar/io/PulsarFunctionE2ETest.java#L131-L138
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lhotari commented on issue #10872: Flaky-test: MetadataCacheTest.crossStoreUpdates
Posted by GitBox <gi...@apache.org>.
lhotari commented on issue #10872:
URL: https://github.com/apache/pulsar/issues/10872#issuecomment-866529954
@sursingh Would you be able to take another look? Thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lhotari commented on issue #10872: Flaky-test: MetadataCacheTest.crossStoreUpdates
Posted by GitBox <gi...@apache.org>.
lhotari commented on issue #10872:
URL: https://github.com/apache/pulsar/issues/10872#issuecomment-866547368
[One more recent failure](https://github.com/apache/pulsar/pull/11026/checks?check_run_id=2889407530#step:9:1489)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] codelipenghui commented on issue #10872: Flaky-test: MetadataCacheTest.crossStoreUpdates
Posted by GitBox <gi...@apache.org>.
codelipenghui commented on issue #10872:
URL: https://github.com/apache/pulsar/issues/10872#issuecomment-1058889600
The issue had no activity for 30 days, mark with Stale label.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] merlimat closed issue #10872: Flaky-test: MetadataCacheTest.crossStoreUpdates
Posted by GitBox <gi...@apache.org>.
merlimat closed issue #10872:
URL: https://github.com/apache/pulsar/issues/10872
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] sursingh commented on issue #10872: Flaky-test: MetadataCacheTest.crossStoreUpdates
Posted by GitBox <gi...@apache.org>.
sursingh commented on issue #10872:
URL: https://github.com/apache/pulsar/issues/10872#issuecomment-857750524
@lhotari : I will take a look.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lhotari commented on issue #10872: Flaky-test: MetadataCacheTest.crossStoreUpdates
Posted by GitBox <gi...@apache.org>.
lhotari commented on issue #10872:
URL: https://github.com/apache/pulsar/issues/10872#issuecomment-866529801
The problem remains. I wonder if this is a real problem in the conflict resolution since I might have seen the same error message in other locations.
[Recent failure:](https://github.com/apache/pulsar/pull/11025/checks?check_run_id=2889196047#step:9:10734)
```
Error: Tests run: 26, Failures: 1, Errors: 0, Skipped: 23, Time elapsed: 11.94 s <<< FAILURE! - in org.apache.pulsar.metadata.MetadataCacheTest
Error: crossStoreUpdates(org.apache.pulsar.metadata.MetadataCacheTest) Time elapsed: 0.341 s <<< FAILURE!
java.util.concurrent.CompletionException: org.apache.pulsar.metadata.api.MetadataStoreException$AlreadyExistsException: org.apache.pulsar.metadata.api.MetadataStoreException$BadVersionException: org.apache.zookeeper.KeeperException$BadVersionException: KeeperErrorCode = BadVersion for /test-key1
at java.base/java.util.concurrent.CompletableFuture.reportJoin(CompletableFuture.java:412)
at java.base/java.util.concurrent.CompletableFuture.join(CompletableFuture.java:2044)
at org.apache.pulsar.metadata.MetadataCacheTest.multiStoreAddDelete(MetadataCacheTest.java:143)
at org.apache.pulsar.metadata.MetadataCacheTest.crossStoreUpdates(MetadataCacheTest.java:125)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:132)
at org.testng.internal.InvokeMethodRunnable.runOne(InvokeMethodRunnable.java:45)
at org.testng.internal.InvokeMethodRunnable.call(InvokeMethodRunnable.java:73)
at org.testng.internal.InvokeMethodRunnable.call(InvokeMethodRunnable.java:11)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: org.apache.pulsar.metadata.api.MetadataStoreException$AlreadyExistsException: org.apache.pulsar.metadata.api.MetadataStoreException$BadVersionException: org.apache.zookeeper.KeeperException$BadVersionException: KeeperErrorCode = BadVersion for /test-key1
at org.apache.pulsar.metadata.cache.impl.MetadataCacheImpl.lambda$create$12(MetadataCacheImpl.java:228)
at java.base/java.util.concurrent.CompletableFuture.uniExceptionally(CompletableFuture.java:986)
at java.base/java.util.concurrent.CompletableFuture$UniExceptionally.tryFire(CompletableFuture.java:970)
at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506)
at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2088)
at org.apache.pulsar.metadata.impl.ZKMetadataStore.lambda$storePut$15(ZKMetadataStore.java:226)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 more
Caused by: org.apache.pulsar.metadata.api.MetadataStoreException$BadVersionException: org.apache.zookeeper.KeeperException$BadVersionException: KeeperErrorCode = BadVersion for /test-key1
at org.apache.pulsar.metadata.impl.ZKMetadataStore.getException(ZKMetadataStore.java:308)
... 5 more
Caused by: org.apache.zookeeper.KeeperException$BadVersionException: KeeperErrorCode = BadVersion for /test-key1
at org.apache.zookeeper.KeeperException.create(KeeperException.java:122)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:54)
at org.apache.pulsar.metadata.impl.ZKMetadataStore.getException(ZKMetadataStore.java:304)
... 5 more
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lhotari commented on issue #10872: Flaky-test: MetadataCacheTest.crossStoreUpdates
Posted by GitBox <gi...@apache.org>.
lhotari commented on issue #10872:
URL: https://github.com/apache/pulsar/issues/10872#issuecomment-866542025
It's also possible that there was a previous failure and TestNG retried the test (In Pulsar, TestNG will retry a failed test once). The original test failure is now lost since I restarted the test run. It would have been in the surefire logs, but that gets lost if the build is restarted.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lhotari edited a comment on issue #10872: Flaky-test: MetadataCacheTest.crossStoreUpdates
Posted by GitBox <gi...@apache.org>.
lhotari edited a comment on issue #10872:
URL: https://github.com/apache/pulsar/issues/10872#issuecomment-857625275
It looks like this flaky test was introduced by #10862 . @sursingh Would you mind taking a look ?
It might help to use Awaitility instead of Thread.sleep in this test.
example of using Awaitility (ignore any exceptions thrown during evaluation) from another test:
https://github.com/apache/pulsar/blob/da71ec2b2a6682ffbab60843756f638bedbd4460/pulsar-broker/src/test/java/org/apache/pulsar/io/PulsarFunctionE2ETest.java#L131-L138
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lhotari commented on issue #10872: Flaky-test: MetadataCacheTest.crossStoreUpdates
Posted by GitBox <gi...@apache.org>.
lhotari commented on issue #10872:
URL: https://github.com/apache/pulsar/issues/10872#issuecomment-884659814
[Yet another recent failure](https://github.com/apache/pulsar/pull/11343/checks?check_run_id=3130775230#step:9:3186)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lhotari commented on issue #10872: Flaky-test: MetadataCacheTest.crossStoreUpdates
Posted by GitBox <gi...@apache.org>.
lhotari commented on issue #10872:
URL: https://github.com/apache/pulsar/issues/10872#issuecomment-866532612
The above exception happens on line 143:
https://github.com/apache/pulsar/blob/4cf1008a7b5faf48e02cc3a68560ff469d700c73/pulsar-metadata/src/test/java/org/apache/pulsar/metadata/MetadataCacheTest.java#L143
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org