You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2022/06/02 11:49:25 UTC
[GitHub] [pulsar] lhotari opened a new issue, #15896: Flaky-test: PulsarLedgerIdGeneratorTest.testGenerateLedgerId
lhotari opened a new issue, #15896:
URL: https://github.com/apache/pulsar/issues/15896
org.apache.pulsar.metadata.bookkeeper.PulsarLedgerIdGeneratorTest is flaky. The testGenerateLedgerId test method fails sporadically.
```
java.lang.AssertionError: Error occur during ledger id generation : expected [0] but found [2]
at org.testng.Assert.fail(Assert.java:99)
at org.testng.Assert.failNotEquals(Assert.java:1037)
at org.testng.Assert.assertEqualsImpl(Assert.java:140)
at org.testng.Assert.assertEquals(Assert.java:122)
at org.testng.Assert.assertEquals(Assert.java:907)
at org.apache.pulsar.metadata.bookkeeper.PulsarLedgerIdGeneratorTest.testGenerateLedgerId(PulsarLedgerIdGeneratorTest.java:113)
```
Usage tip: To enable automatic navigation to failure message, open the following links with CTRL/CMD-click.
[example failure 2022-06-01T14:25:28.8616198Z](https://github.com/apache/pulsar/runs/6691176380?check_suite_focus=true#step:9:1542)
[example failure 2022-05-27T11:16:52.6256505Z](https://github.com/apache/pulsar/runs/6623892528?check_suite_focus=true#step:9:2931)
[example failure 2022-05-27T11:13:47.4439338Z](https://github.com/apache/pulsar/runs/6623892528?check_suite_focus=true#step:9:1534)
<details>
<summary>Full exception stacktrace</summary>
<code><pre>
java.lang.AssertionError: Error occur during ledger id generation : expected [0] but found [2]
at org.testng.Assert.fail(Assert.java:99)
at org.testng.Assert.failNotEquals(Assert.java:1037)
at org.testng.Assert.assertEqualsImpl(Assert.java:140)
at org.testng.Assert.assertEquals(Assert.java:122)
at org.testng.Assert.assertEquals(Assert.java:907)
at org.apache.pulsar.metadata.bookkeeper.PulsarLedgerIdGeneratorTest.testGenerateLedgerId(PulsarLedgerIdGeneratorTest.java:113)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:568)
at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:132)
at org.testng.internal.InvokeMethodRunnable.runOne(InvokeMethodRunnable.java:45)
at org.testng.internal.InvokeMethodRunnable.call(InvokeMethodRunnable.java:73)
at org.testng.internal.InvokeMethodRunnable.call(InvokeMethodRunnable.java:11)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:833)
</pre></code>
</details>
```
java.lang.AssertionError: Error occur during ledger id generation : expected [0] but found [12]
at org.testng.Assert.fail(Assert.java:99)
at org.testng.Assert.failNotEquals(Assert.java:1037)
at org.testng.Assert.assertEqualsImpl(Assert.java:140)
at org.testng.Assert.assertEquals(Assert.java:122)
at org.testng.Assert.assertEquals(Assert.java:907)
at org.apache.pulsar.metadata.bookkeeper.PulsarLedgerIdGeneratorTest.testGenerateLedgerId(PulsarLedgerIdGeneratorTest.java:113)
```
Usage tip: To enable automatic navigation to failure message, open the following links with CTRL/CMD-click.
[example failure 2022-05-30T22:43:38.9123608Z](https://github.com/apache/pulsar/runs/6661894942?check_suite_focus=true#step:10:1517)
<details>
<summary>Full exception stacktrace</summary>
<code><pre>
java.lang.AssertionError: Error occur during ledger id generation : expected [0] but found [12]
at org.testng.Assert.fail(Assert.java:99)
at org.testng.Assert.failNotEquals(Assert.java:1037)
at org.testng.Assert.assertEqualsImpl(Assert.java:140)
at org.testng.Assert.assertEquals(Assert.java:122)
at org.testng.Assert.assertEquals(Assert.java:907)
at org.apache.pulsar.metadata.bookkeeper.PulsarLedgerIdGeneratorTest.testGenerateLedgerId(PulsarLedgerIdGeneratorTest.java:113)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:568)
at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:132)
at org.testng.internal.InvokeMethodRunnable.runOne(InvokeMethodRunnable.java:45)
at org.testng.internal.InvokeMethodRunnable.call(InvokeMethodRunnable.java:73)
at org.testng.internal.InvokeMethodRunnable.call(InvokeMethodRunnable.java:11)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:833)
</pre></code>
</details>
```
java.lang.AssertionError: Error occur during ledger id generation : expected [0] but found [3]
at org.testng.Assert.fail(Assert.java:99)
at org.testng.Assert.failNotEquals(Assert.java:1037)
at org.testng.Assert.assertEqualsImpl(Assert.java:140)
at org.testng.Assert.assertEquals(Assert.java:122)
at org.testng.Assert.assertEquals(Assert.java:907)
at org.apache.pulsar.metadata.bookkeeper.PulsarLedgerIdGeneratorTest.testGenerateLedgerId(PulsarLedgerIdGeneratorTest.java:113)
```
Usage tip: To enable automatic navigation to failure message, open the following links with CTRL/CMD-click.
[example failure 2022-05-31T11:13:55.0547600Z](https://github.com/apache/pulsar/runs/6669032761?check_suite_focus=true#step:9:4932)
<details>
<summary>Full exception stacktrace</summary>
<code><pre>
java.lang.AssertionError: Error occur during ledger id generation : expected [0] but found [3]
at org.testng.Assert.fail(Assert.java:99)
at org.testng.Assert.failNotEquals(Assert.java:1037)
at org.testng.Assert.assertEqualsImpl(Assert.java:140)
at org.testng.Assert.assertEquals(Assert.java:122)
at org.testng.Assert.assertEquals(Assert.java:907)
at org.apache.pulsar.metadata.bookkeeper.PulsarLedgerIdGeneratorTest.testGenerateLedgerId(PulsarLedgerIdGeneratorTest.java:113)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:568)
at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:132)
at org.testng.internal.InvokeMethodRunnable.runOne(InvokeMethodRunnable.java:45)
at org.testng.internal.InvokeMethodRunnable.call(InvokeMethodRunnable.java:73)
at org.testng.internal.InvokeMethodRunnable.call(InvokeMethodRunnable.java:11)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:833)
</pre></code>
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] wuxuanqicn commented on issue #15896: Flaky-test: PulsarLedgerIdGeneratorTest.testGenerateLedgerId
Posted by GitBox <gi...@apache.org>.
wuxuanqicn commented on issue #15896:
URL: https://github.com/apache/pulsar/issues/15896#issuecomment-1150694056
On my local environment, RocksdbMetadataStore.java is the cause, it is transaction lock timout with default value 1 second, and 60 seconds don't work on my environment either.
```
2022-06-09T10:23:45,794 - ERROR - [pool-10-thread-1:PulsarLedgerIdGenerator@68] - Error generating ledger id: org.apache.pulsar.metadata.api.MetadataStoreException: org.rocksdb.RocksDBException: TimedOut(LockTimeout)
```
there are two plans i have,
1. make method RocksdbMetadataStore#storePut synchronized, since Rocksdb as metadata store is mostly used at light-weight environment with light workload
2. make RocksdbMetadataStore extend AbstractBatchedMetadataStore to avoid multi-thread operation
https://github.com/apache/pulsar/blob/c4fcaa6824095028f1d4ba829baa2ff8b3870107/pulsar-metadata/src/main/java/org/apache/pulsar/metadata/impl/RocksdbMetadataStore.java#L513-L526
@Jason918 since you introduced this class at #12776 , what's your opinon?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Jason918 commented on issue #15896: Flaky-test: PulsarLedgerIdGeneratorTest.testGenerateLedgerId
Posted by GitBox <gi...@apache.org>.
Jason918 commented on issue #15896:
URL: https://github.com/apache/pulsar/issues/15896#issuecomment-1151912868
@wuxuanqicn What's the root cause of the lock timeout?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Jason918 commented on issue #15896: Flaky-test: PulsarLedgerIdGeneratorTest.testGenerateLedgerId
Posted by GitBox <gi...@apache.org>.
Jason918 commented on issue #15896:
URL: https://github.com/apache/pulsar/issues/15896#issuecomment-1152288335
@wuxuanqicn I have pushed a PR #16005 to fix this. PTAL.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Flaky-test: PulsarLedgerIdGeneratorTest.testGenerateLedgerId [pulsar]
Posted by "visortelle (via GitHub)" <gi...@apache.org>.
visortelle commented on issue #15896:
URL: https://github.com/apache/pulsar/issues/15896#issuecomment-1951427536
@Jason918 FYI we're still randomly experiencing this issue with Pulsar Standalone when creating resources like namespaces and topics concurrently.
We just switched to sequential resource creation, but finding this GitHub issue and your PR to apply this fix took about a day for us :)
Pulsar 3.1, 3.2.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] merlimat closed issue #15896: Flaky-test: PulsarLedgerIdGeneratorTest.testGenerateLedgerId
Posted by GitBox <gi...@apache.org>.
merlimat closed issue #15896: Flaky-test: PulsarLedgerIdGeneratorTest.testGenerateLedgerId
URL: https://github.com/apache/pulsar/issues/15896
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org