You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2021/08/24 10:43:59 UTC
[GitHub] [pulsar] lhotari opened a new issue #11760: Flaky-test: C++ test BasicEndToEndTest.testLookupThrottling
lhotari opened a new issue #11760:
URL: https://github.com/apache/pulsar/issues/11760
<!---
Instructions for reporting a flaky test using this issue template:
1. Replace [test class] in title and body with the test class name
2. Replace [test method] in title and body with the test method that failed. Multiple methods are flaky, remove the content that refers to the test method.
3. Replace "url here" with a url to an example failure. In the Github Actions workflow run logs, you can right click on the line number to copy a link to the line. Example of such url is https://github.com/apache/pulsar/pull/8892/checks?check_run_id=1531075794#step:9:377 . The logs are available for a limited amount of time (usually for a few weeks).
4. Replace "relevant parts of the exception stacktrace here" with the a few lines of the stack trace that shows at leat the exception message and the line of test code where the stacktrace occured.
5. Replace "full exception stacktrace here" with the full exception stacktrace from logs. This section will be hidded by default.
6. Remove all unused fields / content to unclutter the reported issue. Remove this comment too.
-->
BasicEndToEndTest.testLookupThrottling is flaky.
[example failure](https://github.com/apache/pulsar/pull/11753/checks?check_run_id=3409367305#step:11:387)
try #1
```
[0;33mNote: Google Test filter = BasicEndToEndTest.testLookupThrottling
[m[0;32m[==========] [mRunning 1 test from 1 test case.
[0;32m[----------] [mGlobal test environment set-up.
[0;32m[----------] [m1 test from BasicEndToEndTest
[0;32m[ RUN ] [mBasicEndToEndTest.testLookupThrottling
2021-08-24 09:22:29.917 DEBUG [140347937134592] ReaderTest:126 | Using Binary Lookup
2021-08-24 09:22:29.917 INFO [140347937134592] ReaderTest:181 | [<none> -> pulsar://localhost:6650] Create ClientConnection, timeout=10000
2021-08-24 09:22:29.918 INFO [140347937134592] ReaderTest:84 | Created connection for pulsar://localhost:6650
2021-08-24 09:22:29.919 DEBUG [140347937134592] ReaderTest:514 | [<none> -> pulsar://localhost:6650] Resolving localhost:6650
2021-08-24 09:22:29.919 DEBUG [140347818075904] ReaderTest:542 | [<none> -> pulsar://localhost:6650] Connecting to 127.0.0.1:6650...
2021-08-24 09:22:29.920 DEBUG [140347818075904] ReaderTest:546 | [<none> -> pulsar://localhost:6650] Resolved hostname localhost to 127.0.0.1:6650
2021-08-24 09:22:29.920 INFO [140347818075904] ReaderTest:366 | [127.0.0.1:51390 -> 127.0.0.1:6650] Connected to broker
2021-08-24 09:22:29.926 DEBUG [140347818075904] ReaderTest:742 | [127.0.0.1:51390 -> 127.0.0.1:6650] Handling incoming command: CONNECTED
2021-08-24 09:22:29.927 DEBUG [140347818075904] ReaderTest:265 | Connection has max message size setting: 5242880
2021-08-24 09:22:29.927 DEBUG [140347818075904] ReaderTest:267 | Current max message size is: 5242880
2021-08-24 09:22:29.927 DEBUG [140347818075904] ReaderTest:151 | PartitionMetadataLookup failed for persistent://public/default/testLookupThrottling, result TooManyLookupRequestException
2021-08-24 09:22:29.927 ERROR [140347818075904] ReaderTest:188 | Error Checking/Getting Partition Metadata while creating producer on persistent://public/default/testLookupThrottling -- TooManyLookupRequestException
2021-08-24 09:22:29.927 INFO [140347937134592] ReaderTest:88 | Subscribing on Topic :testLookupThrottling
2021-08-24 09:22:29.927 DEBUG [140347937134592] ReaderTest:69 | Got connection from pool for pulsar://localhost:6650 use_count: 4 @ 0xe1b0d0
2021-08-24 09:22:29.927 DEBUG [140347937134592] ReaderTest:151 | PartitionMetadataLookup failed for persistent://public/default/testLookupThrottling, result TooManyLookupRequestException
2021-08-24 09:22:29.927 ERROR [140347937134592] ReaderTest:392 | Error Checking/Getting Partition Metadata while Subscribing on persistent://public/default/testLookupThrottling -- TooManyLookupRequestException
2021-08-24 09:22:29.927 INFO [140347937134592] ReaderTest:492 | Closing Pulsar client with 0 producers and 0 consumers
2021-08-24 09:22:29.927 DEBUG [140347937134592] ReaderTest:536 | Shutting down producers and consumers for client
2021-08-24 09:22:29.927 INFO [140347937134592] ReaderTest:1495 | [127.0.0.1:51390 -> 127.0.0.1:6650] Connection closed
```
and try #2
```
[0;33mNote: Google Test filter = BasicEndToEndTest.testLookupThrottling
[m[0;32m[==========] [mRunning 1 test from 1 test case.
[0;32m[----------] [mGlobal test environment set-up.
[0;32m[----------] [m1 test from BasicEndToEndTest
[0;32m[ RUN ] [mBasicEndToEndTest.testLookupThrottling
2021-08-24 09:22:30.328 DEBUG [140233890162688] ReaderTest:126 | Using Binary Lookup
2021-08-24 09:22:30.329 INFO [140233890162688] ReaderTest:181 | [<none> -> pulsar://localhost:6650] Create ClientConnection, timeout=10000
2021-08-24 09:22:30.329 INFO [140233890162688] ReaderTest:84 | Created connection for pulsar://localhost:6650
2021-08-24 09:22:30.330 DEBUG [140233890162688] ReaderTest:514 | [<none> -> pulsar://localhost:6650] Resolving localhost:6650
2021-08-24 09:22:30.331 DEBUG [140233771104000] ReaderTest:542 | [<none> -> pulsar://localhost:6650] Connecting to 127.0.0.1:6650...
2021-08-24 09:22:30.331 DEBUG [140233771104000] ReaderTest:546 | [<none> -> pulsar://localhost:6650] Resolved hostname localhost to 127.0.0.1:6650
2021-08-24 09:22:30.331 INFO [140233771104000] ReaderTest:366 | [127.0.0.1:51408 -> 127.0.0.1:6650] Connected to broker
2021-08-24 09:22:30.331 DEBUG [140233771104000] ReaderTest:742 | [127.0.0.1:51408 -> 127.0.0.1:6650] Handling incoming command: CONNECTED
2021-08-24 09:22:30.331 DEBUG [140233771104000] ReaderTest:265 | Connection has max message size setting: 5242880
2021-08-24 09:22:30.331 DEBUG [140233771104000] ReaderTest:267 | Current max message size is: 5242880
2021-08-24 09:22:30.331 DEBUG [140233771104000] ReaderTest:151 | PartitionMetadataLookup failed for persistent://public/default/testLookupThrottling, result TooManyLookupRequestException
2021-08-24 09:22:30.331 ERROR [140233771104000] ReaderTest:188 | Error Checking/Getting Partition Metadata while creating producer on persistent://public/default/testLookupThrottling -- TooManyLookupRequestException
2021-08-24 09:22:30.332 INFO [140233890162688] ReaderTest:88 | Subscribing on Topic :testLookupThrottling
2021-08-24 09:22:30.332 DEBUG [140233890162688] ReaderTest:69 | Got connection from pool for pulsar://localhost:6650 use_count: 4 @ 0x18c50d0
2021-08-24 09:22:30.332 DEBUG [140233890162688] ReaderTest:151 | PartitionMetadataLookup failed for persistent://public/default/testLookupThrottling, result TooManyLookupRequestException
2021-08-24 09:22:30.332 ERROR [140233890162688] ReaderTest:392 | Error Checking/Getting Partition Metadata while Subscribing on persistent://public/default/testLookupThrottling -- TooManyLookupRequestException
2021-08-24 09:22:30.332 INFO [140233890162688] ReaderTest:492 | Closing Pulsar client with 0 producers and 0 consumers
2021-08-24 09:22:30.332 DEBUG [140233890162688] ReaderTest:536 | Shutting down producers and consumers for client
2021-08-24 09:22:30.332 INFO [140233890162688] ReaderTest:1495 | [127.0.0.1:51408 -> 127.0.0.1:6650] Connection closed
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] merlimat closed issue #11760: Flaky-test: C++ test BasicEndToEndTest.testLookupThrottling
Posted by GitBox <gi...@apache.org>.
merlimat closed issue #11760:
URL: https://github.com/apache/pulsar/issues/11760
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] BewareMyPower commented on issue #11760: Flaky-test: C++ test BasicEndToEndTest.testLookupThrottling
Posted by GitBox <gi...@apache.org>.
BewareMyPower commented on issue #11760:
URL: https://github.com/apache/pulsar/issues/11760#issuecomment-904561514
Current not. From the existing logs, it's because C++ client encountered segmentation fault during client's `shutdown()`, more accurately, the `ExecutorProvider`'s close phase (but there's no log).
https://github.com/apache/pulsar/blob/ac5fce55e1394698072d0cbdd0e289bd289a217e/pulsar-client-cpp/lib/ClientImpl.cc#L571-L573
https://github.com/apache/pulsar/blob/ac5fce55e1394698072d0cbdd0e289bd289a217e/pulsar-client-cpp/lib/ExecutorService.cc#L98-L103
The only clue that I found is that `ExecutorServiceProvider::close` is not thread safe because `executors_` is protected by `mutex_` in `ExecutorServiceProvider::get` while `mutex_` is not locked in `close` method.
I'll fix the theoretical thread-safe problem soon, but not guarantee it can fix the flaky test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lhotari commented on issue #11760: Flaky-test: C++ test BasicEndToEndTest.testLookupThrottling
Posted by GitBox <gi...@apache.org>.
lhotari commented on issue #11760:
URL: https://github.com/apache/pulsar/issues/11760#issuecomment-904529721
@BewareMyPower do you have a solution for this flaky C++ test?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org