You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2020/06/30 20:20:52 UTC
[GitHub] [beam] lukecwik opened a new pull request #12137: [BEAM-10392] Attempt to fix/enable detection of rabbitmq:test stuckness.
lukecwik opened a new pull request #12137:
URL: https://github.com/apache/beam/pull/12137
* Ensure that the Consumer uses a thread safe object to pass the messages through (Consumer is invoked on a separate thread as per https://rabbitmq.github.io/rabbitmq-java-client/api/current/com/rabbitmq/client/Consumer.html)
* Add a timeout for the entire test class so we can get a stack trace to narrow down where we are seeing stuckness
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`).
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier).
Post-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
Lang | SDK | Dataflow | Flink | Samza | Spark | Twister2
--- | --- | --- | --- | --- | --- | ---
Go | [](https://ci-beam.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) | ---
Java | [](https://ci-beam.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Twister2/lastCompletedBuild/)
Python | [](https://ci-beam.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow_V2/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Python2_PVR_Flink_Cron/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python35_VR_Flink/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Spark/lastCompletedBuild/) | ---
XLang | --- | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Flink/lastCompletedBuild/) | --- | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Spark/lastCompletedBuild/) | ---
Pre-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
--- |Java | Python | Go | Website
--- | --- | --- | --- | ---
Non-portable | [](https://ci-beam.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PreCommit_PythonLint_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Go_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Website_Cron/lastCompletedBuild/)
Portable | --- | [](https://ci-beam.apache.org/job/beam_PreCommit_Portable_Python_Cron/lastCompletedBuild/) | --- | ---
See [.test-infra/jenkins/README](https://github.com/apache/beam/blob/master/.test-infra/jenkins/README.md) for trigger phrase, status and link of all Jenkins jobs.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] lukecwik commented on pull request #12137: [BEAM-10392] Attempt to fix/enable detection of rabbitmq:test stuckness.
Posted by GitBox <gi...@apache.org>.
lukecwik commented on pull request #12137:
URL: https://github.com/apache/beam/pull/12137#issuecomment-652178160
Run Java PreCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] ajamato commented on pull request #12137: [BEAM-10392] Attempt to fix/enable detection of rabbitmq:test stuckness.
Posted by GitBox <gi...@apache.org>.
ajamato commented on pull request #12137:
URL: https://github.com/apache/beam/pull/12137#issuecomment-652221233
Run Java PreCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] mxm merged pull request #12137: [BEAM-10392] Attempt to fix/enable detection of rabbitmq:test stuckness.
Posted by GitBox <gi...@apache.org>.
mxm merged pull request #12137:
URL: https://github.com/apache/beam/pull/12137
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] lukecwik commented on pull request #12137: [BEAM-10392] Attempt to fix/enable detection of rabbitmq:test stuckness.
Posted by GitBox <gi...@apache.org>.
lukecwik commented on pull request #12137:
URL: https://github.com/apache/beam/pull/12137#issuecomment-652021896
R: @jbonofre @ajamato
CC: @iemejia @mxm
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] mxm commented on pull request #12137: [BEAM-10392] Attempt to fix/enable detection of rabbitmq:test stuckness.
Posted by GitBox <gi...@apache.org>.
mxm commented on pull request #12137:
URL: https://github.com/apache/beam/pull/12137#issuecomment-653570434
Tests are passing now. I think this is good as-is. It enables us to do further debugging of test issues.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] lukecwik commented on pull request #12137: [BEAM-10392] Attempt to fix/enable detection of rabbitmq:test stuckness.
Posted by GitBox <gi...@apache.org>.
lukecwik commented on pull request #12137:
URL: https://github.com/apache/beam/pull/12137#issuecomment-652147585
Run Java PreCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] mxm commented on pull request #12137: [BEAM-10392] Attempt to fix/enable detection of rabbitmq:test stuckness.
Posted by GitBox <gi...@apache.org>.
mxm commented on pull request #12137:
URL: https://github.com/apache/beam/pull/12137#issuecomment-653102269
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] ajamato commented on a change in pull request #12137: [BEAM-10392] Attempt to fix/enable detection of rabbitmq:test stuckness.
Posted by GitBox <gi...@apache.org>.
ajamato commented on a change in pull request #12137:
URL: https://github.com/apache/beam/pull/12137#discussion_r448143856
##########
File path: sdks/java/io/rabbitmq/src/test/java/org/apache/beam/sdk/io/rabbitmq/RabbitMqIOTest.java
##########
@@ -396,18 +394,18 @@ public void testWriteQueue() throws Exception {
connection = connectionFactory.newConnection();
channel = connection.createChannel();
channel.queueDeclare("TEST", true, false, false, null);
- Consumer consumer = new RabbitMqTestUtils.TestConsumer(channel, received);
+ RabbitMqTestUtils.TestConsumer consumer = new RabbitMqTestUtils.TestConsumer(channel);
channel.basicConsume("TEST", true, consumer);
p.run();
- while (received.size() < maxNumRecords) {
+ while (consumer.getReceived().size() < maxNumRecords) {
Thread.sleep(500);
}
- assertEquals(maxNumRecords, received.size());
+ assertEquals(maxNumRecords, consumer.getReceived().size());
Review comment:
nit: I doubt this matters because you only add elements to the received list. but might be safer style to use the same reference instead of calling getReceived() 3 times. Which ensures the exact same snapshot. Up to you if you want to change it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] lukecwik commented on pull request #12137: [BEAM-10392] Attempt to fix/enable detection of rabbitmq:test stuckness.
Posted by GitBox <gi...@apache.org>.
lukecwik commented on pull request #12137:
URL: https://github.com/apache/beam/pull/12137#issuecomment-652709644
> Thanks, looks good!
>
> There is still an issue with the tests:
>
> ```
> java.lang.Exception: Unexpected exception, expected<org.apache.beam.sdk.Pipeline$PipelineExecutionException> but was<java.lang.RuntimeException>
> ```
>
> https://ci-beam.apache.org/job/beam_PreCommit_Java_Phrase/2400/testReport/junit/org.apache.beam.sdk.io.rabbitmq/RabbitMqIOTest/testUseCorrelationIdFailsWhenIdsMissing/
Yeah, once I added the class level timeout rule, it seemed to have exposed an issue. I'll try fixing it since it seems fairly reproducible.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org