You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2023/01/10 17:03:16 UTC
[GitHub] [beam] Abacn opened a new pull request, #24964: Fix kafka performance test write failing due to server overwhelmed
Abacn opened a new pull request, #24964:
URL: https://github.com/apache/beam/pull/24964
Part of #22303
Kafka performance test starts failing again after #24879 in. This is because we removed shuffle=appliance there and then read from shuffle becomes faster. However, the producer is unable to digests data within timeout (see flooding warning log of `send failed : 'Expiring 148 record(s) for beam-sdf-0:130675 ms has passed since batch creation'`)
entered #24963 for the IO connector fix. For now remove ReShuffle and make the test green again.
* Remove ReShuffle before write
* Minor Optimizations on WriteCallBack
**Please** add a meaningful description for your change here
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [ ] Mention the appropriate issue in your description (for example: `addresses #123`), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment `fixes #<ISSUE NUMBER>` instead.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/get-started-contributing/#make-the-reviewers-job-easier).
To check the build health, please visit [https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md](https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md)
GitHub Actions Tests Status (on master branch)
------------------------------------------------------------------------------------------------
[](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Go+tests%22+branch%3Amaster+event%3Aschedule)
See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] Abacn commented on pull request #24964: Fix kafka performance test write failing due to server overwhelmed
Posted by GitBox <gi...@apache.org>.
Abacn commented on PR #24964:
URL: https://github.com/apache/beam/pull/24964#issuecomment-1378001852
Run Java_Kafka_IO_Direct PreCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] Abacn merged pull request #24964: Fix kafka performance test write failing due to server overwhelmed
Posted by GitBox <gi...@apache.org>.
Abacn merged PR #24964:
URL: https://github.com/apache/beam/pull/24964
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] Abacn commented on pull request #24964: Fix kafka performance test write failing due to server overwhelmed
Posted by GitBox <gi...@apache.org>.
Abacn commented on PR #24964:
URL: https://github.com/apache/beam/pull/24964#issuecomment-1377580364
Run Java KafkaIO Performance Test
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] Abacn commented on pull request #24964: Fix kafka performance test write failing due to server overwhelmed
Posted by GitBox <gi...@apache.org>.
Abacn commented on PR #24964:
URL: https://github.com/apache/beam/pull/24964#issuecomment-1378009975
Run Java KafkaIO Performance Test
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] Abacn commented on pull request #24964: Fix kafka performance test write failing due to server overwhelmed
Posted by GitBox <gi...@apache.org>.
Abacn commented on PR #24964:
URL: https://github.com/apache/beam/pull/24964#issuecomment-1378818024
hold on for now. Removing the reshuffle in testbatch write causing batch test has high probability fail (reshuffle added before to fix this issue)
We need to fix the IO connector to make sure it is robust to incoming traffic
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] Abacn commented on pull request #24964: Fix kafka performance test write failing due to server overwhelmed
Posted by GitBox <gi...@apache.org>.
Abacn commented on PR #24964:
URL: https://github.com/apache/beam/pull/24964#issuecomment-1379543289
Run Java KafkaIO Performance Test
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] Abacn commented on pull request #24964: Fix kafka performance test write failing due to server overwhelmed
Posted by GitBox <gi...@apache.org>.
Abacn commented on PR #24964:
URL: https://github.com/apache/beam/pull/24964#issuecomment-1379686826
Run Java KafkaIO Performance Test
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] github-actions[bot] commented on pull request #24964: Fix kafka performance test write failing due to server overwhelmed
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #24964:
URL: https://github.com/apache/beam/pull/24964#issuecomment-1377581114
Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] Abacn commented on a diff in pull request #24964: Fix kafka performance test write failing due to server overwhelmed
Posted by GitBox <gi...@apache.org>.
Abacn commented on code in PR #24964:
URL: https://github.com/apache/beam/pull/24964#discussion_r1067646215
##########
sdks/java/io/kafka/src/test/java/org/apache/beam/sdk/io/kafka/KafkaIOIT.java:
##########
@@ -212,14 +210,16 @@ public void testKafkaIOReadsAndWritesCorrectlyInStreaming() throws IOException {
PipelineResult writeResult = writePipeline.run();
PipelineResult.State writeState = writeResult.waitUntilFinish();
+ // Fail the test if pipeline failed.
+ assertNotEquals(PipelineResult.State.FAILED, writeState);
Review Comment:
Current failing on master is due to the write pipeline continuously hitting OOM and cannot finish the bundle within timeout; then read pipeline does not receive expected number of data, e.g. https://ci-beam.apache.org/view/PerformanceTests/job/beam_PerformanceTests_Kafka_IO/3546/console
```
org.apache.beam.sdk.io.kafka.KafkaIOIT > testKafkaIOReadsAndWritesCorrectlyInStreaming FAILED
20:26:04 java.lang.AssertionError: actual number of records 34569527 smaller than expected: 100000000.
```
put this assertion right after writeResult.waitUntilFinish() to make test fail early if write pipeline fails.
The OOM issue is also a motivation of using single Callback instance (though not have decisive help)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] Abacn commented on pull request #24964: Fix kafka performance test write failing due to server overwhelmed
Posted by GitBox <gi...@apache.org>.
Abacn commented on PR #24964:
URL: https://github.com/apache/beam/pull/24964#issuecomment-1378010127
Run Python_PVR_Flink PreCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] Abacn commented on pull request #24964: Fix kafka performance test write failing due to server overwhelmed
Posted by GitBox <gi...@apache.org>.
Abacn commented on PR #24964:
URL: https://github.com/apache/beam/pull/24964#issuecomment-1378001372
Run RAT PreCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] Abacn commented on pull request #24964: Fix kafka performance test write failing due to server overwhelmed
Posted by GitBox <gi...@apache.org>.
Abacn commented on PR #24964:
URL: https://github.com/apache/beam/pull/24964#issuecomment-1377693470
Performance test succeeded after removing Reshuffle (on test branch)
https://ci-beam.apache.org/job/beam_PerformanceTests_Kafka_IO/3538/
Seems like jenkins trigger broken again
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] Abacn commented on pull request #24964: Fix kafka performance test write failing due to server overwhelmed
Posted by GitBox <gi...@apache.org>.
Abacn commented on PR #24964:
URL: https://github.com/apache/beam/pull/24964#issuecomment-1379735937
test succeeded. Anyway flaky is better than permared. PTAL @johnjcasey
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] Abacn commented on pull request #24964: Fix kafka performance test write failing due to server overwhelmed
Posted by GitBox <gi...@apache.org>.
Abacn commented on PR #24964:
URL: https://github.com/apache/beam/pull/24964#issuecomment-1382459970
GA Playground Examples CI failure irrelevant. merging for now
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] Abacn commented on pull request #24964: Fix kafka performance test write failing due to server overwhelmed
Posted by GitBox <gi...@apache.org>.
Abacn commented on PR #24964:
URL: https://github.com/apache/beam/pull/24964#issuecomment-1377579226
R: @johnjcasey
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] johnjcasey commented on a diff in pull request #24964: Fix kafka performance test write failing due to server overwhelmed
Posted by GitBox <gi...@apache.org>.
johnjcasey commented on code in PR #24964:
URL: https://github.com/apache/beam/pull/24964#discussion_r1066290860
##########
sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaWriter.java:
##########
@@ -42,13 +42,16 @@
*/
class KafkaWriter<K, V> extends DoFn<ProducerRecord<K, V>, Void> {
+ protected transient @Nullable Callback callback;
Review Comment:
what is the motivation for a single callback?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] Abacn commented on a diff in pull request #24964: Fix kafka performance test write failing due to server overwhelmed
Posted by GitBox <gi...@apache.org>.
Abacn commented on code in PR #24964:
URL: https://github.com/apache/beam/pull/24964#discussion_r1066429749
##########
sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaWriter.java:
##########
@@ -42,13 +42,16 @@
*/
class KafkaWriter<K, V> extends DoFn<ProducerRecord<K, V>, Void> {
+ protected transient @Nullable Callback callback;
Review Comment:
SendCallback does not hold any state other than a reference to its outer class (KafkaWriter.this). It modified member variables of KafkaWriter in a synchronized block. This can be set as a transient member of KafkaWriter which has less overhead (no need new object each processElement call, and this is also what flink kafka connector doing)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org