You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2022/09/12 20:00:34 UTC

[GitHub] [beam] Abacn opened a new pull request, #23191: Test fix Kafka Performance test batch

Abacn opened a new pull request, #23191:
URL: https://github.com/apache/beam/pull/23191

   **Please** add a meaningful description for your change here
   
   ------------------------
   
   Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
   
    - [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`).
    - [ ] Mention the appropriate issue in your description (for example: `addresses #123`), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment `fixes #<ISSUE NUMBER>` instead.
    - [ ] Update `CHANGES.md` with noteworthy changes.
    - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/get-started-contributing/#make-the-reviewers-job-easier).
   
   To check the build health, please visit [https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md](https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md)
   
   GitHub Actions Tests Status (on master branch)
   ------------------------------------------------------------------------------------------------
   [![Build python source distribution and wheels](https://github.com/apache/beam/workflows/Build%20python%20source%20distribution%20and%20wheels/badge.svg?branch=master&event=schedule)](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
   [![Python tests](https://github.com/apache/beam/workflows/Python%20tests/badge.svg?branch=master&event=schedule)](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
   [![Java tests](https://github.com/apache/beam/workflows/Java%20Tests/badge.svg?branch=master&event=schedule)](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
   [![Go tests](https://github.com/apache/beam/workflows/Go%20tests/badge.svg?branch=master&event=schedule)](https://github.com/apache/beam/actions?query=workflow%3A%22Go+tests%22+branch%3Amaster+event%3Aschedule)
   
   See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] github-actions[bot] commented on pull request #23191: Test fix Kafka Performance test batch

Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #23191:
URL: https://github.com/apache/beam/pull/23191#issuecomment-1246035642

   Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] johnjcasey commented on pull request #23191: Test fix Kafka Performance test batch

Posted by GitBox <gi...@apache.org>.
johnjcasey commented on PR #23191:
URL: https://github.com/apache/beam/pull/23191#issuecomment-1248228674

   Merging this despite failing performance test. The test never worked, and we are merging these changes in to try and fix it


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] Abacn commented on pull request #23191: Test fix Kafka Performance test batch

Posted by GitBox <gi...@apache.org>.
Abacn commented on PR #23191:
URL: https://github.com/apache/beam/pull/23191#issuecomment-1245847452

   Run Java KafkaIO Performance Test


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] github-actions[bot] commented on pull request #23191: Test fix Kafka Performance test batch

Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #23191:
URL: https://github.com/apache/beam/pull/23191#issuecomment-1245942510

   Assigning reviewers. If you would like to opt out of this review, comment `assign to next reviewer`:
   
   R: @robertwb for label java.
   R: @ahmedabu98 for label io.
   
   Available commands:
   - `stop reviewer notifications` - opt out of the automated review tooling
   - `remind me after tests pass` - tag the comment author after tests pass
   - `waiting on author` - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)
   
   The PR bot will only process comments in the main thread (not review comments).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] Abacn commented on pull request #23191: Test fix Kafka Performance test batch

Posted by GitBox <gi...@apache.org>.
Abacn commented on PR #23191:
URL: https://github.com/apache/beam/pull/23191#issuecomment-1247312754

   The test configuration change still does not take effect. From log it shows "--kafkaTopic=beam","--readTimeout=900" for batch read job. If it takes effect this should be "--kafkaTopic=beam-batch","--readTimeout=1800"


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] Abacn commented on pull request #23191: Test fix Kafka Performance test batch

Posted by GitBox <gi...@apache.org>.
Abacn commented on PR #23191:
URL: https://github.com/apache/beam/pull/23191#issuecomment-1244371323

   Run Java KafkaIO Performance Test


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] Abacn commented on pull request #23191: Test fix Kafka Performance test batch

Posted by GitBox <gi...@apache.org>.
Abacn commented on PR #23191:
URL: https://github.com/apache/beam/pull/23191#issuecomment-1244470562

   Run Java KafkaIO Performance Test


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] johnjcasey commented on pull request #23191: Test fix Kafka Performance test batch

Posted by GitBox <gi...@apache.org>.
johnjcasey commented on PR #23191:
URL: https://github.com/apache/beam/pull/23191#issuecomment-1247212018

   
   Run Java KafkaIO Performance Test
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] Abacn commented on pull request #23191: Test fix Kafka Performance test batch

Posted by GitBox <gi...@apache.org>.
Abacn commented on PR #23191:
URL: https://github.com/apache/beam/pull/23191#issuecomment-1246034700

   Test now (correctly) time out. Running seed job should resolve.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] Abacn commented on pull request #23191: Test fix Kafka Performance test batch

Posted by GitBox <gi...@apache.org>.
Abacn commented on PR #23191:
URL: https://github.com/apache/beam/pull/23191#issuecomment-1248821221

   Pipeline now runs, though get assertion error: "Expected: "0f12c27c9a7672e14775594be66cad9a". but: was "966684bbc4d071c54466678005c86eec"


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] Abacn commented on pull request #23191: Test fix Kafka Performance test batch

Posted by GitBox <gi...@apache.org>.
Abacn commented on PR #23191:
URL: https://github.com/apache/beam/pull/23191#issuecomment-1246034833

   R: @johnjcasey 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] johnjcasey commented on pull request #23191: Test fix Kafka Performance test batch

Posted by GitBox <gi...@apache.org>.
johnjcasey commented on PR #23191:
URL: https://github.com/apache/beam/pull/23191#issuecomment-1247245484

   Run Java KafkaIO Performance Test


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] Abacn commented on pull request #23191: Test fix Kafka Performance test batch

Posted by GitBox <gi...@apache.org>.
Abacn commented on PR #23191:
URL: https://github.com/apache/beam/pull/23191#issuecomment-1244832842

   Run Java KafkaIO Performance Test


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] Abacn commented on a diff in pull request #23191: Test fix Kafka Performance test batch

Posted by GitBox <gi...@apache.org>.
Abacn commented on code in PR #23191:
URL: https://github.com/apache/beam/pull/23191#discussion_r970055394


##########
.test-infra/kubernetes/kafka-cluster/05-kafka/configmap-config.yaml:
##########
@@ -35,7 +35,7 @@ data:
     echo "Applying runtime configuration using confluentinc/cp-kafka:5.0.1"
     kafka-topics --zookeeper zookeeper:2181 --create --if-not-exists --force --topic beam --partitions 1 --replication-factor 3
     kafka-configs --zookeeper zookeeper:2181 --entity-type topics --entity-name beam --describe
-    kafka-topics --zookeeper zookeeper:2181 --create --if-not-exists --force --topic beam-runnerv2 --partitions 1 --replication-factor 3

Review Comment:
   This topic is no longer used as the duplicate test removed in #23027



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] Abacn commented on pull request #23191: Test fix Kafka Performance test batch

Posted by GitBox <gi...@apache.org>.
Abacn commented on PR #23191:
URL: https://github.com/apache/beam/pull/23191#issuecomment-1245922462

   There are three problems:
   1. K8s instance used for testing running out of memory and causing kafka workload fail to initialize. Cleaned up in the backend.
   2. testKafkaIOReadsAndWritesCorrectlyInBatch trying to write/read non-exist kafka topic. Fixed in config. Needs to run seed job to take effect.
   3. Once the first two issues resolved, found that the test does not end in time. Bumped timeout.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] johnjcasey commented on pull request #23191: Test fix Kafka Performance test batch

Posted by GitBox <gi...@apache.org>.
johnjcasey commented on PR #23191:
URL: https://github.com/apache/beam/pull/23191#issuecomment-1247151696

   run seed job


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] Abacn commented on pull request #23191: Test fix Kafka Performance test batch

Posted by GitBox <gi...@apache.org>.
Abacn commented on PR #23191:
URL: https://github.com/apache/beam/pull/23191#issuecomment-1245957189

   Run Java KafkaIO Performance Test


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] Abacn commented on pull request #23191: Test fix Kafka Performance test batch

Posted by GitBox <gi...@apache.org>.
Abacn commented on PR #23191:
URL: https://github.com/apache/beam/pull/23191#issuecomment-1244326548

   Run Java KafkaIO Performance Test


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] Abacn commented on pull request #23191: Test fix Kafka Performance test batch

Posted by GitBox <gi...@apache.org>.
Abacn commented on PR #23191:
URL: https://github.com/apache/beam/pull/23191#issuecomment-1244667625

   Run Java KafkaIO Performance Test


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] johnjcasey merged pull request #23191: Test fix Kafka Performance test batch

Posted by GitBox <gi...@apache.org>.
johnjcasey merged PR #23191:
URL: https://github.com/apache/beam/pull/23191


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org