You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2022/11/11 21:19:45 UTC
[GitHub] [beam] Abacn opened a new pull request, #24129: Test Dataproc 2.1 with Flink load tests
Abacn opened a new pull request, #24129:
URL: https://github.com/apache/beam/pull/24129
Investigating #21696
**Please** add a meaningful description for your change here
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`).
- [ ] Mention the appropriate issue in your description (for example: `addresses #123`), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment `fixes #<ISSUE NUMBER>` instead.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/get-started-contributing/#make-the-reviewers-job-easier).
To check the build health, please visit [https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md](https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md)
GitHub Actions Tests Status (on master branch)
------------------------------------------------------------------------------------------------
[](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Go+tests%22+branch%3Amaster+event%3Aschedule)
See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] Abacn commented on a diff in pull request #24129: Test Dataproc 2.1 with Flink load tests
Posted by GitBox <gi...@apache.org>.
Abacn commented on code in PR #24129:
URL: https://github.com/apache/beam/pull/24129#discussion_r1020670888
##########
.test-infra/dataproc/flink_cluster.sh:
##########
@@ -81,7 +81,7 @@ function get_leader() {
application_ids[$i]=`echo $line | sed "s/ .*//"`
application_masters[$i]=`echo $line | sed "s/.*$CLUSTER_NAME/$CLUSTER_NAME/" | sed "s/ .*//"`
i=$((i+1))
- done <<< $(gcloud compute ssh --zone=$GCLOUD_ZONE --quiet yarn@$MASTER_NAME --command="yarn application -list" | grep "$CLUSTER_NAME")
+ done <<< $(gcloud compute ssh --zone=$GCLOUD_ZONE --quiet yarn@$MASTER_NAME --command="yarn application -list" | grep "Apache Flink")
Review Comment:
The line contains $CLUSTER_NAME if part of the domain contains it. Not always be the case. domain can be an IP address.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] Abacn commented on pull request #24129: Test Dataproc 2.1 with Flink load tests
Posted by GitBox <gi...@apache.org>.
Abacn commented on PR #24129:
URL: https://github.com/apache/beam/pull/24129#issuecomment-1312340058
run seed job
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] Abacn commented on pull request #24129: Test Dataproc 2.1 with Flink load tests
Posted by GitBox <gi...@apache.org>.
Abacn commented on PR #24129:
URL: https://github.com/apache/beam/pull/24129#issuecomment-1312355777
Run Load Tests Python Combine Flink Batch
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] Abacn commented on pull request #24129: Test Dataproc 2.1 with Flink load tests
Posted by GitBox <gi...@apache.org>.
Abacn commented on PR #24129:
URL: https://github.com/apache/beam/pull/24129#issuecomment-1312384677
Run Load Tests Python ParDo Flink Batch
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] commented on pull request #24129: Test Dataproc 2.1 with Flink load tests
Posted by GitBox <gi...@apache.org>.
codecov[bot] commented on PR #24129:
URL: https://github.com/apache/beam/pull/24129#issuecomment-1312359182
# [Codecov](https://codecov.io/gh/apache/beam/pull/24129?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#24129](https://codecov.io/gh/apache/beam/pull/24129?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (c773029) into [master](https://codecov.io/gh/apache/beam/commit/5d2dbf957e4e82fb3980726940df02ac67e563cd?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (5d2dbf9) will **increase** coverage by `0.00%`.
> The diff coverage is `n/a`.
> :exclamation: Current head c773029 differs from pull request most recent head e76ebea. Consider uploading reports for the commit e76ebea to get more accurate results
```diff
@@ Coverage Diff @@
## master #24129 +/- ##
=======================================
Coverage 73.32% 73.32%
=======================================
Files 714 714
Lines 96442 96445 +3
=======================================
+ Hits 70718 70721 +3
Misses 24404 24404
Partials 1320 1320
```
| Flag | Coverage Δ | |
|---|---|---|
| python | `83.20% <ø> (+<0.01%)` | :arrow_up: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/24129?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [sdks/python/apache\_beam/transforms/combiners.py](https://codecov.io/gh/apache/beam/pull/24129/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHJhbnNmb3Jtcy9jb21iaW5lcnMucHk=) | `93.05% <0.00%> (-0.39%)` | :arrow_down: |
| [...ks/python/apache\_beam/runners/worker/sdk\_worker.py](https://codecov.io/gh/apache/beam/pull/24129/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvc2RrX3dvcmtlci5weQ==) | `89.08% <0.00%> (-0.17%)` | :arrow_down: |
| [sdks/python/apache\_beam/ml/inference/\_\_init\_\_.py](https://codecov.io/gh/apache/beam/pull/24129/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vbWwvaW5mZXJlbmNlL19faW5pdF9fLnB5) | `100.00% <0.00%> (ø)` | |
| [...s/python/apache\_beam/runners/portability/stager.py](https://codecov.io/gh/apache/beam/pull/24129/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9wb3J0YWJpbGl0eS9zdGFnZXIucHk=) | `87.22% <0.00%> (+0.94%)` | :arrow_up: |
:mega: We’re building smart automated test selection to slash your CI/CD build times. [Learn more](https://about.codecov.io/iterative-testing/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] tvalentyn commented on pull request #24129: Test Dataproc 2.1 with Flink load tests
Posted by GitBox <gi...@apache.org>.
tvalentyn commented on PR #24129:
URL: https://github.com/apache/beam/pull/24129#issuecomment-1313951788
you can merge your own PRs now, right, @Abacn ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] Abacn commented on pull request #24129: Test Dataproc 2.1 with Flink load tests
Posted by GitBox <gi...@apache.org>.
Abacn commented on PR #24129:
URL: https://github.com/apache/beam/pull/24129#issuecomment-1312370823
Run Load Tests Python Combine Flink Batch
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] github-actions[bot] commented on pull request #24129: Test Dataproc 2.1 with Flink load tests
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #24129:
URL: https://github.com/apache/beam/pull/24129#issuecomment-1313806706
Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] Abacn commented on pull request #24129: Test Dataproc 2.1 with Flink load tests
Posted by GitBox <gi...@apache.org>.
Abacn commented on PR #24129:
URL: https://github.com/apache/beam/pull/24129#issuecomment-1312352467
Run Load Tests Python Combine Flink Batch
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] Abacn commented on pull request #24129: Test Dataproc 2.1 with Flink load tests
Posted by GitBox <gi...@apache.org>.
Abacn commented on PR #24129:
URL: https://github.com/apache/beam/pull/24129#issuecomment-1312483430
Test passed: https://ci-beam.apache.org/job/beam_LoadTests_Python_ParDo_Flink_Batch_PR/30/
Looks like not stuck but pretty slow.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] lostluck commented on pull request #24129: Test Dataproc 2.1 with Flink load tests
Posted by GitBox <gi...@apache.org>.
lostluck commented on PR #24129:
URL: https://github.com/apache/beam/pull/24129#issuecomment-1314090375
I'm very exicited this was figured out! Thank you!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] Abacn commented on pull request #24129: Test Dataproc 2.1 with Flink load tests
Posted by GitBox <gi...@apache.org>.
Abacn commented on PR #24129:
URL: https://github.com/apache/beam/pull/24129#issuecomment-1312363310
Run Load Tests Python Combine Flink Batch
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] Abacn commented on pull request #24129: Test Dataproc 2.1 with Flink load tests
Posted by GitBox <gi...@apache.org>.
Abacn commented on PR #24129:
URL: https://github.com/apache/beam/pull/24129#issuecomment-1313963213
> you can merge your own PRs now, right, @Abacn ?
Yeah, thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] Abacn commented on pull request #24129: Test Dataproc 2.1 with Flink load tests
Posted by GitBox <gi...@apache.org>.
Abacn commented on PR #24129:
URL: https://github.com/apache/beam/pull/24129#issuecomment-1313789899
R: @tvalentyn
R: @lostluck
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] Abacn commented on pull request #24129: Test Dataproc 2.1 with Flink load tests
Posted by GitBox <gi...@apache.org>.
Abacn commented on PR #24129:
URL: https://github.com/apache/beam/pull/24129#issuecomment-1312353126
run seed job
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] Abacn merged pull request #24129: Test Dataproc 2.1 with Flink load tests
Posted by GitBox <gi...@apache.org>.
Abacn merged PR #24129:
URL: https://github.com/apache/beam/pull/24129
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] Abacn commented on pull request #24129: Test Dataproc 2.1 with Flink load tests
Posted by GitBox <gi...@apache.org>.
Abacn commented on PR #24129:
URL: https://github.com/apache/beam/pull/24129#issuecomment-1312367413
run seed job
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] Abacn commented on pull request #24129: Test Dataproc 2.1 with Flink load tests
Posted by GitBox <gi...@apache.org>.
Abacn commented on PR #24129:
URL: https://github.com/apache/beam/pull/24129#issuecomment-1312292752
Need to have a flink 1.15 job server container on gcr.io testing locally. To do this need to run the seedjob against the current change
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] Abacn commented on pull request #24129: Test Dataproc 2.1 with Flink load tests
Posted by GitBox <gi...@apache.org>.
Abacn commented on PR #24129:
URL: https://github.com/apache/beam/pull/24129#issuecomment-1312342474
Run Load Tests Python Combine Flink Batch
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] Abacn commented on pull request #24129: Test Dataproc 2.1 with Flink load tests
Posted by GitBox <gi...@apache.org>.
Abacn commented on PR #24129:
URL: https://github.com/apache/beam/pull/24129#issuecomment-1312368112
run seed job
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] Abacn commented on pull request #24129: Test Dataproc 2.1 with Flink load tests
Posted by GitBox <gi...@apache.org>.
Abacn commented on PR #24129:
URL: https://github.com/apache/beam/pull/24129#issuecomment-1312384454
job now running, but stuck:
<img width="810" alt="image" src="https://user-images.githubusercontent.com/8010435/201460672-c54d9f27-b20d-4aec-ae80-f4136ac239db.png">
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org