You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2022/06/29 17:03:42 UTC
[GitHub] [beam] lostluck opened a new pull request, #22102: Make gradle depdendencies clearer in docker task
lostluck opened a new pull request, #22102:
URL: https://github.com/apache/beam/pull/22102
Attempting to fix the loadtests by fixing #21698
Go Load tests are currently broken because we finally made a backwards incompatible change to the docker container, so the stale containers used by the load tests no longer function properly.
The [last failed build](https://ci-beam.apache.org/job/beam_Publish_Docker_Snapshots/662/) seems to be because the Dockerfile ADD can't find the launcher.
`ADD failed: stat /var/lib/docker/tmp/docker-builder499134148/target/linux_amd64/boot: no such file or directory`
Nothing else seems to be having this issue though, so it's perplexing. This attempts to resolve the missing file.
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`).
- [ ] Mention the appropriate issue in your description (for example: `addresses #123`), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment `fixes #<ISSUE NUMBER>` instead.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier).
To check the build health, please visit [https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md](https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md)
GitHub Actions Tests Status (on master branch)
------------------------------------------------------------------------------------------------
[](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] damccorm commented on pull request #22102: Make gradle depdendencies clearer in docker task
Posted by GitBox <gi...@apache.org>.
damccorm commented on PR #22102:
URL: https://github.com/apache/beam/pull/22102#issuecomment-1170385119
remind me after tests pass
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] github-actions[bot] commented on pull request #22102: Fixes #21698: Use normal Container snapshots for Go Load Tests
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #22102:
URL: https://github.com/apache/beam/pull/22102#issuecomment-1170542531
Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] lostluck commented on pull request #22102: Fixes #21698: Use normal Container snapshots for Go Load Tests
Posted by GitBox <gi...@apache.org>.
lostluck commented on PR #22102:
URL: https://github.com/apache/beam/pull/22102#issuecomment-1170489558
Run Seed Job
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] lostluck merged pull request #22102: Fixes #21698: Use normal Container snapshots for Go Load Tests
Posted by GitBox <gi...@apache.org>.
lostluck merged PR #22102:
URL: https://github.com/apache/beam/pull/22102
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] lostluck commented on pull request #22102: Fixes #21698: Use normal Container snapshots for Go Load Tests
Posted by GitBox <gi...@apache.org>.
lostluck commented on PR #22102:
URL: https://github.com/apache/beam/pull/22102#issuecomment-1170541105
Changing to the normal snapshot container fixed the Dataflow Load test runs.
https://ci-beam.apache.org/job/beam_Publish_Docker_Snapshots/670/ Removing go seems to fix the publish here.
Flink's still busted, but I think it's now getting the same error as the Python runs at least.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] lostluck commented on pull request #22102: Fixes #21698: Use normal Container snapshots for Go Load Tests
Posted by GitBox <gi...@apache.org>.
lostluck commented on PR #22102:
URL: https://github.com/apache/beam/pull/22102#issuecomment-1171442767
Sample load test run and the snapshot publish pass with the chnages, so this is going in. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] lostluck commented on pull request #22102: Make gradle depdendencies clearer in docker task
Posted by GitBox <gi...@apache.org>.
lostluck commented on PR #22102:
URL: https://github.com/apache/beam/pull/22102#issuecomment-1170248141
Beam Publish Docker Snapshots
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] lostluck commented on pull request #22102: Fixes #21698: Use normal Container snapshots for Go Load Tests
Posted by GitBox <gi...@apache.org>.
lostluck commented on PR #22102:
URL: https://github.com/apache/beam/pull/22102#issuecomment-1170541414
R: @damccorm
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] github-actions[bot] commented on pull request #22102: Make gradle depdendencies clearer in docker task
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #22102:
URL: https://github.com/apache/beam/pull/22102#issuecomment-1170381501
Assigning reviewers. If you would like to opt out of this review, comment `assign to next reviewer`:
R: @damccorm for label go.
Available commands:
- `stop reviewer notifications` - opt out of the automated review tooling
- `remind me after tests pass` - tag the comment author after tests pass
- `waiting on author` - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)
The PR bot will only process comments in the main thread (not review comments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] damccorm commented on pull request #22102: Fixes #21698: Use normal Container snapshots for Go Load Tests
Posted by GitBox <gi...@apache.org>.
damccorm commented on PR #22102:
URL: https://github.com/apache/beam/pull/22102#issuecomment-1171379097
Beam Publish Docker Snapshots
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] lostluck commented on a diff in pull request #22102: Fixes #21698: Use normal Container snapshots for Go Load Tests
Posted by GitBox <gi...@apache.org>.
lostluck commented on code in PR #22102:
URL: https://github.com/apache/beam/pull/22102#discussion_r911172120
##########
.test-infra/jenkins/LoadTestsBuilder.groovy:
##########
@@ -25,6 +25,7 @@ import static PythonTestProperties.LOAD_TEST_PYTHON_VERSION
class LoadTestsBuilder {
final static String DOCKER_CONTAINER_REGISTRY = 'gcr.io/apache-beam-testing/beam_portability'
+ final static String DOCKER_CONTAINER_REGISTRY_GO = 'gcr.io/apache-beam-testing/beam-sdk'
Review Comment:
So, I've moved it to a variable, but I'll also point out that there are so many ways these configurations could be refactored to avoid all the repetition, while allowing for one off changes. So further changes of this nature are out of scope for this PR.
Eg. each specific load test has it's configuration repeated in the three languages, with very few changes between them other than the SDK container, and the name.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] lostluck commented on pull request #22102: Make gradle depdendencies clearer in docker task
Posted by GitBox <gi...@apache.org>.
lostluck commented on PR #22102:
URL: https://github.com/apache/beam/pull/22102#issuecomment-1170384328
Run Seed Job
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] damccorm commented on a diff in pull request #22102: Fixes #21698: Use normal Container snapshots for Go Load Tests
Posted by GitBox <gi...@apache.org>.
damccorm commented on code in PR #22102:
URL: https://github.com/apache/beam/pull/22102#discussion_r911181784
##########
.test-infra/jenkins/LoadTestsBuilder.groovy:
##########
@@ -25,6 +25,7 @@ import static PythonTestProperties.LOAD_TEST_PYTHON_VERSION
class LoadTestsBuilder {
final static String DOCKER_CONTAINER_REGISTRY = 'gcr.io/apache-beam-testing/beam_portability'
+ final static String DOCKER_CONTAINER_REGISTRY_GO = 'gcr.io/apache-beam-testing/beam-sdk'
Review Comment:
I agree a refactor would be nice eventually, I like where you landed for this PR though
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] lostluck commented on pull request #22102: Fixes #21698: Use normal Container snapshots for Go Load Tests
Posted by GitBox <gi...@apache.org>.
lostluck commented on PR #22102:
URL: https://github.com/apache/beam/pull/22102#issuecomment-1171378990
Run Seed Job
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] damccorm commented on a diff in pull request #22102: Fixes #21698: Use normal Container snapshots for Go Load Tests
Posted by GitBox <gi...@apache.org>.
damccorm commented on code in PR #22102:
URL: https://github.com/apache/beam/pull/22102#discussion_r910987035
##########
.test-infra/jenkins/LoadTestsBuilder.groovy:
##########
@@ -25,6 +25,7 @@ import static PythonTestProperties.LOAD_TEST_PYTHON_VERSION
class LoadTestsBuilder {
final static String DOCKER_CONTAINER_REGISTRY = 'gcr.io/apache-beam-testing/beam_portability'
+ final static String DOCKER_CONTAINER_REGISTRY_GO = 'gcr.io/apache-beam-testing/beam-sdk'
Review Comment:
Looks like we always exclusively get the go container - could we update to `DOCKER_CONTAINER_GO = 'gcr.io/apache-beam-testing/beam-sdk/beam_go_sdk:latest'` (or maybe drop latest)
##########
.test-infra/jenkins/job_Publish_Docker_Snapshots.groovy:
##########
@@ -42,10 +42,9 @@ job('beam_Publish_Docker_Snapshots') {
rootBuildScriptDir(commonJobProperties.checkoutDir)
commonJobProperties.setGradleSwitches(delegate)
SUPPORTED_CONTAINER_TASKS.each { taskVer ->
- tasks(":sdks:python:container:${taskVer}:dockerPush")
+ tasks(":sdks:python:container:${taskVer}:dockerTagPush")
Review Comment:
I still don't quite get the difference between these two, but why do we need to update the python task? Isn't it just failing on the Go SDK/can we just remove the go one?
Same question about flink as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] lostluck commented on a diff in pull request #22102: Fixes #21698: Use normal Container snapshots for Go Load Tests
Posted by GitBox <gi...@apache.org>.
lostluck commented on code in PR #22102:
URL: https://github.com/apache/beam/pull/22102#discussion_r911172799
##########
.test-infra/jenkins/job_Publish_Docker_Snapshots.groovy:
##########
@@ -42,10 +42,9 @@ job('beam_Publish_Docker_Snapshots') {
rootBuildScriptDir(commonJobProperties.checkoutDir)
commonJobProperties.setGradleSwitches(delegate)
SUPPORTED_CONTAINER_TASKS.each { taskVer ->
- tasks(":sdks:python:container:${taskVer}:dockerPush")
+ tasks(":sdks:python:container:${taskVer}:dockerTagPush")
Review Comment:
Whoops Good catch. This was seeing if it would fix the Go build somehow. It did not and I didn't revert it. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] lostluck commented on pull request #22102: Fixes #21698: Use normal Container snapshots for Go Load Tests
Posted by GitBox <gi...@apache.org>.
lostluck commented on PR #22102:
URL: https://github.com/apache/beam/pull/22102#issuecomment-1171412212
Yup, that's the intent! Needed the Seed job to run, while I breakfasted.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] damccorm commented on pull request #22102: Fixes #21698: Use normal Container snapshots for Go Load Tests
Posted by GitBox <gi...@apache.org>.
damccorm commented on PR #22102:
URL: https://github.com/apache/beam/pull/22102#issuecomment-1171379589
Oh, is mine disabled? Probably
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] damccorm commented on pull request #22102: Make gradle depdendencies clearer in docker task
Posted by GitBox <gi...@apache.org>.
damccorm commented on PR #22102:
URL: https://github.com/apache/beam/pull/22102#issuecomment-1170385909
^to be clear, that's directed at the bot - not me being super passive aggressive 🙃 maybe I need to change that phrase
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] github-actions[bot] commented on pull request #22102: Make gradle depdendencies clearer in docker task
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #22102:
URL: https://github.com/apache/beam/pull/22102#issuecomment-1170385962
Ok - I'll remind @damccorm after tests pass
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] lostluck commented on pull request #22102: Fixes #21698: Use normal Container snapshots for Go Load Tests
Posted by GitBox <gi...@apache.org>.
lostluck commented on PR #22102:
URL: https://github.com/apache/beam/pull/22102#issuecomment-1170490552
Run Seed Job
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] lostluck commented on pull request #22102: Fixes #21698: Use normal Container snapshots for Go Load Tests
Posted by GitBox <gi...@apache.org>.
lostluck commented on PR #22102:
URL: https://github.com/apache/beam/pull/22102#issuecomment-1170509222
Run Seed Job
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org