You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "chamikaramj (via GitHub)" <gi...@apache.org> on 2024/03/12 01:36:58 UTC
[PR] Updates the test suite to use the transform service [beam]
chamikaramj opened a new pull request, #30605:
URL: https://github.com/apache/beam/pull/30605
Tests passes after https://github.com/apache/beam/pull/30491 but they are not actually using the Transform Service (using a jar-based expansion service instead).
This updates the tests to use the Transform Service. Also fixes an issue that causes the test suite to fail originally.
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [ ] Mention the appropriate issue in your description (for example: `addresses #123`), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment `fixes #<ISSUE NUMBER>` instead.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://github.com/apache/beam/blob/master/CONTRIBUTING.md#make-the-reviewers-job-easier).
To check the build health, please visit [https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md](https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md)
GitHub Actions Tests Status (on master branch)
------------------------------------------------------------------------------------------------
[](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Go+tests%22+branch%3Amaster+event%3Aschedule)
See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI or the [workflows README](https://github.com/apache/beam/blob/master/.github/workflows/README.md) to see a list of phrases to trigger workflows.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Updates the test suite to use the transform service [beam]
Posted by "chamikaramj (via GitHub)" <gi...@apache.org>.
chamikaramj commented on PR #30605:
URL: https://github.com/apache/beam/pull/30605#issuecomment-1992235111
assign set of reviewers
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Updates the test suite to use the transform service [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #30605:
URL: https://github.com/apache/beam/pull/30605#issuecomment-1990057353
Checks are failing. Will not request review until checks are succeeding. If you'd like to override that behavior, comment `assign set of reviewers`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Updates the test suite to use the transform service [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #30605:
URL: https://github.com/apache/beam/pull/30605#issuecomment-1992236902
Assigning reviewers. If you would like to opt out of this review, comment `assign to next reviewer`:
R: @shunping for label python.
R: @damccorm for label build.
R: @shunping for label io.
Available commands:
- `stop reviewer notifications` - opt out of the automated review tooling
- `remind me after tests pass` - tag the comment author after tests pass
- `waiting on author` - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)
The PR bot will only process comments in the main thread (not review comments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Updates the test suite to use the transform service [beam]
Posted by "chamikaramj (via GitHub)" <gi...@apache.org>.
chamikaramj commented on code in PR #30605:
URL: https://github.com/apache/beam/pull/30605#discussion_r1522141244
##########
.github/trigger_files/beam_PostCommit_TransformService_Direct.json:
##########
Review Comment:
Done.
##########
sdks/python/expansion-service-container/boot.go:
##########
@@ -145,9 +145,20 @@ func launchExpansionServiceProcess() error {
args := []string{"-m", expansionServiceEntrypoint, "-p", strconv.Itoa(*port), "--fully_qualified_name_glob", "*"}
- if *requirements_file != "" {
- log.Printf("Received the requirements file %v", *requirements_file)
- updatedRequirementsFileName, err := getUpdatedRequirementsFile(*requirements_file, *dependencies_dir)
+ // Requirements file with dependencies to install.
+ // Note that we have to look for the requirements file in the dependencies
+ // volume here not the requirements file at the top level. Latter provides
+ // Beam dependencies.
+ dependencies_requirements_file := filepath.Join(*dependencies_dir, *requirements_file)
+ dependencies_requirements_file_exists := false
+ if _, err := os.Stat(dependencies_requirements_file); err == nil {
+ dependencies_requirements_file_exists = true
+ }
+
+ // We only try to install dependencies, if the requirements file exists.
+ if dependencies_requirements_file_exists {
Review Comment:
Done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Updates the test suite to use the transform service [beam]
Posted by "chamikaramj (via GitHub)" <gi...@apache.org>.
chamikaramj merged PR #30605:
URL: https://github.com/apache/beam/pull/30605
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Updates the test suite to use the transform service [beam]
Posted by "shunping (via GitHub)" <gi...@apache.org>.
shunping commented on PR #30605:
URL: https://github.com/apache/beam/pull/30605#issuecomment-1992409674
LGTM on code change in python
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Updates the test suite to use the transform service [beam]
Posted by "damccorm (via GitHub)" <gi...@apache.org>.
damccorm commented on code in PR #30605:
URL: https://github.com/apache/beam/pull/30605#discussion_r1522087790
##########
sdks/python/expansion-service-container/boot.go:
##########
@@ -145,9 +145,20 @@ func launchExpansionServiceProcess() error {
args := []string{"-m", expansionServiceEntrypoint, "-p", strconv.Itoa(*port), "--fully_qualified_name_glob", "*"}
- if *requirements_file != "" {
- log.Printf("Received the requirements file %v", *requirements_file)
- updatedRequirementsFileName, err := getUpdatedRequirementsFile(*requirements_file, *dependencies_dir)
+ // Requirements file with dependencies to install.
+ // Note that we have to look for the requirements file in the dependencies
+ // volume here not the requirements file at the top level. Latter provides
+ // Beam dependencies.
+ dependencies_requirements_file := filepath.Join(*dependencies_dir, *requirements_file)
+ dependencies_requirements_file_exists := false
+ if _, err := os.Stat(dependencies_requirements_file); err == nil {
+ dependencies_requirements_file_exists = true
+ }
+
+ // We only try to install dependencies, if the requirements file exists.
+ if dependencies_requirements_file_exists {
Review Comment:
The indentation here seems off, this should be in line with the comment, right?
Non-blocking for this PR, but seems like we should catch this with a go fmt check. It would be good to bring all the boot.go files under https://github.com/apache/beam/blob/986f67fd92b27dee0c95c5965de98ba6f84a0143/.github/workflows/go_tests.yml#L61
##########
sdks/python/expansion-service-container/boot.go:
##########
@@ -145,9 +145,20 @@ func launchExpansionServiceProcess() error {
args := []string{"-m", expansionServiceEntrypoint, "-p", strconv.Itoa(*port), "--fully_qualified_name_glob", "*"}
- if *requirements_file != "" {
- log.Printf("Received the requirements file %v", *requirements_file)
- updatedRequirementsFileName, err := getUpdatedRequirementsFile(*requirements_file, *dependencies_dir)
+ // Requirements file with dependencies to install.
+ // Note that we have to look for the requirements file in the dependencies
+ // volume here not the requirements file at the top level. Latter provides
+ // Beam dependencies.
+ dependencies_requirements_file := filepath.Join(*dependencies_dir, *requirements_file)
+ dependencies_requirements_file_exists := false
+ if _, err := os.Stat(dependencies_requirements_file); err == nil {
+ dependencies_requirements_file_exists = true
+ }
+
+ // We only try to install dependencies, if the requirements file exists.
+ if dependencies_requirements_file_exists {
Review Comment:
For this pr, could you run `go fmt boot.go`
##########
.github/trigger_files/beam_PostCommit_TransformService_Direct.json:
##########
Review Comment:
Lets add content here and check it in like https://github.com/apache/beam/commit/97f8dae92f2a3477dee35c38ab55d31bbc63ca63 for consistency
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org