You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2022/03/23 17:04:51 UTC
[GitHub] [beam] deepix opened a new pull request #17167: [BEAM-14160] Parse filesToStage in Java expansion service
deepix opened a new pull request #17167:
URL: https://github.com/apache/beam/pull/17167
### Summary
A filesToStage arg when starting expansion service will let us customize and/or control what files can be staged. This speeds up pipeline execution in environments where we can pre-stage jars in the Java harness SDK.
This is based on guidance by @chamikaramj in a [mailing list thread](https://lists.apache.org/thread/knkf6yn52z1fxzbcgkt22dv48o8055bh).
### Testing
Added two unit tests.
Also ran expansion service separately and a portable Python pipeline that uses Kafka I/O (hence Java expansion service). Pipeline ran successfully.
```
$ java -cp ./runners/flink/1.13/job-server/build/libs/beam-runners-flink-1.13-job-server-2.35.0-SNAPSHOT.jar org.apache.beam.sdk.expansion.service.ExpansionService 8096 --filesToStage="foo.jar"
...
Mar 22, 2022 7:04:30 PM org.apache.beam.sdk.expansion.service.ExpansionService$TransformProvider getDependencies
INFO: Staging to files from the classpath: 1, [foo.jar]
```
One can also prevent staging (useful when we have already pre-staged relevant jars in the Java harness SDK):
```
$ java -cp ./runners/flink/1.13/job-server/build/libs/beam-runners-flink-1.13-job-server-2.35.0-SNAPSHOT.jar org.apache.beam.sdk.expansion.service.ExpansionService 8096 --filesToStage=
...
Mar 22, 2022 7:03:59 PM org.apache.beam.sdk.expansion.service.ExpansionService$TransformProvider getDependencies
INFO: Staging to files from the classpath: 1, []
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] chamikaramj commented on pull request #17167: [BEAM-14160] Parse filesToStage in Java expansion service
Posted by GitBox <gi...@apache.org>.
chamikaramj commented on pull request #17167:
URL: https://github.com/apache/beam/pull/17167#issuecomment-1079233850
Thanks. I was just waiting for the tests to pass.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] chamikaramj commented on pull request #17167: [BEAM-14160] Parse filesToStage in Java expansion service
Posted by GitBox <gi...@apache.org>.
chamikaramj commented on pull request #17167:
URL: https://github.com/apache/beam/pull/17167#issuecomment-1077887348
Thanks. Will take a look.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] deepix commented on pull request #17167: [BEAM-14160] Parse filesToStage in Java expansion service
Posted by GitBox <gi...@apache.org>.
deepix commented on pull request #17167:
URL: https://github.com/apache/beam/pull/17167#issuecomment-1081330623
@lukecwik Thanks, this needed a change in how Python SDK was treating non-local files, fixed by @chamikaramj in #17199.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] chamikaramj commented on pull request #17167: [BEAM-14160] Parse filesToStage in Java expansion service
Posted by GitBox <gi...@apache.org>.
chamikaramj commented on pull request #17167:
URL: https://github.com/apache/beam/pull/17167#issuecomment-1078555857
Run Java PreCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] lukecwik commented on pull request #17167: [BEAM-14160] Parse filesToStage in Java expansion service
Posted by GitBox <gi...@apache.org>.
lukecwik commented on pull request #17167:
URL: https://github.com/apache/beam/pull/17167#issuecomment-1081094872
This change broke the python xlang postcommit, filed https://issues.apache.org/jira/browse/BEAM-14191
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] deepix commented on pull request #17167: [BEAM-14160] Parse filesToStage in Java expansion service
Posted by GitBox <gi...@apache.org>.
deepix commented on pull request #17167:
URL: https://github.com/apache/beam/pull/17167#issuecomment-1076566319
@chamikaramj Please take a look at this PR, thank you.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] deepix commented on pull request #17167: [BEAM-14160] Parse filesToStage in Java expansion service
Posted by GitBox <gi...@apache.org>.
deepix commented on pull request #17167:
URL: https://github.com/apache/beam/pull/17167#issuecomment-1076834278
For the "Run Java PreCommit" failure, I see the failure even when I reset to HEAD, indicating that it's a pre-existing condition. Let me know if I'm mistaken.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] chamikaramj merged pull request #17167: [BEAM-14160] Parse filesToStage in Java expansion service
Posted by GitBox <gi...@apache.org>.
chamikaramj merged pull request #17167:
URL: https://github.com/apache/beam/pull/17167
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org