You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2021/09/28 01:46:33 UTC
[GitHub] [beam] ihji opened a new pull request #15606: [BEAM-12978] Customizable dependency for Java external transform
ihji opened a new pull request #15606:
URL: https://github.com/apache/beam/pull/15606
**Please** add a meaningful description for your change here
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`).
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier).
`ValidatesRunner` compliance status (on master branch)
--------------------------------------------------------
<table>
<thead>
<tr>
<th>Lang</th>
<th>ULR</th>
<th>Dataflow</th>
<th>Flink</th>
<th>Samza</th>
<th>Spark</th>
<th>Twister2</th>
</tr>
</thead>
<tbody>
<tr>
<td>Go</td>
<td>---</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/badge/icon">
</a>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Samza/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Samza/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/badge/icon">
</a>
</td>
<td>---</td>
</tr>
<tr>
<td>Java</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_ULR/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_ULR/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/badge/icon?subject=V1">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Streaming/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Streaming/lastCompletedBuild/badge/icon?subject=V1+Streaming">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/badge/icon?subject=V1+Java+11">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_VR_Dataflow_V2/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_VR_Dataflow_V2/lastCompletedBuild/badge/icon?subject=V2">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_VR_Dataflow_V2_Streaming/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_VR_Dataflow_V2_Streaming/lastCompletedBuild/badge/icon?subject=V2+Streaming">
</a><br>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/badge/icon?subject=Java+8">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/badge/icon?subject=Java+11">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/badge/icon?subject=Portable">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/badge/icon?subject=Portable+Streaming">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/badge/icon">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Samza/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Samza/lastCompletedBuild/badge/icon?subject=Portable">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/badge/icon">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/badge/icon?subject=Portable">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/badge/icon?subject=Structured+Streaming">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Twister2/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Twister2/lastCompletedBuild/badge/icon">
</a>
</td>
</tr>
<tr>
<td>Python</td>
<td>---</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/badge/icon?subject=V1">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow_V2/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow_V2/lastCompletedBuild/badge/icon?subject=V2">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/badge/icon?subject=ValCont">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_Python_PVR_Flink_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_Python_PVR_Flink_Cron/lastCompletedBuild/badge/icon?subject=Portable">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Flink/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Flink/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Samza/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Samza/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Spark/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Spark/lastCompletedBuild/badge/icon">
</a>
</td>
<td>---</td>
</tr>
<tr>
<td>XLang</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_XVR_Direct/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_XVR_Direct/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_XVR_Dataflow/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_XVR_Dataflow/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_XVR_Flink/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_XVR_Flink/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_XVR_Samza/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_XVR_Samza/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_XVR_Spark/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_XVR_Spark/lastCompletedBuild/badge/icon">
</a>
</td>
<td>---</td>
</tr>
</tbody>
</table>
Examples testing status on various runners
--------------------------------------------------------
<table>
<thead>
<tr>
<th>Lang</th>
<th>ULR</th>
<th>Dataflow</th>
<th>Flink</th>
<th>Samza</th>
<th>Spark</th>
<th>Twister2</th>
</tr>
</thead>
<tbody>
<tr>
<td>Go</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
</tr>
<tr>
<td>Java</td>
<td>---</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_Java_Examples_Dataflow_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_Java_Examples_Dataflow_Cron/lastCompletedBuild/badge/icon?subject=V1">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_Java_Examples_Dataflow_Java11_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_Java_Examples_Dataflow_Java11_Cron/lastCompletedBuild/badge/icon?subject=V1+Java11">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_Examples_Dataflow_V2/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_Examples_Dataflow_V2/lastCompletedBuild/badge/icon?subject=V2">
</a><br>
</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
</tr>
<tr>
<td>Python</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
</tr>
<tr>
<td>XLang</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
</tr>
</tbody>
</table>
Post-Commit SDK/Transform Integration Tests Status (on master branch)
------------------------------------------------------------------------------------------------
<table>
<thead>
<tr>
<th>Go</th>
<th>Java</th>
<th>Python</th>
</tr>
</thead>
<tbody>
<tr>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/badge/icon?subject=3.6">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/badge/icon?subject=3.7">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Python38/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Python38/lastCompletedBuild/badge/icon?subject=3.8">
</a>
</td>
</tr>
</tbody>
</table>
Pre-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
<table>
<thead>
<tr>
<th>---</th>
<th>Java</th>
<th>Python</th>
<th>Go</th>
<th>Website</th>
<th>Whitespace</th>
<th>Typescript</th>
</tr>
</thead>
<tbody>
<tr>
<td>Non-portable</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/badge/icon">
</a><br>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/badge/icon?subject=Tests">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_PythonLint_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_PythonLint_Cron/lastCompletedBuild/badge/icon?subject=Lint">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_PythonDocker_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_PythonDocker_Cron/badge/icon?subject=Docker">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_PythonDocs_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_PythonDocs_Cron/badge/icon?subject=Docs">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_Go_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_Go_Cron/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_Website_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_Website_Cron/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_Whitespace_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_Whitespace_Cron/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_Typescript_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_Typescript_Cron/lastCompletedBuild/badge/icon">
</a>
</td>
</tr>
<tr>
<td>Portable</td>
<td>---</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_Portable_Python_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_Portable_Python_Cron/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_GoPortable_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_GoPortable_Cron/lastCompletedBuild/badge/icon">
</a>
</td>
<td>---</td>
<td>---</td>
<td>---</td>
</tr>
</tbody>
</table>
See [.test-infra/jenkins/README](https://github.com/apache/beam/blob/master/.test-infra/jenkins/README.md) for trigger phrase, status and link of all Jenkins jobs.
GitHub Actions Tests Status (on master branch)
------------------------------------------------------------------------------------------------
[](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] ihji removed a comment on pull request #15606: [BEAM-12978] Customizable dependency for Java external transform
Posted by GitBox <gi...@apache.org>.
ihji removed a comment on pull request #15606:
URL: https://github.com/apache/beam/pull/15606#issuecomment-931929883
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] ihji removed a comment on pull request #15606: [BEAM-12978] Customizable dependency for Java external transform
Posted by GitBox <gi...@apache.org>.
ihji removed a comment on pull request #15606:
URL: https://github.com/apache/beam/pull/15606#issuecomment-944612154
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] ihji commented on a change in pull request #15606: [BEAM-12978] Customizable dependency for Java external transform
Posted by GitBox <gi...@apache.org>.
ihji commented on a change in pull request #15606:
URL: https://github.com/apache/beam/pull/15606#discussion_r723731416
##########
File path: sdks/java/expansion-service/src/main/java/org/apache/beam/sdk/expansion/service/ExpansionService.java
##########
@@ -371,6 +371,20 @@ default InputT createInput(Pipeline p, Map<String, PCollection<?>> inputs) {
return extractOutputs(
Pipeline.applyTransform(name, createInput(p, inputs), getTransform(spec)));
}
+
+ default List<String> getDependencies(PipelineOptions options) {
Review comment:
I thought we can use either `ExpansionServiceRegistrar(TransformProvider)` or `ExternalTransformRegistrar(ExternalTransformBuilder)`. Anyhow, it looks like `ExternalTransformBuilder` is the one used with existing Java external IO transforms.
I also added the new method to `ExternalTransformBuilder`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] ihji commented on a change in pull request #15606: [BEAM-12978] Customizable dependency for Java external transform
Posted by GitBox <gi...@apache.org>.
ihji commented on a change in pull request #15606:
URL: https://github.com/apache/beam/pull/15606#discussion_r734220130
##########
File path: sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/ExternalTransformBuilder.java
##########
@@ -44,4 +47,12 @@
/** Builds the transform after it has been configured. */
PTransform<InputT, OutputT> buildExternal(ConfigT configuration);
+
+ /**
+ * List the dependencies needed for this transform. Jars from classpath are used by default when
+ * Optional.empty() is returned.
+ */
Review comment:
Updated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] ihji commented on pull request #15606: [BEAM-12978] Customizable dependency for Java external transform
Posted by GitBox <gi...@apache.org>.
ihji commented on pull request #15606:
URL: https://github.com/apache/beam/pull/15606#issuecomment-944612154
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] chamikaramj commented on a change in pull request #15606: [BEAM-12978] Customizable dependency for Java external transform
Posted by GitBox <gi...@apache.org>.
chamikaramj commented on a change in pull request #15606:
URL: https://github.com/apache/beam/pull/15606#discussion_r718946193
##########
File path: sdks/java/expansion-service/src/main/java/org/apache/beam/sdk/expansion/service/ExpansionService.java
##########
@@ -371,6 +371,20 @@ default InputT createInput(Pipeline p, Map<String, PCollection<?>> inputs) {
return extractOutputs(
Pipeline.applyTransform(name, createInput(p, inputs), getTransform(spec)));
}
+
+ default List<String> getDependencies(PipelineOptions options) {
Review comment:
I don't think a transform author can override this. See here: https://docs.google.com/document/d/1F5CW2UxSE3Jwr3e9u5cFWHMqFVwBO_PgEYYG2CAtrlw/edit?disco=AAAAQhYlvZw
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] ihji commented on a change in pull request #15606: [BEAM-12978] Customizable dependency for Java external transform
Posted by GitBox <gi...@apache.org>.
ihji commented on a change in pull request #15606:
URL: https://github.com/apache/beam/pull/15606#discussion_r736863218
##########
File path: sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/ExternalTransformBuilder.java
##########
@@ -44,4 +47,12 @@
/** Builds the transform after it has been configured. */
PTransform<InputT, OutputT> buildExternal(ConfigT configuration);
+
+ /**
+ * List the dependencies needed for this transform. Jars from classpath are used by default when
Review comment:
> Is this in addition to the dependencies provided through the CLASSPATH/filesToStage option or should this provide the full set of dependencies used for this transform ?
It doesn't include dependencies from classpath by default.
>
> Also, should this be a complete list, including transitive dependencies ?
Yes, it should return a complete list. We just put whatever the method returns into the dependencies field in the environment proto. For better usability, we could provide some utility functions in the future such as automatically downloading jars from artifact specification string or get all transitive dependencies from artifact's pom file.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] ihji commented on pull request #15606: [BEAM-12978] Customizable dependency for Java external transform
Posted by GitBox <gi...@apache.org>.
ihji commented on pull request #15606:
URL: https://github.com/apache/beam/pull/15606#issuecomment-931929809
Run XVR_Flink PostCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] ihji removed a comment on pull request #15606: [BEAM-12978] Customizable dependency for Java external transform
Posted by GitBox <gi...@apache.org>.
ihji removed a comment on pull request #15606:
URL: https://github.com/apache/beam/pull/15606#issuecomment-928596233
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] chamikaramj commented on pull request #15606: [BEAM-12978] Customizable dependency for Java external transform
Posted by GitBox <gi...@apache.org>.
chamikaramj commented on pull request #15606:
URL: https://github.com/apache/beam/pull/15606#issuecomment-953079542
I just realized that there are no unit tests here. Can you add them ?
Also consider adding an integration test for the new feature.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] chamikaramj commented on a change in pull request #15606: [BEAM-12978] Customizable dependency for Java external transform
Posted by GitBox <gi...@apache.org>.
chamikaramj commented on a change in pull request #15606:
URL: https://github.com/apache/beam/pull/15606#discussion_r732248624
##########
File path: sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/ExternalTransformBuilder.java
##########
@@ -44,4 +47,12 @@
/** Builds the transform after it has been configured. */
PTransform<InputT, OutputT> buildExternal(ConfigT configuration);
+
+ /**
+ * List the dependencies needed for this transform. Jars from classpath are used by default when
+ * Optional.empty() is returned.
+ */
Review comment:
I believe you were going to update this by passing in the ConfigT object to getDependencies() call as well ? I think this makes sense since this gives more control by making config parameters available during dependency selection. (for example, select correct set of dependencies for JDBC based on the Database and the version.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] ihji commented on a change in pull request #15606: [BEAM-12978] Customizable dependency for Java external transform
Posted by GitBox <gi...@apache.org>.
ihji commented on a change in pull request #15606:
URL: https://github.com/apache/beam/pull/15606#discussion_r723729076
##########
File path: sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/ExternalTransformBuilder.java
##########
@@ -44,4 +47,12 @@
/** Builds the transform after it has been configured. */
PTransform<InputT, OutputT> buildExternal(ConfigT configuration);
+
+ /**
+ * List the dependencies needed for this transform. Jars from classpath are used by default when
+ * Optional.empty() is returned.
+ */
+ default Optional<List<String>> getDependencies(PipelineOptions options) {
Review comment:
Done.
##########
File path: sdks/java/expansion-service/src/main/java/org/apache/beam/sdk/expansion/service/ExpansionService.java
##########
@@ -371,6 +371,20 @@ default InputT createInput(Pipeline p, Map<String, PCollection<?>> inputs) {
return extractOutputs(
Pipeline.applyTransform(name, createInput(p, inputs), getTransform(spec)));
}
+
+ default List<String> getDependencies(PipelineOptions options) {
Review comment:
I thought we can use either `ExpansionServiceRegistrar(TransformProvider)` or `ExternalTransformRegistrar(ExternalTransformBuilder)`. Anyhow, it looks like `ExternalTransformBuilder` is the one used with existing Java external IO transforms.
I also added the new method to `ExternalTransformBuilder`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] ihji removed a comment on pull request #15606: [BEAM-12978] Customizable dependency for Java external transform
Posted by GitBox <gi...@apache.org>.
ihji removed a comment on pull request #15606:
URL: https://github.com/apache/beam/pull/15606#issuecomment-928596636
Run XVR_Dataflow PostCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] ihji commented on pull request #15606: [BEAM-12978] Customizable dependency for Java external transform
Posted by GitBox <gi...@apache.org>.
ihji commented on pull request #15606:
URL: https://github.com/apache/beam/pull/15606#issuecomment-928595186
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] ihji commented on pull request #15606: [BEAM-12978] Customizable dependency for Java external transform
Posted by GitBox <gi...@apache.org>.
ihji commented on pull request #15606:
URL: https://github.com/apache/beam/pull/15606#issuecomment-931929883
Run XVR_Dataflow PostCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] ihji commented on a change in pull request #15606: [BEAM-12978] Customizable dependency for Java external transform
Posted by GitBox <gi...@apache.org>.
ihji commented on a change in pull request #15606:
URL: https://github.com/apache/beam/pull/15606#discussion_r734220080
##########
File path: sdks/java/expansion-service/src/main/java/org/apache/beam/sdk/expansion/service/ExpansionService.java
##########
@@ -371,6 +382,23 @@ default InputT createInput(Pipeline p, Map<String, PCollection<?>> inputs) {
return extractOutputs(
Pipeline.applyTransform(name, createInput(p, inputs), getTransform(spec)));
}
+
+ default List<String> getDependencies(PipelineOptions options) {
+ List<String> filesToStage = options.as(PortablePipelineOptions.class).getFilesToStage();
+ if (filesToStage == null || filesToStage.isEmpty()) {
Review comment:
already rebased on #15638
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] ihji commented on pull request #15606: [BEAM-12978] Customizable dependency for Java external transform
Posted by GitBox <gi...@apache.org>.
ihji commented on pull request #15606:
URL: https://github.com/apache/beam/pull/15606#issuecomment-932653203
Run Java_Examples_Dataflow PreCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] ihji commented on pull request #15606: [BEAM-12978] Customizable dependency for Java external transform
Posted by GitBox <gi...@apache.org>.
ihji commented on pull request #15606:
URL: https://github.com/apache/beam/pull/15606#issuecomment-932653151
Run XVR_Dataflow PostCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] ihji removed a comment on pull request #15606: [BEAM-12978] Customizable dependency for Java external transform
Posted by GitBox <gi...@apache.org>.
ihji removed a comment on pull request #15606:
URL: https://github.com/apache/beam/pull/15606#issuecomment-932653203
Run Java_Examples_Dataflow PreCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] ihji merged pull request #15606: [BEAM-12978] Customizable dependency for Java external transform
Posted by GitBox <gi...@apache.org>.
ihji merged pull request #15606:
URL: https://github.com/apache/beam/pull/15606
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] chamikaramj commented on a change in pull request #15606: [BEAM-12978] Customizable dependency for Java external transform
Posted by GitBox <gi...@apache.org>.
chamikaramj commented on a change in pull request #15606:
URL: https://github.com/apache/beam/pull/15606#discussion_r732267632
##########
File path: sdks/java/expansion-service/src/main/java/org/apache/beam/sdk/expansion/service/ExpansionService.java
##########
@@ -371,6 +382,23 @@ default InputT createInput(Pipeline p, Map<String, PCollection<?>> inputs) {
return extractOutputs(
Pipeline.applyTransform(name, createInput(p, inputs), getTransform(spec)));
}
+
+ default List<String> getDependencies(PipelineOptions options) {
+ List<String> filesToStage = options.as(PortablePipelineOptions.class).getFilesToStage();
+ if (filesToStage == null || filesToStage.isEmpty()) {
Review comment:
Probably you should sync to reflect changes here: https://github.com/apache/beam/pull/15638
##########
File path: sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/ExternalTransformBuilder.java
##########
@@ -44,4 +47,12 @@
/** Builds the transform after it has been configured. */
PTransform<InputT, OutputT> buildExternal(ConfigT configuration);
+
+ /**
+ * List the dependencies needed for this transform. Jars from classpath are used by default when
Review comment:
Is this in addition to the dependencies provided through the CLASSPATH/filesToStage option or should this provide the full set of dependencies used for this transform ?
Also, should this be a complete list, including transitive dependencies ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] ihji removed a comment on pull request #15606: [BEAM-12978] Customizable dependency for Java external transform
Posted by GitBox <gi...@apache.org>.
ihji removed a comment on pull request #15606:
URL: https://github.com/apache/beam/pull/15606#issuecomment-928596233
Run XVR_Flink PostCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] ihji commented on pull request #15606: [BEAM-12978] Customizable dependency for Java external transform
Posted by GitBox <gi...@apache.org>.
ihji commented on pull request #15606:
URL: https://github.com/apache/beam/pull/15606#issuecomment-928595186
R: @chamikaramj
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] ihji commented on pull request #15606: [BEAM-12978] Customizable dependency for Java external transform
Posted by GitBox <gi...@apache.org>.
ihji commented on pull request #15606:
URL: https://github.com/apache/beam/pull/15606#issuecomment-928596233
Run XVR_Flink PostCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] chamikaramj commented on a change in pull request #15606: [BEAM-12978] Customizable dependency for Java external transform
Posted by GitBox <gi...@apache.org>.
chamikaramj commented on a change in pull request #15606:
URL: https://github.com/apache/beam/pull/15606#discussion_r723565617
##########
File path: sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/ExternalTransformBuilder.java
##########
@@ -44,4 +47,12 @@
/** Builds the transform after it has been configured. */
PTransform<InputT, OutputT> buildExternal(ConfigT configuration);
+
+ /**
+ * List the dependencies needed for this transform. Jars from classpath are used by default when
+ * Optional.empty() is returned.
+ */
+ default Optional<List<String>> getDependencies(PipelineOptions options) {
Review comment:
We should update the dev list regarding this interface change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] ihji removed a comment on pull request #15606: [BEAM-12978] Customizable dependency for Java external transform
Posted by GitBox <gi...@apache.org>.
ihji removed a comment on pull request #15606:
URL: https://github.com/apache/beam/pull/15606#issuecomment-932653151
Run XVR_Dataflow PostCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] chamikaramj commented on a change in pull request #15606: [BEAM-12978] Customizable dependency for Java external transform
Posted by GitBox <gi...@apache.org>.
chamikaramj commented on a change in pull request #15606:
URL: https://github.com/apache/beam/pull/15606#discussion_r718946193
##########
File path: sdks/java/expansion-service/src/main/java/org/apache/beam/sdk/expansion/service/ExpansionService.java
##########
@@ -371,6 +371,20 @@ default InputT createInput(Pipeline p, Map<String, PCollection<?>> inputs) {
return extractOutputs(
Pipeline.applyTransform(name, createInput(p, inputs), getTransform(spec)));
}
+
+ default List<String> getDependencies(PipelineOptions options) {
Review comment:
I don't think a transform author can override this. See here: https://docs.google.com/document/d/1F5CW2UxSE3Jwr3e9u5cFWHMqFVwBO_PgEYYG2CAtrlw/edit?disco=AAAAQhYlvZw
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] ihji commented on a change in pull request #15606: [BEAM-12978] Customizable dependency for Java external transform
Posted by GitBox <gi...@apache.org>.
ihji commented on a change in pull request #15606:
URL: https://github.com/apache/beam/pull/15606#discussion_r723729076
##########
File path: sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/ExternalTransformBuilder.java
##########
@@ -44,4 +47,12 @@
/** Builds the transform after it has been configured. */
PTransform<InputT, OutputT> buildExternal(ConfigT configuration);
+
+ /**
+ * List the dependencies needed for this transform. Jars from classpath are used by default when
+ * Optional.empty() is returned.
+ */
+ default Optional<List<String>> getDependencies(PipelineOptions options) {
Review comment:
Done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] ihji commented on pull request #15606: [BEAM-12978] Customizable dependency for Java external transform
Posted by GitBox <gi...@apache.org>.
ihji commented on pull request #15606:
URL: https://github.com/apache/beam/pull/15606#issuecomment-928596636
Run XVR_Dataflow PostCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org