You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2021/11/11 02:34:12 UTC
[GitHub] [beam] chamikaramj opened a new pull request #15941: [BEAM-13223] Documents two new features of cross-language transforms
chamikaramj opened a new pull request #15941:
URL: https://github.com/apache/beam/pull/15941
Documents following new features.
* Java-class-lookup based expansion when using Java transforms from Python.
* URN convention for cross-language transforms
Also, adds a pointer to cross-language pattern so that users do not confuse that with the main documentation for cross-language transforms.
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`).
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier).
`ValidatesRunner` compliance status (on master branch)
--------------------------------------------------------
<table>
<thead>
<tr>
<th>Lang</th>
<th>ULR</th>
<th>Dataflow</th>
<th>Flink</th>
<th>Samza</th>
<th>Spark</th>
<th>Twister2</th>
</tr>
</thead>
<tbody>
<tr>
<td>Go</td>
<td>---</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/badge/icon">
</a>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Samza/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Samza/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/badge/icon">
</a>
</td>
<td>---</td>
</tr>
<tr>
<td>Java</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_ULR/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_ULR/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/badge/icon?subject=V1">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Streaming/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Streaming/lastCompletedBuild/badge/icon?subject=V1+Streaming">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/badge/icon?subject=V1+Java+11">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_VR_Dataflow_V2/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_VR_Dataflow_V2/lastCompletedBuild/badge/icon?subject=V2">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_VR_Dataflow_V2_Streaming/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_VR_Dataflow_V2_Streaming/lastCompletedBuild/badge/icon?subject=V2+Streaming">
</a><br>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/badge/icon?subject=Java+8">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/badge/icon?subject=Java+11">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/badge/icon?subject=Portable">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/badge/icon?subject=Portable+Streaming">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/badge/icon">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Samza/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Samza/lastCompletedBuild/badge/icon?subject=Portable">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/badge/icon">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/badge/icon?subject=Portable">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/badge/icon?subject=Structured+Streaming">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Twister2/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Twister2/lastCompletedBuild/badge/icon">
</a>
</td>
</tr>
<tr>
<td>Python</td>
<td>---</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/badge/icon?subject=V1">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow_V2/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow_V2/lastCompletedBuild/badge/icon?subject=V2">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/badge/icon?subject=ValCont">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_Python_PVR_Flink_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_Python_PVR_Flink_Cron/lastCompletedBuild/badge/icon?subject=Portable">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Flink/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Flink/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Samza/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Samza/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Spark/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Spark/lastCompletedBuild/badge/icon">
</a>
</td>
<td>---</td>
</tr>
<tr>
<td>XLang</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_XVR_Direct/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_XVR_Direct/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_XVR_Dataflow/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_XVR_Dataflow/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_XVR_Flink/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_XVR_Flink/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_XVR_Samza/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_XVR_Samza/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_XVR_Spark/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_XVR_Spark/lastCompletedBuild/badge/icon">
</a>
</td>
<td>---</td>
</tr>
</tbody>
</table>
Examples testing status on various runners
--------------------------------------------------------
<table>
<thead>
<tr>
<th>Lang</th>
<th>ULR</th>
<th>Dataflow</th>
<th>Flink</th>
<th>Samza</th>
<th>Spark</th>
<th>Twister2</th>
</tr>
</thead>
<tbody>
<tr>
<td>Go</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
</tr>
<tr>
<td>Java</td>
<td>---</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_Java_Examples_Dataflow_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_Java_Examples_Dataflow_Cron/lastCompletedBuild/badge/icon?subject=V1">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_Java_Examples_Dataflow_Java11_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_Java_Examples_Dataflow_Java11_Cron/lastCompletedBuild/badge/icon?subject=V1+Java11">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_Examples_Dataflow_V2/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_Examples_Dataflow_V2/lastCompletedBuild/badge/icon?subject=V2">
</a><br>
</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
</tr>
<tr>
<td>Python</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
</tr>
<tr>
<td>XLang</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
</tr>
</tbody>
</table>
Post-Commit SDK/Transform Integration Tests Status (on master branch)
------------------------------------------------------------------------------------------------
<table>
<thead>
<tr>
<th>Go</th>
<th>Java</th>
<th>Python</th>
</tr>
</thead>
<tbody>
<tr>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/badge/icon?subject=3.6">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/badge/icon?subject=3.7">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Python38/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Python38/lastCompletedBuild/badge/icon?subject=3.8">
</a>
</td>
</tr>
</tbody>
</table>
Pre-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
<table>
<thead>
<tr>
<th>---</th>
<th>Java</th>
<th>Python</th>
<th>Go</th>
<th>Website</th>
<th>Whitespace</th>
<th>Typescript</th>
</tr>
</thead>
<tbody>
<tr>
<td>Non-portable</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/badge/icon">
</a><br>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/badge/icon?subject=Tests">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_PythonLint_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_PythonLint_Cron/lastCompletedBuild/badge/icon?subject=Lint">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_PythonDocker_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_PythonDocker_Cron/badge/icon?subject=Docker">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_PythonDocs_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_PythonDocs_Cron/badge/icon?subject=Docs">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_Go_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_Go_Cron/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_Website_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_Website_Cron/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_Whitespace_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_Whitespace_Cron/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_Typescript_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_Typescript_Cron/lastCompletedBuild/badge/icon">
</a>
</td>
</tr>
<tr>
<td>Portable</td>
<td>---</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_Portable_Python_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_Portable_Python_Cron/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_GoPortable_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_GoPortable_Cron/lastCompletedBuild/badge/icon">
</a>
</td>
<td>---</td>
<td>---</td>
<td>---</td>
</tr>
</tbody>
</table>
See [.test-infra/jenkins/README](https://github.com/apache/beam/blob/master/.test-infra/jenkins/README.md) for trigger phrase, status and link of all Jenkins jobs.
GitHub Actions Tests Status (on master branch)
------------------------------------------------------------------------------------------------
[](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] chamikaramj commented on a change in pull request #15941: [BEAM-13223] Documents two new features of cross-language transforms
Posted by GitBox <gi...@apache.org>.
chamikaramj commented on a change in pull request #15941:
URL: https://github.com/apache/beam/pull/15941#discussion_r748653865
##########
File path: website/www/site/content/en/documentation/programming-guide.md
##########
@@ -6555,6 +6555,115 @@ In this section, we will use [KafkaIO.Read](https://beam.apache.org/releases/jav
#### 13.1.1. Creating cross-language Java transforms
+There are two ways to make Java transforms available to other SDKs.
+
+* Option 1: In some cases, you can use existing Java transforms from other SDKs without writing any additional Java code.
+* Option 2: You can use arbitrary Java Transforms from other SDKs by adding few Java classes.
+
+##### 13.1.1.1 Using Existing Java Transforms from Other SDKs Without Writing more Java Code
+
+Starting with Beam 2.34.0, Python SDK users can use some Java transforms without writing additional Java code. This can be useful in many cases. For example,
+* A developer not familiar with Java may need to use an existing Java transform from a Python pipeline
+* A developer may need to make a Java transform that is already released, available to a Python pipeline without writing/releasing more Java code
Review comment:
Done.
##########
File path: website/www/site/content/en/documentation/programming-guide.md
##########
@@ -6555,6 +6555,115 @@ In this section, we will use [KafkaIO.Read](https://beam.apache.org/releases/jav
#### 13.1.1. Creating cross-language Java transforms
+There are two ways to make Java transforms available to other SDKs.
+
+* Option 1: In some cases, you can use existing Java transforms from other SDKs without writing any additional Java code.
+* Option 2: You can use arbitrary Java Transforms from other SDKs by adding few Java classes.
+
+##### 13.1.1.1 Using Existing Java Transforms from Other SDKs Without Writing more Java Code
+
+Starting with Beam 2.34.0, Python SDK users can use some Java transforms without writing additional Java code. This can be useful in many cases. For example,
+* A developer not familiar with Java may need to use an existing Java transform from a Python pipeline
+* A developer may need to make a Java transform that is already released, available to a Python pipeline without writing/releasing more Java code
+
+> **Note:** This feature is currently only available when using Java transforms from a Python pipeline.
+
+To be eligible for direct usage, API of the Java transforms has to follow the following pattern.
+* Requirement 1: Java transform can be constructed using an available public constructor or a public static method (a constructor method) in the same Java class.
Review comment:
Done.
##########
File path: website/www/site/content/en/documentation/programming-guide.md
##########
@@ -6555,6 +6555,115 @@ In this section, we will use [KafkaIO.Read](https://beam.apache.org/releases/jav
#### 13.1.1. Creating cross-language Java transforms
+There are two ways to make Java transforms available to other SDKs.
+
+* Option 1: In some cases, you can use existing Java transforms from other SDKs without writing any additional Java code.
+* Option 2: You can use arbitrary Java Transforms from other SDKs by adding few Java classes.
+
+##### 13.1.1.1 Using Existing Java Transforms from Other SDKs Without Writing more Java Code
+
+Starting with Beam 2.34.0, Python SDK users can use some Java transforms without writing additional Java code. This can be useful in many cases. For example,
+* A developer not familiar with Java may need to use an existing Java transform from a Python pipeline
+* A developer may need to make a Java transform that is already released, available to a Python pipeline without writing/releasing more Java code
+
+> **Note:** This feature is currently only available when using Java transforms from a Python pipeline.
+
+To be eligible for direct usage, API of the Java transforms has to follow the following pattern.
+* Requirement 1: Java transform can be constructed using an available public constructor or a public static method (a constructor method) in the same Java class.
+* Requirement 2: Java transform can be configured using one or more builder methods. Each builder method should be public and should return an instance of the Java transform.
+
+See below for the structure of an example Java class that can be directly used from the Python API.
+
+{{< highlight >}}
+public class JavaDataGenerator extends PTransform<PBegin, PCollection<String>> {
+ . . .
+
+ // Following method satisfies the Requirement 1.
+ // Note that you may also use a class constructor instead of a static method.
+ public static JavaDataGenerator create(Integer size) {
+ return new JavaDataGenerator(size);
+ }
+
+ static class JavaDataGeneratorConfig implements Serializable {
+ public String prefix;
+ public long length;
+ public String suffix;
+ . . .
+ }
+
+ // Following method conforms to the Requirement 2
+ public JavaDataGenerator withJavaDataGeneratorConfig(JavaDataGeneratorConfig dataConfig) {
+ return new JavaDataGenerator(this.size, javaDataGeneratorConfig);
+ }
+
+ . . .
+}
+{{< /highlight >}}
+
+To use a Java class that conforms to the above requirement from a Python SDK pipeline you may do the following.
+
+* Step 1: create an allowlist file in the _yaml_ format that describes the Java transform classes and methods that will be directly accessed from Python.
+* Step 2: start an Expansion Service with the `javaClassLookupAllowlistFile` option passing path to the allowlist defined in Step 1 as the value.
+* Step 3: Use the Python [JavaExternalTransform](https://github.com/apache/beam/blob/master/sdks/python/apache_beam/transforms/external.py) API to directly
+ access Java transforms defined in the allowlist from the Python side.
+
+Starting with Beam 2.35.0, Step 1 and 2 may be skipped as described in corresponding sections below.
+
+##### Step 1
+
+To use this Java transform from Python, you may define an allowlist file in the _yaml_ format. This allowlist lists the class names,
+constructor methods, and builder methods that are directly available to be used from the Python side.
+
+Starting with Beam 2.35.0, you have the option to specify `*` to the `javaClassLookupAllowlistFile` option instead of defining an actual allowlist which
+denotes that all supported transforms in the classpath of the exapansion service may be accessed through the API.
+
+{{< highlight >}}
+version: v1
+allowedClasses:
+- className: my.beam.transforms.JavaDataGenerator
+ allowedConstructorMethods:
+ - create
+ allowedBuilderMethods:
+ - withJavaDataGeneratorConfig
+{{< /highlight >}}
+
+##### Step 2
+
+The allowlist is provided as an argument when starting up the Java expansion service. For example, the expansion service can be started
+as a local Java process using the following command.
+
+{{< highlight >}}
+java -jar <jar file> <port> --javaClassLookupAllowlistFile=<path to the allowlist file>
+{{< /highlight >}}
+
+Starting with Beam 2.35.0, Beam 'JavaExternalTransform' API will automatically startup an expansion service with a given set of `jar` file dependencies
+if an expansion service address was not provided.
+
+##### Step 3
+
+You can directly use the Java class from your Python pipeline using a stub transform created using the `JavaExternalTransform` API. This API allows you to construct the transform
+using the Java class name and allows you to invoke builder methods to configure the class.
+
+Constructor and method parameter types are mapped between Python and Java using a Beam Schema. The Schema is auto-generated using the object types
+provided in the Python side. If the Java class constructor method or builder method accepts any complex object types, make sure that Beam Schema
Review comment:
Done.
##########
File path: website/www/site/content/en/documentation/programming-guide.md
##########
@@ -6555,6 +6555,115 @@ In this section, we will use [KafkaIO.Read](https://beam.apache.org/releases/jav
#### 13.1.1. Creating cross-language Java transforms
+There are two ways to make Java transforms available to other SDKs.
+
+* Option 1: In some cases, you can use existing Java transforms from other SDKs without writing any additional Java code.
+* Option 2: You can use arbitrary Java Transforms from other SDKs by adding few Java classes.
+
+##### 13.1.1.1 Using Existing Java Transforms from Other SDKs Without Writing more Java Code
+
+Starting with Beam 2.34.0, Python SDK users can use some Java transforms without writing additional Java code. This can be useful in many cases. For example,
+* A developer not familiar with Java may need to use an existing Java transform from a Python pipeline
+* A developer may need to make a Java transform that is already released, available to a Python pipeline without writing/releasing more Java code
+
+> **Note:** This feature is currently only available when using Java transforms from a Python pipeline.
+
+To be eligible for direct usage, API of the Java transforms has to follow the following pattern.
+* Requirement 1: Java transform can be constructed using an available public constructor or a public static method (a constructor method) in the same Java class.
+* Requirement 2: Java transform can be configured using one or more builder methods. Each builder method should be public and should return an instance of the Java transform.
+
+See below for the structure of an example Java class that can be directly used from the Python API.
+
+{{< highlight >}}
+public class JavaDataGenerator extends PTransform<PBegin, PCollection<String>> {
+ . . .
+
+ // Following method satisfies the Requirement 1.
+ // Note that you may also use a class constructor instead of a static method.
+ public static JavaDataGenerator create(Integer size) {
+ return new JavaDataGenerator(size);
+ }
+
+ static class JavaDataGeneratorConfig implements Serializable {
+ public String prefix;
+ public long length;
+ public String suffix;
+ . . .
+ }
+
+ // Following method conforms to the Requirement 2
+ public JavaDataGenerator withJavaDataGeneratorConfig(JavaDataGeneratorConfig dataConfig) {
+ return new JavaDataGenerator(this.size, javaDataGeneratorConfig);
+ }
+
+ . . .
+}
+{{< /highlight >}}
+
+To use a Java class that conforms to the above requirement from a Python SDK pipeline you may do the following.
+
+* Step 1: create an allowlist file in the _yaml_ format that describes the Java transform classes and methods that will be directly accessed from Python.
+* Step 2: start an Expansion Service with the `javaClassLookupAllowlistFile` option passing path to the allowlist defined in Step 1 as the value.
+* Step 3: Use the Python [JavaExternalTransform](https://github.com/apache/beam/blob/master/sdks/python/apache_beam/transforms/external.py) API to directly
+ access Java transforms defined in the allowlist from the Python side.
+
+Starting with Beam 2.35.0, Step 1 and 2 may be skipped as described in corresponding sections below.
+
+##### Step 1
+
+To use this Java transform from Python, you may define an allowlist file in the _yaml_ format. This allowlist lists the class names,
+constructor methods, and builder methods that are directly available to be used from the Python side.
+
+Starting with Beam 2.35.0, you have the option to specify `*` to the `javaClassLookupAllowlistFile` option instead of defining an actual allowlist which
+denotes that all supported transforms in the classpath of the exapansion service may be accessed through the API.
+
+{{< highlight >}}
+version: v1
+allowedClasses:
+- className: my.beam.transforms.JavaDataGenerator
+ allowedConstructorMethods:
+ - create
+ allowedBuilderMethods:
+ - withJavaDataGeneratorConfig
+{{< /highlight >}}
+
+##### Step 2
+
+The allowlist is provided as an argument when starting up the Java expansion service. For example, the expansion service can be started
+as a local Java process using the following command.
+
+{{< highlight >}}
+java -jar <jar file> <port> --javaClassLookupAllowlistFile=<path to the allowlist file>
+{{< /highlight >}}
+
+Starting with Beam 2.35.0, Beam 'JavaExternalTransform' API will automatically startup an expansion service with a given set of `jar` file dependencies
+if an expansion service address was not provided.
+
+##### Step 3
+
+You can directly use the Java class from your Python pipeline using a stub transform created using the `JavaExternalTransform` API. This API allows you to construct the transform
+using the Java class name and allows you to invoke builder methods to configure the class.
+
+Constructor and method parameter types are mapped between Python and Java using a Beam Schema. The Schema is auto-generated using the object types
+provided in the Python side. If the Java class constructor method or builder method accepts any complex object types, make sure that Beam Schema
+for these objects are registered and available for the Java expansion service. If a schema has not been registered, Java expansion service will
Review comment:
Done.
##########
File path: website/www/site/content/en/documentation/programming-guide.md
##########
@@ -6555,6 +6555,115 @@ In this section, we will use [KafkaIO.Read](https://beam.apache.org/releases/jav
#### 13.1.1. Creating cross-language Java transforms
+There are two ways to make Java transforms available to other SDKs.
+
+* Option 1: In some cases, you can use existing Java transforms from other SDKs without writing any additional Java code.
+* Option 2: You can use arbitrary Java Transforms from other SDKs by adding few Java classes.
+
+##### 13.1.1.1 Using Existing Java Transforms from Other SDKs Without Writing more Java Code
+
+Starting with Beam 2.34.0, Python SDK users can use some Java transforms without writing additional Java code. This can be useful in many cases. For example,
+* A developer not familiar with Java may need to use an existing Java transform from a Python pipeline
+* A developer may need to make a Java transform that is already released, available to a Python pipeline without writing/releasing more Java code
+
+> **Note:** This feature is currently only available when using Java transforms from a Python pipeline.
+
+To be eligible for direct usage, API of the Java transforms has to follow the following pattern.
Review comment:
Done.
##########
File path: website/www/site/content/en/documentation/programming-guide.md
##########
@@ -6555,6 +6555,115 @@ In this section, we will use [KafkaIO.Read](https://beam.apache.org/releases/jav
#### 13.1.1. Creating cross-language Java transforms
+There are two ways to make Java transforms available to other SDKs.
+
+* Option 1: In some cases, you can use existing Java transforms from other SDKs without writing any additional Java code.
+* Option 2: You can use arbitrary Java Transforms from other SDKs by adding few Java classes.
+
+##### 13.1.1.1 Using Existing Java Transforms from Other SDKs Without Writing more Java Code
+
+Starting with Beam 2.34.0, Python SDK users can use some Java transforms without writing additional Java code. This can be useful in many cases. For example,
+* A developer not familiar with Java may need to use an existing Java transform from a Python pipeline
+* A developer may need to make a Java transform that is already released, available to a Python pipeline without writing/releasing more Java code
+
+> **Note:** This feature is currently only available when using Java transforms from a Python pipeline.
+
+To be eligible for direct usage, API of the Java transforms has to follow the following pattern.
+* Requirement 1: Java transform can be constructed using an available public constructor or a public static method (a constructor method) in the same Java class.
+* Requirement 2: Java transform can be configured using one or more builder methods. Each builder method should be public and should return an instance of the Java transform.
+
+See below for the structure of an example Java class that can be directly used from the Python API.
+
+{{< highlight >}}
+public class JavaDataGenerator extends PTransform<PBegin, PCollection<String>> {
+ . . .
+
+ // Following method satisfies the Requirement 1.
+ // Note that you may also use a class constructor instead of a static method.
+ public static JavaDataGenerator create(Integer size) {
+ return new JavaDataGenerator(size);
+ }
+
+ static class JavaDataGeneratorConfig implements Serializable {
+ public String prefix;
+ public long length;
+ public String suffix;
+ . . .
+ }
+
+ // Following method conforms to the Requirement 2
+ public JavaDataGenerator withJavaDataGeneratorConfig(JavaDataGeneratorConfig dataConfig) {
+ return new JavaDataGenerator(this.size, javaDataGeneratorConfig);
+ }
+
+ . . .
+}
+{{< /highlight >}}
+
+To use a Java class that conforms to the above requirement from a Python SDK pipeline you may do the following.
+
+* Step 1: create an allowlist file in the _yaml_ format that describes the Java transform classes and methods that will be directly accessed from Python.
+* Step 2: start an Expansion Service with the `javaClassLookupAllowlistFile` option passing path to the allowlist defined in Step 1 as the value.
+* Step 3: Use the Python [JavaExternalTransform](https://github.com/apache/beam/blob/master/sdks/python/apache_beam/transforms/external.py) API to directly
+ access Java transforms defined in the allowlist from the Python side.
+
+Starting with Beam 2.35.0, Step 1 and 2 may be skipped as described in corresponding sections below.
+
+##### Step 1
+
+To use this Java transform from Python, you may define an allowlist file in the _yaml_ format. This allowlist lists the class names,
+constructor methods, and builder methods that are directly available to be used from the Python side.
+
+Starting with Beam 2.35.0, you have the option to specify `*` to the `javaClassLookupAllowlistFile` option instead of defining an actual allowlist which
+denotes that all supported transforms in the classpath of the exapansion service may be accessed through the API.
+
+{{< highlight >}}
+version: v1
+allowedClasses:
+- className: my.beam.transforms.JavaDataGenerator
+ allowedConstructorMethods:
+ - create
+ allowedBuilderMethods:
+ - withJavaDataGeneratorConfig
+{{< /highlight >}}
+
+##### Step 2
+
+The allowlist is provided as an argument when starting up the Java expansion service. For example, the expansion service can be started
+as a local Java process using the following command.
+
+{{< highlight >}}
+java -jar <jar file> <port> --javaClassLookupAllowlistFile=<path to the allowlist file>
+{{< /highlight >}}
+
+Starting with Beam 2.35.0, Beam 'JavaExternalTransform' API will automatically startup an expansion service with a given set of `jar` file dependencies
Review comment:
Done.
##########
File path: website/www/site/content/en/documentation/programming-guide.md
##########
@@ -6555,6 +6555,115 @@ In this section, we will use [KafkaIO.Read](https://beam.apache.org/releases/jav
#### 13.1.1. Creating cross-language Java transforms
+There are two ways to make Java transforms available to other SDKs.
+
+* Option 1: In some cases, you can use existing Java transforms from other SDKs without writing any additional Java code.
+* Option 2: You can use arbitrary Java Transforms from other SDKs by adding few Java classes.
+
+##### 13.1.1.1 Using Existing Java Transforms from Other SDKs Without Writing more Java Code
+
+Starting with Beam 2.34.0, Python SDK users can use some Java transforms without writing additional Java code. This can be useful in many cases. For example,
+* A developer not familiar with Java may need to use an existing Java transform from a Python pipeline
+* A developer may need to make a Java transform that is already released, available to a Python pipeline without writing/releasing more Java code
+
+> **Note:** This feature is currently only available when using Java transforms from a Python pipeline.
+
+To be eligible for direct usage, API of the Java transforms has to follow the following pattern.
+* Requirement 1: Java transform can be constructed using an available public constructor or a public static method (a constructor method) in the same Java class.
+* Requirement 2: Java transform can be configured using one or more builder methods. Each builder method should be public and should return an instance of the Java transform.
+
+See below for the structure of an example Java class that can be directly used from the Python API.
+
+{{< highlight >}}
+public class JavaDataGenerator extends PTransform<PBegin, PCollection<String>> {
+ . . .
+
+ // Following method satisfies the Requirement 1.
+ // Note that you may also use a class constructor instead of a static method.
+ public static JavaDataGenerator create(Integer size) {
+ return new JavaDataGenerator(size);
+ }
+
+ static class JavaDataGeneratorConfig implements Serializable {
+ public String prefix;
+ public long length;
+ public String suffix;
+ . . .
+ }
+
+ // Following method conforms to the Requirement 2
+ public JavaDataGenerator withJavaDataGeneratorConfig(JavaDataGeneratorConfig dataConfig) {
+ return new JavaDataGenerator(this.size, javaDataGeneratorConfig);
+ }
+
+ . . .
+}
+{{< /highlight >}}
+
+To use a Java class that conforms to the above requirement from a Python SDK pipeline you may do the following.
+
+* Step 1: create an allowlist file in the _yaml_ format that describes the Java transform classes and methods that will be directly accessed from Python.
+* Step 2: start an Expansion Service with the `javaClassLookupAllowlistFile` option passing path to the allowlist defined in Step 1 as the value.
+* Step 3: Use the Python [JavaExternalTransform](https://github.com/apache/beam/blob/master/sdks/python/apache_beam/transforms/external.py) API to directly
+ access Java transforms defined in the allowlist from the Python side.
+
+Starting with Beam 2.35.0, Step 1 and 2 may be skipped as described in corresponding sections below.
+
+##### Step 1
+
+To use this Java transform from Python, you may define an allowlist file in the _yaml_ format. This allowlist lists the class names,
+constructor methods, and builder methods that are directly available to be used from the Python side.
+
+Starting with Beam 2.35.0, you have the option to specify `*` to the `javaClassLookupAllowlistFile` option instead of defining an actual allowlist which
+denotes that all supported transforms in the classpath of the exapansion service may be accessed through the API.
+
+{{< highlight >}}
+version: v1
+allowedClasses:
+- className: my.beam.transforms.JavaDataGenerator
+ allowedConstructorMethods:
+ - create
+ allowedBuilderMethods:
+ - withJavaDataGeneratorConfig
+{{< /highlight >}}
+
+##### Step 2
+
+The allowlist is provided as an argument when starting up the Java expansion service. For example, the expansion service can be started
+as a local Java process using the following command.
+
+{{< highlight >}}
+java -jar <jar file> <port> --javaClassLookupAllowlistFile=<path to the allowlist file>
+{{< /highlight >}}
+
+Starting with Beam 2.35.0, Beam 'JavaExternalTransform' API will automatically startup an expansion service with a given set of `jar` file dependencies
+if an expansion service address was not provided.
+
+##### Step 3
+
+You can directly use the Java class from your Python pipeline using a stub transform created using the `JavaExternalTransform` API. This API allows you to construct the transform
+using the Java class name and allows you to invoke builder methods to configure the class.
+
+Constructor and method parameter types are mapped between Python and Java using a Beam Schema. The Schema is auto-generated using the object types
+provided in the Python side. If the Java class constructor method or builder method accepts any complex object types, make sure that Beam Schema
Review comment:
Done.
##########
File path: website/www/site/content/en/documentation/programming-guide.md
##########
@@ -6555,6 +6555,115 @@ In this section, we will use [KafkaIO.Read](https://beam.apache.org/releases/jav
#### 13.1.1. Creating cross-language Java transforms
+There are two ways to make Java transforms available to other SDKs.
+
+* Option 1: In some cases, you can use existing Java transforms from other SDKs without writing any additional Java code.
+* Option 2: You can use arbitrary Java Transforms from other SDKs by adding few Java classes.
+
+##### 13.1.1.1 Using Existing Java Transforms from Other SDKs Without Writing more Java Code
+
+Starting with Beam 2.34.0, Python SDK users can use some Java transforms without writing additional Java code. This can be useful in many cases. For example,
+* A developer not familiar with Java may need to use an existing Java transform from a Python pipeline
+* A developer may need to make a Java transform that is already released, available to a Python pipeline without writing/releasing more Java code
+
+> **Note:** This feature is currently only available when using Java transforms from a Python pipeline.
+
+To be eligible for direct usage, API of the Java transforms has to follow the following pattern.
+* Requirement 1: Java transform can be constructed using an available public constructor or a public static method (a constructor method) in the same Java class.
+* Requirement 2: Java transform can be configured using one or more builder methods. Each builder method should be public and should return an instance of the Java transform.
Review comment:
Done.
##########
File path: website/www/site/content/en/documentation/programming-guide.md
##########
@@ -6555,6 +6555,115 @@ In this section, we will use [KafkaIO.Read](https://beam.apache.org/releases/jav
#### 13.1.1. Creating cross-language Java transforms
+There are two ways to make Java transforms available to other SDKs.
+
+* Option 1: In some cases, you can use existing Java transforms from other SDKs without writing any additional Java code.
+* Option 2: You can use arbitrary Java Transforms from other SDKs by adding few Java classes.
Review comment:
Done.
##########
File path: website/www/site/content/en/documentation/programming-guide.md
##########
@@ -6752,6 +6861,43 @@ Currently Python external transforms are limited to dependencies available in co
Go currently does not support creating cross-language transforms, only using cross-language
transforms from other languages; see more at [BEAM-9923](https://issues.apache.org/jira/browse/BEAM-9923).
+#### 13.1.4. Selecting a URN for Cross-language Transforms
+
+Developing a cross-language transform involves defining a URN for registering the transform with an expansion service. In this section
+we provide a convention for defining such URNs. Following this convention is optional but following it will make sure that your transform
+will not run into conflicts when registering in an expansion service along with transforms developed by other developers.
+
+##### Schema
+
+Suggested URN consists of the components given below.
+* ns-id: A namespace identifier. Default recommendation is `beam:transform`.
+* org-identifier: Identifies the organization where the transform was defined. Transforms defined in Apache Beam use `org.apache.beam` for this.
+* functionality-identifier - Identifies the functionality of the cross-language transform.
+* version - a version number for the transform
+
+We provide the schema from the URN convention in [augmented Backus–Naur](https://en.wikipedia.org/wiki/Augmented_Backus%E2%80%93Naur_form) form.
+Keywords in upper case are from the [URN spec](https://datatracker.ietf.org/doc/html/rfc8141).
+
+{{< highlight >}}
+transform-urn = ns-id “:” org-identifier “:” functionality-identifier “:” version
+ns-id = (“beam” / NID) “:” “transform”
+id-char = ALPHA / DIGIT / "-" / "." / "_" / "~" ; A subset of characters allowed in a URN
+org-identifier = 1*id-char
+functionality-identifier = 1*id-char
+version = “v” 1*(DIGIT / “.”) ; For example, ‘v1.2’
+{{< /highlight >}}
+
+##### Examples
+
+Below we’ve given some example transforms classes and corresponding URNs to be used.
Review comment:
Done.
##########
File path: website/www/site/content/en/documentation/programming-guide.md
##########
@@ -6752,6 +6861,43 @@ Currently Python external transforms are limited to dependencies available in co
Go currently does not support creating cross-language transforms, only using cross-language
transforms from other languages; see more at [BEAM-9923](https://issues.apache.org/jira/browse/BEAM-9923).
+#### 13.1.4. Selecting a URN for Cross-language Transforms
+
+Developing a cross-language transform involves defining a URN for registering the transform with an expansion service. In this section
+we provide a convention for defining such URNs. Following this convention is optional but following it will make sure that your transform
+will not run into conflicts when registering in an expansion service along with transforms developed by other developers.
+
+##### Schema
+
+Suggested URN consists of the components given below.
Review comment:
Done.
##########
File path: website/www/site/content/en/documentation/programming-guide.md
##########
@@ -6555,6 +6555,115 @@ In this section, we will use [KafkaIO.Read](https://beam.apache.org/releases/jav
#### 13.1.1. Creating cross-language Java transforms
+There are two ways to make Java transforms available to other SDKs.
+
+* Option 1: In some cases, you can use existing Java transforms from other SDKs without writing any additional Java code.
+* Option 2: You can use arbitrary Java Transforms from other SDKs by adding few Java classes.
+
+##### 13.1.1.1 Using Existing Java Transforms from Other SDKs Without Writing more Java Code
+
+Starting with Beam 2.34.0, Python SDK users can use some Java transforms without writing additional Java code. This can be useful in many cases. For example,
+* A developer not familiar with Java may need to use an existing Java transform from a Python pipeline
+* A developer may need to make a Java transform that is already released, available to a Python pipeline without writing/releasing more Java code
+
+> **Note:** This feature is currently only available when using Java transforms from a Python pipeline.
+
+To be eligible for direct usage, API of the Java transforms has to follow the following pattern.
+* Requirement 1: Java transform can be constructed using an available public constructor or a public static method (a constructor method) in the same Java class.
+* Requirement 2: Java transform can be configured using one or more builder methods. Each builder method should be public and should return an instance of the Java transform.
+
+See below for the structure of an example Java class that can be directly used from the Python API.
+
+{{< highlight >}}
+public class JavaDataGenerator extends PTransform<PBegin, PCollection<String>> {
+ . . .
+
+ // Following method satisfies the Requirement 1.
+ // Note that you may also use a class constructor instead of a static method.
+ public static JavaDataGenerator create(Integer size) {
+ return new JavaDataGenerator(size);
+ }
+
+ static class JavaDataGeneratorConfig implements Serializable {
+ public String prefix;
+ public long length;
+ public String suffix;
+ . . .
+ }
+
+ // Following method conforms to the Requirement 2
+ public JavaDataGenerator withJavaDataGeneratorConfig(JavaDataGeneratorConfig dataConfig) {
+ return new JavaDataGenerator(this.size, javaDataGeneratorConfig);
+ }
+
+ . . .
+}
+{{< /highlight >}}
+
+To use a Java class that conforms to the above requirement from a Python SDK pipeline you may do the following.
+
+* Step 1: create an allowlist file in the _yaml_ format that describes the Java transform classes and methods that will be directly accessed from Python.
+* Step 2: start an Expansion Service with the `javaClassLookupAllowlistFile` option passing path to the allowlist defined in Step 1 as the value.
+* Step 3: Use the Python [JavaExternalTransform](https://github.com/apache/beam/blob/master/sdks/python/apache_beam/transforms/external.py) API to directly
+ access Java transforms defined in the allowlist from the Python side.
+
+Starting with Beam 2.35.0, Step 1 and 2 may be skipped as described in corresponding sections below.
+
+##### Step 1
+
+To use this Java transform from Python, you may define an allowlist file in the _yaml_ format. This allowlist lists the class names,
+constructor methods, and builder methods that are directly available to be used from the Python side.
+
+Starting with Beam 2.35.0, you have the option to specify `*` to the `javaClassLookupAllowlistFile` option instead of defining an actual allowlist which
+denotes that all supported transforms in the classpath of the exapansion service may be accessed through the API.
+
+{{< highlight >}}
+version: v1
+allowedClasses:
+- className: my.beam.transforms.JavaDataGenerator
+ allowedConstructorMethods:
+ - create
+ allowedBuilderMethods:
+ - withJavaDataGeneratorConfig
+{{< /highlight >}}
+
+##### Step 2
+
+The allowlist is provided as an argument when starting up the Java expansion service. For example, the expansion service can be started
+as a local Java process using the following command.
+
+{{< highlight >}}
+java -jar <jar file> <port> --javaClassLookupAllowlistFile=<path to the allowlist file>
+{{< /highlight >}}
+
+Starting with Beam 2.35.0, Beam 'JavaExternalTransform' API will automatically startup an expansion service with a given set of `jar` file dependencies
+if an expansion service address was not provided.
+
+##### Step 3
+
+You can directly use the Java class from your Python pipeline using a stub transform created using the `JavaExternalTransform` API. This API allows you to construct the transform
+using the Java class name and allows you to invoke builder methods to configure the class.
+
+Constructor and method parameter types are mapped between Python and Java using a Beam Schema. The Schema is auto-generated using the object types
+provided in the Python side. If the Java class constructor method or builder method accepts any complex object types, make sure that Beam Schema
+for these objects are registered and available for the Java expansion service. If a schema has not been registered, Java expansion service will
Review comment:
Done.
##########
File path: website/www/site/content/en/documentation/programming-guide.md
##########
@@ -6555,6 +6555,115 @@ In this section, we will use [KafkaIO.Read](https://beam.apache.org/releases/jav
#### 13.1.1. Creating cross-language Java transforms
+There are two ways to make Java transforms available to other SDKs.
+
+* Option 1: In some cases, you can use existing Java transforms from other SDKs without writing any additional Java code.
+* Option 2: You can use arbitrary Java Transforms from other SDKs by adding few Java classes.
+
+##### 13.1.1.1 Using Existing Java Transforms from Other SDKs Without Writing more Java Code
+
+Starting with Beam 2.34.0, Python SDK users can use some Java transforms without writing additional Java code. This can be useful in many cases. For example,
+* A developer not familiar with Java may need to use an existing Java transform from a Python pipeline
+* A developer may need to make a Java transform that is already released, available to a Python pipeline without writing/releasing more Java code
+
+> **Note:** This feature is currently only available when using Java transforms from a Python pipeline.
+
+To be eligible for direct usage, API of the Java transforms has to follow the following pattern.
+* Requirement 1: Java transform can be constructed using an available public constructor or a public static method (a constructor method) in the same Java class.
+* Requirement 2: Java transform can be configured using one or more builder methods. Each builder method should be public and should return an instance of the Java transform.
+
+See below for the structure of an example Java class that can be directly used from the Python API.
+
+{{< highlight >}}
+public class JavaDataGenerator extends PTransform<PBegin, PCollection<String>> {
+ . . .
+
+ // Following method satisfies the Requirement 1.
+ // Note that you may also use a class constructor instead of a static method.
+ public static JavaDataGenerator create(Integer size) {
+ return new JavaDataGenerator(size);
+ }
+
+ static class JavaDataGeneratorConfig implements Serializable {
+ public String prefix;
+ public long length;
+ public String suffix;
+ . . .
+ }
+
+ // Following method conforms to the Requirement 2
+ public JavaDataGenerator withJavaDataGeneratorConfig(JavaDataGeneratorConfig dataConfig) {
+ return new JavaDataGenerator(this.size, javaDataGeneratorConfig);
+ }
+
+ . . .
+}
+{{< /highlight >}}
+
+To use a Java class that conforms to the above requirement from a Python SDK pipeline you may do the following.
+
+* Step 1: create an allowlist file in the _yaml_ format that describes the Java transform classes and methods that will be directly accessed from Python.
+* Step 2: start an Expansion Service with the `javaClassLookupAllowlistFile` option passing path to the allowlist defined in Step 1 as the value.
+* Step 3: Use the Python [JavaExternalTransform](https://github.com/apache/beam/blob/master/sdks/python/apache_beam/transforms/external.py) API to directly
+ access Java transforms defined in the allowlist from the Python side.
+
+Starting with Beam 2.35.0, Step 1 and 2 may be skipped as described in corresponding sections below.
+
+##### Step 1
+
+To use this Java transform from Python, you may define an allowlist file in the _yaml_ format. This allowlist lists the class names,
+constructor methods, and builder methods that are directly available to be used from the Python side.
+
+Starting with Beam 2.35.0, you have the option to specify `*` to the `javaClassLookupAllowlistFile` option instead of defining an actual allowlist which
+denotes that all supported transforms in the classpath of the exapansion service may be accessed through the API.
Review comment:
Done.
##########
File path: website/www/site/content/en/documentation/programming-guide.md
##########
@@ -6752,6 +6861,43 @@ Currently Python external transforms are limited to dependencies available in co
Go currently does not support creating cross-language transforms, only using cross-language
transforms from other languages; see more at [BEAM-9923](https://issues.apache.org/jira/browse/BEAM-9923).
+#### 13.1.4. Selecting a URN for Cross-language Transforms
+
+Developing a cross-language transform involves defining a URN for registering the transform with an expansion service. In this section
+we provide a convention for defining such URNs. Following this convention is optional but following it will make sure that your transform
Review comment:
Done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pcoet commented on a change in pull request #15941: [BEAM-13223] Documents two new features of cross-language transforms
Posted by GitBox <gi...@apache.org>.
pcoet commented on a change in pull request #15941:
URL: https://github.com/apache/beam/pull/15941#discussion_r748572408
##########
File path: website/www/site/content/en/documentation/programming-guide.md
##########
@@ -6555,6 +6555,115 @@ In this section, we will use [KafkaIO.Read](https://beam.apache.org/releases/jav
#### 13.1.1. Creating cross-language Java transforms
+There are two ways to make Java transforms available to other SDKs.
+
+* Option 1: In some cases, you can use existing Java transforms from other SDKs without writing any additional Java code.
+* Option 2: You can use arbitrary Java Transforms from other SDKs by adding few Java classes.
Review comment:
"adding few" -> "adding a few"
##########
File path: website/www/site/content/en/documentation/programming-guide.md
##########
@@ -6555,6 +6555,115 @@ In this section, we will use [KafkaIO.Read](https://beam.apache.org/releases/jav
#### 13.1.1. Creating cross-language Java transforms
+There are two ways to make Java transforms available to other SDKs.
+
+* Option 1: In some cases, you can use existing Java transforms from other SDKs without writing any additional Java code.
+* Option 2: You can use arbitrary Java Transforms from other SDKs by adding few Java classes.
+
+##### 13.1.1.1 Using Existing Java Transforms from Other SDKs Without Writing more Java Code
+
+Starting with Beam 2.34.0, Python SDK users can use some Java transforms without writing additional Java code. This can be useful in many cases. For example,
+* A developer not familiar with Java may need to use an existing Java transform from a Python pipeline
+* A developer may need to make a Java transform that is already released, available to a Python pipeline without writing/releasing more Java code
+
+> **Note:** This feature is currently only available when using Java transforms from a Python pipeline.
+
+To be eligible for direct usage, API of the Java transforms has to follow the following pattern.
+* Requirement 1: Java transform can be constructed using an available public constructor or a public static method (a constructor method) in the same Java class.
Review comment:
"Java" -> "The Java"
##########
File path: website/www/site/content/en/documentation/programming-guide.md
##########
@@ -6752,6 +6861,43 @@ Currently Python external transforms are limited to dependencies available in co
Go currently does not support creating cross-language transforms, only using cross-language
transforms from other languages; see more at [BEAM-9923](https://issues.apache.org/jira/browse/BEAM-9923).
+#### 13.1.4. Selecting a URN for Cross-language Transforms
+
+Developing a cross-language transform involves defining a URN for registering the transform with an expansion service. In this section
+we provide a convention for defining such URNs. Following this convention is optional but following it will make sure that your transform
+will not run into conflicts when registering in an expansion service along with transforms developed by other developers.
+
+##### Schema
+
+Suggested URN consists of the components given below.
Review comment:
"Suggested URN ... below." -> "A URN should consist of the following components:"
##########
File path: website/www/site/content/en/documentation/programming-guide.md
##########
@@ -6555,6 +6555,115 @@ In this section, we will use [KafkaIO.Read](https://beam.apache.org/releases/jav
#### 13.1.1. Creating cross-language Java transforms
+There are two ways to make Java transforms available to other SDKs.
+
+* Option 1: In some cases, you can use existing Java transforms from other SDKs without writing any additional Java code.
+* Option 2: You can use arbitrary Java Transforms from other SDKs by adding few Java classes.
+
+##### 13.1.1.1 Using Existing Java Transforms from Other SDKs Without Writing more Java Code
+
+Starting with Beam 2.34.0, Python SDK users can use some Java transforms without writing additional Java code. This can be useful in many cases. For example,
+* A developer not familiar with Java may need to use an existing Java transform from a Python pipeline
+* A developer may need to make a Java transform that is already released, available to a Python pipeline without writing/releasing more Java code
+
+> **Note:** This feature is currently only available when using Java transforms from a Python pipeline.
+
+To be eligible for direct usage, API of the Java transforms has to follow the following pattern.
+* Requirement 1: Java transform can be constructed using an available public constructor or a public static method (a constructor method) in the same Java class.
+* Requirement 2: Java transform can be configured using one or more builder methods. Each builder method should be public and should return an instance of the Java transform.
+
+See below for the structure of an example Java class that can be directly used from the Python API.
+
+{{< highlight >}}
+public class JavaDataGenerator extends PTransform<PBegin, PCollection<String>> {
+ . . .
+
+ // Following method satisfies the Requirement 1.
+ // Note that you may also use a class constructor instead of a static method.
+ public static JavaDataGenerator create(Integer size) {
+ return new JavaDataGenerator(size);
+ }
+
+ static class JavaDataGeneratorConfig implements Serializable {
+ public String prefix;
+ public long length;
+ public String suffix;
+ . . .
+ }
+
+ // Following method conforms to the Requirement 2
+ public JavaDataGenerator withJavaDataGeneratorConfig(JavaDataGeneratorConfig dataConfig) {
+ return new JavaDataGenerator(this.size, javaDataGeneratorConfig);
+ }
+
+ . . .
+}
+{{< /highlight >}}
+
+To use a Java class that conforms to the above requirement from a Python SDK pipeline you may do the following.
+
+* Step 1: create an allowlist file in the _yaml_ format that describes the Java transform classes and methods that will be directly accessed from Python.
+* Step 2: start an Expansion Service with the `javaClassLookupAllowlistFile` option passing path to the allowlist defined in Step 1 as the value.
+* Step 3: Use the Python [JavaExternalTransform](https://github.com/apache/beam/blob/master/sdks/python/apache_beam/transforms/external.py) API to directly
+ access Java transforms defined in the allowlist from the Python side.
+
+Starting with Beam 2.35.0, Step 1 and 2 may be skipped as described in corresponding sections below.
+
+##### Step 1
+
+To use this Java transform from Python, you may define an allowlist file in the _yaml_ format. This allowlist lists the class names,
+constructor methods, and builder methods that are directly available to be used from the Python side.
+
+Starting with Beam 2.35.0, you have the option to specify `*` to the `javaClassLookupAllowlistFile` option instead of defining an actual allowlist which
+denotes that all supported transforms in the classpath of the exapansion service may be accessed through the API.
+
+{{< highlight >}}
+version: v1
+allowedClasses:
+- className: my.beam.transforms.JavaDataGenerator
+ allowedConstructorMethods:
+ - create
+ allowedBuilderMethods:
+ - withJavaDataGeneratorConfig
+{{< /highlight >}}
+
+##### Step 2
+
+The allowlist is provided as an argument when starting up the Java expansion service. For example, the expansion service can be started
+as a local Java process using the following command.
+
+{{< highlight >}}
+java -jar <jar file> <port> --javaClassLookupAllowlistFile=<path to the allowlist file>
+{{< /highlight >}}
+
+Starting with Beam 2.35.0, Beam 'JavaExternalTransform' API will automatically startup an expansion service with a given set of `jar` file dependencies
+if an expansion service address was not provided.
+
+##### Step 3
+
+You can directly use the Java class from your Python pipeline using a stub transform created using the `JavaExternalTransform` API. This API allows you to construct the transform
+using the Java class name and allows you to invoke builder methods to configure the class.
+
+Constructor and method parameter types are mapped between Python and Java using a Beam Schema. The Schema is auto-generated using the object types
+provided in the Python side. If the Java class constructor method or builder method accepts any complex object types, make sure that Beam Schema
+for these objects are registered and available for the Java expansion service. If a schema has not been registered, Java expansion service will
Review comment:
"objects are" -> "objects is" (because of subject "schema")
##########
File path: website/www/site/content/en/documentation/programming-guide.md
##########
@@ -6752,6 +6861,43 @@ Currently Python external transforms are limited to dependencies available in co
Go currently does not support creating cross-language transforms, only using cross-language
transforms from other languages; see more at [BEAM-9923](https://issues.apache.org/jira/browse/BEAM-9923).
+#### 13.1.4. Selecting a URN for Cross-language Transforms
+
+Developing a cross-language transform involves defining a URN for registering the transform with an expansion service. In this section
+we provide a convention for defining such URNs. Following this convention is optional but following it will make sure that your transform
Review comment:
"but following it will make sure" -> "but it will ensure"
##########
File path: website/www/site/content/en/documentation/programming-guide.md
##########
@@ -6752,6 +6861,43 @@ Currently Python external transforms are limited to dependencies available in co
Go currently does not support creating cross-language transforms, only using cross-language
transforms from other languages; see more at [BEAM-9923](https://issues.apache.org/jira/browse/BEAM-9923).
+#### 13.1.4. Selecting a URN for Cross-language Transforms
+
+Developing a cross-language transform involves defining a URN for registering the transform with an expansion service. In this section
+we provide a convention for defining such URNs. Following this convention is optional but following it will make sure that your transform
+will not run into conflicts when registering in an expansion service along with transforms developed by other developers.
+
+##### Schema
+
+Suggested URN consists of the components given below.
+* ns-id: A namespace identifier. Default recommendation is `beam:transform`.
+* org-identifier: Identifies the organization where the transform was defined. Transforms defined in Apache Beam use `org.apache.beam` for this.
+* functionality-identifier - Identifies the functionality of the cross-language transform.
+* version - a version number for the transform
+
+We provide the schema from the URN convention in [augmented Backus–Naur](https://en.wikipedia.org/wiki/Augmented_Backus%E2%80%93Naur_form) form.
+Keywords in upper case are from the [URN spec](https://datatracker.ietf.org/doc/html/rfc8141).
+
+{{< highlight >}}
+transform-urn = ns-id “:” org-identifier “:” functionality-identifier “:” version
+ns-id = (“beam” / NID) “:” “transform”
+id-char = ALPHA / DIGIT / "-" / "." / "_" / "~" ; A subset of characters allowed in a URN
+org-identifier = 1*id-char
+functionality-identifier = 1*id-char
+version = “v” 1*(DIGIT / “.”) ; For example, ‘v1.2’
+{{< /highlight >}}
+
+##### Examples
+
+Below we’ve given some example transforms classes and corresponding URNs to be used.
Review comment:
"transforms classes" -> "transform classes"
##########
File path: website/www/site/content/en/documentation/programming-guide.md
##########
@@ -6555,6 +6555,115 @@ In this section, we will use [KafkaIO.Read](https://beam.apache.org/releases/jav
#### 13.1.1. Creating cross-language Java transforms
+There are two ways to make Java transforms available to other SDKs.
+
+* Option 1: In some cases, you can use existing Java transforms from other SDKs without writing any additional Java code.
+* Option 2: You can use arbitrary Java Transforms from other SDKs by adding few Java classes.
+
+##### 13.1.1.1 Using Existing Java Transforms from Other SDKs Without Writing more Java Code
+
+Starting with Beam 2.34.0, Python SDK users can use some Java transforms without writing additional Java code. This can be useful in many cases. For example,
+* A developer not familiar with Java may need to use an existing Java transform from a Python pipeline
+* A developer may need to make a Java transform that is already released, available to a Python pipeline without writing/releasing more Java code
+
+> **Note:** This feature is currently only available when using Java transforms from a Python pipeline.
+
+To be eligible for direct usage, API of the Java transforms has to follow the following pattern.
Review comment:
"API" -> "the API"
"Java transforms" -> "Java transform"
##########
File path: website/www/site/content/en/documentation/programming-guide.md
##########
@@ -6555,6 +6555,115 @@ In this section, we will use [KafkaIO.Read](https://beam.apache.org/releases/jav
#### 13.1.1. Creating cross-language Java transforms
+There are two ways to make Java transforms available to other SDKs.
+
+* Option 1: In some cases, you can use existing Java transforms from other SDKs without writing any additional Java code.
+* Option 2: You can use arbitrary Java Transforms from other SDKs by adding few Java classes.
+
+##### 13.1.1.1 Using Existing Java Transforms from Other SDKs Without Writing more Java Code
+
+Starting with Beam 2.34.0, Python SDK users can use some Java transforms without writing additional Java code. This can be useful in many cases. For example,
+* A developer not familiar with Java may need to use an existing Java transform from a Python pipeline
+* A developer may need to make a Java transform that is already released, available to a Python pipeline without writing/releasing more Java code
Review comment:
"a Java transform that is already released, available" -> "an existing Java transform available"
##########
File path: website/www/site/content/en/documentation/programming-guide.md
##########
@@ -6555,6 +6555,115 @@ In this section, we will use [KafkaIO.Read](https://beam.apache.org/releases/jav
#### 13.1.1. Creating cross-language Java transforms
+There are two ways to make Java transforms available to other SDKs.
+
+* Option 1: In some cases, you can use existing Java transforms from other SDKs without writing any additional Java code.
+* Option 2: You can use arbitrary Java Transforms from other SDKs by adding few Java classes.
+
+##### 13.1.1.1 Using Existing Java Transforms from Other SDKs Without Writing more Java Code
+
+Starting with Beam 2.34.0, Python SDK users can use some Java transforms without writing additional Java code. This can be useful in many cases. For example,
+* A developer not familiar with Java may need to use an existing Java transform from a Python pipeline
+* A developer may need to make a Java transform that is already released, available to a Python pipeline without writing/releasing more Java code
+
+> **Note:** This feature is currently only available when using Java transforms from a Python pipeline.
+
+To be eligible for direct usage, API of the Java transforms has to follow the following pattern.
+* Requirement 1: Java transform can be constructed using an available public constructor or a public static method (a constructor method) in the same Java class.
+* Requirement 2: Java transform can be configured using one or more builder methods. Each builder method should be public and should return an instance of the Java transform.
+
+See below for the structure of an example Java class that can be directly used from the Python API.
+
+{{< highlight >}}
+public class JavaDataGenerator extends PTransform<PBegin, PCollection<String>> {
+ . . .
+
+ // Following method satisfies the Requirement 1.
+ // Note that you may also use a class constructor instead of a static method.
+ public static JavaDataGenerator create(Integer size) {
+ return new JavaDataGenerator(size);
+ }
+
+ static class JavaDataGeneratorConfig implements Serializable {
+ public String prefix;
+ public long length;
+ public String suffix;
+ . . .
+ }
+
+ // Following method conforms to the Requirement 2
+ public JavaDataGenerator withJavaDataGeneratorConfig(JavaDataGeneratorConfig dataConfig) {
+ return new JavaDataGenerator(this.size, javaDataGeneratorConfig);
+ }
+
+ . . .
+}
+{{< /highlight >}}
+
+To use a Java class that conforms to the above requirement from a Python SDK pipeline you may do the following.
+
+* Step 1: create an allowlist file in the _yaml_ format that describes the Java transform classes and methods that will be directly accessed from Python.
+* Step 2: start an Expansion Service with the `javaClassLookupAllowlistFile` option passing path to the allowlist defined in Step 1 as the value.
+* Step 3: Use the Python [JavaExternalTransform](https://github.com/apache/beam/blob/master/sdks/python/apache_beam/transforms/external.py) API to directly
+ access Java transforms defined in the allowlist from the Python side.
+
+Starting with Beam 2.35.0, Step 1 and 2 may be skipped as described in corresponding sections below.
+
+##### Step 1
+
+To use this Java transform from Python, you may define an allowlist file in the _yaml_ format. This allowlist lists the class names,
+constructor methods, and builder methods that are directly available to be used from the Python side.
+
+Starting with Beam 2.35.0, you have the option to specify `*` to the `javaClassLookupAllowlistFile` option instead of defining an actual allowlist which
+denotes that all supported transforms in the classpath of the exapansion service may be accessed through the API.
+
+{{< highlight >}}
+version: v1
+allowedClasses:
+- className: my.beam.transforms.JavaDataGenerator
+ allowedConstructorMethods:
+ - create
+ allowedBuilderMethods:
+ - withJavaDataGeneratorConfig
+{{< /highlight >}}
+
+##### Step 2
+
+The allowlist is provided as an argument when starting up the Java expansion service. For example, the expansion service can be started
+as a local Java process using the following command.
+
+{{< highlight >}}
+java -jar <jar file> <port> --javaClassLookupAllowlistFile=<path to the allowlist file>
+{{< /highlight >}}
+
+Starting with Beam 2.35.0, Beam 'JavaExternalTransform' API will automatically startup an expansion service with a given set of `jar` file dependencies
+if an expansion service address was not provided.
+
+##### Step 3
+
+You can directly use the Java class from your Python pipeline using a stub transform created using the `JavaExternalTransform` API. This API allows you to construct the transform
+using the Java class name and allows you to invoke builder methods to configure the class.
+
+Constructor and method parameter types are mapped between Python and Java using a Beam Schema. The Schema is auto-generated using the object types
+provided in the Python side. If the Java class constructor method or builder method accepts any complex object types, make sure that Beam Schema
Review comment:
"Beam Schema" -> "the Beam schema"
##########
File path: website/www/site/content/en/documentation/programming-guide.md
##########
@@ -6555,6 +6555,115 @@ In this section, we will use [KafkaIO.Read](https://beam.apache.org/releases/jav
#### 13.1.1. Creating cross-language Java transforms
+There are two ways to make Java transforms available to other SDKs.
+
+* Option 1: In some cases, you can use existing Java transforms from other SDKs without writing any additional Java code.
+* Option 2: You can use arbitrary Java Transforms from other SDKs by adding few Java classes.
+
+##### 13.1.1.1 Using Existing Java Transforms from Other SDKs Without Writing more Java Code
+
+Starting with Beam 2.34.0, Python SDK users can use some Java transforms without writing additional Java code. This can be useful in many cases. For example,
+* A developer not familiar with Java may need to use an existing Java transform from a Python pipeline
+* A developer may need to make a Java transform that is already released, available to a Python pipeline without writing/releasing more Java code
+
+> **Note:** This feature is currently only available when using Java transforms from a Python pipeline.
+
+To be eligible for direct usage, API of the Java transforms has to follow the following pattern.
+* Requirement 1: Java transform can be constructed using an available public constructor or a public static method (a constructor method) in the same Java class.
+* Requirement 2: Java transform can be configured using one or more builder methods. Each builder method should be public and should return an instance of the Java transform.
Review comment:
"Java" -> "The Java"
##########
File path: website/www/site/content/en/documentation/programming-guide.md
##########
@@ -6555,6 +6555,115 @@ In this section, we will use [KafkaIO.Read](https://beam.apache.org/releases/jav
#### 13.1.1. Creating cross-language Java transforms
+There are two ways to make Java transforms available to other SDKs.
+
+* Option 1: In some cases, you can use existing Java transforms from other SDKs without writing any additional Java code.
+* Option 2: You can use arbitrary Java Transforms from other SDKs by adding few Java classes.
+
+##### 13.1.1.1 Using Existing Java Transforms from Other SDKs Without Writing more Java Code
+
+Starting with Beam 2.34.0, Python SDK users can use some Java transforms without writing additional Java code. This can be useful in many cases. For example,
+* A developer not familiar with Java may need to use an existing Java transform from a Python pipeline
+* A developer may need to make a Java transform that is already released, available to a Python pipeline without writing/releasing more Java code
+
+> **Note:** This feature is currently only available when using Java transforms from a Python pipeline.
+
+To be eligible for direct usage, API of the Java transforms has to follow the following pattern.
+* Requirement 1: Java transform can be constructed using an available public constructor or a public static method (a constructor method) in the same Java class.
+* Requirement 2: Java transform can be configured using one or more builder methods. Each builder method should be public and should return an instance of the Java transform.
+
+See below for the structure of an example Java class that can be directly used from the Python API.
+
+{{< highlight >}}
+public class JavaDataGenerator extends PTransform<PBegin, PCollection<String>> {
+ . . .
+
+ // Following method satisfies the Requirement 1.
+ // Note that you may also use a class constructor instead of a static method.
+ public static JavaDataGenerator create(Integer size) {
+ return new JavaDataGenerator(size);
+ }
+
+ static class JavaDataGeneratorConfig implements Serializable {
+ public String prefix;
+ public long length;
+ public String suffix;
+ . . .
+ }
+
+ // Following method conforms to the Requirement 2
+ public JavaDataGenerator withJavaDataGeneratorConfig(JavaDataGeneratorConfig dataConfig) {
+ return new JavaDataGenerator(this.size, javaDataGeneratorConfig);
+ }
+
+ . . .
+}
+{{< /highlight >}}
+
+To use a Java class that conforms to the above requirement from a Python SDK pipeline you may do the following.
+
+* Step 1: create an allowlist file in the _yaml_ format that describes the Java transform classes and methods that will be directly accessed from Python.
+* Step 2: start an Expansion Service with the `javaClassLookupAllowlistFile` option passing path to the allowlist defined in Step 1 as the value.
+* Step 3: Use the Python [JavaExternalTransform](https://github.com/apache/beam/blob/master/sdks/python/apache_beam/transforms/external.py) API to directly
+ access Java transforms defined in the allowlist from the Python side.
+
+Starting with Beam 2.35.0, Step 1 and 2 may be skipped as described in corresponding sections below.
+
+##### Step 1
+
+To use this Java transform from Python, you may define an allowlist file in the _yaml_ format. This allowlist lists the class names,
+constructor methods, and builder methods that are directly available to be used from the Python side.
+
+Starting with Beam 2.35.0, you have the option to specify `*` to the `javaClassLookupAllowlistFile` option instead of defining an actual allowlist which
+denotes that all supported transforms in the classpath of the exapansion service may be accessed through the API.
Review comment:
typo: "exapansion"
##########
File path: website/www/site/content/en/documentation/programming-guide.md
##########
@@ -6555,6 +6555,115 @@ In this section, we will use [KafkaIO.Read](https://beam.apache.org/releases/jav
#### 13.1.1. Creating cross-language Java transforms
+There are two ways to make Java transforms available to other SDKs.
+
+* Option 1: In some cases, you can use existing Java transforms from other SDKs without writing any additional Java code.
+* Option 2: You can use arbitrary Java Transforms from other SDKs by adding few Java classes.
+
+##### 13.1.1.1 Using Existing Java Transforms from Other SDKs Without Writing more Java Code
+
+Starting with Beam 2.34.0, Python SDK users can use some Java transforms without writing additional Java code. This can be useful in many cases. For example,
+* A developer not familiar with Java may need to use an existing Java transform from a Python pipeline
+* A developer may need to make a Java transform that is already released, available to a Python pipeline without writing/releasing more Java code
+
+> **Note:** This feature is currently only available when using Java transforms from a Python pipeline.
+
+To be eligible for direct usage, API of the Java transforms has to follow the following pattern.
+* Requirement 1: Java transform can be constructed using an available public constructor or a public static method (a constructor method) in the same Java class.
+* Requirement 2: Java transform can be configured using one or more builder methods. Each builder method should be public and should return an instance of the Java transform.
+
+See below for the structure of an example Java class that can be directly used from the Python API.
+
+{{< highlight >}}
+public class JavaDataGenerator extends PTransform<PBegin, PCollection<String>> {
+ . . .
+
+ // Following method satisfies the Requirement 1.
+ // Note that you may also use a class constructor instead of a static method.
+ public static JavaDataGenerator create(Integer size) {
+ return new JavaDataGenerator(size);
+ }
+
+ static class JavaDataGeneratorConfig implements Serializable {
+ public String prefix;
+ public long length;
+ public String suffix;
+ . . .
+ }
+
+ // Following method conforms to the Requirement 2
+ public JavaDataGenerator withJavaDataGeneratorConfig(JavaDataGeneratorConfig dataConfig) {
+ return new JavaDataGenerator(this.size, javaDataGeneratorConfig);
+ }
+
+ . . .
+}
+{{< /highlight >}}
+
+To use a Java class that conforms to the above requirement from a Python SDK pipeline you may do the following.
+
+* Step 1: create an allowlist file in the _yaml_ format that describes the Java transform classes and methods that will be directly accessed from Python.
+* Step 2: start an Expansion Service with the `javaClassLookupAllowlistFile` option passing path to the allowlist defined in Step 1 as the value.
+* Step 3: Use the Python [JavaExternalTransform](https://github.com/apache/beam/blob/master/sdks/python/apache_beam/transforms/external.py) API to directly
+ access Java transforms defined in the allowlist from the Python side.
+
+Starting with Beam 2.35.0, Step 1 and 2 may be skipped as described in corresponding sections below.
+
+##### Step 1
+
+To use this Java transform from Python, you may define an allowlist file in the _yaml_ format. This allowlist lists the class names,
+constructor methods, and builder methods that are directly available to be used from the Python side.
+
+Starting with Beam 2.35.0, you have the option to specify `*` to the `javaClassLookupAllowlistFile` option instead of defining an actual allowlist which
+denotes that all supported transforms in the classpath of the exapansion service may be accessed through the API.
+
+{{< highlight >}}
+version: v1
+allowedClasses:
+- className: my.beam.transforms.JavaDataGenerator
+ allowedConstructorMethods:
+ - create
+ allowedBuilderMethods:
+ - withJavaDataGeneratorConfig
+{{< /highlight >}}
+
+##### Step 2
+
+The allowlist is provided as an argument when starting up the Java expansion service. For example, the expansion service can be started
+as a local Java process using the following command.
+
+{{< highlight >}}
+java -jar <jar file> <port> --javaClassLookupAllowlistFile=<path to the allowlist file>
+{{< /highlight >}}
+
+Starting with Beam 2.35.0, Beam 'JavaExternalTransform' API will automatically startup an expansion service with a given set of `jar` file dependencies
+if an expansion service address was not provided.
+
+##### Step 3
+
+You can directly use the Java class from your Python pipeline using a stub transform created using the `JavaExternalTransform` API. This API allows you to construct the transform
+using the Java class name and allows you to invoke builder methods to configure the class.
+
+Constructor and method parameter types are mapped between Python and Java using a Beam Schema. The Schema is auto-generated using the object types
+provided in the Python side. If the Java class constructor method or builder method accepts any complex object types, make sure that Beam Schema
+for these objects are registered and available for the Java expansion service. If a schema has not been registered, Java expansion service will
Review comment:
"Java expansion service" -> "the Java expansion service"
##########
File path: website/www/site/content/en/documentation/programming-guide.md
##########
@@ -6555,6 +6555,115 @@ In this section, we will use [KafkaIO.Read](https://beam.apache.org/releases/jav
#### 13.1.1. Creating cross-language Java transforms
+There are two ways to make Java transforms available to other SDKs.
+
+* Option 1: In some cases, you can use existing Java transforms from other SDKs without writing any additional Java code.
+* Option 2: You can use arbitrary Java Transforms from other SDKs by adding few Java classes.
+
+##### 13.1.1.1 Using Existing Java Transforms from Other SDKs Without Writing more Java Code
+
+Starting with Beam 2.34.0, Python SDK users can use some Java transforms without writing additional Java code. This can be useful in many cases. For example,
+* A developer not familiar with Java may need to use an existing Java transform from a Python pipeline
+* A developer may need to make a Java transform that is already released, available to a Python pipeline without writing/releasing more Java code
+
+> **Note:** This feature is currently only available when using Java transforms from a Python pipeline.
+
+To be eligible for direct usage, API of the Java transforms has to follow the following pattern.
+* Requirement 1: Java transform can be constructed using an available public constructor or a public static method (a constructor method) in the same Java class.
+* Requirement 2: Java transform can be configured using one or more builder methods. Each builder method should be public and should return an instance of the Java transform.
+
+See below for the structure of an example Java class that can be directly used from the Python API.
+
+{{< highlight >}}
+public class JavaDataGenerator extends PTransform<PBegin, PCollection<String>> {
+ . . .
+
+ // Following method satisfies the Requirement 1.
+ // Note that you may also use a class constructor instead of a static method.
+ public static JavaDataGenerator create(Integer size) {
+ return new JavaDataGenerator(size);
+ }
+
+ static class JavaDataGeneratorConfig implements Serializable {
+ public String prefix;
+ public long length;
+ public String suffix;
+ . . .
+ }
+
+ // Following method conforms to the Requirement 2
+ public JavaDataGenerator withJavaDataGeneratorConfig(JavaDataGeneratorConfig dataConfig) {
+ return new JavaDataGenerator(this.size, javaDataGeneratorConfig);
+ }
+
+ . . .
+}
+{{< /highlight >}}
+
+To use a Java class that conforms to the above requirement from a Python SDK pipeline you may do the following.
+
+* Step 1: create an allowlist file in the _yaml_ format that describes the Java transform classes and methods that will be directly accessed from Python.
+* Step 2: start an Expansion Service with the `javaClassLookupAllowlistFile` option passing path to the allowlist defined in Step 1 as the value.
+* Step 3: Use the Python [JavaExternalTransform](https://github.com/apache/beam/blob/master/sdks/python/apache_beam/transforms/external.py) API to directly
+ access Java transforms defined in the allowlist from the Python side.
+
+Starting with Beam 2.35.0, Step 1 and 2 may be skipped as described in corresponding sections below.
+
+##### Step 1
+
+To use this Java transform from Python, you may define an allowlist file in the _yaml_ format. This allowlist lists the class names,
+constructor methods, and builder methods that are directly available to be used from the Python side.
+
+Starting with Beam 2.35.0, you have the option to specify `*` to the `javaClassLookupAllowlistFile` option instead of defining an actual allowlist which
+denotes that all supported transforms in the classpath of the exapansion service may be accessed through the API.
+
+{{< highlight >}}
+version: v1
+allowedClasses:
+- className: my.beam.transforms.JavaDataGenerator
+ allowedConstructorMethods:
+ - create
+ allowedBuilderMethods:
+ - withJavaDataGeneratorConfig
+{{< /highlight >}}
+
+##### Step 2
+
+The allowlist is provided as an argument when starting up the Java expansion service. For example, the expansion service can be started
+as a local Java process using the following command.
+
+{{< highlight >}}
+java -jar <jar file> <port> --javaClassLookupAllowlistFile=<path to the allowlist file>
+{{< /highlight >}}
+
+Starting with Beam 2.35.0, Beam 'JavaExternalTransform' API will automatically startup an expansion service with a given set of `jar` file dependencies
Review comment:
Use backticks instead of single quotes for syntax highlighting: `JavaExternalTransform` (I think this is actually a class name, right?)
##########
File path: website/www/site/content/en/documentation/programming-guide.md
##########
@@ -6555,6 +6555,115 @@ In this section, we will use [KafkaIO.Read](https://beam.apache.org/releases/jav
#### 13.1.1. Creating cross-language Java transforms
+There are two ways to make Java transforms available to other SDKs.
+
+* Option 1: In some cases, you can use existing Java transforms from other SDKs without writing any additional Java code.
+* Option 2: You can use arbitrary Java Transforms from other SDKs by adding few Java classes.
+
+##### 13.1.1.1 Using Existing Java Transforms from Other SDKs Without Writing more Java Code
+
+Starting with Beam 2.34.0, Python SDK users can use some Java transforms without writing additional Java code. This can be useful in many cases. For example,
+* A developer not familiar with Java may need to use an existing Java transform from a Python pipeline
+* A developer may need to make a Java transform that is already released, available to a Python pipeline without writing/releasing more Java code
+
+> **Note:** This feature is currently only available when using Java transforms from a Python pipeline.
+
+To be eligible for direct usage, API of the Java transforms has to follow the following pattern.
+* Requirement 1: Java transform can be constructed using an available public constructor or a public static method (a constructor method) in the same Java class.
+* Requirement 2: Java transform can be configured using one or more builder methods. Each builder method should be public and should return an instance of the Java transform.
+
+See below for the structure of an example Java class that can be directly used from the Python API.
+
+{{< highlight >}}
+public class JavaDataGenerator extends PTransform<PBegin, PCollection<String>> {
+ . . .
+
+ // Following method satisfies the Requirement 1.
+ // Note that you may also use a class constructor instead of a static method.
+ public static JavaDataGenerator create(Integer size) {
+ return new JavaDataGenerator(size);
+ }
+
+ static class JavaDataGeneratorConfig implements Serializable {
+ public String prefix;
+ public long length;
+ public String suffix;
+ . . .
+ }
+
+ // Following method conforms to the Requirement 2
+ public JavaDataGenerator withJavaDataGeneratorConfig(JavaDataGeneratorConfig dataConfig) {
+ return new JavaDataGenerator(this.size, javaDataGeneratorConfig);
+ }
+
+ . . .
+}
+{{< /highlight >}}
+
+To use a Java class that conforms to the above requirement from a Python SDK pipeline you may do the following.
+
+* Step 1: create an allowlist file in the _yaml_ format that describes the Java transform classes and methods that will be directly accessed from Python.
+* Step 2: start an Expansion Service with the `javaClassLookupAllowlistFile` option passing path to the allowlist defined in Step 1 as the value.
+* Step 3: Use the Python [JavaExternalTransform](https://github.com/apache/beam/blob/master/sdks/python/apache_beam/transforms/external.py) API to directly
+ access Java transforms defined in the allowlist from the Python side.
+
+Starting with Beam 2.35.0, Step 1 and 2 may be skipped as described in corresponding sections below.
+
+##### Step 1
+
+To use this Java transform from Python, you may define an allowlist file in the _yaml_ format. This allowlist lists the class names,
+constructor methods, and builder methods that are directly available to be used from the Python side.
+
+Starting with Beam 2.35.0, you have the option to specify `*` to the `javaClassLookupAllowlistFile` option instead of defining an actual allowlist which
+denotes that all supported transforms in the classpath of the exapansion service may be accessed through the API.
+
+{{< highlight >}}
+version: v1
+allowedClasses:
+- className: my.beam.transforms.JavaDataGenerator
+ allowedConstructorMethods:
+ - create
+ allowedBuilderMethods:
+ - withJavaDataGeneratorConfig
+{{< /highlight >}}
+
+##### Step 2
+
+The allowlist is provided as an argument when starting up the Java expansion service. For example, the expansion service can be started
+as a local Java process using the following command.
+
+{{< highlight >}}
+java -jar <jar file> <port> --javaClassLookupAllowlistFile=<path to the allowlist file>
+{{< /highlight >}}
+
+Starting with Beam 2.35.0, Beam 'JavaExternalTransform' API will automatically startup an expansion service with a given set of `jar` file dependencies
+if an expansion service address was not provided.
+
+##### Step 3
+
+You can directly use the Java class from your Python pipeline using a stub transform created using the `JavaExternalTransform` API. This API allows you to construct the transform
+using the Java class name and allows you to invoke builder methods to configure the class.
+
+Constructor and method parameter types are mapped between Python and Java using a Beam Schema. The Schema is auto-generated using the object types
+provided in the Python side. If the Java class constructor method or builder method accepts any complex object types, make sure that Beam Schema
Review comment:
"in the Python" -> "on the Python"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] chamikaramj merged pull request #15941: [BEAM-13223] Documents two new features of cross-language transforms
Posted by GitBox <gi...@apache.org>.
chamikaramj merged pull request #15941:
URL: https://github.com/apache/beam/pull/15941
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pcoet commented on a change in pull request #15941: [BEAM-13223] Documents two new features of cross-language transforms
Posted by GitBox <gi...@apache.org>.
pcoet commented on a change in pull request #15941:
URL: https://github.com/apache/beam/pull/15941#discussion_r748574358
##########
File path: website/www/site/content/en/documentation/programming-guide.md
##########
@@ -6555,6 +6555,115 @@ In this section, we will use [KafkaIO.Read](https://beam.apache.org/releases/jav
#### 13.1.1. Creating cross-language Java transforms
+There are two ways to make Java transforms available to other SDKs.
+
+* Option 1: In some cases, you can use existing Java transforms from other SDKs without writing any additional Java code.
+* Option 2: You can use arbitrary Java Transforms from other SDKs by adding few Java classes.
+
+##### 13.1.1.1 Using Existing Java Transforms from Other SDKs Without Writing more Java Code
+
+Starting with Beam 2.34.0, Python SDK users can use some Java transforms without writing additional Java code. This can be useful in many cases. For example,
+* A developer not familiar with Java may need to use an existing Java transform from a Python pipeline
+* A developer may need to make a Java transform that is already released, available to a Python pipeline without writing/releasing more Java code
+
+> **Note:** This feature is currently only available when using Java transforms from a Python pipeline.
+
+To be eligible for direct usage, API of the Java transforms has to follow the following pattern.
+* Requirement 1: Java transform can be constructed using an available public constructor or a public static method (a constructor method) in the same Java class.
Review comment:
"Requirement 1: Java" -> "Requirement 1: The Java"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] chamikaramj commented on pull request #15941: [BEAM-13223] Documents two new features of cross-language transforms
Posted by GitBox <gi...@apache.org>.
chamikaramj commented on pull request #15941:
URL: https://github.com/apache/beam/pull/15941#issuecomment-965937731
R: @pcoet
CC: @robertwb
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org