You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "Abacn (via GitHub)" <gi...@apache.org> on 2025/03/13 21:24:28 UTC

[PR] Expose add-modules JVM option in SDKHarnessOptions [beam]

Abacn opened a new pull request, #34289:
URL: https://github.com/apache/beam/pull/34289

   Fix #30281
   
   **Please** add a meaningful description for your change here
   
   ------------------------
   
   Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
   
    - [ ] Mention the appropriate issue in your description (for example: `addresses #123`), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment `fixes #<ISSUE NUMBER>` instead.
    - [ ] Update `CHANGES.md` with noteworthy changes.
    - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://github.com/apache/beam/blob/master/CONTRIBUTING.md#make-the-reviewers-job-easier).
   
   To check the build health, please visit [https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md](https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md)
   
   GitHub Actions Tests Status (on master branch)
   ------------------------------------------------------------------------------------------------
   [![Build python source distribution and wheels](https://github.com/apache/beam/actions/workflows/build_wheels.yml/badge.svg?event=schedule&&?branch=master)](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
   [![Python tests](https://github.com/apache/beam/actions/workflows/python_tests.yml/badge.svg?event=schedule&&?branch=master)](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
   [![Java tests](https://github.com/apache/beam/actions/workflows/java_tests.yml/badge.svg?event=schedule&&?branch=master)](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
   [![Go tests](https://github.com/apache/beam/actions/workflows/go_tests.yml/badge.svg?event=schedule&&?branch=master)](https://github.com/apache/beam/actions?query=workflow%3A%22Go+tests%22+branch%3Amaster+event%3Aschedule)
   
   See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI or the [workflows README](https://github.com/apache/beam/blob/master/.github/workflows/README.md) to see a list of phrases to trigger workflows.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] Expose add-modules JVM option in SDKHarnessOptions [beam]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #34289:
URL: https://github.com/apache/beam/pull/34289#issuecomment-2722836565

   Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control. If you'd like to restart, comment `assign set of reviewers`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] Expose add-modules JVM option in SDKHarnessOptions [beam]

Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on PR #34289:
URL: https://github.com/apache/beam/pull/34289#issuecomment-2734331259

   failed Dataflow worker test:
   
   - testCommit_handlesFailedCommits
   - testStop_successfullyTerminates
   - testLatencyAttributionToCommittingState
   
   not related to the change


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] Expose add-modules JVM option in SDKHarnessOptions [beam]

Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn commented on code in PR #34289:
URL: https://github.com/apache/beam/pull/34289#discussion_r2001455921


##########
CHANGES.md:
##########
@@ -68,8 +68,9 @@
 
 ## New Features / Improvements
 
-* Support custom coders in Reshuffle ([#29908](https://github.com/apache/beam/issues/29908), [#33356](https://github.com/apache/beam/issues/33356)).
+* [Python] Support custom coders in Reshuffle ([#29908](https://github.com/apache/beam/issues/29908), [#33356](https://github.com/apache/beam/issues/33356)).
 * [Java] Upgrade SLF4J to 2.0.16. Update default Spark version to 3.5.0. ([#33574](https://github.com/apache/beam/pull/33574))
+* [Java] Support `--add-modules` JVM option is added. This allows SDK incubator modules such as jdk.incubator.vector ([#30281](https://github.com/apache/beam/issues/30281)).

Review Comment:
   Consider the following note:
   
   ```suggestion
   * [Java] Support for `--add-modules` JVM option is added through a new pipeline option `JdkAddRootModules`. This allows extending the module graph with optional modules such as Java SDK incubator modules. Sample usage:  `<pipeline invocation> --jdkAddRootModules=jdk.incubator.vector` ([#30281](https://github.com/apache/beam/issues/30281)).
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] Expose add-modules JVM option in SDKHarnessOptions [beam]

Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn merged PR #34289:
URL: https://github.com/apache/beam/pull/34289


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] Expose add-modules JVM option in SDKHarnessOptions [beam]

Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn commented on code in PR #34289:
URL: https://github.com/apache/beam/pull/34289#discussion_r2001455921


##########
CHANGES.md:
##########
@@ -68,8 +68,9 @@
 
 ## New Features / Improvements
 
-* Support custom coders in Reshuffle ([#29908](https://github.com/apache/beam/issues/29908), [#33356](https://github.com/apache/beam/issues/33356)).
+* [Python] Support custom coders in Reshuffle ([#29908](https://github.com/apache/beam/issues/29908), [#33356](https://github.com/apache/beam/issues/33356)).
 * [Java] Upgrade SLF4J to 2.0.16. Update default Spark version to 3.5.0. ([#33574](https://github.com/apache/beam/pull/33574))
+* [Java] Support `--add-modules` JVM option is added. This allows SDK incubator modules such as jdk.incubator.vector ([#30281](https://github.com/apache/beam/issues/30281)).

Review Comment:
   ```suggestion
   * [Java] Support for `--add-modules` JVM option is added through a new pipeline option `JdkAddRootModules`. This allows extending the module graph with optional modules such as SDK  incubator modules. Sample usage:  `<pipeline invocation > --jdkAddRootModules=jdk.incubator.vector` ([#30281](https://github.com/apache/beam/issues/30281)).
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] Expose add-modules JVM option in SDKHarnessOptions [beam]

Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on PR #34289:
URL: https://github.com/apache/beam/pull/34289#issuecomment-2722822617

   Tested on Dataflow v2 + Java 21 with a pipeline uses "jdk.incubator.vector"
   
   Beam 2.63.0 (job Id `2025-03-13_15_08_35-746015370466363900`):
   
   ```
   "org.apache.beam.sdk.util.UserCodeException: java.lang.NoClassDefFoundError: jdk/incubator/vector/Vector
   	at org.apache.beam.sdk.util.UserCodeException.wrap(UserCodeException.java:39)
   	at com.github.abacn.VectorTest$1$DoFnInvoker.invokeProcessElement(Unknown Source)
   ```
   
   with this change (build java core locally, build a custom container, add pipeline option `--sdkContainerImage=gcr.io/.../beam_java21_sdk:2.63.0-custom --jdkAddRootModules=jdk.incubator.vector`), it succeeded. job id: `2025-03-13_15_22_36-15628134386308323283`
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] Expose add-modules JVM option in SDKHarnessOptions [beam]

Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn commented on code in PR #34289:
URL: https://github.com/apache/beam/pull/34289#discussion_r2001455921


##########
CHANGES.md:
##########
@@ -68,8 +68,9 @@
 
 ## New Features / Improvements
 
-* Support custom coders in Reshuffle ([#29908](https://github.com/apache/beam/issues/29908), [#33356](https://github.com/apache/beam/issues/33356)).
+* [Python] Support custom coders in Reshuffle ([#29908](https://github.com/apache/beam/issues/29908), [#33356](https://github.com/apache/beam/issues/33356)).
 * [Java] Upgrade SLF4J to 2.0.16. Update default Spark version to 3.5.0. ([#33574](https://github.com/apache/beam/pull/33574))
+* [Java] Support `--add-modules` JVM option is added. This allows SDK incubator modules such as jdk.incubator.vector ([#30281](https://github.com/apache/beam/issues/30281)).

Review Comment:
   ```suggestion
   * [Java] Support for `--add-modules` JVM option is added through a new pipeline option `JdkAddRootModules`. This allows extending the module graph with optional modules such as Java SDK incubator modules. Sample usage:  `<pipeline invocation > --jdkAddRootModules=jdk.incubator.vector` ([#30281](https://github.com/apache/beam/issues/30281)).
   ```



##########
CHANGES.md:
##########
@@ -68,8 +68,9 @@
 
 ## New Features / Improvements
 
-* Support custom coders in Reshuffle ([#29908](https://github.com/apache/beam/issues/29908), [#33356](https://github.com/apache/beam/issues/33356)).
+* [Python] Support custom coders in Reshuffle ([#29908](https://github.com/apache/beam/issues/29908), [#33356](https://github.com/apache/beam/issues/33356)).
 * [Java] Upgrade SLF4J to 2.0.16. Update default Spark version to 3.5.0. ([#33574](https://github.com/apache/beam/pull/33574))
+* [Java] Support `--add-modules` JVM option is added. This allows SDK incubator modules such as jdk.incubator.vector ([#30281](https://github.com/apache/beam/issues/30281)).

Review Comment:
   ```suggestion
   * [Java] Support for `--add-modules` JVM option is added through a new pipeline option `JdkAddRootModules`. This allows extending the module graph with optional modules such as Java SDK incubator modules. Sample usage:  `<pipeline invocation> --jdkAddRootModules=jdk.incubator.vector` ([#30281](https://github.com/apache/beam/issues/30281)).
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] Expose add-modules JVM option in SDKHarnessOptions [beam]

Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on PR #34289:
URL: https://github.com/apache/beam/pull/34289#issuecomment-2733522593

   R: @tvalentyn (internal tracker: b/337016101)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] Expose add-modules JVM option in SDKHarnessOptions [beam]

Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn commented on code in PR #34289:
URL: https://github.com/apache/beam/pull/34289#discussion_r2001456361


##########
CHANGES.md:
##########
@@ -68,8 +68,9 @@
 
 ## New Features / Improvements
 
-* Support custom coders in Reshuffle ([#29908](https://github.com/apache/beam/issues/29908), [#33356](https://github.com/apache/beam/issues/33356)).
+* [Python] Support custom coders in Reshuffle ([#29908](https://github.com/apache/beam/issues/29908), [#33356](https://github.com/apache/beam/issues/33356)).
 * [Java] Upgrade SLF4J to 2.0.16. Update default Spark version to 3.5.0. ([#33574](https://github.com/apache/beam/pull/33574))
+* [Java] Support `--add-modules` JVM option is added. This allows SDK incubator modules such as jdk.incubator.vector ([#30281](https://github.com/apache/beam/issues/30281)).

Review Comment:
   (Please correct as necessary)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] Expose add-modules JVM option in SDKHarnessOptions [beam]

Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn commented on PR #34289:
URL: https://github.com/apache/beam/pull/34289#issuecomment-2733906267

   (Updated pr description)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] Expose add-modules JVM option in SDKHarnessOptions [beam]

Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on PR #34289:
URL: https://github.com/apache/beam/pull/34289#issuecomment-2725147468

   R: @dannymccormick 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] Expose add-modules JVM option in SDKHarnessOptions [beam]

Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn commented on code in PR #34289:
URL: https://github.com/apache/beam/pull/34289#discussion_r2001466799


##########
CHANGES.md:
##########
@@ -68,8 +68,9 @@
 
 ## New Features / Improvements
 
-* Support custom coders in Reshuffle ([#29908](https://github.com/apache/beam/issues/29908), [#33356](https://github.com/apache/beam/issues/33356)).
+* [Python] Support custom coders in Reshuffle ([#29908](https://github.com/apache/beam/issues/29908), [#33356](https://github.com/apache/beam/issues/33356)).
 * [Java] Upgrade SLF4J to 2.0.16. Update default Spark version to 3.5.0. ([#33574](https://github.com/apache/beam/pull/33574))
+* [Java] Support for `--add-modules` JVM option is added through a new pipeline option `JdkAddRootModules`. This allows extending the module graph with optional modules such as SDK incubator modules. Sample usage:  `<pipeline invocation > --jdkAddRootModules=jdk.incubator.vector` ([#30281](https://github.com/apache/beam/issues/30281)).

Review Comment:
   ```suggestion
   * [Java] Support for `--add-modules` JVM option is added through a new pipeline option `JdkAddRootModules`. This allows extending the module graph with optional modules such as SDK incubator modules. Sample usage: `<pipeline invocation> --jdkAddRootModules=jdk.incubator.vector` ([#30281](https://github.com/apache/beam/issues/30281)).
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] Expose add-modules JVM option in SDKHarnessOptions [beam]

Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on PR #34289:
URL: https://github.com/apache/beam/pull/34289#issuecomment-2722834989

   R: @kennknowles 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] Expose add-modules JVM option in SDKHarnessOptions [beam]

Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on PR #34289:
URL: https://github.com/apache/beam/pull/34289#issuecomment-2725140094

   also worked for Dataflow legacy runner (after internal change)
   
   job id: `2025-03-14_08_57_18-6904344628307499835`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org