You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "Abacn (via GitHub)" <gi...@apache.org> on 2025/09/30 21:29:21 UTC

[PR] Set user agent when downloading jars with local_jar [beam]

Abacn opened a new pull request, #36342:
URL: https://github.com/apache/beam/pull/36342

   **Please** add a meaningful description for your change here
   
   ------------------------
   
   Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
   
    - [ ] Mention the appropriate issue in your description (for example: `addresses #123`), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment `fixes #<ISSUE NUMBER>` instead.
    - [ ] Update `CHANGES.md` with noteworthy changes.
    - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://github.com/apache/beam/blob/master/CONTRIBUTING.md#make-the-reviewers-job-easier).
   
   To check the build health, please visit [https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md](https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md)
   
   GitHub Actions Tests Status (on master branch)
   ------------------------------------------------------------------------------------------------
   [![Build python source distribution and wheels](https://github.com/apache/beam/actions/workflows/build_wheels.yml/badge.svg?event=schedule&&?branch=master)](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
   [![Python tests](https://github.com/apache/beam/actions/workflows/python_tests.yml/badge.svg?event=schedule&&?branch=master)](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
   [![Java tests](https://github.com/apache/beam/actions/workflows/java_tests.yml/badge.svg?event=schedule&&?branch=master)](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
   [![Go tests](https://github.com/apache/beam/actions/workflows/go_tests.yml/badge.svg?event=schedule&&?branch=master)](https://github.com/apache/beam/actions?query=workflow%3A%22Go+tests%22+branch%3Amaster+event%3Aschedule)
   
   See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI or the [workflows README](https://github.com/apache/beam/blob/master/.github/workflows/README.md) to see a list of phrases to trigger workflows.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Set user agent when downloading jars with local_jar [beam]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #36342:
URL: https://github.com/apache/beam/pull/36342#issuecomment-3357229753

   Assigning reviewers:
   
   R: @liferoad for label python.
   
   
   Note: If you would like to opt out of this review, comment `assign to next reviewer`.
   
   Available commands:
   - `stop reviewer notifications` - opt out of the automated review tooling
   - `remind me after tests pass` - tag the comment author after tests pass
   - `waiting on author` - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)
   
   The PR bot will only process comments in the main thread (not review comments).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Set user agent when downloading jars with local_jar [beam]

Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on code in PR #36342:
URL: https://github.com/apache/beam/pull/36342#discussion_r2395515088


##########
sdks/python/apache_beam/utils/transform_service_launcher.py:
##########
@@ -57,13 +57,13 @@ class TransformServiceLauncher(object):
 
   # Maintaining a static list of launchers to prevent temporary resources
   # from being created unnecessarily.
-  def __new__(cls, project_name, port, beam_version=None):
+  def __new__(cls, project_name, port, beam_version=None, user_agent=None):
     if project_name not in TransformServiceLauncher._launchers:
       TransformServiceLauncher._launchers[project_name] = super(
           TransformServiceLauncher, cls).__new__(cls)
     return TransformServiceLauncher._launchers[project_name]

Review Comment:
   existing logic and internal API, closing this one.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Set user agent when downloading jars with local_jar [beam]

Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on PR #36342:
URL: https://github.com/apache/beam/pull/36342#issuecomment-3361252211

   MilvusException in Python3.12 ML PreCommit, not related to this change


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Set user agent when downloading jars with local_jar [beam]

Posted by "stankiewicz (via GitHub)" <gi...@apache.org>.
stankiewicz commented on code in PR #36342:
URL: https://github.com/apache/beam/pull/36342#discussion_r2393969102


##########
sdks/python/apache_beam/options/pipeline_options.py:
##########
@@ -1716,6 +1716,12 @@ def _add_argparse_args(cls, parser):
         help=(
             'Docker registry url to use for tagging and pushing the prebuilt '
             'sdk worker container image.'))
+    parser.add_argument(
+        '--user_agent',
+        default=None,

Review Comment:
   can we set new default e.g. "Apache Beam" ?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Set user agent when downloading jars with local_jar [beam]

Posted by "gemini-code-assist[bot] (via GitHub)" <gi...@apache.org>.
gemini-code-assist[bot] commented on code in PR #36342:
URL: https://github.com/apache/beam/pull/36342#discussion_r2395307342


##########
sdks/python/apache_beam/runners/portability/spark_uber_jar_job_server.py:
##########
@@ -53,6 +53,7 @@ def __init__(self, rest_url, options):
     spark_options = options.view_as(pipeline_options.SparkRunnerOptions)
     self._executable_jar = spark_options.spark_job_server_jar
     self._spark_version = spark_options.spark_version
+    self._user_agnet = options.view_as(pipeline_options.SetupOptions).user_agent

Review Comment:
   ![high](https://www.gstatic.com/codereviewagent/high-priority.svg)
   
   There's a typo in the variable name. It should be `_user_agent` instead of `_user_agnet`.
   
   ```suggestion
       self._user_agent = options.view_as(pipeline_options.SetupOptions).user_agent
   ```



##########
sdks/python/apache_beam/runners/portability/spark_uber_jar_job_server.py:
##########
@@ -78,7 +79,8 @@ def executable_jar(self):
       else:
         url = job_server.JavaJarJobServer.path_to_beam_jar(
             ':runners:spark:3:job-server:shadowJar')
-    return job_server.JavaJarJobServer.local_jar(url)
+    return job_server.JavaJarJobServer.local_jar(
+        url, user_agent=self._user_agnet)

Review Comment:
   ![high](https://www.gstatic.com/codereviewagent/high-priority.svg)
   
   This variable name has a typo due to its definition earlier in the file. It should be `_user_agent`.
   
   ```suggestion
       return job_server.JavaJarJobServer.local_jar(
           url, user_agent=self._user_agent)
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Set user agent when downloading jars with local_jar [beam]

Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on code in PR #36342:
URL: https://github.com/apache/beam/pull/36342#discussion_r2395520584


##########
sdks/python/apache_beam/transforms/external.py:
##########
@@ -1080,10 +1088,14 @@ def _default_args(self):
       args.append('--alsoStartLoopbackWorker')
     return args
 
+  def with_user_agent(self, user_agent: str):
+    self._user_agent = user_agent
+    return self

Review Comment:
   user agent only known from pipeline option and expected to remain the same for a pipeline, closing this



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Set user agent when downloading jars with local_jar [beam]

Posted by "damccorm (via GitHub)" <gi...@apache.org>.
damccorm commented on code in PR #36342:
URL: https://github.com/apache/beam/pull/36342#discussion_r2394624714


##########
sdks/python/apache_beam/options/pipeline_options.py:
##########
@@ -1716,6 +1716,12 @@ def _add_argparse_args(cls, parser):
         help=(
             'Docker registry url to use for tagging and pushing the prebuilt '
             'sdk worker container image.'))
+    parser.add_argument(
+        '--user_agent',
+        default=None,

Review Comment:
   I agree - from conversations with the sonatype folks, it sounds like having a default would be very helpful



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Set user agent when downloading jars with local_jar [beam]

Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on PR #36342:
URL: https://github.com/apache/beam/pull/36342#issuecomment-3357612709

   /gemini review


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Set user agent when downloading jars with local_jar [beam]

Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on code in PR #36342:
URL: https://github.com/apache/beam/pull/36342#discussion_r2394765626


##########
sdks/python/apache_beam/options/pipeline_options.py:
##########
@@ -1716,6 +1716,12 @@ def _add_argparse_args(cls, parser):
         help=(
             'Docker registry url to use for tagging and pushing the prebuilt '
             'sdk worker container image.'))
+    parser.add_argument(
+        '--user_agent',
+        default=None,

Review Comment:
   The default is set here: https://github.com/apache/beam/pull/36342/files#diff-fd140abcfcbb9b91db28c94a0458d02b4eb21f7f61db4935bf501464b27e1cfeR291 if user not setting this value the user agent is set to "Apache Beam Python SDK/2.xx.0"
   
   The problem of default pipeline options is that Java already has a --userAgent option defaults to "Apache Beam Java SDK/2.xx.0". This change is essentially port the option to Python. However Java side default is handled by a DefaultFactory class, but if I set a non-empty str as default in Python, I afraid am it could pass to Java side when running portable pipelines / xlang pipelines, or other unintended result



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Set user agent when downloading jars with local_jar [beam]

Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn merged PR #36342:
URL: https://github.com/apache/beam/pull/36342


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Set user agent when downloading jars with local_jar [beam]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #36342:
URL: https://github.com/apache/beam/pull/36342#issuecomment-3354246113

   Checks are failing. Will not request review until checks are succeeding. If you'd like to override that behavior, comment `assign set of reviewers`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Set user agent when downloading jars with local_jar [beam]

Posted by "gemini-code-assist[bot] (via GitHub)" <gi...@apache.org>.
gemini-code-assist[bot] commented on code in PR #36342:
URL: https://github.com/apache/beam/pull/36342#discussion_r2395486948


##########
sdks/python/apache_beam/utils/transform_service_launcher.py:
##########
@@ -57,13 +57,13 @@ class TransformServiceLauncher(object):
 
   # Maintaining a static list of launchers to prevent temporary resources
   # from being created unnecessarily.
-  def __new__(cls, project_name, port, beam_version=None):
+  def __new__(cls, project_name, port, beam_version=None, user_agent=None):
     if project_name not in TransformServiceLauncher._launchers:
       TransformServiceLauncher._launchers[project_name] = super(
           TransformServiceLauncher, cls).__new__(cls)
     return TransformServiceLauncher._launchers[project_name]

Review Comment:
   ![high](https://www.gstatic.com/codereviewagent/high-priority.svg)
   
   The `__new__` method caches `TransformServiceLauncher` instances using only `project_name` as the key. However, other parameters like `port` and `beam_version` are crucial for the launcher's configuration. This can lead to issues where a cached launcher for a given `project_name` is reused with a different `port` or `beam_version`, causing `__init__` to be re-run with new parameters on an old object. This can lead to incorrect behavior, such as attempting to connect to a service on the wrong port. The cache key should include all parameters that define a unique launcher configuration to prevent such state corruption.



##########
sdks/python/apache_beam/transforms/external.py:
##########
@@ -1080,10 +1088,14 @@ def _default_args(self):
       args.append('--alsoStartLoopbackWorker')
     return args
 
+  def with_user_agent(self, user_agent: str):
+    self._user_agent = user_agent
+    return self

Review Comment:
   ![medium](https://www.gstatic.com/codereviewagent/medium-priority.svg)
   
   The `with_user_agent` method modifies the instance in-place. This can lead to unexpected side-effects if the `JavaJarExpansionService` instance is shared, for example, across multiple pipelines with different `user_agent` options. It's safer to treat `JavaJarExpansionService` instances as immutable and return a new instance with the updated configuration.
   
   ```python
     def with_user_agent(self, user_agent: str):
       return JavaJarExpansionService(
           self.path_to_jar,
           self._extra_args,
           self._classpath,
           self._append_args,
           user_agent=user_agent)
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Set user agent when downloading jars with local_jar [beam]

Posted by "liferoad (via GitHub)" <gi...@apache.org>.
liferoad commented on PR #36342:
URL: https://github.com/apache/beam/pull/36342#issuecomment-3357344399

   /gemini review


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Set user agent when downloading jars with local_jar [beam]

Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on PR #36342:
URL: https://github.com/apache/beam/pull/36342#issuecomment-3354000085

   It's tricky to enter e2e tests (from pipeline option to check user agent in request) because the code path is excluded from dev Beam:
   
   https://github.com/apache/beam/blob/449ab577d3211b6fdfe9e9f3d231f31b44680377/sdks/python/apache_beam/utils/subprocess_server.py#L407
   
   needed to comment out these lines to make test script pass over this check. Tested that `--user_agent` pipeline option is honored when expanding a pipeline contains kafkaio. For this PR, added a check on request for default user agent (Apache Beam Python SDK/...)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org