You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "tvalentyn (via GitHub)" <gi...@apache.org> on 2023/02/04 02:55:15 UTC

[GitHub] [beam] tvalentyn opened a new pull request, #25313: Fail the pipeline when a mismatched Python version is detected.

tvalentyn opened a new pull request, #25313:
URL: https://github.com/apache/beam/pull/25313

   Addresses one of the main painpoints of #25312.
   
   ------------------------
   
   Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
   
    - [ ] Mention the appropriate issue in your description (for example: `addresses #123`), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment `fixes #<ISSUE NUMBER>` instead.
    - [ ] Update `CHANGES.md` with noteworthy changes.
    - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/get-started-contributing/#make-the-reviewers-job-easier).
   
   To check the build health, please visit [https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md](https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md)
   
   GitHub Actions Tests Status (on master branch)
   ------------------------------------------------------------------------------------------------
   [![Build python source distribution and wheels](https://github.com/apache/beam/workflows/Build%20python%20source%20distribution%20and%20wheels/badge.svg?branch=master&event=schedule)](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
   [![Python tests](https://github.com/apache/beam/workflows/Python%20tests/badge.svg?branch=master&event=schedule)](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
   [![Java tests](https://github.com/apache/beam/workflows/Java%20Tests/badge.svg?branch=master&event=schedule)](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
   [![Go tests](https://github.com/apache/beam/workflows/Go%20tests/badge.svg?branch=master&event=schedule)](https://github.com/apache/beam/actions?query=workflow%3A%22Go+tests%22+branch%3Amaster+event%3Aschedule)
   
   See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] codecov[bot] commented on pull request #25313: Fail the pipeline when a mismatched Python version is detected.

Posted by "codecov[bot] (via GitHub)" <gi...@apache.org>.
codecov[bot] commented on PR #25313:
URL: https://github.com/apache/beam/pull/25313#issuecomment-1416646482

   # [Codecov](https://codecov.io/gh/apache/beam/pull/25313?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
   > Merging [#25313](https://codecov.io/gh/apache/beam/pull/25313?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (7ae8071) into [master](https://codecov.io/gh/apache/beam/commit/86480f33397bd7984ae5cbb06ddcb3ffa1043ffe?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (86480f3) will **increase** coverage by `0.00%`.
   > The diff coverage is `100.00%`.
   
   ```diff
   @@           Coverage Diff           @@
   ##           master   #25313   +/-   ##
   =======================================
     Coverage   72.95%   72.95%           
   =======================================
     Files         745      745           
     Lines       99174    99189   +15     
   =======================================
   + Hits        72348    72367   +19     
   + Misses      25461    25457    -4     
     Partials     1365     1365           
   ```
   
   | Flag | Coverage Δ | |
   |---|---|---|
   | python | `82.46% <100.00%> (+<0.01%)` | :arrow_up: |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
   
   | [Impacted Files](https://codecov.io/gh/apache/beam/pull/25313?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
   |---|---|---|
   | [...hon/apache\_beam/runners/worker/bundle\_processor.py](https://codecov.io/gh/apache/beam/pull/25313?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvYnVuZGxlX3Byb2Nlc3Nvci5weQ==) | `93.66% <100.00%> (+0.36%)` | :arrow_up: |
   | [.../python/apache\_beam/testing/test\_stream\_service.py](https://codecov.io/gh/apache/beam/pull/25313?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdGVzdGluZy90ZXN0X3N0cmVhbV9zZXJ2aWNlLnB5) | `88.09% <0.00%> (-4.77%)` | :arrow_down: |
   | [sdks/python/apache\_beam/utils/interactive\_utils.py](https://codecov.io/gh/apache/beam/pull/25313?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdXRpbHMvaW50ZXJhY3RpdmVfdXRpbHMucHk=) | `95.12% <0.00%> (-2.44%)` | :arrow_down: |
   | [...che\_beam/runners/interactive/interactive\_runner.py](https://codecov.io/gh/apache/beam/pull/25313?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9pbnRlcmFjdGl2ZV9ydW5uZXIucHk=) | `90.50% <0.00%> (-1.27%)` | :arrow_down: |
   | [...ks/python/apache\_beam/runners/worker/sdk\_worker.py](https://codecov.io/gh/apache/beam/pull/25313?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvc2RrX3dvcmtlci5weQ==) | `89.24% <0.00%> (+0.16%)` | :arrow_up: |
   | [...ks/python/apache\_beam/runners/worker/data\_plane.py](https://codecov.io/gh/apache/beam/pull/25313?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvZGF0YV9wbGFuZS5weQ==) | `89.26% <0.00%> (+1.69%)` | :arrow_up: |
   
   :mega: We’re building smart automated test selection to slash your CI/CD build times. [Learn more](https://about.codecov.io/iterative-testing/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] tvalentyn commented on a diff in pull request #25313: Fail the pipeline when a mismatched Python version is detected.

Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn commented on code in PR #25313:
URL: https://github.com/apache/beam/pull/25313#discussion_r1098248699


##########
sdks/python/apache_beam/runners/worker/bundle_processor.py:
##########
@@ -823,6 +825,38 @@ def only_element(iterable):
   return element
 
 
+def _extract_py_version(capability):
+  return capability[len(_PY_SDK_CAPABILITY_PREFIX):].split("_sdk:")[0]
+
+
+def _verify_descriptor_created_in_a_compatible_env(process_bundle_descriptor):
+  # type: beam_fn_api_pb2.ProcessBundleDescriptor -> None
+
+  py_envs_at_submission = set()
+  for _, env in process_bundle_descriptor.environments.items():

Review Comment:
   > Rather than enumerating over all environments, just enumerate over the environments associated with the transforms of this bundle descriptor such that we can require an exact match.
   
   Made the changes, logic is simpler now. However: looks like Dataflow's runner v2 is not setting environment_id field on transforms in ProcessBundleDescriptor. Perhaps, due to an implicit assumption that there is only 1 environment per SDK harness? I also noticed that UW harness starts with a mapping of {SDK_harness_id: environment_id}. However this mapping is not visible to SDK harness. There is still a list of environments in PBD.  
    
    Works with PortableRunner.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] tvalentyn merged pull request #25313: Fail the pipeline when a mismatched Python or Beam version is detected.

Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn merged PR #25313:
URL: https://github.com/apache/beam/pull/25313


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] tvalentyn commented on a diff in pull request #25313: Fail the pipeline when a mismatched Python version is detected.

Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn commented on code in PR #25313:
URL: https://github.com/apache/beam/pull/25313#discussion_r1098263435


##########
sdks/python/apache_beam/runners/worker/bundle_processor.py:
##########
@@ -823,6 +825,38 @@ def only_element(iterable):
   return element
 
 
+def _extract_py_version(capability):
+  return capability[len(_PY_SDK_CAPABILITY_PREFIX):].split("_sdk:")[0]
+
+
+def _verify_descriptor_created_in_a_compatible_env(process_bundle_descriptor):
+  # type: beam_fn_api_pb2.ProcessBundleDescriptor -> None
+
+  py_envs_at_submission = set()
+  for _, env in process_bundle_descriptor.environments.items():

Review Comment:
   >  There is still a list of environments in PBD.
   
   ... and it includes an "empty" environment:
   
   ```
   environments {
     key: ""
     value {
     }
   }
   environments {
     key: "ref_Environment_default_environment_1"
     value {
       urn: "beam:env:docker:v1"
       payload: "\n apache/beam_python3.9_sdk:2.44.0"
       capabilities: "beam:coder:bytes:v1"
       capabilities: "beam:coder:string_utf8:v1"
       capabilities: "beam:coder:kv:v1"
       capabilities: "beam:coder:bool:v1"
       capabilities: "beam:coder:varint:v1"
       capabilities: "beam:coder:double:v1"
       capabilities: "beam:coder:iterable:v1"
       capabilities: "beam:coder:timer:v1"
       capabilities: "beam:coder:interval_window:v1"
       capabilities: "beam:coder:length_prefix:v1"
       capabilities: "beam:coder:global_window:v1"
       capabilities: "beam:coder:windowed_value:v1"
       capabilities: "beam:coder:param_windowed_value:v1"
       capabilities: "beam:coder:state_backed_iterable:v1"
       capabilities: "beam:coder:custom_window:v1"
       capabilities: "beam:coder:row:v1"
       capabilities: "beam:coder:sharded_key:v1"
       capabilities: "beam:coder:nullable:v1"
       capabilities: "beam:protocol:progress_reporting:v0"
       capabilities: "beam:protocol:harness_monitoring_infos:v1"
       capabilities: "beam:protocol:worker_status:v1"
       capabilities: "beam:combinefn:packed_python:v1"
       capabilities: "beam:version:sdk_base:apache/beam_python3.8_sdk:2.46.0.dev"
       capabilities: "beam:transform:sdf_truncate_sized_restrictions:v1"
       capabilities: "beam:transform:to_string:v1"
       capabilities: "beam:protocol:sibling_workers:v1"
       dependencies {
         type_urn: "beam:artifact:type:url:v1"
         type_payload: "\nlgs://clouddfe-valentyn/beamapp-valentyn-0207064357-172319-kxdzeic6.1675752237.172464/dataflow_python_sdk.tar"
         role_urn: "beam:artifact:role:staging_to:v1"
         role_payload: "\n\027dataflow_python_sdk.tar"
       }
       dependencies {
         type_urn: "beam:artifact:type:url:v1"
         type_payload: "\nigs://clouddfe-valentyn/beamapp-valentyn-0207064357-172319-kxdzeic6.1675752237.172464/pickled_main_session"
         role_urn: "beam:artifact:role:staging_to:v1"
         role_payload: "\n\024pickled_main_session"
       }
     }
   }
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] tvalentyn commented on a diff in pull request #25313: Fail the pipeline when a mismatched Python version is detected.

Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn commented on code in PR #25313:
URL: https://github.com/apache/beam/pull/25313#discussion_r1098263435


##########
sdks/python/apache_beam/runners/worker/bundle_processor.py:
##########
@@ -823,6 +825,38 @@ def only_element(iterable):
   return element
 
 
+def _extract_py_version(capability):
+  return capability[len(_PY_SDK_CAPABILITY_PREFIX):].split("_sdk:")[0]
+
+
+def _verify_descriptor_created_in_a_compatible_env(process_bundle_descriptor):
+  # type: beam_fn_api_pb2.ProcessBundleDescriptor -> None
+
+  py_envs_at_submission = set()
+  for _, env in process_bundle_descriptor.environments.items():

Review Comment:
   >  There is still a list of environments in PBD.
   
   ... and it includes an "empty" environment:
   
   (from [logs](https://pantheon.corp.google.com/logs/query;query=resource.type%3D%22dataflow_step%22%0Aresource.labels.job_id%3D%222023-02-06_22_44_06-7899562700680057780%22%0Aref_Environment_default_environment_1%0A;timeRange=2023-02-07T06:44:06.936Z%2F2023-02-07T06:50:27.603Z;cursorTimestamp=2023-02-07T06:48:10.892833709Z?project=google.com:clouddfe))
   
   ```
   environments {
     key: ""
     value {
     }
   }
   environments {
     key: "ref_Environment_default_environment_1"
     value {
       urn: "beam:env:docker:v1"
       payload: "\n apache/beam_python3.9_sdk:2.44.0"
       capabilities: "beam:coder:bytes:v1"
       capabilities: "beam:coder:string_utf8:v1"
       capabilities: "beam:coder:kv:v1"
       capabilities: "beam:coder:bool:v1"
       capabilities: "beam:coder:varint:v1"
       capabilities: "beam:coder:double:v1"
       capabilities: "beam:coder:iterable:v1"
       capabilities: "beam:coder:timer:v1"
       capabilities: "beam:coder:interval_window:v1"
       capabilities: "beam:coder:length_prefix:v1"
       capabilities: "beam:coder:global_window:v1"
       capabilities: "beam:coder:windowed_value:v1"
       capabilities: "beam:coder:param_windowed_value:v1"
       capabilities: "beam:coder:state_backed_iterable:v1"
       capabilities: "beam:coder:custom_window:v1"
       capabilities: "beam:coder:row:v1"
       capabilities: "beam:coder:sharded_key:v1"
       capabilities: "beam:coder:nullable:v1"
       capabilities: "beam:protocol:progress_reporting:v0"
       capabilities: "beam:protocol:harness_monitoring_infos:v1"
       capabilities: "beam:protocol:worker_status:v1"
       capabilities: "beam:combinefn:packed_python:v1"
       capabilities: "beam:version:sdk_base:apache/beam_python3.8_sdk:2.46.0.dev"
       capabilities: "beam:transform:sdf_truncate_sized_restrictions:v1"
       capabilities: "beam:transform:to_string:v1"
       capabilities: "beam:protocol:sibling_workers:v1"
       dependencies {
         type_urn: "beam:artifact:type:url:v1"
         type_payload: "\nlgs://clouddfe-valentyn/beamapp-valentyn-0207064357-172319-kxdzeic6.1675752237.172464/dataflow_python_sdk.tar"
         role_urn: "beam:artifact:role:staging_to:v1"
         role_payload: "\n\027dataflow_python_sdk.tar"
       }
       dependencies {
         type_urn: "beam:artifact:type:url:v1"
         type_payload: "\nigs://clouddfe-valentyn/beamapp-valentyn-0207064357-172319-kxdzeic6.1675752237.172464/pickled_main_session"
         role_urn: "beam:artifact:role:staging_to:v1"
         role_payload: "\n\024pickled_main_session"
       }
     }
   }
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] tvalentyn commented on a diff in pull request #25313: Fail the pipeline when a mismatched Python version is detected.

Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn commented on code in PR #25313:
URL: https://github.com/apache/beam/pull/25313#discussion_r1097790713


##########
sdks/python/apache_beam/runners/worker/bundle_processor.py:
##########
@@ -823,6 +825,38 @@ def only_element(iterable):
   return element
 
 
+def _extract_py_version(capability):
+  return capability[len(_PY_SDK_CAPABILITY_PREFIX):].split("_sdk:")[0]
+
+
+def _verify_descriptor_created_in_a_compatible_env(process_bundle_descriptor):
+  # type: beam_fn_api_pb2.ProcessBundleDescriptor -> None
+
+  py_envs_at_submission = set()
+  for _, env in process_bundle_descriptor.environments.items():

Review Comment:
   thanks for feedback. i am also thinking to stop checking as soon as one environment successfully matched, and also considering doing the matches at most once in process lifetime. let me know if you think it's over-optimizing.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] robertwb commented on a diff in pull request #25313: Fail the pipeline when a mismatched Python version is detected.

Posted by "robertwb (via GitHub)" <gi...@apache.org>.
robertwb commented on code in PR #25313:
URL: https://github.com/apache/beam/pull/25313#discussion_r1097866767


##########
sdks/python/apache_beam/runners/worker/bundle_processor.py:
##########
@@ -823,6 +825,38 @@ def only_element(iterable):
   return element
 
 
+def _extract_py_version(capability):
+  return capability[len(_PY_SDK_CAPABILITY_PREFIX):].split("_sdk:")[0]
+
+
+def _verify_descriptor_created_in_a_compatible_env(process_bundle_descriptor):
+  # type: beam_fn_api_pb2.ProcessBundleDescriptor -> None
+
+  py_envs_at_submission = set()
+  for _, env in process_bundle_descriptor.environments.items():

Review Comment:
   > thanks for feedback. i am also thinking to stop checking as soon as one environment successfully matched, 
   
   We should require that all the environments (referenced in this bundles transforms) match. 
   
   > and also considering doing the matches at most once in process lifetime. let me know if you think it's over-optimizing.
   
   We need to repeatedly raise the error to get the pipeline to fail. This happens once per (already expensive and rare) bundle processor initialization, so I wouldn't worry about trying to limit it to once per process (though a stronger case could be made about warnings about package version mismatch).



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] tvalentyn commented on a diff in pull request #25313: Fail the pipeline when a mismatched Python version is detected.

Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn commented on code in PR #25313:
URL: https://github.com/apache/beam/pull/25313#discussion_r1097790713


##########
sdks/python/apache_beam/runners/worker/bundle_processor.py:
##########
@@ -823,6 +825,38 @@ def only_element(iterable):
   return element
 
 
+def _extract_py_version(capability):
+  return capability[len(_PY_SDK_CAPABILITY_PREFIX):].split("_sdk:")[0]
+
+
+def _verify_descriptor_created_in_a_compatible_env(process_bundle_descriptor):
+  # type: beam_fn_api_pb2.ProcessBundleDescriptor -> None
+
+  py_envs_at_submission = set()
+  for _, env in process_bundle_descriptor.environments.items():

Review Comment:
   thanks for feedback. i am also thinking to stop checking as soon as one environment successfully matched, and also considering doing the matches at most once per process. let me know if you think it's over-optimizing.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] robertwb commented on a diff in pull request #25313: Fail the pipeline when a mismatched Python version is detected.

Posted by "robertwb (via GitHub)" <gi...@apache.org>.
robertwb commented on code in PR #25313:
URL: https://github.com/apache/beam/pull/25313#discussion_r1099239277


##########
sdks/python/apache_beam/runners/worker/bundle_processor.py:
##########
@@ -823,6 +824,27 @@ def only_element(iterable):
   return element
 
 
+def _verify_descriptor_created_in_a_compatible_env(process_bundle_descriptor):
+  # type: (beam_fn_api_pb2.ProcessBundleDescriptor) -> None
+
+  runtime_sdk = environments.sdk_base_version_capability()
+  for _, t in process_bundle_descriptor.transforms.items():

Review Comment:
   You can also do `for t in process_bundle_descriptor.transforms.values()`



##########
sdks/python/apache_beam/runners/worker/bundle_processor.py:
##########
@@ -823,6 +825,38 @@ def only_element(iterable):
   return element
 
 
+def _extract_py_version(capability):
+  return capability[len(_PY_SDK_CAPABILITY_PREFIX):].split("_sdk:")[0]
+
+
+def _verify_descriptor_created_in_a_compatible_env(process_bundle_descriptor):
+  # type: beam_fn_api_pb2.ProcessBundleDescriptor -> None
+
+  py_envs_at_submission = set()
+  for _, env in process_bundle_descriptor.environments.items():

Review Comment:
   Interesting. I can take a look on the UW side. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] robertwb commented on a diff in pull request #25313: Fail the pipeline when a mismatched Python version is detected.

Posted by "robertwb (via GitHub)" <gi...@apache.org>.
robertwb commented on code in PR #25313:
URL: https://github.com/apache/beam/pull/25313#discussion_r1097760889


##########
sdks/python/apache_beam/runners/worker/bundle_processor.py:
##########
@@ -823,6 +825,38 @@ def only_element(iterable):
   return element
 
 
+def _extract_py_version(capability):
+  return capability[len(_PY_SDK_CAPABILITY_PREFIX):].split("_sdk:")[0]
+
+
+def _verify_descriptor_created_in_a_compatible_env(process_bundle_descriptor):
+  # type: beam_fn_api_pb2.ProcessBundleDescriptor -> None
+
+  py_envs_at_submission = set()
+  for _, env in process_bundle_descriptor.environments.items():

Review Comment:
   Rather than enumerating over all environments, just enumerate over the environments associated with the transforms of this bundle descriptor such that we can require an exact match. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] tvalentyn commented on a diff in pull request #25313: Fail the pipeline when a mismatched Python version is detected.

Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn commented on code in PR #25313:
URL: https://github.com/apache/beam/pull/25313#discussion_r1099439239


##########
sdks/python/apache_beam/runners/worker/bundle_processor.py:
##########
@@ -823,6 +824,27 @@ def only_element(iterable):
   return element
 
 
+def _verify_descriptor_created_in_a_compatible_env(process_bundle_descriptor):
+  # type: (beam_fn_api_pb2.ProcessBundleDescriptor) -> None
+
+  runtime_sdk = environments.sdk_base_version_capability()
+  for _, t in process_bundle_descriptor.transforms.items():

Review Comment:
   fixed, thanks.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] github-actions[bot] commented on pull request #25313: Fail the pipeline when a mismatched Python version is detected.

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #25313:
URL: https://github.com/apache/beam/pull/25313#issuecomment-1416668251

   Assigning reviewers. If you would like to opt out of this review, comment `assign to next reviewer`:
   
   R: @pabloem for label python.
   
   Available commands:
   - `stop reviewer notifications` - opt out of the automated review tooling
   - `remind me after tests pass` - tag the comment author after tests pass
   - `waiting on author` - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)
   
   The PR bot will only process comments in the main thread (not review comments).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] tvalentyn commented on pull request #25313: Fail the pipeline when a mismatched Python version is detected.

Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn commented on PR #25313:
URL: https://github.com/apache/beam/pull/25313#issuecomment-1421032446

   Run Python_Transforms PreCommit


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org