You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "riteshghorse (via GitHub)" <gi...@apache.org> on 2023/09/20 17:50:46 UTC
[GitHub] [beam] riteshghorse opened a new pull request, #28564: [Python] Log dependencies at runtime and at submission environment
riteshghorse opened a new pull request, #28564:
URL: https://github.com/apache/beam/pull/28564
Saves the submission environment dependencies and stage it. Logs it along with the runtime dependencies.
Fixes #28563
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [ ] Mention the appropriate issue in your description (for example: `addresses #123`), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment `fixes #<ISSUE NUMBER>` instead.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://github.com/apache/beam/blob/master/CONTRIBUTING.md#make-the-reviewers-job-easier).
To check the build health, please visit [https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md](https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md)
GitHub Actions Tests Status (on master branch)
------------------------------------------------------------------------------------------------
[](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Go+tests%22+branch%3Amaster+event%3Aschedule)
See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI or the [workflows README](https://github.com/apache/beam/blob/master/.github/workflows/README.md) to see a list of phrases to trigger workflows.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Python] Log dependencies at runtime and at submission environment [beam]
Posted by "riteshghorse (via GitHub)" <gi...@apache.org>.
riteshghorse commented on code in PR #28564:
URL: https://github.com/apache/beam/pull/28564#discussion_r1344557204
##########
sdks/python/apache_beam/transforms/environments.py:
##########
@@ -124,9 +124,9 @@ def __init__(self,
dict(resource_hints) if resource_hints else {})
def __eq__(self, other):
+ # don't compare artifacts since they have different hashes in their names.
return (
- self.__class__ == other.__class__ and
- self._artifacts == other._artifacts
Review Comment:
I agree that if two environment have different dependencies then they are not equal. But unit tests where we compare the two environments will fail with the addition of a new staging file. Even though this file is same (name and content), it has a a different hash in name:
```
first: [type_urn: "beam:artifact:type:file:v1"
type_payload: "\nd/var/folders/kk/jk392txd7x14sg93r8sjnzkh00tp20/T/tmpmxw3v4d9/submission_environment_dependencies.txt"
role_urn: "beam:artifact:role:staging_to:v1"
role_payload: "\n\'submission_environment_dependencies.txt"
]
second: [type_urn: "beam:artifact:type:file:v1"
type_payload: "\nd/var/folders/kk/jk392txd7x14sg93r8sjnzkh00tp20/T/tmpni8k_038/submission_environment_dependencies.txt"
role_urn: "beam:artifact:role:staging_to:v1"
role_payload: "\n\'submission_environment_dependencies.txt"
]
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Python] Log dependencies at runtime and at submission environment [beam]
Posted by "riteshghorse (via GitHub)" <gi...@apache.org>.
riteshghorse commented on code in PR #28564:
URL: https://github.com/apache/beam/pull/28564#discussion_r1344636199
##########
sdks/python/apache_beam/transforms/environments.py:
##########
@@ -124,9 +124,9 @@ def __init__(self,
dict(resource_hints) if resource_hints else {})
def __eq__(self, other):
+ # don't compare artifacts since they have different hashes in their names.
return (
- self.__class__ == other.__class__ and
- self._artifacts == other._artifacts
Review Comment:
> Do you have a stack trace for the failure ?
>
> The way it works now is that we stage all artifacts at job submission and each Environment carries the correct set of dependencies for that particular environment.
>
> At runtime, the runner (UW for Dataflow) would look up the correct set of dependencies from the Runner API proto an serve them to the corresponding SDKs.
Stack trace for `ExternalTransformTest` failure:
```
apache_beam/transforms/external_test.py::ExternalTransformTest::test_pipeline_generation FAILED [100%]
============================================================================================== FAILURES ==============================================================================================
___________________________________________________________________________ ExternalTransformTest.test_pipeline_generation ___________________________________________________________________________
self = <apache_beam.transforms.external_test.ExternalTransformTest testMethod=test_pipeline_generation>
def test_pipeline_generation(self):
pipeline = beam.Pipeline()
_ = (
> pipeline
| beam.Create(['a', 'b'])
| beam.ExternalTransform(
'beam:transforms:xlang:test:prefix',
ImplicitSchemaPayloadBuilder({'data': '0'}),
expansion_service.ExpansionServiceServicer()))
apache_beam/transforms/external_test.py:181:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
apache_beam/pvalue.py:137: in __or__
return self.pipeline.apply(ptransform, self)
apache_beam/pipeline.py:712: in apply
pvalueish_result = self.runner.apply(transform, pvalueish, self._options)
apache_beam/runners/runner.py:203: in apply
return self.apply_PTransform(transform, input, options)
apache_beam/runners/runner.py:207: in apply_PTransform
return transform.expand(input)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = <ExternalTransform(PTransform) label=[ExternalTransform(beam:transforms:xlang:test:prefix)] at 0x11feb4d90>, pvalueish = <PCollection[Create/Map(decode).None] at 0x11feb4700>
def expand(self, pvalueish):
# type: (pvalue.PCollection) -> pvalue.PCollection
if isinstance(pvalueish, pvalue.PBegin):
self._inputs = {}
elif isinstance(pvalueish, (list, tuple)):
self._inputs = {str(ix): pvalue for ix, pvalue in enumerate(pvalueish)}
elif isinstance(pvalueish, dict):
self._inputs = pvalueish
else:
self._inputs = {'input': pvalueish}
pipeline = (
next(iter(self._inputs.values())).pipeline
if self._inputs else pvalueish.pipeline)
context = pipeline_context.PipelineContext(
component_id_map=pipeline.component_id_map)
transform_proto = beam_runner_api_pb2.PTransform(
unique_name=pipeline._current_transform().full_label,
spec=beam_runner_api_pb2.FunctionSpec(
urn=self._urn, payload=self._payload))
for tag, pcoll in self._inputs.items():
transform_proto.inputs[tag] = context.pcollections.get_id(pcoll)
# Conversion to/from proto assumes producers.
# TODO: Possibly loosen this.
context.transforms.put_proto(
'%s_%s' % (self._IMPULSE_PREFIX, tag),
beam_runner_api_pb2.PTransform(
unique_name='%s_%s' % (self._IMPULSE_PREFIX, tag),
spec=beam_runner_api_pb2.FunctionSpec(
urn=common_urns.primitives.IMPULSE.urn),
outputs={'out': transform_proto.inputs[tag]}))
output_coders = None
if self._type_hints.output_types:
if self._type_hints.output_types[0]:
output_coders = dict(
(str(k), context.coder_id_from_element_type(v))
for (k, v) in enumerate(self._type_hints.output_types[0]))
elif self._type_hints.output_types[1]:
output_coders = {
k: context.coder_id_from_element_type(v)
for (k, v) in self._type_hints.output_types[1].items()
}
components = context.to_runner_api()
request = beam_expansion_api_pb2.ExpansionRequest(
components=components,
namespace=self._external_namespace,
transform=transform_proto,
output_coder_requests=output_coders)
expansion_service = _maybe_use_transform_service(
self._expansion_service, pipeline.options)
with ExternalTransform.service(expansion_service) as service:
response = service.Expand(request)
if response.error:
raise RuntimeError(response.error)
self._expanded_components = response.components
if any(env.dependencies
for env in self._expanded_components.environments.values()):
self._expanded_components = self._resolve_artifacts(
self._expanded_components,
> service.artifact_service(),
pipeline.local_tempdir)
E AttributeError: 'ExpansionServiceServicer' object has no attribute 'artifact_service'
apache_beam/transforms/external.py:736: AttributeError
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Python] Log dependencies at runtime and at submission environment [beam]
Posted by "riteshghorse (via GitHub)" <gi...@apache.org>.
riteshghorse commented on PR #28564:
URL: https://github.com/apache/beam/pull/28564#issuecomment-1850163808
> Hey @riteshghorse , could we extract the commits that log the runtime dependencies and merge those before next release cut while submission dependencies portion is being sorted out? Thanks!
Sounds good
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Python] Log dependencies at runtime and at submission environment [beam]
Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn commented on code in PR #28564:
URL: https://github.com/apache/beam/pull/28564#discussion_r1344626645
##########
sdks/python/apache_beam/transforms/environments.py:
##########
@@ -124,9 +124,9 @@ def __init__(self,
dict(resource_hints) if resource_hints else {})
def __eq__(self, other):
+ # don't compare artifacts since they have different hashes in their names.
return (
- self.__class__ == other.__class__ and
- self._artifacts == other._artifacts
Review Comment:
that is a bizzare behavior. we should work on fixing comparison. I am not sure what is 'type_payload', but as far as file payloads are concerned, i think it's reasonably safe to compare file hashes + destination names; alternatively we can look and see why the file path is different and if we should fix that instead. without any additional context and digging further, first option sounds more promising, second might be simpler but hackier and result in some other tech debt.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Python] Log dependencies at runtime and at submission environment [beam]
Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn commented on code in PR #28564:
URL: https://github.com/apache/beam/pull/28564#discussion_r1344626645
##########
sdks/python/apache_beam/transforms/environments.py:
##########
@@ -124,9 +124,9 @@ def __init__(self,
dict(resource_hints) if resource_hints else {})
def __eq__(self, other):
+ # don't compare artifacts since they have different hashes in their names.
return (
- self.__class__ == other.__class__ and
- self._artifacts == other._artifacts
Review Comment:
that is a bizzare behavior. we should work on fixing comparison. I am not sure what is 'type_payload', but as far as file payloads are concerned, i think it's reasonably safe to compare file hashes (sha) + destination names; alternatively we can look and see why the file path is different and if we should fix that instead. without any additional context and digging further, first option sounds more promising, second might be simpler but hackier and result in some other tech debt.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Python] Log dependencies installed in submission environment [beam]
Posted by "riteshghorse (via GitHub)" <gi...@apache.org>.
riteshghorse commented on code in PR #28564:
URL: https://github.com/apache/beam/pull/28564#discussion_r1484380995
##########
sdks/python/apache_beam/transforms/environments_test.py:
##########
@@ -21,20 +21,40 @@
# pytype: skip-file
import logging
+import tempfile
import unittest
from apache_beam.options.pipeline_options import PortableOptions
+from apache_beam.options.pipeline_options import SetupOptions
from apache_beam.portability import common_urns
from apache_beam.runners import pipeline_context
+from apache_beam.runners.portability import stager
from apache_beam.transforms import environments
from apache_beam.transforms.environments import DockerEnvironment
from apache_beam.transforms.environments import EmbeddedPythonEnvironment
from apache_beam.transforms.environments import EmbeddedPythonGrpcEnvironment
from apache_beam.transforms.environments import Environment
from apache_beam.transforms.environments import ExternalEnvironment
from apache_beam.transforms.environments import ProcessEnvironment
+from apache_beam.transforms.environments import PyPIArtifactRegistry
from apache_beam.transforms.environments import SubprocessSDKEnvironment
+# create a temp directory so that all artifacts are put in the same directory.
+tmp_dir = tempfile.mkdtemp()
Review Comment:
sounds good
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] tvalentyn commented on a diff in pull request #28564: [Python] Log dependencies at runtime and at submission environment
Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn commented on code in PR #28564:
URL: https://github.com/apache/beam/pull/28564#discussion_r1339020668
##########
sdks/python/apache_beam/runners/portability/stager.py:
##########
@@ -837,3 +846,20 @@ def _create_beam_sdk(sdk_remote_location, temp_dir):
return [
Stager._create_file_stage_to_artifact(local_download_file, staged_name)
]
+
+ @staticmethod
+ def _create_stage_submission_env_dependencies(temp_dir):
Review Comment:
Let's add a comment here in case someone is wondering why we do this staging. The purpose is to compare the submission and runtime dependencies at runtime: make it possible for runners to warn users about possible mismatches and help debug issues related to environment mismatches .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] riteshghorse commented on a diff in pull request #28564: [Python] Log dependencies at runtime and at submission environment
Posted by "riteshghorse (via GitHub)" <gi...@apache.org>.
riteshghorse commented on code in PR #28564:
URL: https://github.com/apache/beam/pull/28564#discussion_r1340627982
##########
sdks/python/apache_beam/transforms/environments.py:
##########
@@ -124,9 +124,9 @@ def __init__(self,
dict(resource_hints) if resource_hints else {})
def __eq__(self, other):
+ # don't compare artifacts since they have different hashes in their names.
return (
- self.__class__ == other.__class__ and
- self._artifacts == other._artifacts
Review Comment:
The `Stager.create_job_resources` is called from `environments.python_sdk_dependencies` which gets call when instantiating the environments. Other than this it is called in `sdk_container_builder`. I tried a run with `log_submission_env_dependencies=False` from environments and it didn't stage the file. Apparently, it has to be a part of it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] chamikaramj commented on a diff in pull request #28564: [Python] Log dependencies at runtime and at submission environment
Posted by "chamikaramj (via GitHub)" <gi...@apache.org>.
chamikaramj commented on code in PR #28564:
URL: https://github.com/apache/beam/pull/28564#discussion_r1340666197
##########
sdks/python/apache_beam/transforms/environments.py:
##########
@@ -124,9 +124,9 @@ def __init__(self,
dict(resource_hints) if resource_hints else {})
def __eq__(self, other):
+ # don't compare artifacts since they have different hashes in their names.
return (
- self.__class__ == other.__class__ and
- self._artifacts == other._artifacts
Review Comment:
Do you have a stack trace for the failure ?
The way it works now is that we stage all artifacts at job submission and each Environment carries the correct set of dependencies for that particular environment.
At runtime, the runner (UW for Dataflow) would look up the correct set of dependencies from the Runner API proto an serve them to the corresponding SDKs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Python] Log dependencies at runtime and at submission environment [beam]
Posted by "riteshghorse (via GitHub)" <gi...@apache.org>.
riteshghorse commented on code in PR #28564:
URL: https://github.com/apache/beam/pull/28564#discussion_r1344636199
##########
sdks/python/apache_beam/transforms/environments.py:
##########
@@ -124,9 +124,9 @@ def __init__(self,
dict(resource_hints) if resource_hints else {})
def __eq__(self, other):
+ # don't compare artifacts since they have different hashes in their names.
return (
- self.__class__ == other.__class__ and
- self._artifacts == other._artifacts
Review Comment:
> Do you have a stack trace for the failure ?
>
> The way it works now is that we stage all artifacts at job submission and each Environment carries the correct set of dependencies for that particular environment.
>
> At runtime, the runner (UW for Dataflow) would look up the correct set of dependencies from the Runner API proto an serve them to the corresponding SDKs.
@chamikaramj Stack trace for `ExternalTransformTest` failure (sorry for late followup here):
```
apache_beam/transforms/external_test.py::ExternalTransformTest::test_pipeline_generation FAILED [100%]
============================================================================================== FAILURES ==============================================================================================
___________________________________________________________________________ ExternalTransformTest.test_pipeline_generation ___________________________________________________________________________
self = <apache_beam.transforms.external_test.ExternalTransformTest testMethod=test_pipeline_generation>
def test_pipeline_generation(self):
pipeline = beam.Pipeline()
_ = (
> pipeline
| beam.Create(['a', 'b'])
| beam.ExternalTransform(
'beam:transforms:xlang:test:prefix',
ImplicitSchemaPayloadBuilder({'data': '0'}),
expansion_service.ExpansionServiceServicer()))
apache_beam/transforms/external_test.py:181:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
apache_beam/pvalue.py:137: in __or__
return self.pipeline.apply(ptransform, self)
apache_beam/pipeline.py:712: in apply
pvalueish_result = self.runner.apply(transform, pvalueish, self._options)
apache_beam/runners/runner.py:203: in apply
return self.apply_PTransform(transform, input, options)
apache_beam/runners/runner.py:207: in apply_PTransform
return transform.expand(input)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = <ExternalTransform(PTransform) label=[ExternalTransform(beam:transforms:xlang:test:prefix)] at 0x11feb4d90>, pvalueish = <PCollection[Create/Map(decode).None] at 0x11feb4700>
def expand(self, pvalueish):
# type: (pvalue.PCollection) -> pvalue.PCollection
if isinstance(pvalueish, pvalue.PBegin):
self._inputs = {}
elif isinstance(pvalueish, (list, tuple)):
self._inputs = {str(ix): pvalue for ix, pvalue in enumerate(pvalueish)}
elif isinstance(pvalueish, dict):
self._inputs = pvalueish
else:
self._inputs = {'input': pvalueish}
pipeline = (
next(iter(self._inputs.values())).pipeline
if self._inputs else pvalueish.pipeline)
context = pipeline_context.PipelineContext(
component_id_map=pipeline.component_id_map)
transform_proto = beam_runner_api_pb2.PTransform(
unique_name=pipeline._current_transform().full_label,
spec=beam_runner_api_pb2.FunctionSpec(
urn=self._urn, payload=self._payload))
for tag, pcoll in self._inputs.items():
transform_proto.inputs[tag] = context.pcollections.get_id(pcoll)
# Conversion to/from proto assumes producers.
# TODO: Possibly loosen this.
context.transforms.put_proto(
'%s_%s' % (self._IMPULSE_PREFIX, tag),
beam_runner_api_pb2.PTransform(
unique_name='%s_%s' % (self._IMPULSE_PREFIX, tag),
spec=beam_runner_api_pb2.FunctionSpec(
urn=common_urns.primitives.IMPULSE.urn),
outputs={'out': transform_proto.inputs[tag]}))
output_coders = None
if self._type_hints.output_types:
if self._type_hints.output_types[0]:
output_coders = dict(
(str(k), context.coder_id_from_element_type(v))
for (k, v) in enumerate(self._type_hints.output_types[0]))
elif self._type_hints.output_types[1]:
output_coders = {
k: context.coder_id_from_element_type(v)
for (k, v) in self._type_hints.output_types[1].items()
}
components = context.to_runner_api()
request = beam_expansion_api_pb2.ExpansionRequest(
components=components,
namespace=self._external_namespace,
transform=transform_proto,
output_coder_requests=output_coders)
expansion_service = _maybe_use_transform_service(
self._expansion_service, pipeline.options)
with ExternalTransform.service(expansion_service) as service:
response = service.Expand(request)
if response.error:
raise RuntimeError(response.error)
self._expanded_components = response.components
if any(env.dependencies
for env in self._expanded_components.environments.values()):
self._expanded_components = self._resolve_artifacts(
self._expanded_components,
> service.artifact_service(),
pipeline.local_tempdir)
E AttributeError: 'ExpansionServiceServicer' object has no attribute 'artifact_service'
apache_beam/transforms/external.py:736: AttributeError
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Python] Log dependencies at runtime and at submission environment [beam]
Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn commented on code in PR #28564:
URL: https://github.com/apache/beam/pull/28564#discussion_r1344626645
##########
sdks/python/apache_beam/transforms/environments.py:
##########
@@ -124,9 +124,9 @@ def __init__(self,
dict(resource_hints) if resource_hints else {})
def __eq__(self, other):
+ # don't compare artifacts since they have different hashes in their names.
return (
- self.__class__ == other.__class__ and
- self._artifacts == other._artifacts
Review Comment:
that is a bizzare behavior. we should work on fixing comparison. I am not sure what is 'type_payload', but as far as file payloads are concerned, i think it's reasonably safe to compare file hashes; alternatively we can look and see why the file path is different and if we should fix that instead. without any additional context and digging further, first options sounds more promising.
##########
sdks/python/apache_beam/transforms/environments.py:
##########
@@ -124,9 +124,9 @@ def __init__(self,
dict(resource_hints) if resource_hints else {})
def __eq__(self, other):
+ # don't compare artifacts since they have different hashes in their names.
return (
- self.__class__ == other.__class__ and
- self._artifacts == other._artifacts
Review Comment:
that is a bizzare behavior. we should work on fixing comparison. I am not sure what is 'type_payload', but as far as file payloads are concerned, i think it's reasonably safe to compare file hashes; alternatively we can look and see why the file path is different and if we should fix that instead. without any additional context and digging further, first option sounds more promising.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Python] Log dependencies installed in submission environment [beam]
Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn commented on code in PR #28564:
URL: https://github.com/apache/beam/pull/28564#discussion_r1483765959
##########
sdks/python/apache_beam/transforms/environments_test.py:
##########
@@ -21,20 +21,40 @@
# pytype: skip-file
import logging
+import tempfile
import unittest
from apache_beam.options.pipeline_options import PortableOptions
+from apache_beam.options.pipeline_options import SetupOptions
from apache_beam.portability import common_urns
from apache_beam.runners import pipeline_context
+from apache_beam.runners.portability import stager
from apache_beam.transforms import environments
from apache_beam.transforms.environments import DockerEnvironment
from apache_beam.transforms.environments import EmbeddedPythonEnvironment
from apache_beam.transforms.environments import EmbeddedPythonGrpcEnvironment
from apache_beam.transforms.environments import Environment
from apache_beam.transforms.environments import ExternalEnvironment
from apache_beam.transforms.environments import ProcessEnvironment
+from apache_beam.transforms.environments import PyPIArtifactRegistry
from apache_beam.transforms.environments import SubprocessSDKEnvironment
+# create a temp directory so that all artifacts are put in the same directory.
+tmp_dir = tempfile.mkdtemp()
+
+
+def mock_python_sdk_dependencies(options, tmp_dir=tmp_dir):
Review Comment:
Would mocking tempfile.mkdtemp() be sufficient? In this case we avoid maintaining a copy of a larger fuction.
(optionally), see: https://docs.python.org/3/library/unittest.mock.html - it provides decorators that allows applying and reverting mocks.
##########
sdks/python/apache_beam/transforms/environments_test.py:
##########
@@ -21,20 +21,40 @@
# pytype: skip-file
import logging
+import tempfile
import unittest
from apache_beam.options.pipeline_options import PortableOptions
+from apache_beam.options.pipeline_options import SetupOptions
from apache_beam.portability import common_urns
from apache_beam.runners import pipeline_context
+from apache_beam.runners.portability import stager
from apache_beam.transforms import environments
from apache_beam.transforms.environments import DockerEnvironment
from apache_beam.transforms.environments import EmbeddedPythonEnvironment
from apache_beam.transforms.environments import EmbeddedPythonGrpcEnvironment
from apache_beam.transforms.environments import Environment
from apache_beam.transforms.environments import ExternalEnvironment
from apache_beam.transforms.environments import ProcessEnvironment
+from apache_beam.transforms.environments import PyPIArtifactRegistry
from apache_beam.transforms.environments import SubprocessSDKEnvironment
+# create a temp directory so that all artifacts are put in the same directory.
+tmp_dir = tempfile.mkdtemp()
Review Comment:
I'd move this inside some TestClass.setUp(). Also consider `class tempfile.TemporaryDirectory` instead - then it will also clean up after the test.
##########
sdks/python/apache_beam/runners/dataflow/dataflow_runner_test.py:
##########
@@ -211,15 +211,13 @@ def test_environment_override_translation_legacy_worker_harness_image(self):
p | ptransform.Create([1, 2, 3])
| 'Do' >> ptransform.FlatMap(lambda x: [(x, x)])
| ptransform.GroupByKey())
- self.assertEqual(
- list(remote_runner.proto_pipeline.components.environments.values()),
- [
- beam_runner_api_pb2.Environment(
- urn=common_urns.environments.DOCKER.urn,
- payload=beam_runner_api_pb2.DockerPayload(
- container_image='LEGACY').SerializeToString(),
- capabilities=environments.python_sdk_docker_capabilities())
- ])
+ first = remote_runner.proto_pipeline.components.environments.values()
+ second = beam_runner_api_pb2.Environment(
+ urn=common_urns.environments.DOCKER.urn,
+ payload=beam_runner_api_pb2.DockerPayload(
+ container_image='LEGACY').SerializeToString(),
+ capabilities=environments.python_sdk_docker_capabilities())
+ self.assertTrue(first.__eq__(second))
Review Comment:
do we still need to make this change? same question for line 232 below
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Python] Log dependencies at runtime and at submission environment [beam]
Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn commented on code in PR #28564:
URL: https://github.com/apache/beam/pull/28564#discussion_r1340713624
##########
sdks/python/apache_beam/runners/portability/stager.py:
##########
@@ -84,6 +85,8 @@
WORKFLOW_TARBALL_FILE = 'workflow.tar.gz'
REQUIREMENTS_FILE = 'requirements.txt'
EXTRA_PACKAGES_FILE = 'extra_packages.txt'
+# Filename that stores the submission environment dependencies.
+SUBMISSION_ENV_DEPENDENCIES_FILE = 'submission_environment_dependencies.txt'
Review Comment:
There may be conflation of what we mean by environment.
The intent is to capture the content of environment that creates the translates job graph from Beam Graph to RunnerAPI proto. In a pure-python pipeline, it is the the envirionment of Python interpreter that calls `pipeline.run()`. In xlang, it is where transform expansion happens. Good point that there might be more than 1 environment involve (if someone had a XLang transform with two python environments).
On the execution side, we should make `pip freeze` content available to respective runtime environment that will run transform code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Python] Log dependencies at runtime and at submission environment [beam]
Posted by "riteshghorse (via GitHub)" <gi...@apache.org>.
riteshghorse commented on code in PR #28564:
URL: https://github.com/apache/beam/pull/28564#discussion_r1344557204
##########
sdks/python/apache_beam/transforms/environments.py:
##########
@@ -124,9 +124,9 @@ def __init__(self,
dict(resource_hints) if resource_hints else {})
def __eq__(self, other):
+ # don't compare artifacts since they have different hashes in their names.
return (
- self.__class__ == other.__class__ and
- self._artifacts == other._artifacts
Review Comment:
I agree that if two environment have different dependencies then they are not equal. But unit tests where we compare the two environments will fail (For example: [EnvironmentOptionsTest::test_environments_with_same_hints_are_equal](https://github.com/apache/beam/blob/12b071402035a0404c4f6e4a83124b3839b5de40/sdks/python/apache_beam/transforms/environments_test.py#L121)) with the addition of a new staging file. Even though this file is same, it has a a different hash in name:
```
first: [type_urn: "beam:artifact:type:file:v1"
type_payload: "\nd/var/folders/kk/jk392txd7x14sg93r8sjnzkh00tp20/T/tmpmxw3v4d9/submission_environment_dependencies.txt"
role_urn: "beam:artifact:role:staging_to:v1"
role_payload: "\n\'submission_environment_dependencies.txt"
]
second: [type_urn: "beam:artifact:type:file:v1"
type_payload: "\nd/var/folders/kk/jk392txd7x14sg93r8sjnzkh00tp20/T/tmpni8k_038/submission_environment_dependencies.txt"
role_urn: "beam:artifact:role:staging_to:v1"
role_payload: "\n\'submission_environment_dependencies.txt"
]
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] riteshghorse commented on a diff in pull request #28564: [Python] Log dependencies at runtime and at submission environment
Posted by "riteshghorse (via GitHub)" <gi...@apache.org>.
riteshghorse commented on code in PR #28564:
URL: https://github.com/apache/beam/pull/28564#discussion_r1334462772
##########
sdks/python/apache_beam/runners/portability/stager.py:
##########
@@ -62,6 +62,7 @@
from urllib.parse import urlparse
from packaging import version
+from pip._internal.operations import freeze
Review Comment:
sounds good, did with cmd pip
##########
sdks/python/apache_beam/runners/portability/stager.py:
##########
@@ -84,6 +85,8 @@
WORKFLOW_TARBALL_FILE = 'workflow.tar.gz'
REQUIREMENTS_FILE = 'requirements.txt'
EXTRA_PACKAGES_FILE = 'extra_packages.txt'
+# Filename that stores the submission environment dependencies.
+SUBMISSION_ENV_DEPENDENCIES_FILENAME = 'submission_environment_dependencies.txt'
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Python] Log dependencies installed in submission environment [beam]
Posted by "riteshghorse (via GitHub)" <gi...@apache.org>.
riteshghorse commented on code in PR #28564:
URL: https://github.com/apache/beam/pull/28564#discussion_r1491673729
##########
sdks/python/apache_beam/runners/dataflow/dataflow_runner_test.py:
##########
@@ -16,10 +16,13 @@
#
"""Unit tests for the DataflowRunner class."""
-
# pytype: skip-file
+import os
+
+import tempfile
import unittest
+from unittest.mock import MagicMock
Review Comment:
```suggestion
import tempfile
import unittest
from unittest.mock import MagicMock
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] riteshghorse commented on pull request #28564: [Python] Log dependencies at runtime and at submission environment
Posted by "riteshghorse (via GitHub)" <gi...@apache.org>.
riteshghorse commented on PR #28564:
URL: https://github.com/apache/beam/pull/28564#issuecomment-1729753729
Some unit tests are failing because there is additional staging file now which will always be present. Got the solution. I'll update the PR. Defer review until then.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] chamikaramj commented on a diff in pull request #28564: [Python] Log dependencies at runtime and at submission environment
Posted by "chamikaramj (via GitHub)" <gi...@apache.org>.
chamikaramj commented on code in PR #28564:
URL: https://github.com/apache/beam/pull/28564#discussion_r1340668395
##########
sdks/python/apache_beam/runners/portability/stager.py:
##########
@@ -84,6 +85,8 @@
WORKFLOW_TARBALL_FILE = 'workflow.tar.gz'
REQUIREMENTS_FILE = 'requirements.txt'
EXTRA_PACKAGES_FILE = 'extra_packages.txt'
+# Filename that stores the submission environment dependencies.
+SUBMISSION_ENV_DEPENDENCIES_FILE = 'submission_environment_dependencies.txt'
Review Comment:
Why do we special case submission environment. Should we document/log dependencies of every environment that is used in a pipeline ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Python] Log dependencies installed in submission environment [beam]
Posted by "riteshghorse (via GitHub)" <gi...@apache.org>.
riteshghorse commented on PR #28564:
URL: https://github.com/apache/beam/pull/28564#issuecomment-1927705837
R: @tvalentyn
Barring the lint failure, all tests pass.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Python] Log dependencies installed in submission environment [beam]
Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn commented on code in PR #28564:
URL: https://github.com/apache/beam/pull/28564#discussion_r1484578197
##########
sdks/python/apache_beam/runners/dataflow/dataflow_runner_test.py:
##########
@@ -211,15 +211,13 @@ def test_environment_override_translation_legacy_worker_harness_image(self):
p | ptransform.Create([1, 2, 3])
| 'Do' >> ptransform.FlatMap(lambda x: [(x, x)])
| ptransform.GroupByKey())
- self.assertEqual(
- list(remote_runner.proto_pipeline.components.environments.values()),
- [
- beam_runner_api_pb2.Environment(
- urn=common_urns.environments.DOCKER.urn,
- payload=beam_runner_api_pb2.DockerPayload(
- container_image='LEGACY').SerializeToString(),
- capabilities=environments.python_sdk_docker_capabilities())
- ])
+ first = remote_runner.proto_pipeline.components.environments.values()
+ second = beam_runner_api_pb2.Environment(
+ urn=common_urns.environments.DOCKER.urn,
+ payload=beam_runner_api_pb2.DockerPayload(
+ container_image='LEGACY').SerializeToString(),
+ capabilities=environments.python_sdk_docker_capabilities())
+ self.assertTrue(first.__eq__(second))
Review Comment:
strange, have you seen what's the diff? New condition looks as though we compare a list of environments (first), with a single environment (second)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Python] Log dependencies installed in submission environment [beam]
Posted by "riteshghorse (via GitHub)" <gi...@apache.org>.
riteshghorse commented on code in PR #28564:
URL: https://github.com/apache/beam/pull/28564#discussion_r1516624016
##########
sdks/python/apache_beam/runners/portability/stager.py:
##########
@@ -365,6 +371,11 @@ def create_job_resources(options, # type: PipelineOptions
Stager._create_file_stage_to_artifact(
pickled_session_file, names.PICKLED_MAIN_SESSION_FILE))
+ # stage the submission environment dependencies, if enabled.
Review Comment:
Done, this is better idea.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] riteshghorse commented on pull request #28564: [Python] Log dependencies at runtime and at submission environment
Posted by "riteshghorse (via GitHub)" <gi...@apache.org>.
riteshghorse commented on PR #28564:
URL: https://github.com/apache/beam/pull/28564#issuecomment-1728356996
Run Python_Integration PreCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] riteshghorse commented on pull request #28564: [Python] Log dependencies at runtime and at submission environment
Posted by "riteshghorse (via GitHub)" <gi...@apache.org>.
riteshghorse commented on PR #28564:
URL: https://github.com/apache/beam/pull/28564#issuecomment-1737798133
R: @chamikaramj could you comment of the external transform environment. The external_transform_tests would fail if there is an additional staging file by default.
It complains about no artifact service when it tries to resolve that artifact.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] github-actions[bot] commented on pull request #28564: [Python] Log dependencies at runtime and at submission environment
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #28564:
URL: https://github.com/apache/beam/pull/28564#issuecomment-1728282878
Checks are failing. Will not request review until checks are succeeding. If you'd like to override that behavior, comment `assign set of reviewers`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] riteshghorse commented on a diff in pull request #28564: [Python] Log dependencies at runtime and at submission environment
Posted by "riteshghorse (via GitHub)" <gi...@apache.org>.
riteshghorse commented on code in PR #28564:
URL: https://github.com/apache/beam/pull/28564#discussion_r1334462311
##########
sdks/python/apache_beam/runners/portability/stager.py:
##########
@@ -365,6 +368,16 @@ def create_job_resources(options, # type: PipelineOptions
Stager._create_file_stage_to_artifact(
pickled_session_file, names.PICKLED_MAIN_SESSION_FILE))
+ # stage the submission environment dependencies
+ local_dependency_file_path = os.path.join(
+ temp_dir, SUBMISSION_ENV_DEPENDENCIES_FILENAME)
+ dependencies = freeze.freeze()
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] chamikaramj commented on a diff in pull request #28564: [Python] Log dependencies at runtime and at submission environment
Posted by "chamikaramj (via GitHub)" <gi...@apache.org>.
chamikaramj commented on code in PR #28564:
URL: https://github.com/apache/beam/pull/28564#discussion_r1340669991
##########
sdks/python/apache_beam/transforms/environments.py:
##########
@@ -124,9 +124,9 @@ def __init__(self,
dict(resource_hints) if resource_hints else {})
def __eq__(self, other):
+ # don't compare artifacts since they have different hashes in their names.
return (
- self.__class__ == other.__class__ and
- self._artifacts == other._artifacts
Review Comment:
So if you add a new dependencies, you probably have to update the runner API proto (before being staged) to include the new dependency in the corresponding environment(s).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Python] Log dependencies installed in submission environment [beam]
Posted by "riteshghorse (via GitHub)" <gi...@apache.org>.
riteshghorse commented on code in PR #28564:
URL: https://github.com/apache/beam/pull/28564#discussion_r1491228413
##########
sdks/python/apache_beam/runners/dataflow/dataflow_runner_test.py:
##########
@@ -211,15 +211,13 @@ def test_environment_override_translation_legacy_worker_harness_image(self):
p | ptransform.Create([1, 2, 3])
| 'Do' >> ptransform.FlatMap(lambda x: [(x, x)])
| ptransform.GroupByKey())
- self.assertEqual(
- list(remote_runner.proto_pipeline.components.environments.values()),
- [
- beam_runner_api_pb2.Environment(
- urn=common_urns.environments.DOCKER.urn,
- payload=beam_runner_api_pb2.DockerPayload(
- container_image='LEGACY').SerializeToString(),
- capabilities=environments.python_sdk_docker_capabilities())
- ])
+ first = remote_runner.proto_pipeline.components.environments.values()
+ second = beam_runner_api_pb2.Environment(
+ urn=common_urns.environments.DOCKER.urn,
+ payload=beam_runner_api_pb2.DockerPayload(
+ container_image='LEGACY').SerializeToString(),
+ capabilities=environments.python_sdk_docker_capabilities())
+ self.assertTrue(first.__eq__(second))
Review Comment:
https://www.toptal.com/developers/paste-gd/fkyZTsfi
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Python] Log dependencies at runtime and at submission environment [beam]
Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn commented on PR #28564:
URL: https://github.com/apache/beam/pull/28564#issuecomment-1849646526
Hey @riteshghorse , could we extract the commits that log the runtime dependencies and merge those before next release cut while submission dependencies portion is being sorted out? Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Python] Log dependencies at runtime and at submission environment [beam]
Posted by "riteshghorse (via GitHub)" <gi...@apache.org>.
riteshghorse commented on PR #28564:
URL: https://github.com/apache/beam/pull/28564#issuecomment-1850236083
Created #29705
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] tvalentyn commented on a diff in pull request #28564: [Python] Log dependencies at runtime and at submission environment
Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn commented on code in PR #28564:
URL: https://github.com/apache/beam/pull/28564#discussion_r1333326458
##########
sdks/python/apache_beam/runners/portability/stager.py:
##########
@@ -62,6 +62,7 @@
from urllib.parse import urlparse
from packaging import version
+from pip._internal.operations import freeze
Review Comment:
let's not use internal APIs. a future upgrade to pip may break this api call in previously released versions of Beam, this happened before: https://github.com/pypa/pip/issues/5243#issuecomment-381422449 . Running `pip freeze` command line would be more reliable.
Quick search, but feel free to look more:
https://stackoverflow.com/questions/49923671/are-there-any-function-replacement-for-pip-get-installed-distributions-in-pip
##########
sdks/python/apache_beam/runners/portability/stager.py:
##########
@@ -84,6 +85,8 @@
WORKFLOW_TARBALL_FILE = 'workflow.tar.gz'
REQUIREMENTS_FILE = 'requirements.txt'
EXTRA_PACKAGES_FILE = 'extra_packages.txt'
+# Filename that stores the submission environment dependencies.
+SUBMISSION_ENV_DEPENDENCIES_FILENAME = 'submission_environment_dependencies.txt'
Review Comment:
consistency nit:
```suggestion
SUBMISSION_ENV_DEPENDENCIES_FILE = 'submission_environment_dependencies.txt'
```
##########
sdks/python/apache_beam/runners/portability/stager.py:
##########
@@ -365,6 +368,16 @@ def create_job_resources(options, # type: PipelineOptions
Stager._create_file_stage_to_artifact(
pickled_session_file, names.PICKLED_MAIN_SESSION_FILE))
+ # stage the submission environment dependencies
+ local_dependency_file_path = os.path.join(
+ temp_dir, SUBMISSION_ENV_DEPENDENCIES_FILENAME)
+ dependencies = freeze.freeze()
Review Comment:
Let's make this best effort: if for whatever reason this fails, don't fail the job submission.
You could also consider moving this portion of code into a helper since this method keeps growing (no strong opinion).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] tvalentyn commented on a diff in pull request #28564: [Python] Log dependencies at runtime and at submission environment
Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn commented on code in PR #28564:
URL: https://github.com/apache/beam/pull/28564#discussion_r1339013982
##########
sdks/python/apache_beam/runners/portability/stager.py:
##########
@@ -837,3 +846,20 @@ def _create_beam_sdk(sdk_remote_location, temp_dir):
return [
Stager._create_file_stage_to_artifact(local_download_file, staged_name)
]
+
+ @staticmethod
+ def _create_stage_submission_env_dependencies(temp_dir):
+ try:
+ local_dependency_file_path = os.path.join(
+ temp_dir, SUBMISSION_ENV_DEPENDENCIES_FILE)
+ dependencies = subprocess.check_output(
+ [sys.executable, '-m', 'pip', 'freeze'])
+ with open(local_dependency_file_path, 'w') as f:
+ f.write(str(dependencies))
+ return [
+ Stager._create_file_stage_to_artifact(
+ local_dependency_file_path, SUBMISSION_ENV_DEPENDENCIES_FILE)
+ ]
+ except Exception:
+ _LOGGER.debug("couldn't stage dependencies from submission environment")
Review Comment:
nit: wording suggestion since we are not staging deps themselves, just a list of them. during requirements.txt processing we actually stage the dependencies.
```suggestion
_LOGGER.warning("Couldn't stage a list of installed dependencies in submission environment.")
```
also you could log the exception.
##########
sdks/python/apache_beam/runners/portability/stager.py:
##########
@@ -837,3 +846,20 @@ def _create_beam_sdk(sdk_remote_location, temp_dir):
return [
Stager._create_file_stage_to_artifact(local_download_file, staged_name)
]
+
+ @staticmethod
+ def _create_stage_submission_env_dependencies(temp_dir):
Review Comment:
Let's add a comment here in case someone is wondering why we do this staging. The purpose is to compare the both lists at runtime, make it possible for runners to warn users about possible mismatches and help debug issues related to environment mismatches .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] jrmccluskey commented on a diff in pull request #28564: [Python] Log dependencies at runtime and at submission environment
Posted by "jrmccluskey (via GitHub)" <gi...@apache.org>.
jrmccluskey commented on code in PR #28564:
URL: https://github.com/apache/beam/pull/28564#discussion_r1337335992
##########
sdks/python/container/boot.go:
##########
@@ -409,7 +410,40 @@ func installSetupPackages(ctx context.Context, logger *tools.Logger, files []str
if err := pipInstallPackage(ctx, logger, files, workDir, workflowFile, false, true, nil); err != nil {
return fmt.Errorf("failed to install workflow: %v", err)
}
+ if err := logRuntimeDependencies(ctx, logger); err != nil {
+ logger.Warnf(ctx, "couldn't fetch the runtime dependencies: %v", err)
+ }
+ if err := logSubmissionEnvDependencies(ctx, logger, workDir); err != nil {
+ logger.Warnf(ctx, "couldn't fetch the submission environment dependencies: %v", err)
+ }
+
+ return nil
+}
+func logSubmissionEnvDependencies(ctx context.Context, logger *tools.Logger, dir string) error {
+ logger.Log(ctx, fnexecution_v1.LogEntry_Severity_DEBUG, "Logging submission environment dependencies:")
Review Comment:
To log at debug over the logging API you can invoke logger.Printf(), debug is the default.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] tvalentyn commented on a diff in pull request #28564: [Python] Log dependencies at runtime and at submission environment
Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn commented on code in PR #28564:
URL: https://github.com/apache/beam/pull/28564#discussion_r1340617198
##########
sdks/python/apache_beam/transforms/environments.py:
##########
@@ -124,9 +124,9 @@ def __init__(self,
dict(resource_hints) if resource_hints else {})
def __eq__(self, other):
+ # don't compare artifacts since they have different hashes in their names.
return (
- self.__class__ == other.__class__ and
- self._artifacts == other._artifacts
Review Comment:
https://github.com/apache/beam/blob/a4ee8548424b335cb3ffdf6aeda899fd1e3ba8ad/model/pipeline/src/main/proto/org/apache/beam/model/pipeline/v1/beam_runner_api.proto#L1554
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] chamikaramj commented on a diff in pull request #28564: [Python] Log dependencies at runtime and at submission environment
Posted by "chamikaramj (via GitHub)" <gi...@apache.org>.
chamikaramj commented on code in PR #28564:
URL: https://github.com/apache/beam/pull/28564#discussion_r1340666022
##########
sdks/python/apache_beam/transforms/environments.py:
##########
@@ -124,9 +124,9 @@ def __init__(self,
dict(resource_hints) if resource_hints else {})
def __eq__(self, other):
+ # don't compare artifacts since they have different hashes in their names.
return (
- self.__class__ == other.__class__ and
- self._artifacts == other._artifacts
Review Comment:
Yeah, dependencies are a part of the environment definition so I don't think this line should be reverted.
https://github.com/apache/beam/blob/a9516ba18a7569524fd4f8092caeb12ff4c02bf6/model/pipeline/src/main/proto/org/apache/beam/model/pipeline/v1/beam_runner_api.proto#L1554
##########
sdks/python/apache_beam/transforms/environments.py:
##########
@@ -124,9 +124,9 @@ def __init__(self,
dict(resource_hints) if resource_hints else {})
def __eq__(self, other):
+ # don't compare artifacts since they have different hashes in their names.
return (
- self.__class__ == other.__class__ and
- self._artifacts == other._artifacts
Review Comment:
Do you have a stack trace for the failure ?
The way it works now is that we stage all artifacts at job submission and each Environment carries the correct set of dependencies for that particular environment.
https://github.com/apache/beam/blob/a9516ba18a7569524fd4f8092caeb12ff4c02bf6/model/pipeline/src/main/proto/org/apache/beam/model/pipeline/v1/beam_runner_api.proto#L1554
At runtime, the runner (UW for Dataflow) would look up the correct set of dependencies from the Runner API proto an serve them to the corresponding SDKs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Python] Log dependencies installed in submission environment [beam]
Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn commented on code in PR #28564:
URL: https://github.com/apache/beam/pull/28564#discussion_r1508325095
##########
sdks/python/apache_beam/runners/portability/stager.py:
##########
@@ -365,6 +371,11 @@ def create_job_resources(options, # type: PipelineOptions
Stager._create_file_stage_to_artifact(
pickled_session_file, names.PICKLED_MAIN_SESSION_FILE))
+ # stage the submission environment dependencies, if enabled.
Review Comment:
let's provide a way to disable this behavior via experiment. For example:
```
if (log_submission_env_dependencies and not
options.view_as(DebugOptions).lookup_experiment('disable_logging_submission_environment'))
```
you can use that experiment in one of the tests that you had issues.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Python] Log dependencies at runtime and at submission environment [beam]
Posted by "riteshghorse (via GitHub)" <gi...@apache.org>.
riteshghorse commented on PR #28564:
URL: https://github.com/apache/beam/pull/28564#issuecomment-1864599021
I've verified that a multi-language from python to java works - [Job](https://pantheon.corp.google.com/dataflow/jobs/us-central1/2023-12-19_12_49_40-3339738506115945308;bottomTab=WORKER_LOGS;graphView=0;logsSeverity=INFO?pageState=(%22dfTime%22:(%22s%22:%222023-12-19T20:49:41.203Z%22,%22e%22:%222023-12-19T20:56:48.774Z%22))&project=google.com:clouddfe)
So it is a problem with just the test expansion service
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Python] Log dependencies at runtime and at submission environment [beam]
Posted by "riteshghorse (via GitHub)" <gi...@apache.org>.
riteshghorse commented on PR #28564:
URL: https://github.com/apache/beam/pull/28564#issuecomment-1864600023
I'll a add fake artifact_service method to the test expansion service, that should get us going here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Python] Log dependencies installed in submission environment [beam]
Posted by "riteshghorse (via GitHub)" <gi...@apache.org>.
riteshghorse commented on PR #28564:
URL: https://github.com/apache/beam/pull/28564#issuecomment-1864975748
R: @tvalentyn this is ready for review
Changes to note:
1. Changed the artifact comparison logic to ignore the type payload field since that has unique hashes
2. The external transform test failure was because of the `ExpansionServiceServicer` not having artifact service method. So added that. Confirmed this by running a multi-language pipleine successfully - [Job Link](https://pantheon.corp.google.com/dataflow/jobs/us-central1/2023-12-20_10_40_53-3162726322772484431;bottomTab=WORKER_LOGS;graphView=0;logsSeverity=INFO?pageState=(%22dfTime%22:(%22s%22:%222023-12-20T18:40:53.940Z%22,%22e%22:%222023-12-20T18:44:14.315Z%22))&project=google.com:clouddfe)
3. The staging logic stays in `stager.py` since we ulitmately call `create_job_resources` from `python_sdk_dependencies()` which is invoked during environment creation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Python] Log dependencies installed in submission environment [beam]
Posted by "riteshghorse (via GitHub)" <gi...@apache.org>.
riteshghorse commented on code in PR #28564:
URL: https://github.com/apache/beam/pull/28564#discussion_r1484380585
##########
sdks/python/apache_beam/runners/dataflow/dataflow_runner_test.py:
##########
@@ -211,15 +211,13 @@ def test_environment_override_translation_legacy_worker_harness_image(self):
p | ptransform.Create([1, 2, 3])
| 'Do' >> ptransform.FlatMap(lambda x: [(x, x)])
| ptransform.GroupByKey())
- self.assertEqual(
- list(remote_runner.proto_pipeline.components.environments.values()),
- [
- beam_runner_api_pb2.Environment(
- urn=common_urns.environments.DOCKER.urn,
- payload=beam_runner_api_pb2.DockerPayload(
- container_image='LEGACY').SerializeToString(),
- capabilities=environments.python_sdk_docker_capabilities())
- ])
+ first = remote_runner.proto_pipeline.components.environments.values()
+ second = beam_runner_api_pb2.Environment(
+ urn=common_urns.environments.DOCKER.urn,
+ payload=beam_runner_api_pb2.DockerPayload(
+ container_image='LEGACY').SerializeToString(),
+ capabilities=environments.python_sdk_docker_capabilities())
+ self.assertTrue(first.__eq__(second))
Review Comment:
yeah, on recent run there is this error - https://www.toptal.com/developers/paste-gd/xxNhmlAf
However, with `.__eq__()` it runs fine
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Python] Log dependencies installed in submission environment [beam]
Posted by "riteshghorse (via GitHub)" <gi...@apache.org>.
riteshghorse commented on PR #28564:
URL: https://github.com/apache/beam/pull/28564#issuecomment-1984165926
Done, I'll merge once the check passes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] github-actions[bot] commented on pull request #28564: [Python] Log dependencies at runtime and at submission environment
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #28564:
URL: https://github.com/apache/beam/pull/28564#issuecomment-1737800439
Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] tvalentyn commented on pull request #28564: [Python] Log dependencies at runtime and at submission environment
Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn commented on PR #28564:
URL: https://github.com/apache/beam/pull/28564#issuecomment-1737909333
> t. The external_transform environment tests would fail if there is an additional staging file by default.
Can we stage job submission dependencies without including them in the runtime environment defintion?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Python] Log dependencies at runtime and at submission environment [beam]
Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn commented on code in PR #28564:
URL: https://github.com/apache/beam/pull/28564#discussion_r1344626645
##########
sdks/python/apache_beam/transforms/environments.py:
##########
@@ -124,9 +124,9 @@ def __init__(self,
dict(resource_hints) if resource_hints else {})
def __eq__(self, other):
+ # don't compare artifacts since they have different hashes in their names.
return (
- self.__class__ == other.__class__ and
- self._artifacts == other._artifacts
Review Comment:
that is a bizzare behavior. we should work on fixing comparison. I am not sure what is 'type_payload', but as far as file payloads are concerned, i think it's reasonably safe to compare file hashes (sha) + destination names; alternatively we can look and see why the file path is different and fix that instead. without any additional context and digging further, first option sounds more promising, second might be simpler but hackier and result in some other tech debt.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Python] Log dependencies at runtime and at submission environment [beam]
Posted by "riteshghorse (via GitHub)" <gi...@apache.org>.
riteshghorse commented on PR #28564:
URL: https://github.com/apache/beam/pull/28564#issuecomment-1745653822
Summary:
We have two choices to make:
1. Resolve artifact comparison in environments's `__eq__` method
2. Log/Skip dependency logging for External Environments
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] commented on pull request #28564: [Python] Log dependencies at runtime and at submission environment
Posted by "codecov[bot] (via GitHub)" <gi...@apache.org>.
codecov[bot] commented on PR #28564:
URL: https://github.com/apache/beam/pull/28564#issuecomment-1728211352
## [Codecov](https://app.codecov.io/gh/apache/beam/pull/28564?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
> Merging [#28564](https://app.codecov.io/gh/apache/beam/pull/28564?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) (b696f6f) into [master](https://app.codecov.io/gh/apache/beam/commit/82228873f413de07f9eebcd430ef739c3940ba26?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) (8222887) will **decrease** coverage by `0.01%`.
> Report is 31 commits behind head on master.
> The diff coverage is `n/a`.
```diff
@@ Coverage Diff @@
## master #28564 +/- ##
==========================================
- Coverage 72.24% 72.23% -0.01%
==========================================
Files 684 684
Lines 100952 100963 +11
==========================================
- Hits 72933 72931 -2
- Misses 26441 26453 +12
- Partials 1578 1579 +1
```
| [Flag](https://app.codecov.io/gh/apache/beam/pull/28564/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [go](https://app.codecov.io/gh/apache/beam/pull/28564/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `53.41% <ø> (-0.03%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Files Changed](https://app.codecov.io/gh/apache/beam/pull/28564?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [sdks/python/apache\_beam/runners/internal/names.py](https://app.codecov.io/gh/apache/beam/pull/28564?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcm5hbC9uYW1lcy5weQ==) | `100.00% <ø> (ø)` | |
| [...s/python/apache\_beam/runners/portability/stager.py](https://app.codecov.io/gh/apache/beam/pull/28564?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9wb3J0YWJpbGl0eS9zdGFnZXIucHk=) | `84.39% <ø> (ø)` | |
... and [7 files with indirect coverage changes](https://app.codecov.io/gh/apache/beam/pull/28564/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
:mega: We’re building smart automated test selection to slash your CI/CD build times. [Learn more](https://about.codecov.io/iterative-testing/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] tvalentyn commented on a diff in pull request #28564: [Python] Log dependencies at runtime and at submission environment
Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn commented on code in PR #28564:
URL: https://github.com/apache/beam/pull/28564#discussion_r1340618756
##########
sdks/python/apache_beam/transforms/environments.py:
##########
@@ -124,9 +124,9 @@ def __init__(self,
dict(resource_hints) if resource_hints else {})
def __eq__(self, other):
+ # don't compare artifacts since they have different hashes in their names.
return (
- self.__class__ == other.__class__ and
- self._artifacts == other._artifacts
Review Comment:
also, it is reasonable to assume that if two environments have different set of dependencies, they are different. Why do we
##########
sdks/python/apache_beam/transforms/environments.py:
##########
@@ -124,9 +124,9 @@ def __init__(self,
dict(resource_hints) if resource_hints else {})
def __eq__(self, other):
+ # don't compare artifacts since they have different hashes in their names.
return (
- self.__class__ == other.__class__ and
- self._artifacts == other._artifacts
Review Comment:
also, it is reasonable to assume that if two environments have different set of dependencies, they are different.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] tvalentyn commented on a diff in pull request #28564: [Python] Log dependencies at runtime and at submission environment
Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn commented on code in PR #28564:
URL: https://github.com/apache/beam/pull/28564#discussion_r1340713624
##########
sdks/python/apache_beam/runners/portability/stager.py:
##########
@@ -84,6 +85,8 @@
WORKFLOW_TARBALL_FILE = 'workflow.tar.gz'
REQUIREMENTS_FILE = 'requirements.txt'
EXTRA_PACKAGES_FILE = 'extra_packages.txt'
+# Filename that stores the submission environment dependencies.
+SUBMISSION_ENV_DEPENDENCIES_FILE = 'submission_environment_dependencies.txt'
Review Comment:
There maybe conflation of what we mean by environment.
The intent is to capture the content of environment that creates the translates job graph from Beam Graph to RunnerAPI proto. In a pure-python pipeline, it is the the envirionment of Python interpreter that calls `pipeline.run()`. In xlang, it is where transform expansion happens. Good point that there might be more than 1 environment involve (if someone had a XLang transform with two python environments).
On the execution side, we should make `pip freeze` content available to respective runtime environment that will run transform code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Python] Log dependencies installed in submission environment [beam]
Posted by "riteshghorse (via GitHub)" <gi...@apache.org>.
riteshghorse commented on code in PR #28564:
URL: https://github.com/apache/beam/pull/28564#discussion_r1478696633
##########
sdks/python/apache_beam/transforms/environments.py:
##########
@@ -125,9 +125,17 @@ def __init__(self,
dict(resource_hints) if resource_hints else {})
def __eq__(self, other):
+ equal_artifacts = True
+ for first, second in zip(self._artifacts, other._artifacts):
+ # do not compare type payload since it contains a unique hash.
Review Comment:
Okay, so I reverted this back to how it was originally. creating a mock for the tests that writes artifacts to the same directory resolves this issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Python] Log dependencies installed in submission environment [beam]
Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn commented on PR #28564:
URL: https://github.com/apache/beam/pull/28564#issuecomment-1976940999
i left one comment, after that it should be ready to merge.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Python] Log dependencies installed in submission environment [beam]
Posted by "riteshghorse (via GitHub)" <gi...@apache.org>.
riteshghorse commented on code in PR #28564:
URL: https://github.com/apache/beam/pull/28564#discussion_r1434233735
##########
sdks/python/apache_beam/runners/dataflow/dataflow_runner_test.py:
##########
@@ -211,15 +211,13 @@ def test_environment_override_translation_legacy_worker_harness_image(self):
p | ptransform.Create([1, 2, 3])
| 'Do' >> ptransform.FlatMap(lambda x: [(x, x)])
| ptransform.GroupByKey())
- self.assertEqual(
- list(remote_runner.proto_pipeline.components.environments.values()),
- [
- beam_runner_api_pb2.Environment(
- urn=common_urns.environments.DOCKER.urn,
- payload=beam_runner_api_pb2.DockerPayload(
- container_image='LEGACY').SerializeToString(),
- capabilities=environments.python_sdk_docker_capabilities())
- ])
+ first = remote_runner.proto_pipeline.components.environments.values()
+ second = beam_runner_api_pb2.Environment(
+ urn=common_urns.environments.DOCKER.urn,
+ payload=beam_runner_api_pb2.DockerPayload(
+ container_image='LEGACY').SerializeToString(),
+ capabilities=environments.python_sdk_docker_capabilities())
+ self.assertTrue(first.__eq__(second))
Review Comment:
yeah that's weird. it was failing earlier which led me to do this. unfortunately that check's logs has expired
##########
sdks/python/apache_beam/runners/portability/expansion_service.py:
##########
@@ -128,3 +130,13 @@ def with_pipeline(component, pcoll_id=None):
except Exception: # pylint: disable=broad-except
return beam_expansion_api_pb2.ExpansionResponse(
error=traceback.format_exc())
+
+ def artifact_service(self):
+ return artifact_service.ArtifactRetrievalService(file_reader)
Review Comment:
Sure!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Python] Log dependencies installed in submission environment [beam]
Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn commented on code in PR #28564:
URL: https://github.com/apache/beam/pull/28564#discussion_r1433809471
##########
sdks/python/apache_beam/runners/portability/expansion_service.py:
##########
@@ -128,3 +130,13 @@ def with_pipeline(component, pcoll_id=None):
except Exception: # pylint: disable=broad-except
return beam_expansion_api_pb2.ExpansionResponse(
error=traceback.format_exc())
+
+ def artifact_service(self):
+ return artifact_service.ArtifactRetrievalService(file_reader)
Review Comment:
I think you can use this file_reader:
`artifact_service.BeamFilesystemHandler(None).file_reader` instead of reimplementing.
Also let's CC Cham to review this bit after we iron out other issues.
##########
sdks/python/apache_beam/runners/dataflow/dataflow_runner_test.py:
##########
@@ -211,15 +211,13 @@ def test_environment_override_translation_legacy_worker_harness_image(self):
p | ptransform.Create([1, 2, 3])
| 'Do' >> ptransform.FlatMap(lambda x: [(x, x)])
| ptransform.GroupByKey())
- self.assertEqual(
- list(remote_runner.proto_pipeline.components.environments.values()),
- [
- beam_runner_api_pb2.Environment(
- urn=common_urns.environments.DOCKER.urn,
- payload=beam_runner_api_pb2.DockerPayload(
- container_image='LEGACY').SerializeToString(),
- capabilities=environments.python_sdk_docker_capabilities())
- ])
+ first = remote_runner.proto_pipeline.components.environments.values()
+ second = beam_runner_api_pb2.Environment(
+ urn=common_urns.environments.DOCKER.urn,
+ payload=beam_runner_api_pb2.DockerPayload(
+ container_image='LEGACY').SerializeToString(),
+ capabilities=environments.python_sdk_docker_capabilities())
+ self.assertTrue(first.__eq__(second))
Review Comment:
could we phrase it with `self.assertEqual(expected, actual)`? why did we have to change the helper?
##########
sdks/python/apache_beam/runners/portability/stager.py:
##########
@@ -850,3 +861,40 @@ def _create_beam_sdk(sdk_remote_location, temp_dir):
return [
Stager._create_file_stage_to_artifact(local_download_file, staged_name)
]
+
+ @staticmethod
+ def _create_stage_submission_env_dependencies(temp_dir):
+ """Create and stage a file with list of dependencies installed in the
+ submission environment.
+
+ This staged file is used at runtime to compare the dependencies in the
Review Comment:
```suggestion
This list can be used at runtime to compare against the dependencies in the
```
##########
sdks/python/apache_beam/transforms/environments.py:
##########
@@ -125,9 +125,17 @@ def __init__(self,
dict(resource_hints) if resource_hints else {})
def __eq__(self, other):
+ equal_artifacts = True
+ for first, second in zip(self._artifacts, other._artifacts):
+ # do not compare type payload since it contains a unique hash.
Review Comment:
This comparison is brittle because:
1) we rely on a particular ordering of self._artifacts. It sounds like the reason you are changing it is that you have two essentially equal environments that include an artifact with identical payload/sha256 but different source file location. The source file location is affected by temp folder that introduces randomness. This randomness will be reflected in `artifact.SerializeToString()`, which is used as a key to `sort self._artifacts` (line 121). Because of that, there is no guarantee that when you iterate over `self._artifacts` and `other._artifacts`, multiple artifacts will be ordered in a consistent manner.
2) ignoring entire type.payload might backfire. I would rather use a different key for staged files, than `x.SerializeToString()`. For local files, it could be evaluated based on sha256 and destination name but ignore the source location. Then, you can use such key when sorting artifacts, and when comparing them. Having such key would mean that we'd have to decode ArtifactInformation, and potentially compare different artifact types/roles differently. We could still fall back to `x.SerializeToString()`.
Alternatively you could adjust requirement generation to use the same temp directory for generating the pip freeze over the course of the python process or the test. I think we could do something to prevent the issue you are dealing with here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Python] Log dependencies at runtime and at submission environment [beam]
Posted by "riteshghorse (via GitHub)" <gi...@apache.org>.
riteshghorse commented on code in PR #28564:
URL: https://github.com/apache/beam/pull/28564#discussion_r1344557204
##########
sdks/python/apache_beam/transforms/environments.py:
##########
@@ -124,9 +124,9 @@ def __init__(self,
dict(resource_hints) if resource_hints else {})
def __eq__(self, other):
+ # don't compare artifacts since they have different hashes in their names.
return (
- self.__class__ == other.__class__ and
- self._artifacts == other._artifacts
Review Comment:
I agree that if two environment have different dependencies then they are not equal. But unit tests where we compare the two environments will fail (For example: [EnvironmentOptionsTest::test_environments_with_same_hints_are_equal](https://github.com/apache/beam/blob/12b071402035a0404c4f6e4a83124b3839b5de40/sdks/python/apache_beam/transforms/environments_test.py#L121)) with the addition of a new staging file. Even though this file is same (name and content), it has a a different hash in name:
```
first: [type_urn: "beam:artifact:type:file:v1"
type_payload: "\nd/var/folders/kk/jk392txd7x14sg93r8sjnzkh00tp20/T/tmpmxw3v4d9/submission_environment_dependencies.txt"
role_urn: "beam:artifact:role:staging_to:v1"
role_payload: "\n\'submission_environment_dependencies.txt"
]
second: [type_urn: "beam:artifact:type:file:v1"
type_payload: "\nd/var/folders/kk/jk392txd7x14sg93r8sjnzkh00tp20/T/tmpni8k_038/submission_environment_dependencies.txt"
role_urn: "beam:artifact:role:staging_to:v1"
role_payload: "\n\'submission_environment_dependencies.txt"
]
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Python] Log dependencies at runtime and at submission environment [beam]
Posted by "riteshghorse (via GitHub)" <gi...@apache.org>.
riteshghorse commented on code in PR #28564:
URL: https://github.com/apache/beam/pull/28564#discussion_r1344645172
##########
sdks/python/apache_beam/transforms/environments.py:
##########
@@ -124,9 +124,9 @@ def __init__(self,
dict(resource_hints) if resource_hints else {})
def __eq__(self, other):
+ # don't compare artifacts since they have different hashes in their names.
return (
- self.__class__ == other.__class__ and
- self._artifacts == other._artifacts
Review Comment:
> that is a bizzare behavior. we should work on fixing comparison. I am not sure what is 'type_payload', but as far as file payloads are concerned, i think it's reasonably safe to compare file hashes; alternatively we can look and see why the file path is different and if we should fix that instead. without any additional context and digging further, first option sounds more promising.
File paths are different because we are ultimately calling [`python_sdk_dependencies(options)`](https://github.com/apache/beam/blob/c01b41f9758392a00fd96965c91edea7b61d98bb/sdks/python/apache_beam/transforms/environments.py#L494) which creates a unique temp directory with [`tmp_dir = tempfile.mkdtemp()`](https://github.com/apache/beam/blob/c01b41f9758392a00fd96965c91edea7b61d98bb/sdks/python/apache_beam/transforms/environments.py#L843) every time
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Python] Log dependencies installed in submission environment [beam]
Posted by "riteshghorse (via GitHub)" <gi...@apache.org>.
riteshghorse commented on code in PR #28564:
URL: https://github.com/apache/beam/pull/28564#discussion_r1484380585
##########
sdks/python/apache_beam/runners/dataflow/dataflow_runner_test.py:
##########
@@ -211,15 +211,13 @@ def test_environment_override_translation_legacy_worker_harness_image(self):
p | ptransform.Create([1, 2, 3])
| 'Do' >> ptransform.FlatMap(lambda x: [(x, x)])
| ptransform.GroupByKey())
- self.assertEqual(
- list(remote_runner.proto_pipeline.components.environments.values()),
- [
- beam_runner_api_pb2.Environment(
- urn=common_urns.environments.DOCKER.urn,
- payload=beam_runner_api_pb2.DockerPayload(
- container_image='LEGACY').SerializeToString(),
- capabilities=environments.python_sdk_docker_capabilities())
- ])
+ first = remote_runner.proto_pipeline.components.environments.values()
+ second = beam_runner_api_pb2.Environment(
+ urn=common_urns.environments.DOCKER.urn,
+ payload=beam_runner_api_pb2.DockerPayload(
+ container_image='LEGACY').SerializeToString(),
+ capabilities=environments.python_sdk_docker_capabilities())
+ self.assertTrue(first.__eq__(second))
Review Comment:
yeah, on recent run there is this error - https://www.toptal.com/developers/paste-gd/xxNhmlAf
However, with `.__eq__` it runs fine
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Python] Log dependencies installed in submission environment [beam]
Posted by "kennknowles (via GitHub)" <gi...@apache.org>.
kennknowles commented on PR #28564:
URL: https://github.com/apache/beam/pull/28564#issuecomment-1976924084
Is this ready to merge?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Python] Log dependencies installed in submission environment [beam]
Posted by "riteshghorse (via GitHub)" <gi...@apache.org>.
riteshghorse merged PR #28564:
URL: https://github.com/apache/beam/pull/28564
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Python] Log dependencies at runtime and at submission environment [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #28564:
URL: https://github.com/apache/beam/pull/28564#issuecomment-1837468754
This pull request has been marked as stale due to 60 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@beam.apache.org list. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Python] Log dependencies at runtime and at submission environment [beam]
Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn commented on code in PR #28564:
URL: https://github.com/apache/beam/pull/28564#discussion_r1344626645
##########
sdks/python/apache_beam/transforms/environments.py:
##########
@@ -124,9 +124,9 @@ def __init__(self,
dict(resource_hints) if resource_hints else {})
def __eq__(self, other):
+ # don't compare artifacts since they have different hashes in their names.
return (
- self.__class__ == other.__class__ and
- self._artifacts == other._artifacts
Review Comment:
that is a bizzare behavior. we should work on fixing comparison. I am not sure what is 'type_payload', but as far as file payloads are concerned, i think it's reasonably safe to compare file hashes + destination names; alternatively we can look and see why the file path is different and if we should fix that instead. without any additional context and digging further, first option sounds more promising.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Python] Log dependencies at runtime and at submission environment [beam]
Posted by "riteshghorse (via GitHub)" <gi...@apache.org>.
riteshghorse commented on code in PR #28564:
URL: https://github.com/apache/beam/pull/28564#discussion_r1344542429
##########
sdks/python/apache_beam/runners/portability/stager.py:
##########
@@ -837,3 +846,20 @@ def _create_beam_sdk(sdk_remote_location, temp_dir):
return [
Stager._create_file_stage_to_artifact(local_download_file, staged_name)
]
+
+ @staticmethod
+ def _create_stage_submission_env_dependencies(temp_dir):
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Python] Log dependencies at runtime and at submission environment [beam]
Posted by "riteshghorse (via GitHub)" <gi...@apache.org>.
riteshghorse commented on code in PR #28564:
URL: https://github.com/apache/beam/pull/28564#discussion_r1344645172
##########
sdks/python/apache_beam/transforms/environments.py:
##########
@@ -124,9 +124,9 @@ def __init__(self,
dict(resource_hints) if resource_hints else {})
def __eq__(self, other):
+ # don't compare artifacts since they have different hashes in their names.
return (
- self.__class__ == other.__class__ and
- self._artifacts == other._artifacts
Review Comment:
> that is a bizzare behavior. we should work on fixing comparison. I am not sure what is 'type_payload', but as far as file payloads are concerned, i think it's reasonably safe to compare file hashes; alternatively we can look and see why the file path is different and if we should fix that instead. without any additional context and digging further, first option sounds more promising.
File paths are different because we are ultimately calling `python_sdk_dependencies(options)` which creates a unique temp directory with `tmp_dir = tempfile.mkdtemp()` every time
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] github-actions[bot] commented on pull request #28564: [Python] Log dependencies at runtime and at submission environment
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #28564:
URL: https://github.com/apache/beam/pull/28564#issuecomment-1731625595
Assigning reviewers. If you would like to opt out of this review, comment `assign to next reviewer`:
R: @jrmccluskey for label python.
Available commands:
- `stop reviewer notifications` - opt out of the automated review tooling
- `remind me after tests pass` - tag the comment author after tests pass
- `waiting on author` - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)
The PR bot will only process comments in the main thread (not review comments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] tvalentyn commented on a diff in pull request #28564: [Python] Log dependencies at runtime and at submission environment
Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn commented on code in PR #28564:
URL: https://github.com/apache/beam/pull/28564#discussion_r1339013982
##########
sdks/python/apache_beam/runners/portability/stager.py:
##########
@@ -837,3 +846,20 @@ def _create_beam_sdk(sdk_remote_location, temp_dir):
return [
Stager._create_file_stage_to_artifact(local_download_file, staged_name)
]
+
+ @staticmethod
+ def _create_stage_submission_env_dependencies(temp_dir):
+ try:
+ local_dependency_file_path = os.path.join(
+ temp_dir, SUBMISSION_ENV_DEPENDENCIES_FILE)
+ dependencies = subprocess.check_output(
+ [sys.executable, '-m', 'pip', 'freeze'])
+ with open(local_dependency_file_path, 'w') as f:
+ f.write(str(dependencies))
+ return [
+ Stager._create_file_stage_to_artifact(
+ local_dependency_file_path, SUBMISSION_ENV_DEPENDENCIES_FILE)
+ ]
+ except Exception:
+ _LOGGER.debug("couldn't stage dependencies from submission environment")
Review Comment:
nit: wording suggestion since we are not staging deps themselves like during requirements.txt processing.
```suggestion
_LOGGER.warning("Couldn't stage a list of installed dependencies in submission environment.")
```
also you could log the exception.
##########
sdks/python/apache_beam/runners/portability/stager.py:
##########
@@ -837,3 +846,20 @@ def _create_beam_sdk(sdk_remote_location, temp_dir):
return [
Stager._create_file_stage_to_artifact(local_download_file, staged_name)
]
+
+ @staticmethod
+ def _create_stage_submission_env_dependencies(temp_dir):
Review Comment:
Let's add a here in case someone is wondering why we do this staging. The purpose is to compare the both lists at runtime, make it possible for runners to warn users about possible mismatches and help debug issues related to environment mismatches .
##########
sdks/python/apache_beam/transforms/environments.py:
##########
@@ -124,9 +124,9 @@ def __init__(self,
dict(resource_hints) if resource_hints else {})
def __eq__(self, other):
+ # don't compare artifacts since they have different hashes in their names.
return (
- self.__class__ == other.__class__ and
- self._artifacts == other._artifacts
Review Comment:
this change is not obvious to me. I think we shouldn't include the staged job submission dependencies into the list of environment artifacts.
The transform environment describes the runtime environment the transforms should have on the runner. Job submission dependencies currently do not impose any requirements on runtime environment, so they should not be a part of environment definition. Would it resolve the issue?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] riteshghorse commented on a diff in pull request #28564: [Python] Log dependencies at runtime and at submission environment
Posted by "riteshghorse (via GitHub)" <gi...@apache.org>.
riteshghorse commented on code in PR #28564:
URL: https://github.com/apache/beam/pull/28564#discussion_r1337578790
##########
sdks/python/container/boot.go:
##########
@@ -409,7 +410,40 @@ func installSetupPackages(ctx context.Context, logger *tools.Logger, files []str
if err := pipInstallPackage(ctx, logger, files, workDir, workflowFile, false, true, nil); err != nil {
return fmt.Errorf("failed to install workflow: %v", err)
}
+ if err := logRuntimeDependencies(ctx, logger); err != nil {
+ logger.Warnf(ctx, "couldn't fetch the runtime dependencies: %v", err)
+ }
+ if err := logSubmissionEnvDependencies(ctx, logger, workDir); err != nil {
+ logger.Warnf(ctx, "couldn't fetch the submission environment dependencies: %v", err)
+ }
+
+ return nil
+}
+func logSubmissionEnvDependencies(ctx context.Context, logger *tools.Logger, dir string) error {
+ logger.Log(ctx, fnexecution_v1.LogEntry_Severity_DEBUG, "Logging submission environment dependencies:")
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] tvalentyn commented on a diff in pull request #28564: [Python] Log dependencies at runtime and at submission environment
Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn commented on code in PR #28564:
URL: https://github.com/apache/beam/pull/28564#discussion_r1340595246
##########
sdks/python/apache_beam/transforms/environments.py:
##########
@@ -124,9 +124,9 @@ def __init__(self,
dict(resource_hints) if resource_hints else {})
def __eq__(self, other):
+ # don't compare artifacts since they have different hashes in their names.
return (
- self.__class__ == other.__class__ and
- self._artifacts == other._artifacts
Review Comment:
> Job submission dependencies currently do not impose any requirements on runtime environment...
Actually, I may be wrong here - we don't want to install the submission dependencies on the workers, but we still need to stage the file, in that sense the file is still part of the runtime environment, and it might be that list of staged artifacts is communicated via environment definition.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Python] Log dependencies installed in submission environment [beam]
Posted by "riteshghorse (via GitHub)" <gi...@apache.org>.
riteshghorse commented on code in PR #28564:
URL: https://github.com/apache/beam/pull/28564#discussion_r1434236712
##########
sdks/python/apache_beam/transforms/environments.py:
##########
@@ -125,9 +125,17 @@ def __init__(self,
dict(resource_hints) if resource_hints else {})
def __eq__(self, other):
+ equal_artifacts = True
+ for first, second in zip(self._artifacts, other._artifacts):
+ # do not compare type payload since it contains a unique hash.
Review Comment:
with same temp directory, won't it overwrite the file with same name? because the two values are like this:
```
first: [type_urn: "beam:artifact:type:file:v1"
type_payload: "\nd/var/folders/kk/jk392txd7x14sg93r8sjnzkh00tp20/T/tmpmxw3v4d9/submission_environment_dependencies.txt"
role_urn: "beam:artifact:role:staging_to:v1"
role_payload: "\n\'submission_environment_dependencies.txt"
]
second: [type_urn: "beam:artifact:type:file:v1"
type_payload: "\nd/var/folders/kk/jk392txd7x14sg93r8sjnzkh00tp20/T/tmpni8k_038/submission_environment_dependencies.txt"
role_urn: "beam:artifact:role:staging_to:v1"
role_payload: "\n\'submission_environment_dependencies.txt"
]
```
But yeah I agree that we need to fix the artifact comparison function, somehow this wasn't caught earlier!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Python] Log dependencies installed in submission environment [beam]
Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn commented on code in PR #28564:
URL: https://github.com/apache/beam/pull/28564#discussion_r1434204041
##########
sdks/python/apache_beam/transforms/environments.py:
##########
@@ -125,9 +125,17 @@ def __init__(self,
dict(resource_hints) if resource_hints else {})
def __eq__(self, other):
+ equal_artifacts = True
+ for first, second in zip(self._artifacts, other._artifacts):
+ # do not compare type payload since it contains a unique hash.
Review Comment:
This comparison is brittle because:
1) we rely on a particular ordering of self._artifacts. It sounds like the reason you are changing it is that you have two essentially equal environments that include an artifact with identical payload/sha256 but different source file location. The source file location is affected by temp folder that introduces randomness. This randomness will be reflected in `artifact.SerializeToString()`, which is used as a key to `sort self._artifacts` (line 121). Because of that, there is no guarantee that when you iterate over `self._artifacts` and `other._artifacts`, the lists will be in a consistent order.
2) ignoring entire type.payload might backfire. I would rather use a different key for staged files, than `x.SerializeToString()`. For local files, it could be evaluated based on sha256 and destination name but ignore the source location. Then, you can use such key when sorting artifacts, and when comparing them. Having such key would mean that we'd have to decode ArtifactInformation, and potentially compare different artifact types/roles differently. We could still fall back to `x.SerializeToString()`.
Alternatively you could adjust requirement generation to use the same temp directory for generating the pip freeze over the course of the python process or the test. I think we could do something to prevent the issue you are dealing with here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Python] Log dependencies installed in submission environment [beam]
Posted by "riteshghorse (via GitHub)" <gi...@apache.org>.
riteshghorse commented on code in PR #28564:
URL: https://github.com/apache/beam/pull/28564#discussion_r1484374771
##########
sdks/python/apache_beam/transforms/environments_test.py:
##########
@@ -21,20 +21,40 @@
# pytype: skip-file
import logging
+import tempfile
import unittest
from apache_beam.options.pipeline_options import PortableOptions
+from apache_beam.options.pipeline_options import SetupOptions
from apache_beam.portability import common_urns
from apache_beam.runners import pipeline_context
+from apache_beam.runners.portability import stager
from apache_beam.transforms import environments
from apache_beam.transforms.environments import DockerEnvironment
from apache_beam.transforms.environments import EmbeddedPythonEnvironment
from apache_beam.transforms.environments import EmbeddedPythonGrpcEnvironment
from apache_beam.transforms.environments import Environment
from apache_beam.transforms.environments import ExternalEnvironment
from apache_beam.transforms.environments import ProcessEnvironment
+from apache_beam.transforms.environments import PyPIArtifactRegistry
from apache_beam.transforms.environments import SubprocessSDKEnvironment
+# create a temp directory so that all artifacts are put in the same directory.
+tmp_dir = tempfile.mkdtemp()
+
+
+def mock_python_sdk_dependencies(options, tmp_dir=tmp_dir):
Review Comment:
yeah I think this will look more clean, thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org