You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@beam.apache.org by "Sergei Lebedev (Jira)" <ji...@apache.org> on 2021/12/06 12:44:00 UTC
[jira] [Updated] (BEAM-13384) apache_beam does not explicitly re-export metrics
[ https://issues.apache.org/jira/browse/BEAM-13384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sergei Lebedev updated BEAM-13384:
----------------------------------
Description:
{{apache_beam/__init__.py}} does not re-export the {{metrics}} subpackage and thus forces users to rely on import time side-effects. Specifically, it seems fairly common to create metrics as
{code:java}
import apache_beam as beam
# ...
c = beam.metrics.Metrics.counter("ns", "counter"){code}
This works at runtime, because {{apache_beam}} imports the {{metrics}} subpackage indirectly through one of its dependencies and get the "metrics" attribute as a side-effect of that import (see [last paragraph|https://docs.python.org/3/reference/import.html#submodules] in "The import system" for an explanation of how that works).
Examples from GitHub
* [tensorflow/transform|https://github.com/tensorflow/transform/blob/e0331bda765b7cc34347b38da74270dd3c01939b/tensorflow_transform/beam/impl.py#L282]
* [tensorflow/tfx|https://github.com/tensorflow/tfx/blob/578c40cc23d5dd661826f5ed0e16db58479db1b4/tfx/components/transform/executor.py#L600]
* [google-research/tapas|https://github.com/google-research/tapas/blob/f3d9f068e6eedb252883049b582516a1294ff951/tapas/utils/pretrain_utils.py#L65]
* [google/tensorflow-recorder|https://github.com/google/tensorflow-recorder/blob/231b2c5422593d4eb7fc8502d5023425d08bedd6/tfrecorder/beam_pipeline.py#L141]
* [kubeflow/examples|https://github.com/kubeflow/examples/blob/fcd2ef38027c1c0a91af2370c2beedb4c24298a8/code_search/src/code_search/dataflow/do_fns/prediction_do_fn.py#L112]
I think we should re-export metrics in {{apache_beam/__init__.py,}} similarly to coders and io. Apart from legalizing a common way of defining metrics, this would also make {{apache_beam}} more tooling friendly, since type checkers and IDEs would be able to understand what {{apache_beam.metrics}} refers to without any Beam-specific logic/hacks.
was:
apache_beam/__init__.py does not re-export the metrics subpackage and thus forces users to rely on import time side-effects. Specifically, it seems fairly common to create metrics as
{code:java}
import apache_beam as beam
# ...
c = beam.metrics.Metrics.counter("ns", "counter"){code}
This works at runtime, because apache_beam imports the metrics subpackage indirectly through one of its dependencies and get the "metrics" attribute as a side-effect of that import (see [last paragraph|https://docs.python.org/3/reference/import.html#submodules] in "The import system" for an explanation of how that works).
Examples from GitHub
* [tensorflow/transform|https://github.com/tensorflow/transform/blob/e0331bda765b7cc34347b38da74270dd3c01939b/tensorflow_transform/beam/impl.py#L282]
* [tensorflow/tfx|https://github.com/tensorflow/tfx/blob/578c40cc23d5dd661826f5ed0e16db58479db1b4/tfx/components/transform/executor.py#L600]
* [google-research/tapas|https://github.com/google-research/tapas/blob/f3d9f068e6eedb252883049b582516a1294ff951/tapas/utils/pretrain_utils.py#L65]
* [google/tensorflow-recorder|https://github.com/google/tensorflow-recorder/blob/231b2c5422593d4eb7fc8502d5023425d08bedd6/tfrecorder/beam_pipeline.py#L141]
* [kubeflow/examples|https://github.com/kubeflow/examples/blob/fcd2ef38027c1c0a91af2370c2beedb4c24298a8/code_search/src/code_search/dataflow/do_fns/prediction_do_fn.py#L112]
I think we should re-export metrics in apache_beam/__init__.py, similarly to coders and io.
> apache_beam does not explicitly re-export metrics
> -------------------------------------------------
>
> Key: BEAM-13384
> URL: https://issues.apache.org/jira/browse/BEAM-13384
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core
> Reporter: Sergei Lebedev
> Priority: P2
>
> {{apache_beam/__init__.py}} does not re-export the {{metrics}} subpackage and thus forces users to rely on import time side-effects. Specifically, it seems fairly common to create metrics as
> {code:java}
> import apache_beam as beam
> # ...
> c = beam.metrics.Metrics.counter("ns", "counter"){code}
> This works at runtime, because {{apache_beam}} imports the {{metrics}} subpackage indirectly through one of its dependencies and get the "metrics" attribute as a side-effect of that import (see [last paragraph|https://docs.python.org/3/reference/import.html#submodules] in "The import system" for an explanation of how that works).
> Examples from GitHub
> * [tensorflow/transform|https://github.com/tensorflow/transform/blob/e0331bda765b7cc34347b38da74270dd3c01939b/tensorflow_transform/beam/impl.py#L282]
> * [tensorflow/tfx|https://github.com/tensorflow/tfx/blob/578c40cc23d5dd661826f5ed0e16db58479db1b4/tfx/components/transform/executor.py#L600]
> * [google-research/tapas|https://github.com/google-research/tapas/blob/f3d9f068e6eedb252883049b582516a1294ff951/tapas/utils/pretrain_utils.py#L65]
> * [google/tensorflow-recorder|https://github.com/google/tensorflow-recorder/blob/231b2c5422593d4eb7fc8502d5023425d08bedd6/tfrecorder/beam_pipeline.py#L141]
> * [kubeflow/examples|https://github.com/kubeflow/examples/blob/fcd2ef38027c1c0a91af2370c2beedb4c24298a8/code_search/src/code_search/dataflow/do_fns/prediction_do_fn.py#L112]
> I think we should re-export metrics in {{apache_beam/__init__.py,}} similarly to coders and io. Apart from legalizing a common way of defining metrics, this would also make {{apache_beam}} more tooling friendly, since type checkers and IDEs would be able to understand what {{apache_beam.metrics}} refers to without any Beam-specific logic/hacks.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)