You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2020/06/29 21:04:30 UTC
[GitHub] [beam] rainwoodman opened a new pull request #12123: MetricFilter matching by Metric objects.
rainwoodman opened a new pull request #12123:
URL: https://github.com/apache/beam/pull/12123
This change would encourage reusing predefined metric objects in
metric queries.
Querying with the metric object may result more maintainable code
than duplicating the name and namespace strings. Or storeing the
names elsewhere as CONSTANT_NAME.
**Please** add a meaningful description for your change here
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`).
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier).
Post-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
Lang | SDK | Dataflow | Flink | Samza | Spark | Twister2
--- | --- | --- | --- | --- | --- | ---
Go | [](https://ci-beam.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) | ---
Java | [](https://ci-beam.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Twister2/lastCompletedBuild/)
Python | [](https://ci-beam.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow_V2/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Python2_PVR_Flink_Cron/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python35_VR_Flink/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Spark/lastCompletedBuild/) | ---
XLang | --- | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Flink/lastCompletedBuild/) | --- | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Spark/lastCompletedBuild/) | ---
Pre-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
--- |Java | Python | Go | Website
--- | --- | --- | --- | ---
Non-portable | [](https://ci-beam.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PreCommit_PythonLint_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Go_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Website_Cron/lastCompletedBuild/)
Portable | --- | [](https://ci-beam.apache.org/job/beam_PreCommit_Portable_Python_Cron/lastCompletedBuild/) | --- | ---
See [.test-infra/jenkins/README](https://github.com/apache/beam/blob/master/.test-infra/jenkins/README.md) for trigger phrase, status and link of all Jenkins jobs.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] rainwoodman commented on pull request #12123: MetricFilter matching by Metric objects.
Posted by GitBox <gi...@apache.org>.
rainwoodman commented on pull request #12123:
URL: https://github.com/apache/beam/pull/12123#issuecomment-651366435
R: @aaltay
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem merged pull request #12123: MetricFilter matching by Metric objects.
Posted by GitBox <gi...@apache.org>.
pabloem merged pull request #12123:
URL: https://github.com/apache/beam/pull/12123
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem commented on pull request #12123: MetricFilter matching by Metric objects.
Posted by GitBox <gi...@apache.org>.
pabloem commented on pull request #12123:
URL: https://github.com/apache/beam/pull/12123#issuecomment-651999437
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem commented on a change in pull request #12123: MetricFilter matching by Metric objects.
Posted by GitBox <gi...@apache.org>.
pabloem commented on a change in pull request #12123:
URL: https://github.com/apache/beam/pull/12123#discussion_r448128821
##########
File path: sdks/python/apache_beam/metrics/metric_test.py
##########
@@ -159,15 +163,26 @@ def process(self, element):
res = pipeline.run()
res.wait_until_finish()
+ # Verify static counter.
+ metric_results = (
+ res.metrics().query(
+ MetricsFilter().with_metric(SomeDoFn.static_counter_elements)))
+ outputs_static_counter = metric_results['counters'][0]
+
+ self.assertEqual(
+ outputs_static_counter.key.metric.name,
+ 'metrics_static_counter_element')
+ self.assertEqual(outputs_static_counter.committed, 8)
+
# Verify user counter.
metric_results = (
res.metrics().query(
MetricsFilter().with_name('metrics_user_counter_element')))
- outputs_counter = metric_results['counters'][0]
+ outputs_user_counter = metric_results['counters'][0]
Review comment:
you're right. I would not call this the canonical way. Though tbh this API should be improved to be less JAva-like.
In this case, we query it this way only for purposes of the unittest, because we know what to expect.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem commented on a change in pull request #12123: MetricFilter matching by Metric objects.
Posted by GitBox <gi...@apache.org>.
pabloem commented on a change in pull request #12123:
URL: https://github.com/apache/beam/pull/12123#discussion_r447325724
##########
File path: sdks/python/apache_beam/metrics/metric_test.py
##########
@@ -162,7 +161,7 @@ def process(self, element):
# Verify user counter.
metric_results = (
res.metrics().query(
- MetricsFilter().with_name('metrics_user_counter_element')))
+ MetricsFilter().with_metric(SomeDoFn.user_counter_elements)))
Review comment:
yes, can you add both to the test, please?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aaltay commented on a change in pull request #12123: MetricFilter matching by Metric objects.
Posted by GitBox <gi...@apache.org>.
aaltay commented on a change in pull request #12123:
URL: https://github.com/apache/beam/pull/12123#discussion_r447282378
##########
File path: sdks/python/apache_beam/metrics/metric_test.py
##########
@@ -162,7 +161,7 @@ def process(self, element):
# Verify user counter.
metric_results = (
res.metrics().query(
- MetricsFilter().with_name('metrics_user_counter_element')))
+ MetricsFilter().with_metric(SomeDoFn.user_counter_elements)))
Review comment:
Should we add a new test for with_metric instead of replacing the with_name test?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org