You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@beam.apache.org by GitBox <gi...@apache.org> on 2020/03/02 12:51:28 UTC
[GitHub] [beam] echauchot opened a new pull request #11013: Add metrics
export to documentation on the website.
echauchot opened a new pull request #11013: Add metrics export to documentation on the website.
URL: https://github.com/apache/beam/pull/11013
**Please** add a meaningful description for your change here
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`).
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier).
Post-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
--- | --- | --- | --- | --- | --- | --- | ---
Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/)
Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/)
Python | [](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow_V2/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Python2_PVR_Flink_Cron/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python35_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/lastCompletedBuild/)
XLang | --- | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_XVR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_XVR_Spark/lastCompletedBuild/)
Pre-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
--- |Java | Python | Go | Website
--- | --- | --- | --- | ---
Non-portable | [](https://builds.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PreCommit_PythonLint_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Go_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Website_Cron/lastCompletedBuild/)
Portable | --- | [](https://builds.apache.org/job/beam_PreCommit_Portable_Python_Cron/lastCompletedBuild/) | --- | ---
See [.test-infra/jenkins/README](https://github.com/apache/beam/blob/master/.test-infra/jenkins/README.md) for trigger phrase, status and link of all Jenkins jobs.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] aromanenko-dev commented on a change in pull request #11013:
Add metrics export to documentation on the website.
Posted by GitBox <gi...@apache.org>.
aromanenko-dev commented on a change in pull request #11013: Add metrics export to documentation on the website.
URL: https://github.com/apache/beam/pull/11013#discussion_r387122980
##########
File path: website/src/documentation/programming-guide.md
##########
@@ -3121,6 +3121,15 @@ public class MyMetricsDoFn extends DoFn<Integer, Integer> {
}
}
```
+### 9.5 Export metrics {#export-metrics}
+Beam metrics can be exported to external sinks. If set up the runner will push metrics to a configured sink at a configured period.
+The configuration is held in the [MetricsOptions](https://beam.apache.org/releases/javadoc/2.19.0/org/apache/beam/sdk/metrics/MetricsOptions.html) class.
+It contains push period configuration and also sink specific options such as URL. As for now only REST http sink and graphite sinks are supported and only
+Flink and Spark runners support metrics export.
+
+Also Beam metrics are exported to inner Spark and Flink dashboards to be consulted in their respective UI.
+
+
Review comment:
I was thinking about any useful examples that will help users to use it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] echauchot commented on a change in pull request #11013: Add
metrics export to documentation on the website.
Posted by GitBox <gi...@apache.org>.
echauchot commented on a change in pull request #11013: Add metrics export to documentation on the website.
URL: https://github.com/apache/beam/pull/11013#discussion_r387119462
##########
File path: website/src/documentation/programming-guide.md
##########
@@ -3121,6 +3121,15 @@ public class MyMetricsDoFn extends DoFn<Integer, Integer> {
}
}
```
+### 9.5 Export metrics {#export-metrics}
+Beam metrics can be exported to external sinks. If set up the runner will push metrics to a configured sink at a configured period.
+The configuration is held in the [MetricsOptions](https://beam.apache.org/releases/javadoc/2.19.0/org/apache/beam/sdk/metrics/MetricsOptions.html) class.
+It contains push period configuration and also sink specific options such as URL. As for now only REST http sink and graphite sinks are supported and only
+Flink and Spark runners support metrics export.
+
+Also Beam metrics are exported to inner Spark and Flink dashboards to be consulted in their respective UI.
+
+
Review comment:
there is no more code than what is present in the example in previous sections. The user only needs to configure the export to export the metrics he defined in his pipeline, what do you mean ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] echauchot commented on a change in pull request #11013: Add
metrics export to documentation on the website.
Posted by GitBox <gi...@apache.org>.
echauchot commented on a change in pull request #11013: Add metrics export to documentation on the website.
URL: https://github.com/apache/beam/pull/11013#discussion_r387118412
##########
File path: website/src/documentation/programming-guide.md
##########
@@ -3121,6 +3121,15 @@ public class MyMetricsDoFn extends DoFn<Integer, Integer> {
}
}
```
+### 9.5 Export metrics {#export-metrics}
+Beam metrics can be exported to external sinks. If set up the runner will push metrics to a configured sink at a configured period.
+The configuration is held in the [MetricsOptions](https://beam.apache.org/releases/javadoc/2.19.0/org/apache/beam/sdk/metrics/MetricsOptions.html) class.
+It contains push period configuration and also sink specific options such as URL. As for now only REST http sink and graphite sinks are supported and only
Review comment:
+1
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] echauchot commented on a change in pull request #11013: Add
metrics export to documentation on the website.
Posted by GitBox <gi...@apache.org>.
echauchot commented on a change in pull request #11013: Add metrics export to documentation on the website.
URL: https://github.com/apache/beam/pull/11013#discussion_r387119462
##########
File path: website/src/documentation/programming-guide.md
##########
@@ -3121,6 +3121,15 @@ public class MyMetricsDoFn extends DoFn<Integer, Integer> {
}
}
```
+### 9.5 Export metrics {#export-metrics}
+Beam metrics can be exported to external sinks. If set up the runner will push metrics to a configured sink at a configured period.
+The configuration is held in the [MetricsOptions](https://beam.apache.org/releases/javadoc/2.19.0/org/apache/beam/sdk/metrics/MetricsOptions.html) class.
+It contains push period configuration and also sink specific options such as URL. As for now only REST http sink and graphite sinks are supported and only
+Flink and Spark runners support metrics export.
+
+Also Beam metrics are exported to inner Spark and Flink dashboards to be consulted in their respective UI.
+
+
Review comment:
there is no more code than what is present in the examples in previous sections. The user only needs to configure the export to export the metrics he defined in his pipeline, what do you mean ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] aromanenko-dev commented on a change in pull request #11013:
Add metrics export to documentation on the website.
Posted by GitBox <gi...@apache.org>.
aromanenko-dev commented on a change in pull request #11013: Add metrics export to documentation on the website.
URL: https://github.com/apache/beam/pull/11013#discussion_r387060666
##########
File path: website/src/documentation/programming-guide.md
##########
@@ -3121,6 +3121,15 @@ public class MyMetricsDoFn extends DoFn<Integer, Integer> {
}
}
```
+### 9.5 Export metrics {#export-metrics}
+Beam metrics can be exported to external sinks. If set up the runner will push metrics to a configured sink at a configured period.
Review comment:
> If set up the runner will push metrics to a configured sink at a configured period.
Could you elaborate or rephrase this sentence? I think it misses "then" somewhere since it starts from "if".
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] aromanenko-dev merged pull request #11013: Add metrics
export to documentation on the website.
Posted by GitBox <gi...@apache.org>.
aromanenko-dev merged pull request #11013: Add metrics export to documentation on the website.
URL: https://github.com/apache/beam/pull/11013
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] echauchot commented on a change in pull request #11013: Add
metrics export to documentation on the website.
Posted by GitBox <gi...@apache.org>.
echauchot commented on a change in pull request #11013: Add metrics export to documentation on the website.
URL: https://github.com/apache/beam/pull/11013#discussion_r387122201
##########
File path: website/src/documentation/programming-guide.md
##########
@@ -3121,6 +3121,15 @@ public class MyMetricsDoFn extends DoFn<Integer, Integer> {
}
}
```
+### 9.5 Export metrics {#export-metrics}
+Beam metrics can be exported to external sinks. If set up the runner will push metrics to a configured sink at a configured period.
Review comment:
I asked a native, it is correct but lacks the , after set up. But I'll elaborate a bit to clarify.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] echauchot commented on issue #11013: Add metrics export to
documentation on the website.
Posted by GitBox <gi...@apache.org>.
echauchot commented on issue #11013: Add metrics export to documentation on the website.
URL: https://github.com/apache/beam/pull/11013#issuecomment-594031820
@aromanenko-dev thanks for the review ! I addressed your comments and squashed. PTAL
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] aromanenko-dev commented on a change in pull request #11013:
Add metrics export to documentation on the website.
Posted by GitBox <gi...@apache.org>.
aromanenko-dev commented on a change in pull request #11013: Add metrics export to documentation on the website.
URL: https://github.com/apache/beam/pull/11013#discussion_r387064086
##########
File path: website/src/documentation/programming-guide.md
##########
@@ -3121,6 +3121,15 @@ public class MyMetricsDoFn extends DoFn<Integer, Integer> {
}
}
```
+### 9.5 Export metrics {#export-metrics}
+Beam metrics can be exported to external sinks. If set up the runner will push metrics to a configured sink at a configured period.
+The configuration is held in the [MetricsOptions](https://beam.apache.org/releases/javadoc/2.19.0/org/apache/beam/sdk/metrics/MetricsOptions.html) class.
+It contains push period configuration and also sink specific options such as URL. As for now only REST http sink and graphite sinks are supported and only
+Flink and Spark runners support metrics export.
+
+Also Beam metrics are exported to inner Spark and Flink dashboards to be consulted in their respective UI.
+
+
Review comment:
Do you have any code examples of metrics export? It will worth to add if any.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] aromanenko-dev commented on a change in pull request #11013:
Add metrics export to documentation on the website.
Posted by GitBox <gi...@apache.org>.
aromanenko-dev commented on a change in pull request #11013: Add metrics export to documentation on the website.
URL: https://github.com/apache/beam/pull/11013#discussion_r387062584
##########
File path: website/src/documentation/programming-guide.md
##########
@@ -3121,6 +3121,15 @@ public class MyMetricsDoFn extends DoFn<Integer, Integer> {
}
}
```
+### 9.5 Export metrics {#export-metrics}
+Beam metrics can be exported to external sinks. If set up the runner will push metrics to a configured sink at a configured period.
+The configuration is held in the [MetricsOptions](https://beam.apache.org/releases/javadoc/2.19.0/org/apache/beam/sdk/metrics/MetricsOptions.html) class.
+It contains push period configuration and also sink specific options such as URL. As for now only REST http sink and graphite sinks are supported and only
Review comment:
> As for now only REST http sink and graphite sinks are supported
I's suggest to drop the first "sink", and leave it like:
"As for now, the only REST HTTP and Graphite sinks are supported"
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services