You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2020/06/27 11:21:16 UTC
[GitHub] [hudi] shenh062326 opened a new pull request #1769: [DOC] Add document for the use of metrics system in Hudi.
shenh062326 opened a new pull request #1769:
URL: https://github.com/apache/hudi/pull/1769
## *Tips*
- *Thank you very much for contributing to Apache Hudi.*
- *Please review https://hudi.apache.org/contributing.html before opening a pull request.*
## What is the purpose of the pull request
*(For example: This pull request adds quick-start document.)*
## Brief change log
*(for example:)*
- *Modify AnnotationLocation checkstyle rule in checkstyle.xml*
## Verify this pull request
*(Please pick either of the following options)*
This pull request is a trivial rework / code cleanup without any test coverage.
*(or)*
This pull request is already covered by existing tests, such as *(please describe tests)*.
(or)
This change added tests and can be verified as follows:
*(example:)*
- *Added integration tests for end-to-end.*
- *Added HoodieClientWriteTest to verify the change.*
- *Manually verified the change by running a job locally.*
## Committer checklist
- [ ] Has a corresponding JIRA in PR title & commit
- [ ] Commit message is descriptive of the change
- [ ] CI is green
- [ ] Necessary doc changes done or have another open PR
- [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xushiyan commented on pull request #1769: [DOC] Add document for the use of metrics system in Hudi.
Posted by GitBox <gi...@apache.org>.
xushiyan commented on pull request #1769:
URL: https://github.com/apache/hudi/pull/1769#issuecomment-653825894
> @shenh062326 Thanks for you contributing! would you please clarify why put the metrics in a separate section?
@leesf This was discussed in https://github.com/apache/hudi/pull/1672#issuecomment-634841513
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] shenh062326 commented on a change in pull request #1769: [DOC] Add document for the use of metrics system in Hudi.
Posted by GitBox <gi...@apache.org>.
shenh062326 commented on a change in pull request #1769:
URL: https://github.com/apache/hudi/pull/1769#discussion_r456859789
##########
File path: docs/_docs/2_8_metrics.md
##########
@@ -0,0 +1,108 @@
+---
+title: Metrics Guide
+keywords: hudi, administration, operation, devops, metrics
+permalink: /docs/metrics.html
+summary: This section offers an overview of metrics in Hudi
+toc: true
+last_modified_at: 2020-06-20T15:59:57-04:00
+---
+
+In this section, We will introduce the metrics and metricsReporter in Hudi. You can view the metrics configuration [here](configurations.html#metrics-configs).
+
+## Metrics
Review comment:
Yes, I will use HoodieMetrics instand of Metrics.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] leesf commented on a change in pull request #1769: [DOC] Add document for the use of metrics system in Hudi.
Posted by GitBox <gi...@apache.org>.
leesf commented on a change in pull request #1769:
URL: https://github.com/apache/hudi/pull/1769#discussion_r453203145
##########
File path: docs/_docs/2_8_metrics.md
##########
@@ -0,0 +1,108 @@
+---
+title: Metrics Guide
+keywords: hudi, administration, operation, devops, metrics
+permalink: /docs/metrics.html
+summary: This section offers an overview of metrics in Hudi
+toc: true
+last_modified_at: 2020-06-20T15:59:57-04:00
+---
+
+In this section, We will introduce the metrics and metricsReporter in Hudi. You can view the metrics configuration [here](configurations.html#metrics-configs).
+
+## Metrics
+
+Once the Hudi writer is configured with the right table and environment for metrics, it produces the following graphite metrics, that aid in debugging hudi tables
+
+ - **Commit Duration** - This is amount of time it took to successfully commit a batch of records
+ - **Rollback Duration** - Similarly, amount of time taken to undo partial data left over by a failed commit (happens everytime automatically after a failing write)
+ - **File Level metrics** - Shows the amount of new files added, versions, deleted (cleaned) in each commit
+ - **Record Level Metrics** - Total records inserted/updated etc per commit
+ - **Partition Level metrics** - number of partitions upserted (super useful to understand sudden spikes in commit duration)
+
+These metrics can then be plotted on a standard tool like grafana. Below is a sample commit duration chart.
+
+<figure>
+ <img class="docimage" src="/assets/images/hudi_commit_duration.png" alt="hudi_commit_duration.png" style="max-width: 100%" />
+</figure>
+
+## MetricsReporter
+
+MetricsReporter is a interface for report metrics to user specified place. Currently, it's implementations has InMemoryMetricsReporter, JmxMetricsReporter, MetricsGraphiteReporter and DatadogMetricsReporter. Since InMemoryMetricsReporter is only used for testing, we will introduce the other three implementations.
+
+### JmxMetricsReporter
+
+JmxMetricsReporter is a implementation of Jmx reporter, which used to report jmx metric.
+
+#### Configurations
+The following is an example of configuration as JXM. The detailed configuration can refer to [here](configurations.html#jmx).
+
+ ```properties
+ hoodie.metrics.on=true
+ hoodie.metrics.reporter.type=JMX
+ hoodie.metrics.jmx.host=192.168.0.106
+ hoodie.metrics.jmx.port=4001
+ ```
+
+#### Demo
+As configuration above, Hudi metrics will started JMX server on port 4001. Then we can start jconsole to connect to 192.168.0.106:4001. Below is a sample of monitoring hudi jmx metrics through jconsole.
+<figure>
+ <img class="docimage" src="/assets/images/hudi_jxm_metrics.png" alt="hudi_jxm_metrics.png" style="max-width: 100%" />
+</figure>
+
+### MetricsGraphiteReporter
+
+MetricsGraphiteReporter is a implementation of Graphite reporter, which connects to the Graphite server, and send metrics to that server.
+
+#### Configurations
+The following is an example of configuration as GRAPHITE. The detailed configuration can refer to [here](configurations.html#graphite).
+
+ ```properties
+ hoodie.metrics.on=true
+ hoodie.metrics.reporter.type=GRAPHITE
+ hoodie.metrics.graphite.host=192.168.0.106
+ hoodie.metrics.graphite.port=2003
+ hoodie.metrics.graphite.metric.prefix=<your metrics prefix>
+ ```
+#### Demo
+As configuration above, we should first start graphite server on host 192.168.0.106 and port 2003, Hudi metrics will connect to graphite server, and report Hudi metrics to graphite server. Below is a sample of monitoring hudi metrics through graphite.
+ <figure>
+ <img class="docimage" src="/assets/images/hudi_graphite_metrics.png" alt="hudi_graphite_metrics.png" style="max-width: 100%" />
+ </figure>
+
+### DatadogMetricsReporter
+
+DatadogMetricsReporter is a implementation of Datadog reporter.
+A reporter which publishes metric values to Datadog monitoring service via Datadog HTTP API.
+
+#### Configurations
+The following is an example of configuration as Datadog. The detailed configuration can refer to [here](configurations.html#datadog).
+
+```properties
+hoodie.metrics.on=true
+hoodie.metrics.reporter.type=DATADOG
+hoodie.metrics.datadog.api.site=EU # or US
+hoodie.metrics.datadog.api.key=<your api key>
+hoodie.metrics.datadog.metric.prefix=<your metrics prefix>
+```
+
+ * `hoodie.metrics.datadog.api.site` will sets the Datadog API site, It determines whether the requests will be sent to api.datadoghq.eu (EU) or api.datadoghq.com (US). Set this according to your Datadog account settings.
Review comment:
`will sets the Datadog API site, `-> `will set the Datadog API site.`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] shenh062326 commented on pull request #1769: [DOC] Add document for the use of metrics system in Hudi.
Posted by GitBox <gi...@apache.org>.
shenh062326 commented on pull request #1769:
URL: https://github.com/apache/hudi/pull/1769#issuecomment-657011592
> > > @shenh062326 Thanks for you contributing! would you please clarify why put the metrics in a separate section?
> >
> >
> > @leesf This was discussed in [#1672 (comment)](https://github.com/apache/hudi/pull/1672#issuecomment-634841513)
>
> @xushiyan ack, seems like we would also move Datadog here?
Done.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] leesf commented on a change in pull request #1769: [DOC] Add document for the use of metrics system in Hudi.
Posted by GitBox <gi...@apache.org>.
leesf commented on a change in pull request #1769:
URL: https://github.com/apache/hudi/pull/1769#discussion_r453203202
##########
File path: docs/_docs/2_8_metrics.md
##########
@@ -0,0 +1,108 @@
+---
+title: Metrics Guide
+keywords: hudi, administration, operation, devops, metrics
+permalink: /docs/metrics.html
+summary: This section offers an overview of metrics in Hudi
+toc: true
+last_modified_at: 2020-06-20T15:59:57-04:00
+---
+
+In this section, We will introduce the metrics and metricsReporter in Hudi. You can view the metrics configuration [here](configurations.html#metrics-configs).
+
+## Metrics
+
+Once the Hudi writer is configured with the right table and environment for metrics, it produces the following graphite metrics, that aid in debugging hudi tables
+
+ - **Commit Duration** - This is amount of time it took to successfully commit a batch of records
+ - **Rollback Duration** - Similarly, amount of time taken to undo partial data left over by a failed commit (happens everytime automatically after a failing write)
+ - **File Level metrics** - Shows the amount of new files added, versions, deleted (cleaned) in each commit
+ - **Record Level Metrics** - Total records inserted/updated etc per commit
+ - **Partition Level metrics** - number of partitions upserted (super useful to understand sudden spikes in commit duration)
+
+These metrics can then be plotted on a standard tool like grafana. Below is a sample commit duration chart.
+
+<figure>
+ <img class="docimage" src="/assets/images/hudi_commit_duration.png" alt="hudi_commit_duration.png" style="max-width: 100%" />
+</figure>
+
+## MetricsReporter
+
+MetricsReporter is a interface for report metrics to user specified place. Currently, it's implementations has InMemoryMetricsReporter, JmxMetricsReporter, MetricsGraphiteReporter and DatadogMetricsReporter. Since InMemoryMetricsReporter is only used for testing, we will introduce the other three implementations.
+
+### JmxMetricsReporter
+
+JmxMetricsReporter is a implementation of Jmx reporter, which used to report jmx metric.
+
+#### Configurations
+The following is an example of configuration as JXM. The detailed configuration can refer to [here](configurations.html#jmx).
+
+ ```properties
+ hoodie.metrics.on=true
+ hoodie.metrics.reporter.type=JMX
+ hoodie.metrics.jmx.host=192.168.0.106
+ hoodie.metrics.jmx.port=4001
+ ```
+
+#### Demo
+As configuration above, Hudi metrics will started JMX server on port 4001. Then we can start jconsole to connect to 192.168.0.106:4001. Below is a sample of monitoring hudi jmx metrics through jconsole.
+<figure>
+ <img class="docimage" src="/assets/images/hudi_jxm_metrics.png" alt="hudi_jxm_metrics.png" style="max-width: 100%" />
+</figure>
+
+### MetricsGraphiteReporter
+
+MetricsGraphiteReporter is a implementation of Graphite reporter, which connects to the Graphite server, and send metrics to that server.
+
+#### Configurations
+The following is an example of configuration as GRAPHITE. The detailed configuration can refer to [here](configurations.html#graphite).
+
+ ```properties
+ hoodie.metrics.on=true
+ hoodie.metrics.reporter.type=GRAPHITE
+ hoodie.metrics.graphite.host=192.168.0.106
+ hoodie.metrics.graphite.port=2003
+ hoodie.metrics.graphite.metric.prefix=<your metrics prefix>
+ ```
+#### Demo
+As configuration above, we should first start graphite server on host 192.168.0.106 and port 2003, Hudi metrics will connect to graphite server, and report Hudi metrics to graphite server. Below is a sample of monitoring hudi metrics through graphite.
+ <figure>
+ <img class="docimage" src="/assets/images/hudi_graphite_metrics.png" alt="hudi_graphite_metrics.png" style="max-width: 100%" />
+ </figure>
+
+### DatadogMetricsReporter
+
+DatadogMetricsReporter is a implementation of Datadog reporter.
+A reporter which publishes metric values to Datadog monitoring service via Datadog HTTP API.
+
+#### Configurations
+The following is an example of configuration as Datadog. The detailed configuration can refer to [here](configurations.html#datadog).
+
+```properties
+hoodie.metrics.on=true
+hoodie.metrics.reporter.type=DATADOG
+hoodie.metrics.datadog.api.site=EU # or US
+hoodie.metrics.datadog.api.key=<your api key>
+hoodie.metrics.datadog.metric.prefix=<your metrics prefix>
+```
+
+ * `hoodie.metrics.datadog.api.site` will sets the Datadog API site, It determines whether the requests will be sent to api.datadoghq.eu (EU) or api.datadoghq.com (US). Set this according to your Datadog account settings.
+ * `hoodie.metrics.datadog.api.key` will set set the api key.
Review comment:
`will set set the api key.` -> ``will set the api key.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] leesf commented on a change in pull request #1769: [DOC] Add document for the use of metrics system in Hudi.
Posted by GitBox <gi...@apache.org>.
leesf commented on a change in pull request #1769:
URL: https://github.com/apache/hudi/pull/1769#discussion_r453203202
##########
File path: docs/_docs/2_8_metrics.md
##########
@@ -0,0 +1,108 @@
+---
+title: Metrics Guide
+keywords: hudi, administration, operation, devops, metrics
+permalink: /docs/metrics.html
+summary: This section offers an overview of metrics in Hudi
+toc: true
+last_modified_at: 2020-06-20T15:59:57-04:00
+---
+
+In this section, We will introduce the metrics and metricsReporter in Hudi. You can view the metrics configuration [here](configurations.html#metrics-configs).
+
+## Metrics
+
+Once the Hudi writer is configured with the right table and environment for metrics, it produces the following graphite metrics, that aid in debugging hudi tables
+
+ - **Commit Duration** - This is amount of time it took to successfully commit a batch of records
+ - **Rollback Duration** - Similarly, amount of time taken to undo partial data left over by a failed commit (happens everytime automatically after a failing write)
+ - **File Level metrics** - Shows the amount of new files added, versions, deleted (cleaned) in each commit
+ - **Record Level Metrics** - Total records inserted/updated etc per commit
+ - **Partition Level metrics** - number of partitions upserted (super useful to understand sudden spikes in commit duration)
+
+These metrics can then be plotted on a standard tool like grafana. Below is a sample commit duration chart.
+
+<figure>
+ <img class="docimage" src="/assets/images/hudi_commit_duration.png" alt="hudi_commit_duration.png" style="max-width: 100%" />
+</figure>
+
+## MetricsReporter
+
+MetricsReporter is a interface for report metrics to user specified place. Currently, it's implementations has InMemoryMetricsReporter, JmxMetricsReporter, MetricsGraphiteReporter and DatadogMetricsReporter. Since InMemoryMetricsReporter is only used for testing, we will introduce the other three implementations.
+
+### JmxMetricsReporter
+
+JmxMetricsReporter is a implementation of Jmx reporter, which used to report jmx metric.
+
+#### Configurations
+The following is an example of configuration as JXM. The detailed configuration can refer to [here](configurations.html#jmx).
+
+ ```properties
+ hoodie.metrics.on=true
+ hoodie.metrics.reporter.type=JMX
+ hoodie.metrics.jmx.host=192.168.0.106
+ hoodie.metrics.jmx.port=4001
+ ```
+
+#### Demo
+As configuration above, Hudi metrics will started JMX server on port 4001. Then we can start jconsole to connect to 192.168.0.106:4001. Below is a sample of monitoring hudi jmx metrics through jconsole.
+<figure>
+ <img class="docimage" src="/assets/images/hudi_jxm_metrics.png" alt="hudi_jxm_metrics.png" style="max-width: 100%" />
+</figure>
+
+### MetricsGraphiteReporter
+
+MetricsGraphiteReporter is a implementation of Graphite reporter, which connects to the Graphite server, and send metrics to that server.
+
+#### Configurations
+The following is an example of configuration as GRAPHITE. The detailed configuration can refer to [here](configurations.html#graphite).
+
+ ```properties
+ hoodie.metrics.on=true
+ hoodie.metrics.reporter.type=GRAPHITE
+ hoodie.metrics.graphite.host=192.168.0.106
+ hoodie.metrics.graphite.port=2003
+ hoodie.metrics.graphite.metric.prefix=<your metrics prefix>
+ ```
+#### Demo
+As configuration above, we should first start graphite server on host 192.168.0.106 and port 2003, Hudi metrics will connect to graphite server, and report Hudi metrics to graphite server. Below is a sample of monitoring hudi metrics through graphite.
+ <figure>
+ <img class="docimage" src="/assets/images/hudi_graphite_metrics.png" alt="hudi_graphite_metrics.png" style="max-width: 100%" />
+ </figure>
+
+### DatadogMetricsReporter
+
+DatadogMetricsReporter is a implementation of Datadog reporter.
+A reporter which publishes metric values to Datadog monitoring service via Datadog HTTP API.
+
+#### Configurations
+The following is an example of configuration as Datadog. The detailed configuration can refer to [here](configurations.html#datadog).
+
+```properties
+hoodie.metrics.on=true
+hoodie.metrics.reporter.type=DATADOG
+hoodie.metrics.datadog.api.site=EU # or US
+hoodie.metrics.datadog.api.key=<your api key>
+hoodie.metrics.datadog.metric.prefix=<your metrics prefix>
+```
+
+ * `hoodie.metrics.datadog.api.site` will sets the Datadog API site, It determines whether the requests will be sent to api.datadoghq.eu (EU) or api.datadoghq.com (US). Set this according to your Datadog account settings.
+ * `hoodie.metrics.datadog.api.key` will set set the api key.
Review comment:
`will set set the api key.` -> `will set the api key.`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] leesf commented on pull request #1769: [DOC] Add document for the use of metrics system in Hudi.
Posted by GitBox <gi...@apache.org>.
leesf commented on pull request #1769:
URL: https://github.com/apache/hudi/pull/1769#issuecomment-653891474
> > @shenh062326 Thanks for you contributing! would you please clarify why put the metrics in a separate section?
>
> @leesf This was discussed in [#1672 (comment)](https://github.com/apache/hudi/pull/1672#issuecomment-634841513)
@xushiyan ack, seems like we would also move Datadog here?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] leesf commented on a change in pull request #1769: [DOC] Add document for the use of metrics system in Hudi.
Posted by GitBox <gi...@apache.org>.
leesf commented on a change in pull request #1769:
URL: https://github.com/apache/hudi/pull/1769#discussion_r453203250
##########
File path: docs/_docs/2_8_metrics.md
##########
@@ -0,0 +1,108 @@
+---
+title: Metrics Guide
+keywords: hudi, administration, operation, devops, metrics
+permalink: /docs/metrics.html
+summary: This section offers an overview of metrics in Hudi
+toc: true
+last_modified_at: 2020-06-20T15:59:57-04:00
+---
+
+In this section, We will introduce the metrics and metricsReporter in Hudi. You can view the metrics configuration [here](configurations.html#metrics-configs).
+
+## Metrics
+
+Once the Hudi writer is configured with the right table and environment for metrics, it produces the following graphite metrics, that aid in debugging hudi tables
+
+ - **Commit Duration** - This is amount of time it took to successfully commit a batch of records
+ - **Rollback Duration** - Similarly, amount of time taken to undo partial data left over by a failed commit (happens everytime automatically after a failing write)
+ - **File Level metrics** - Shows the amount of new files added, versions, deleted (cleaned) in each commit
+ - **Record Level Metrics** - Total records inserted/updated etc per commit
+ - **Partition Level metrics** - number of partitions upserted (super useful to understand sudden spikes in commit duration)
+
+These metrics can then be plotted on a standard tool like grafana. Below is a sample commit duration chart.
+
+<figure>
+ <img class="docimage" src="/assets/images/hudi_commit_duration.png" alt="hudi_commit_duration.png" style="max-width: 100%" />
+</figure>
+
+## MetricsReporter
+
+MetricsReporter is a interface for report metrics to user specified place. Currently, it's implementations has InMemoryMetricsReporter, JmxMetricsReporter, MetricsGraphiteReporter and DatadogMetricsReporter. Since InMemoryMetricsReporter is only used for testing, we will introduce the other three implementations.
+
+### JmxMetricsReporter
+
+JmxMetricsReporter is a implementation of Jmx reporter, which used to report jmx metric.
+
+#### Configurations
+The following is an example of configuration as JXM. The detailed configuration can refer to [here](configurations.html#jmx).
+
+ ```properties
+ hoodie.metrics.on=true
+ hoodie.metrics.reporter.type=JMX
+ hoodie.metrics.jmx.host=192.168.0.106
+ hoodie.metrics.jmx.port=4001
+ ```
+
+#### Demo
+As configuration above, Hudi metrics will started JMX server on port 4001. Then we can start jconsole to connect to 192.168.0.106:4001. Below is a sample of monitoring hudi jmx metrics through jconsole.
+<figure>
+ <img class="docimage" src="/assets/images/hudi_jxm_metrics.png" alt="hudi_jxm_metrics.png" style="max-width: 100%" />
+</figure>
+
+### MetricsGraphiteReporter
+
+MetricsGraphiteReporter is a implementation of Graphite reporter, which connects to the Graphite server, and send metrics to that server.
+
+#### Configurations
+The following is an example of configuration as GRAPHITE. The detailed configuration can refer to [here](configurations.html#graphite).
+
+ ```properties
+ hoodie.metrics.on=true
+ hoodie.metrics.reporter.type=GRAPHITE
+ hoodie.metrics.graphite.host=192.168.0.106
+ hoodie.metrics.graphite.port=2003
+ hoodie.metrics.graphite.metric.prefix=<your metrics prefix>
+ ```
+#### Demo
+As configuration above, we should first start graphite server on host 192.168.0.106 and port 2003, Hudi metrics will connect to graphite server, and report Hudi metrics to graphite server. Below is a sample of monitoring hudi metrics through graphite.
+ <figure>
+ <img class="docimage" src="/assets/images/hudi_graphite_metrics.png" alt="hudi_graphite_metrics.png" style="max-width: 100%" />
+ </figure>
+
+### DatadogMetricsReporter
+
+DatadogMetricsReporter is a implementation of Datadog reporter.
+A reporter which publishes metric values to Datadog monitoring service via Datadog HTTP API.
+
+#### Configurations
+The following is an example of configuration as Datadog. The detailed configuration can refer to [here](configurations.html#datadog).
+
+```properties
+hoodie.metrics.on=true
+hoodie.metrics.reporter.type=DATADOG
+hoodie.metrics.datadog.api.site=EU # or US
+hoodie.metrics.datadog.api.key=<your api key>
+hoodie.metrics.datadog.metric.prefix=<your metrics prefix>
+```
+
+ * `hoodie.metrics.datadog.api.site` will sets the Datadog API site, It determines whether the requests will be sent to api.datadoghq.eu (EU) or api.datadoghq.com (US). Set this according to your Datadog account settings.
+ * `hoodie.metrics.datadog.api.key` will set set the api key.
+ * `hoodie.metrics.datadog.metric.prefix` will helps segregate metrics by setting different prefixes for different jobs.Note that it will use `.` to delimit the prefix and the metric name. For example, if the prefix is set to foo, then foo. will be prepended to the metric name.
Review comment:
helps -> help and one blank before `Note that`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] shenh062326 commented on a change in pull request #1769: [DOC] Add document for the use of metrics system in Hudi.
Posted by GitBox <gi...@apache.org>.
shenh062326 commented on a change in pull request #1769:
URL: https://github.com/apache/hudi/pull/1769#discussion_r456852020
##########
File path: docs/_docs/2_8_metrics.md
##########
@@ -0,0 +1,108 @@
+---
+title: Metrics Guide
+keywords: hudi, administration, operation, devops, metrics
+permalink: /docs/metrics.html
+summary: This section offers an overview of metrics in Hudi
+toc: true
+last_modified_at: 2020-06-20T15:59:57-04:00
+---
+
+In this section, We will introduce the metrics and metricsReporter in Hudi. You can view the metrics configuration [here](configurations.html#metrics-configs).
Review comment:
Yes, thanks for you comments, I will fix it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xushiyan commented on a change in pull request #1769: [DOC] Add document for the use of metrics system in Hudi.
Posted by GitBox <gi...@apache.org>.
xushiyan commented on a change in pull request #1769:
URL: https://github.com/apache/hudi/pull/1769#discussion_r456772276
##########
File path: docs/_docs/2_8_metrics.md
##########
@@ -0,0 +1,108 @@
+---
+title: Metrics Guide
+keywords: hudi, administration, operation, devops, metrics
+permalink: /docs/metrics.html
+summary: This section offers an overview of metrics in Hudi
+toc: true
+last_modified_at: 2020-06-20T15:59:57-04:00
+---
+
+In this section, We will introduce the metrics and metricsReporter in Hudi. You can view the metrics configuration [here](configurations.html#metrics-configs).
Review comment:
```suggestion
In this section, we will introduce `HoodieMetrics` and `MetricsReporter` in Hudi. You can view the metrics-related configurations [here](configurations.html#metrics-configs).
```
##########
File path: docs/_docs/2_8_metrics.md
##########
@@ -0,0 +1,108 @@
+---
+title: Metrics Guide
+keywords: hudi, administration, operation, devops, metrics
+permalink: /docs/metrics.html
+summary: This section offers an overview of metrics in Hudi
+toc: true
+last_modified_at: 2020-06-20T15:59:57-04:00
+---
+
+In this section, We will introduce the metrics and metricsReporter in Hudi. You can view the metrics configuration [here](configurations.html#metrics-configs).
+
+## Metrics
+
+Once the Hudi writer is configured with the right table and environment for metrics, it produces the following graphite metrics, that aid in debugging hudi tables
Review comment:
> configured with the right table and environment for metrics
Do you think it'll be clearer to illustrate how to achieve this before moving on to the followings?
##########
File path: docs/_docs/2_8_metrics.md
##########
@@ -0,0 +1,108 @@
+---
+title: Metrics Guide
+keywords: hudi, administration, operation, devops, metrics
+permalink: /docs/metrics.html
+summary: This section offers an overview of metrics in Hudi
+toc: true
+last_modified_at: 2020-06-20T15:59:57-04:00
+---
+
+In this section, We will introduce the metrics and metricsReporter in Hudi. You can view the metrics configuration [here](configurations.html#metrics-configs).
+
+## Metrics
Review comment:
Did you mean to introduce `HoodieMetrics` ? if so, please align on the class name for the section
##########
File path: docs/_docs/2_8_metrics.md
##########
@@ -0,0 +1,108 @@
+---
+title: Metrics Guide
+keywords: hudi, administration, operation, devops, metrics
+permalink: /docs/metrics.html
+summary: This section offers an overview of metrics in Hudi
+toc: true
+last_modified_at: 2020-06-20T15:59:57-04:00
+---
+
+In this section, We will introduce the metrics and metricsReporter in Hudi. You can view the metrics configuration [here](configurations.html#metrics-configs).
+
+## Metrics
+
+Once the Hudi writer is configured with the right table and environment for metrics, it produces the following graphite metrics, that aid in debugging hudi tables
+
+ - **Commit Duration** - This is amount of time it took to successfully commit a batch of records
+ - **Rollback Duration** - Similarly, amount of time taken to undo partial data left over by a failed commit (happens everytime automatically after a failing write)
+ - **File Level metrics** - Shows the amount of new files added, versions, deleted (cleaned) in each commit
+ - **Record Level Metrics** - Total records inserted/updated etc per commit
+ - **Partition Level metrics** - number of partitions upserted (super useful to understand sudden spikes in commit duration)
+
+These metrics can then be plotted on a standard tool like grafana. Below is a sample commit duration chart.
+
+<figure>
+ <img class="docimage" src="/assets/images/hudi_commit_duration.png" alt="hudi_commit_duration.png" style="max-width: 100%" />
+</figure>
+
+## MetricsReporter
+
+MetricsReporter is a interface for report metrics to user specified place. Currently, it's implementations has InMemoryMetricsReporter, JmxMetricsReporter, MetricsGraphiteReporter and DatadogMetricsReporter. Since InMemoryMetricsReporter is only used for testing, we will introduce the other three implementations.
+
+### JmxMetricsReporter
+
+JmxMetricsReporter is a implementation of Jmx reporter, which used to report jmx metric.
+
+#### Configurations
+The following is an example of configuration as JXM. The detailed configuration can refer to [here](configurations.html#jmx).
+
+ ```properties
+ hoodie.metrics.on=true
+ hoodie.metrics.reporter.type=JMX
+ hoodie.metrics.jmx.host=192.168.0.106
+ hoodie.metrics.jmx.port=4001
+ ```
+
+#### Demo
+As configuration above, Hudi metrics will started JMX server on port 4001. Then we can start jconsole to connect to 192.168.0.106:4001. Below is a sample of monitoring hudi jmx metrics through jconsole.
+<figure>
+ <img class="docimage" src="/assets/images/hudi_jxm_metrics.png" alt="hudi_jxm_metrics.png" style="max-width: 100%" />
+</figure>
+
+### MetricsGraphiteReporter
+
+MetricsGraphiteReporter is a implementation of Graphite reporter, which connects to the Graphite server, and send metrics to that server.
+
+#### Configurations
+The following is an example of configuration as GRAPHITE. The detailed configuration can refer to [here](configurations.html#graphite).
+
+ ```properties
+ hoodie.metrics.on=true
+ hoodie.metrics.reporter.type=GRAPHITE
+ hoodie.metrics.graphite.host=192.168.0.106
+ hoodie.metrics.graphite.port=2003
+ hoodie.metrics.graphite.metric.prefix=<your metrics prefix>
+ ```
+#### Demo
+As configuration above, we should first start graphite server on host 192.168.0.106 and port 2003, Hudi metrics will connect to graphite server, and report Hudi metrics to graphite server. Below is a sample of monitoring hudi metrics through graphite.
+ <figure>
+ <img class="docimage" src="/assets/images/hudi_graphite_metrics.png" alt="hudi_graphite_metrics.png" style="max-width: 100%" />
+ </figure>
+
+### DatadogMetricsReporter
+
+DatadogMetricsReporter is a implementation of Datadog reporter.
+A reporter which publishes metric values to Datadog monitoring service via Datadog HTTP API.
+
+#### Configurations
+The following is an example of configuration as Datadog. The detailed configuration can refer to [here](configurations.html#datadog).
+
+```properties
+hoodie.metrics.on=true
+hoodie.metrics.reporter.type=DATADOG
+hoodie.metrics.datadog.api.site=EU # or US
+hoodie.metrics.datadog.api.key=<your api key>
+hoodie.metrics.datadog.metric.prefix=<your metrics prefix>
+```
+
+ * `hoodie.metrics.datadog.api.site` will set the Datadog API site, It determines whether the requests will be sent to api.datadoghq.eu (EU) or api.datadoghq.com (US). Set this according to your Datadog account settings.
+ * `hoodie.metrics.datadog.api.key` will set the api key.
+ * `hoodie.metrics.datadog.metric.prefix` will help segregate metrics by setting different prefixes for different jobs.Note that it will use `.` to delimit the prefix and the metric name. For example, if the prefix is set to foo, then foo. will be prepended to the metric name.
Review comment:
```suggestion
* `hoodie.metrics.datadog.metric.prefix` will help segregate metrics by setting different prefixes for different jobs. Note that it will use `.` to delimit the prefix and the metric name. For example, if the prefix is set to `foo`, then `foo.` will be prepended to the metric name.
```
##########
File path: docs/_docs/2_8_metrics.md
##########
@@ -0,0 +1,108 @@
+---
+title: Metrics Guide
+keywords: hudi, administration, operation, devops, metrics
+permalink: /docs/metrics.html
+summary: This section offers an overview of metrics in Hudi
+toc: true
+last_modified_at: 2020-06-20T15:59:57-04:00
+---
+
+In this section, We will introduce the metrics and metricsReporter in Hudi. You can view the metrics configuration [here](configurations.html#metrics-configs).
+
+## Metrics
+
+Once the Hudi writer is configured with the right table and environment for metrics, it produces the following graphite metrics, that aid in debugging hudi tables
+
+ - **Commit Duration** - This is amount of time it took to successfully commit a batch of records
+ - **Rollback Duration** - Similarly, amount of time taken to undo partial data left over by a failed commit (happens everytime automatically after a failing write)
Review comment:
```suggestion
- **Rollback Duration** - Similarly, the amount of time taken to undo partial data left over by a failed commit (rollback happens automatically after a failing write)
```
##########
File path: docs/_docs/2_8_metrics.md
##########
@@ -0,0 +1,108 @@
+---
+title: Metrics Guide
+keywords: hudi, administration, operation, devops, metrics
+permalink: /docs/metrics.html
+summary: This section offers an overview of metrics in Hudi
+toc: true
+last_modified_at: 2020-06-20T15:59:57-04:00
+---
+
+In this section, We will introduce the metrics and metricsReporter in Hudi. You can view the metrics configuration [here](configurations.html#metrics-configs).
+
+## Metrics
+
+Once the Hudi writer is configured with the right table and environment for metrics, it produces the following graphite metrics, that aid in debugging hudi tables
+
+ - **Commit Duration** - This is amount of time it took to successfully commit a batch of records
+ - **Rollback Duration** - Similarly, amount of time taken to undo partial data left over by a failed commit (happens everytime automatically after a failing write)
+ - **File Level metrics** - Shows the amount of new files added, versions, deleted (cleaned) in each commit
+ - **Record Level Metrics** - Total records inserted/updated etc per commit
+ - **Partition Level metrics** - number of partitions upserted (super useful to understand sudden spikes in commit duration)
+
+These metrics can then be plotted on a standard tool like grafana. Below is a sample commit duration chart.
+
+<figure>
+ <img class="docimage" src="/assets/images/hudi_commit_duration.png" alt="hudi_commit_duration.png" style="max-width: 100%" />
+</figure>
+
+## MetricsReporter
+
+MetricsReporter is a interface for report metrics to user specified place. Currently, it's implementations has InMemoryMetricsReporter, JmxMetricsReporter, MetricsGraphiteReporter and DatadogMetricsReporter. Since InMemoryMetricsReporter is only used for testing, we will introduce the other three implementations.
Review comment:
```suggestion
MetricsReporter provides APIs for reporting metrics to user-specified backends. Currently, the implementations include InMemoryMetricsReporter, JmxMetricsReporter, MetricsGraphiteReporter and DatadogMetricsReporter. Since InMemoryMetricsReporter is only used for testing, we will introduce the other three implementations.
```
##########
File path: docs/_docs/2_8_metrics.md
##########
@@ -0,0 +1,108 @@
+---
+title: Metrics Guide
+keywords: hudi, administration, operation, devops, metrics
+permalink: /docs/metrics.html
+summary: This section offers an overview of metrics in Hudi
+toc: true
+last_modified_at: 2020-06-20T15:59:57-04:00
+---
+
+In this section, We will introduce the metrics and metricsReporter in Hudi. You can view the metrics configuration [here](configurations.html#metrics-configs).
+
+## Metrics
+
+Once the Hudi writer is configured with the right table and environment for metrics, it produces the following graphite metrics, that aid in debugging hudi tables
+
+ - **Commit Duration** - This is amount of time it took to successfully commit a batch of records
Review comment:
```suggestion
- **Commit Duration** - the amount of time it took to successfully commit a batch of records
```
##########
File path: docs/_docs/2_8_metrics.md
##########
@@ -0,0 +1,108 @@
+---
+title: Metrics Guide
+keywords: hudi, administration, operation, devops, metrics
+permalink: /docs/metrics.html
+summary: This section offers an overview of metrics in Hudi
+toc: true
+last_modified_at: 2020-06-20T15:59:57-04:00
+---
+
+In this section, We will introduce the metrics and metricsReporter in Hudi. You can view the metrics configuration [here](configurations.html#metrics-configs).
+
+## Metrics
+
+Once the Hudi writer is configured with the right table and environment for metrics, it produces the following graphite metrics, that aid in debugging hudi tables
+
+ - **Commit Duration** - This is amount of time it took to successfully commit a batch of records
+ - **Rollback Duration** - Similarly, amount of time taken to undo partial data left over by a failed commit (happens everytime automatically after a failing write)
+ - **File Level metrics** - Shows the amount of new files added, versions, deleted (cleaned) in each commit
+ - **Record Level Metrics** - Total records inserted/updated etc per commit
+ - **Partition Level metrics** - number of partitions upserted (super useful to understand sudden spikes in commit duration)
+
+These metrics can then be plotted on a standard tool like grafana. Below is a sample commit duration chart.
+
+<figure>
+ <img class="docimage" src="/assets/images/hudi_commit_duration.png" alt="hudi_commit_duration.png" style="max-width: 100%" />
+</figure>
+
+## MetricsReporter
+
+MetricsReporter is a interface for report metrics to user specified place. Currently, it's implementations has InMemoryMetricsReporter, JmxMetricsReporter, MetricsGraphiteReporter and DatadogMetricsReporter. Since InMemoryMetricsReporter is only used for testing, we will introduce the other three implementations.
+
+### JmxMetricsReporter
+
+JmxMetricsReporter is a implementation of Jmx reporter, which used to report jmx metric.
+
+#### Configurations
+The following is an example of configuration as JXM. The detailed configuration can refer to [here](configurations.html#jmx).
+
+ ```properties
+ hoodie.metrics.on=true
+ hoodie.metrics.reporter.type=JMX
+ hoodie.metrics.jmx.host=192.168.0.106
+ hoodie.metrics.jmx.port=4001
+ ```
+
+#### Demo
+As configuration above, Hudi metrics will started JMX server on port 4001. Then we can start jconsole to connect to 192.168.0.106:4001. Below is a sample of monitoring hudi jmx metrics through jconsole.
+<figure>
+ <img class="docimage" src="/assets/images/hudi_jxm_metrics.png" alt="hudi_jxm_metrics.png" style="max-width: 100%" />
+</figure>
+
+### MetricsGraphiteReporter
+
+MetricsGraphiteReporter is a implementation of Graphite reporter, which connects to the Graphite server, and send metrics to that server.
+
+#### Configurations
+The following is an example of configuration as GRAPHITE. The detailed configuration can refer to [here](configurations.html#graphite).
+
+ ```properties
+ hoodie.metrics.on=true
+ hoodie.metrics.reporter.type=GRAPHITE
+ hoodie.metrics.graphite.host=192.168.0.106
+ hoodie.metrics.graphite.port=2003
+ hoodie.metrics.graphite.metric.prefix=<your metrics prefix>
+ ```
+#### Demo
+As configuration above, we should first start graphite server on host 192.168.0.106 and port 2003, Hudi metrics will connect to graphite server, and report Hudi metrics to graphite server. Below is a sample of monitoring hudi metrics through graphite.
+ <figure>
+ <img class="docimage" src="/assets/images/hudi_graphite_metrics.png" alt="hudi_graphite_metrics.png" style="max-width: 100%" />
+ </figure>
+
+### DatadogMetricsReporter
+
+DatadogMetricsReporter is a implementation of Datadog reporter.
+A reporter which publishes metric values to Datadog monitoring service via Datadog HTTP API.
+
+#### Configurations
+The following is an example of configuration as Datadog. The detailed configuration can refer to [here](configurations.html#datadog).
+
+```properties
+hoodie.metrics.on=true
+hoodie.metrics.reporter.type=DATADOG
+hoodie.metrics.datadog.api.site=EU # or US
+hoodie.metrics.datadog.api.key=<your api key>
+hoodie.metrics.datadog.metric.prefix=<your metrics prefix>
+```
+
+ * `hoodie.metrics.datadog.api.site` will set the Datadog API site, It determines whether the requests will be sent to api.datadoghq.eu (EU) or api.datadoghq.com (US). Set this according to your Datadog account settings.
Review comment:
```suggestion
* `hoodie.metrics.datadog.api.site` will set the Datadog API site, which determines whether the requests will be sent to api.datadoghq.eu (EU) or api.datadoghq.com (US). Set this according to your Datadog account settings.
```
##########
File path: docs/_docs/2_8_metrics.md
##########
@@ -0,0 +1,108 @@
+---
+title: Metrics Guide
+keywords: hudi, administration, operation, devops, metrics
+permalink: /docs/metrics.html
+summary: This section offers an overview of metrics in Hudi
+toc: true
+last_modified_at: 2020-06-20T15:59:57-04:00
+---
+
+In this section, We will introduce the metrics and metricsReporter in Hudi. You can view the metrics configuration [here](configurations.html#metrics-configs).
+
+## Metrics
+
+Once the Hudi writer is configured with the right table and environment for metrics, it produces the following graphite metrics, that aid in debugging hudi tables
Review comment:
> produces the following graphite metrics
Why is it graphite metrics? this is implementation-agnostic metrics right?
##########
File path: docs/_docs/2_8_metrics.md
##########
@@ -0,0 +1,108 @@
+---
+title: Metrics Guide
+keywords: hudi, administration, operation, devops, metrics
+permalink: /docs/metrics.html
+summary: This section offers an overview of metrics in Hudi
+toc: true
+last_modified_at: 2020-06-20T15:59:57-04:00
+---
+
+In this section, We will introduce the metrics and metricsReporter in Hudi. You can view the metrics configuration [here](configurations.html#metrics-configs).
Review comment:
you meant to introduce `HoodieMetrics` right?
##########
File path: docs/_docs/2_8_metrics.md
##########
@@ -0,0 +1,108 @@
+---
+title: Metrics Guide
+keywords: hudi, administration, operation, devops, metrics
+permalink: /docs/metrics.html
+summary: This section offers an overview of metrics in Hudi
+toc: true
+last_modified_at: 2020-06-20T15:59:57-04:00
+---
+
+In this section, We will introduce the metrics and metricsReporter in Hudi. You can view the metrics configuration [here](configurations.html#metrics-configs).
+
+## Metrics
+
+Once the Hudi writer is configured with the right table and environment for metrics, it produces the following graphite metrics, that aid in debugging hudi tables
+
+ - **Commit Duration** - This is amount of time it took to successfully commit a batch of records
+ - **Rollback Duration** - Similarly, amount of time taken to undo partial data left over by a failed commit (happens everytime automatically after a failing write)
+ - **File Level metrics** - Shows the amount of new files added, versions, deleted (cleaned) in each commit
+ - **Record Level Metrics** - Total records inserted/updated etc per commit
+ - **Partition Level metrics** - number of partitions upserted (super useful to understand sudden spikes in commit duration)
+
+These metrics can then be plotted on a standard tool like grafana. Below is a sample commit duration chart.
+
+<figure>
+ <img class="docimage" src="/assets/images/hudi_commit_duration.png" alt="hudi_commit_duration.png" style="max-width: 100%" />
+</figure>
+
+## MetricsReporter
+
+MetricsReporter is a interface for report metrics to user specified place. Currently, it's implementations has InMemoryMetricsReporter, JmxMetricsReporter, MetricsGraphiteReporter and DatadogMetricsReporter. Since InMemoryMetricsReporter is only used for testing, we will introduce the other three implementations.
+
+### JmxMetricsReporter
+
+JmxMetricsReporter is a implementation of Jmx reporter, which used to report jmx metric.
+
+#### Configurations
+The following is an example of configuration as JXM. The detailed configuration can refer to [here](configurations.html#jmx).
+
+ ```properties
+ hoodie.metrics.on=true
+ hoodie.metrics.reporter.type=JMX
+ hoodie.metrics.jmx.host=192.168.0.106
+ hoodie.metrics.jmx.port=4001
+ ```
+
+#### Demo
+As configuration above, Hudi metrics will started JMX server on port 4001. Then we can start jconsole to connect to 192.168.0.106:4001. Below is a sample of monitoring hudi jmx metrics through jconsole.
+<figure>
+ <img class="docimage" src="/assets/images/hudi_jxm_metrics.png" alt="hudi_jxm_metrics.png" style="max-width: 100%" />
+</figure>
+
+### MetricsGraphiteReporter
+
+MetricsGraphiteReporter is a implementation of Graphite reporter, which connects to the Graphite server, and send metrics to that server.
Review comment:
```suggestion
MetricsGraphiteReporter is an implementation of Graphite reporter, which connects to a Graphite server and send metrics to it.
```
##########
File path: docs/_docs/2_8_metrics.md
##########
@@ -0,0 +1,108 @@
+---
+title: Metrics Guide
+keywords: hudi, administration, operation, devops, metrics
+permalink: /docs/metrics.html
+summary: This section offers an overview of metrics in Hudi
+toc: true
+last_modified_at: 2020-06-20T15:59:57-04:00
+---
+
+In this section, We will introduce the metrics and metricsReporter in Hudi. You can view the metrics configuration [here](configurations.html#metrics-configs).
+
+## Metrics
+
+Once the Hudi writer is configured with the right table and environment for metrics, it produces the following graphite metrics, that aid in debugging hudi tables
+
+ - **Commit Duration** - This is amount of time it took to successfully commit a batch of records
+ - **Rollback Duration** - Similarly, amount of time taken to undo partial data left over by a failed commit (happens everytime automatically after a failing write)
+ - **File Level metrics** - Shows the amount of new files added, versions, deleted (cleaned) in each commit
+ - **Record Level Metrics** - Total records inserted/updated etc per commit
+ - **Partition Level metrics** - number of partitions upserted (super useful to understand sudden spikes in commit duration)
+
+These metrics can then be plotted on a standard tool like grafana. Below is a sample commit duration chart.
+
+<figure>
+ <img class="docimage" src="/assets/images/hudi_commit_duration.png" alt="hudi_commit_duration.png" style="max-width: 100%" />
+</figure>
+
+## MetricsReporter
+
+MetricsReporter is a interface for report metrics to user specified place. Currently, it's implementations has InMemoryMetricsReporter, JmxMetricsReporter, MetricsGraphiteReporter and DatadogMetricsReporter. Since InMemoryMetricsReporter is only used for testing, we will introduce the other three implementations.
+
+### JmxMetricsReporter
+
+JmxMetricsReporter is a implementation of Jmx reporter, which used to report jmx metric.
+
+#### Configurations
+The following is an example of configuration as JXM. The detailed configuration can refer to [here](configurations.html#jmx).
+
+ ```properties
+ hoodie.metrics.on=true
+ hoodie.metrics.reporter.type=JMX
+ hoodie.metrics.jmx.host=192.168.0.106
+ hoodie.metrics.jmx.port=4001
+ ```
+
+#### Demo
+As configuration above, Hudi metrics will started JMX server on port 4001. Then we can start jconsole to connect to 192.168.0.106:4001. Below is a sample of monitoring hudi jmx metrics through jconsole.
+<figure>
+ <img class="docimage" src="/assets/images/hudi_jxm_metrics.png" alt="hudi_jxm_metrics.png" style="max-width: 100%" />
+</figure>
+
+### MetricsGraphiteReporter
+
+MetricsGraphiteReporter is a implementation of Graphite reporter, which connects to the Graphite server, and send metrics to that server.
+
+#### Configurations
+The following is an example of configuration as GRAPHITE. The detailed configuration can refer to [here](configurations.html#graphite).
Review comment:
```suggestion
The following is an example of `MetricsGraphiteReporter`. More detailed configurations can be referenced [here](configurations.html#graphite).
```
##########
File path: docs/_docs/2_8_metrics.md
##########
@@ -0,0 +1,108 @@
+---
+title: Metrics Guide
+keywords: hudi, administration, operation, devops, metrics
+permalink: /docs/metrics.html
+summary: This section offers an overview of metrics in Hudi
+toc: true
+last_modified_at: 2020-06-20T15:59:57-04:00
+---
+
+In this section, We will introduce the metrics and metricsReporter in Hudi. You can view the metrics configuration [here](configurations.html#metrics-configs).
+
+## Metrics
+
+Once the Hudi writer is configured with the right table and environment for metrics, it produces the following graphite metrics, that aid in debugging hudi tables
+
+ - **Commit Duration** - This is amount of time it took to successfully commit a batch of records
+ - **Rollback Duration** - Similarly, amount of time taken to undo partial data left over by a failed commit (happens everytime automatically after a failing write)
+ - **File Level metrics** - Shows the amount of new files added, versions, deleted (cleaned) in each commit
+ - **Record Level Metrics** - Total records inserted/updated etc per commit
+ - **Partition Level metrics** - number of partitions upserted (super useful to understand sudden spikes in commit duration)
+
+These metrics can then be plotted on a standard tool like grafana. Below is a sample commit duration chart.
+
+<figure>
+ <img class="docimage" src="/assets/images/hudi_commit_duration.png" alt="hudi_commit_duration.png" style="max-width: 100%" />
+</figure>
+
+## MetricsReporter
+
+MetricsReporter is a interface for report metrics to user specified place. Currently, it's implementations has InMemoryMetricsReporter, JmxMetricsReporter, MetricsGraphiteReporter and DatadogMetricsReporter. Since InMemoryMetricsReporter is only used for testing, we will introduce the other three implementations.
+
+### JmxMetricsReporter
+
+JmxMetricsReporter is a implementation of Jmx reporter, which used to report jmx metric.
+
+#### Configurations
+The following is an example of configuration as JXM. The detailed configuration can refer to [here](configurations.html#jmx).
+
+ ```properties
+ hoodie.metrics.on=true
+ hoodie.metrics.reporter.type=JMX
+ hoodie.metrics.jmx.host=192.168.0.106
+ hoodie.metrics.jmx.port=4001
+ ```
+
+#### Demo
+As configuration above, Hudi metrics will started JMX server on port 4001. Then we can start jconsole to connect to 192.168.0.106:4001. Below is a sample of monitoring hudi jmx metrics through jconsole.
+<figure>
+ <img class="docimage" src="/assets/images/hudi_jxm_metrics.png" alt="hudi_jxm_metrics.png" style="max-width: 100%" />
+</figure>
+
+### MetricsGraphiteReporter
+
+MetricsGraphiteReporter is a implementation of Graphite reporter, which connects to the Graphite server, and send metrics to that server.
+
+#### Configurations
+The following is an example of configuration as GRAPHITE. The detailed configuration can refer to [here](configurations.html#graphite).
+
+ ```properties
+ hoodie.metrics.on=true
+ hoodie.metrics.reporter.type=GRAPHITE
+ hoodie.metrics.graphite.host=192.168.0.106
+ hoodie.metrics.graphite.port=2003
+ hoodie.metrics.graphite.metric.prefix=<your metrics prefix>
+ ```
+#### Demo
+As configuration above, we should first start graphite server on host 192.168.0.106 and port 2003, Hudi metrics will connect to graphite server, and report Hudi metrics to graphite server. Below is a sample of monitoring hudi metrics through graphite.
Review comment:
```suggestion
As configured above, assuming a Graphite server is running on host 192.168.0.106 and port 2003, a running Hudi job will connect and report metrics data to it. Below is an illustration of monitoring Hudi metrics through Graphite.
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] shenh062326 commented on pull request #1769: [DOC] Add document for the use of metrics system in Hudi.
Posted by GitBox <gi...@apache.org>.
shenh062326 commented on pull request #1769:
URL: https://github.com/apache/hudi/pull/1769#issuecomment-653892441
@leesf Yes, I will add document for Datadog.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] leesf merged pull request #1769: [DOC] Add document for the use of metrics system in Hudi.
Posted by GitBox <gi...@apache.org>.
leesf merged pull request #1769:
URL: https://github.com/apache/hudi/pull/1769
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] leesf commented on a change in pull request #1769: [DOC] Add document for the use of metrics system in Hudi.
Posted by GitBox <gi...@apache.org>.
leesf commented on a change in pull request #1769:
URL: https://github.com/apache/hudi/pull/1769#discussion_r453203355
##########
File path: docs/_docs/2_8_metrics.md
##########
@@ -0,0 +1,108 @@
+---
+title: Metrics Guide
+keywords: hudi, administration, operation, devops, metrics
+permalink: /docs/metrics.html
+summary: This section offers an overview of metrics in Hudi
+toc: true
+last_modified_at: 2020-06-20T15:59:57-04:00
+---
+
+In this section, We will introduce the metrics and metricsReporter in Hudi. You can view the metrics configuration [here](configurations.html#metrics-configs).
+
+## Metrics
+
+Once the Hudi writer is configured with the right table and environment for metrics, it produces the following graphite metrics, that aid in debugging hudi tables
+
+ - **Commit Duration** - This is amount of time it took to successfully commit a batch of records
+ - **Rollback Duration** - Similarly, amount of time taken to undo partial data left over by a failed commit (happens everytime automatically after a failing write)
+ - **File Level metrics** - Shows the amount of new files added, versions, deleted (cleaned) in each commit
+ - **Record Level Metrics** - Total records inserted/updated etc per commit
+ - **Partition Level metrics** - number of partitions upserted (super useful to understand sudden spikes in commit duration)
+
+These metrics can then be plotted on a standard tool like grafana. Below is a sample commit duration chart.
+
+<figure>
+ <img class="docimage" src="/assets/images/hudi_commit_duration.png" alt="hudi_commit_duration.png" style="max-width: 100%" />
+</figure>
+
+## MetricsReporter
+
+MetricsReporter is a interface for report metrics to user specified place. Currently, it's implementations has InMemoryMetricsReporter, JmxMetricsReporter, MetricsGraphiteReporter and DatadogMetricsReporter. Since InMemoryMetricsReporter is only used for testing, we will introduce the other three implementations.
+
+### JmxMetricsReporter
+
+JmxMetricsReporter is a implementation of Jmx reporter, which used to report jmx metric.
+
+#### Configurations
+The following is an example of configuration as JXM. The detailed configuration can refer to [here](configurations.html#jmx).
+
+ ```properties
+ hoodie.metrics.on=true
+ hoodie.metrics.reporter.type=JMX
+ hoodie.metrics.jmx.host=192.168.0.106
+ hoodie.metrics.jmx.port=4001
+ ```
+
+#### Demo
+As configuration above, Hudi metrics will started JMX server on port 4001. Then we can start jconsole to connect to 192.168.0.106:4001. Below is a sample of monitoring hudi jmx metrics through jconsole.
+<figure>
+ <img class="docimage" src="/assets/images/hudi_jxm_metrics.png" alt="hudi_jxm_metrics.png" style="max-width: 100%" />
+</figure>
+
+### MetricsGraphiteReporter
+
+MetricsGraphiteReporter is a implementation of Graphite reporter, which connects to the Graphite server, and send metrics to that server.
+
+#### Configurations
+The following is an example of configuration as GRAPHITE. The detailed configuration can refer to [here](configurations.html#graphite).
+
+ ```properties
+ hoodie.metrics.on=true
+ hoodie.metrics.reporter.type=GRAPHITE
+ hoodie.metrics.graphite.host=192.168.0.106
+ hoodie.metrics.graphite.port=2003
+ hoodie.metrics.graphite.metric.prefix=<your metrics prefix>
+ ```
+#### Demo
+As configuration above, we should first start graphite server on host 192.168.0.106 and port 2003, Hudi metrics will connect to graphite server, and report Hudi metrics to graphite server. Below is a sample of monitoring hudi metrics through graphite.
+ <figure>
+ <img class="docimage" src="/assets/images/hudi_graphite_metrics.png" alt="hudi_graphite_metrics.png" style="max-width: 100%" />
+ </figure>
+
+### DatadogMetricsReporter
+
+DatadogMetricsReporter is a implementation of Datadog reporter.
+A reporter which publishes metric values to Datadog monitoring service via Datadog HTTP API.
+
+#### Configurations
+The following is an example of configuration as Datadog. The detailed configuration can refer to [here](configurations.html#datadog).
+
+```properties
+hoodie.metrics.on=true
+hoodie.metrics.reporter.type=DATADOG
+hoodie.metrics.datadog.api.site=EU # or US
+hoodie.metrics.datadog.api.key=<your api key>
+hoodie.metrics.datadog.metric.prefix=<your metrics prefix>
+```
+
+ * `hoodie.metrics.datadog.api.site` will sets the Datadog API site, It determines whether the requests will be sent to api.datadoghq.eu (EU) or api.datadoghq.com (US). Set this according to your Datadog account settings.
+ * `hoodie.metrics.datadog.api.key` will set set the api key.
+ * `hoodie.metrics.datadog.metric.prefix` will helps segregate metrics by setting different prefixes for different jobs.Note that it will use `.` to delimit the prefix and the metric name. For example, if the prefix is set to foo, then foo. will be prepended to the metric name.
+
+#### Demo
+In this demo, we ran a `HoodieDeltaStreamer` job with metrics turn on and other configurations set properly.
Review comment:
turn on -> turned on
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] shenh062326 commented on a change in pull request #1769: [DOC] Add document for the use of metrics system in Hudi.
Posted by GitBox <gi...@apache.org>.
shenh062326 commented on a change in pull request #1769:
URL: https://github.com/apache/hudi/pull/1769#discussion_r456859658
##########
File path: docs/_docs/2_8_metrics.md
##########
@@ -0,0 +1,108 @@
+---
+title: Metrics Guide
+keywords: hudi, administration, operation, devops, metrics
+permalink: /docs/metrics.html
+summary: This section offers an overview of metrics in Hudi
+toc: true
+last_modified_at: 2020-06-20T15:59:57-04:00
+---
+
+In this section, We will introduce the metrics and metricsReporter in Hudi. You can view the metrics configuration [here](configurations.html#metrics-configs).
+
+## Metrics
+
+Once the Hudi writer is configured with the right table and environment for metrics, it produces the following graphite metrics, that aid in debugging hudi tables
Review comment:
> > configured with the right table and environment for metrics
>
> Do you think it'll be clearer to illustrate how to achieve this before moving on to the followings?
I will move this part to the end.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] leesf commented on a change in pull request #1769: [DOC] Add document for the use of metrics system in Hudi.
Posted by GitBox <gi...@apache.org>.
leesf commented on a change in pull request #1769:
URL: https://github.com/apache/hudi/pull/1769#discussion_r453203250
##########
File path: docs/_docs/2_8_metrics.md
##########
@@ -0,0 +1,108 @@
+---
+title: Metrics Guide
+keywords: hudi, administration, operation, devops, metrics
+permalink: /docs/metrics.html
+summary: This section offers an overview of metrics in Hudi
+toc: true
+last_modified_at: 2020-06-20T15:59:57-04:00
+---
+
+In this section, We will introduce the metrics and metricsReporter in Hudi. You can view the metrics configuration [here](configurations.html#metrics-configs).
+
+## Metrics
+
+Once the Hudi writer is configured with the right table and environment for metrics, it produces the following graphite metrics, that aid in debugging hudi tables
+
+ - **Commit Duration** - This is amount of time it took to successfully commit a batch of records
+ - **Rollback Duration** - Similarly, amount of time taken to undo partial data left over by a failed commit (happens everytime automatically after a failing write)
+ - **File Level metrics** - Shows the amount of new files added, versions, deleted (cleaned) in each commit
+ - **Record Level Metrics** - Total records inserted/updated etc per commit
+ - **Partition Level metrics** - number of partitions upserted (super useful to understand sudden spikes in commit duration)
+
+These metrics can then be plotted on a standard tool like grafana. Below is a sample commit duration chart.
+
+<figure>
+ <img class="docimage" src="/assets/images/hudi_commit_duration.png" alt="hudi_commit_duration.png" style="max-width: 100%" />
+</figure>
+
+## MetricsReporter
+
+MetricsReporter is a interface for report metrics to user specified place. Currently, it's implementations has InMemoryMetricsReporter, JmxMetricsReporter, MetricsGraphiteReporter and DatadogMetricsReporter. Since InMemoryMetricsReporter is only used for testing, we will introduce the other three implementations.
+
+### JmxMetricsReporter
+
+JmxMetricsReporter is a implementation of Jmx reporter, which used to report jmx metric.
+
+#### Configurations
+The following is an example of configuration as JXM. The detailed configuration can refer to [here](configurations.html#jmx).
+
+ ```properties
+ hoodie.metrics.on=true
+ hoodie.metrics.reporter.type=JMX
+ hoodie.metrics.jmx.host=192.168.0.106
+ hoodie.metrics.jmx.port=4001
+ ```
+
+#### Demo
+As configuration above, Hudi metrics will started JMX server on port 4001. Then we can start jconsole to connect to 192.168.0.106:4001. Below is a sample of monitoring hudi jmx metrics through jconsole.
+<figure>
+ <img class="docimage" src="/assets/images/hudi_jxm_metrics.png" alt="hudi_jxm_metrics.png" style="max-width: 100%" />
+</figure>
+
+### MetricsGraphiteReporter
+
+MetricsGraphiteReporter is a implementation of Graphite reporter, which connects to the Graphite server, and send metrics to that server.
+
+#### Configurations
+The following is an example of configuration as GRAPHITE. The detailed configuration can refer to [here](configurations.html#graphite).
+
+ ```properties
+ hoodie.metrics.on=true
+ hoodie.metrics.reporter.type=GRAPHITE
+ hoodie.metrics.graphite.host=192.168.0.106
+ hoodie.metrics.graphite.port=2003
+ hoodie.metrics.graphite.metric.prefix=<your metrics prefix>
+ ```
+#### Demo
+As configuration above, we should first start graphite server on host 192.168.0.106 and port 2003, Hudi metrics will connect to graphite server, and report Hudi metrics to graphite server. Below is a sample of monitoring hudi metrics through graphite.
+ <figure>
+ <img class="docimage" src="/assets/images/hudi_graphite_metrics.png" alt="hudi_graphite_metrics.png" style="max-width: 100%" />
+ </figure>
+
+### DatadogMetricsReporter
+
+DatadogMetricsReporter is a implementation of Datadog reporter.
+A reporter which publishes metric values to Datadog monitoring service via Datadog HTTP API.
+
+#### Configurations
+The following is an example of configuration as Datadog. The detailed configuration can refer to [here](configurations.html#datadog).
+
+```properties
+hoodie.metrics.on=true
+hoodie.metrics.reporter.type=DATADOG
+hoodie.metrics.datadog.api.site=EU # or US
+hoodie.metrics.datadog.api.key=<your api key>
+hoodie.metrics.datadog.metric.prefix=<your metrics prefix>
+```
+
+ * `hoodie.metrics.datadog.api.site` will sets the Datadog API site, It determines whether the requests will be sent to api.datadoghq.eu (EU) or api.datadoghq.com (US). Set this according to your Datadog account settings.
+ * `hoodie.metrics.datadog.api.key` will set set the api key.
+ * `hoodie.metrics.datadog.metric.prefix` will helps segregate metrics by setting different prefixes for different jobs.Note that it will use `.` to delimit the prefix and the metric name. For example, if the prefix is set to foo, then foo. will be prepended to the metric name.
Review comment:
helps -> help
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org