You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@beam.apache.org by "Jean-Baptiste Onofré (JIRA)" <ji...@apache.org> on 2017/06/16 07:06:00 UTC
[jira] [Created] (BEAM-2456) Introduce generic metric poll thread
interceptor
Jean-Baptiste Onofré created BEAM-2456:
------------------------------------------
Summary: Introduce generic metric poll thread interceptor
Key: BEAM-2456
URL: https://issues.apache.org/jira/browse/BEAM-2456
Project: Beam
Issue Type: New Feature
Components: sdk-ideas
Reporter: Jean-Baptiste Onofré
Assignee: Jean-Baptiste Onofré
The Spark runner provides a convenient feature which is the metric sink.
By configuration, it allows us to configure a metric sink using {{metrics.properties}} configuration containing:
{code}
driver.sink.graphite.class=org.apache.beam.runners.spark.metrics.sink.GraphiteSink
driver.sink.graphite.host=localhost
driver.sink.graphite.port=2003
driver.sink.graphite.prefix=spark
driver.sink.graphite.period=1
driver.sink.graphite.unit=SECONDS
{code}
This approach is very convenient to send the metric to any sink. I think we can apply this logic in generic way working with any runner.
The idea is to use a metric poll thread in the pipeline (that we can enable via {{PipelineOptions}}) and send to a sink.
I started a PoC about that.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)