Fwd: Accessing Custom Beam Metrics in Dataproc

[Rion Williams <ri...@gmail.com>]

+user

Begin forwarded message:

> From: Rion Williams <ri...@gmail.com>
> Date: January 12, 2021 at 4:09:34 PM CST
> To: dev@beam.apache.org
> Subject: Accessing Custom Beam Metrics in Dataproc
> Reply-To: dev@beam.apache.org
> 
> Hi all,
> 
> I'm currently in the process of adding some metrics to an existing pipeline that runs on Google Dataproc via Spark and I'm trying to determine how to access these metrics and eventually expose them to Stackdriver (to be used downstream in Grafana dashboards).
> 
> The metrics themselves are fairly simple (a series of counters) and are defined as such (and accessed in DoFns throughout the pipeline):
> 
> ```
> /** Metrics gathered during Event-related transforms */
>    private object Metrics {
>        // This is used to keep track of any dynamically added data sources and their counts
>        val totalMessages: Counter = BeamMetrics.counter(Events::class.qualifiedName, "messages_total")
>    }
> ```
> 
> After initially running the pipeline in Dataproc, I wasn't able to see anything that specifically indicated that the metrics were being exposed at all. I haven't added any specific configuration to handle this within the pipeline itself, however I did notice an interface that I may need to consider implementing called MetricOptions:
> 
> ```
> interface MyPipelineOptions : ... MetricsOptions { ... }
> ```
> 
> So my questions primarily center around:
> 
> - Will Metrics be emitted automatically? Or do I need to explicitly implement the MetricsOptions interface for the pipeline?
> - Does anyone have any experience with handling this (i.e. Pipeline > Metrics > Stackdriver)? I'd imagine since this is all self-contained within GCP (Dataproc + Stackdriver), that it wouldn't be too rough to hand that baton off.
> 
> Any advice / articles / examples would be greatly appreciated!
> 
> Thanks,
> 
> Rion