You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@beam.apache.org by "Ismaël Mejía (Jira)" <ji...@apache.org> on 2020/01/08 11:07:00 UTC

[jira] [Updated] (BEAM-9065) Spark runner accumulates metrics (incorrectly) between runs

     [ https://issues.apache.org/jira/browse/BEAM-9065?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ismaël Mejía updated BEAM-9065:
-------------------------------
    Summary: Spark runner accumulates metrics (incorrectly) between runs  (was: metrics are not reset upon init)

> Spark runner accumulates metrics (incorrectly) between runs
> -----------------------------------------------------------
>
>                 Key: BEAM-9065
>                 URL: https://issues.apache.org/jira/browse/BEAM-9065
>             Project: Beam
>          Issue Type: Bug
>          Components: runner-spark
>            Reporter: Etienne Chauchot
>            Assignee: Etienne Chauchot
>            Priority: Major
>             Fix For: 2.18.0
>
>          Time Spent: 10m
>  Remaining Estimate: 0h
>
> When pipeline.run() is called, MetricsAccumulator (wrapper of MetricsContainerStepMap spark accumulator) is initialized. Spark needs this class to be a singleton for failover. The problem is that when several pipelines are run inside the same JVM, the initialization of MetricsAccumulator singleton does not reset the underlying spark accumulator  causing metrics to be accumulated between runs.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)