[jira] [Closed] (FLINK-10761) MetricGroup#getAllVariables can deadlock

["Chesnay Schepler (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/FLINK-10761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Chesnay Schepler closed FLINK-10761.
------------------------------------
       Resolution: Fixed
    Fix Version/s: 1.7.2
                   1.6.4

master: aa92438c4aa6218c5d7e67083152733271697089
1.7: 45d53d27e4dec934057404674e38c6eab2e66e62
1.6: cb85e62b5b122b7cbb9eeb72a116dd133467a920

> MetricGroup#getAllVariables can deadlock
> ----------------------------------------
>
>                 Key: FLINK-10761
>                 URL: https://issues.apache.org/jira/browse/FLINK-10761
>             Project: Flink
>          Issue Type: Bug
>          Components: Metrics
>    Affects Versions: 1.5.5, 1.6.2, 1.7.0
>            Reporter: Chesnay Schepler
>            Assignee: Chesnay Schepler
>            Priority: Critical
>              Labels: pull-request-available
>             Fix For: 1.6.4, 1.7.2, 1.8.0
>
>          Time Spent: 20m
>  Remaining Estimate: 0h
>
> {{AbstractMetricGroup#getAllVariables}} acquires the locks of both the current and all parent groups when assembling the variables map. This can lead to a deadlock if metrics are registered concurrently on a child and parent if the child registration is applied first and the reporter uses said method (which many do).
> Assume we have a MetricGroup Mc(hild) and Mp(arent).
> 2 separate threads Tc and Tp each register a metric on their respective group, acquiring the lock.
> Let's assume that Tc has a slight headstart.
> Tc will now call {{MetricRegistry#register}} first, acquiring the MR lock.
> Tp will block on this lock.
> Tc now iterates over all reporters calling {{MetricReporter#notifyOfAddedMetric}}. Assume that in this method {{MetricGroup#getAllVariables}} is called on Mc by Tc.
> Tc still holds the lock to Mc, and attempts to acquire the lock to Mp.
> The lock to Mp is still held by Tp however, which waits for the MR lock to be released by Tc.
> Thus a deadlock is created. This may deadlock anything, be it minor threads, tasks, or entire components.
> This has not surfaced so far since usually metrics are no longer added to a group once children have been created (since the component initialization at that point is complete).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)