[jira] [Commented] (HDDS-4404) Datanode can go OOM when a Recon or SCM Server is very slow in processing reports.

["Siyao Meng (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HDDS-4404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17224888#comment-17224888 ] 

Siyao Meng commented on HDDS-4404:
----------------------------------

In a discussion, [~avijayan] mentioned we could solve the OOM issue simply by adding passive RPC endpoint check here:

{code:java|title=Current HeartbeatEndpointTask#call}
    } catch (IOException ex) {
      // put back the reports which failed to be sent
      putBackReports(requestBuilder);

      rpcEndpoint.logIfNeeded(ex);
    } finally {
{code}

{code:java|title=Suggested}
    } catch (IOException ex) {
      if (!rpcEndpoint.isPassive()) {
        // put back the reports which failed to be sent
        putBackReports(requestBuilder);
      }

      rpcEndpoint.logIfNeeded(ex);
    } finally {
{code}

This way if a Recon is registered and down for days. The DN won't be piling up and adding all the reports in the queue.

Any insight on why we put the report back to the queue on exception in the first place? [~nanda619]

> Datanode can go OOM when a Recon or SCM Server is very slow in processing reports.
> ----------------------------------------------------------------------------------
>
>                 Key: HDDS-4404
>                 URL: https://issues.apache.org/jira/browse/HDDS-4404
>             Project: Hadoop Distributed Data Store
>          Issue Type: Task
>          Components: Ozone Datanode
>    Affects Versions: 1.0.0
>            Reporter: Aravindan Vijayan
>            Assignee: Siyao Meng
>            Priority: Critical
>         Attachments: Screen Shot 2020-10-26 at 11.24.09 PM.png
>
>
> From [~nanda619]'s analysis.
> ContainerReportPublisher thread runs periodically (default interval 60s) in Datanode and adds ContainerReport to StateContext (Queue).
> Heartbeat thread runs periodically (default interval 30s), picks up the ContainerReport (if any) from StateContext.
> For short time, the ContainerReport will be held in Datanode StateContext.
> For Recon, a change was made in datanode such that the ContainerReport will be cached in Datanode StateContext separately for each endpoint (i.e. SCM and Recon). As I see, if Recon is configured in the Datanode and all the reports that are to be sent to Recon will be pending in the StateContextQueue (LinkedList)



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org