You are viewing a plain text version of this content. The canonical link for it is here.

Posted to commits@samza.apache.org by "Abhishek Shivanna (JIRA)" <ji...@apache.org> on 2017/10/06 18:59:00 UTC

[jira] [Created] (SAMZA-1447) YarnCliJobStatusProvider cannot handle a high QPS to the JobsResource REST endpoint

Abhishek Shivanna created SAMZA-1447:
----------------------------------------

             Summary: YarnCliJobStatusProvider cannot handle a high QPS to the JobsResource REST endpoint
                 Key: SAMZA-1447
                 URL: https://issues.apache.org/jira/browse/SAMZA-1447
             Project: Samza
          Issue Type: Bug
            Reporter: Abhishek Shivanna
            Assignee: Abhishek Shivanna


YarnCliJobStatusProvider shells out to a new process (running org.apache.hadoop.yarn.client.cli.ApplicationCLI) to fetch metadata about applications in the cluster.
This leads to the host running out of memory when the number of requests being made to the JobsResource endpoint is high.
The fix should be to use the Resource Manager REST endpoint instead which provides the same data without the new process per call overhead.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)