You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ambari.apache.org by "Jonathan Hurley (JIRA)" <ji...@apache.org> on 2016/08/05 20:34:20 UTC

[jira] [Created] (AMBARI-18052) Starting a Component After Pausing An Upgrade Can Take 9 Minutes

Jonathan Hurley created AMBARI-18052:
----------------------------------------

             Summary: Starting a Component After Pausing An Upgrade Can Take 9 Minutes
                 Key: AMBARI-18052
                 URL: https://issues.apache.org/jira/browse/AMBARI-18052
             Project: Ambari
          Issue Type: Bug
          Components: ambari-server
    Affects Versions: 2.2.0
            Reporter: Jonathan Hurley
            Assignee: Jonathan Hurley
            Priority: Blocker
             Fix For: 2.4.0


STR:

- Begin an EU on a large cluster (say 900 hosts) with a full stack.
- At finalize, pause the upgrade and then attempt to start a component

Upon clicking on start the request was accepted, but it was not scheduled for the next 9 minutes. After which it was scheduled and started correctly. Further requests were responsive.

Request:
{code}
  "href" : "http://perf-a-1:8080/api/v1/clusters/perf/requests/77",
  "Requests" : {
    "aborted_task_count" : 0,
    "cluster_name" : "perf",
    "completed_task_count" : 1,
    "create_time" : 1468613056719,
    "end_time" : 1468613636151,
    "exclusive" : false,
    "failed_task_count" : 0,
    "id" : 77,
    "inputs" : null,
    "operation_level" : null,
    "progress_percent" : 100.0,
    "queued_task_count" : 0,
    "request_context" : "Start NodeManager",
    "request_schedule" : null,
    "request_status" : "COMPLETED",
    "resource_filters" : [ ],
    "start_time" : 1468613625415,
    "task_count" : 1,
    "timed_out_task_count" : 0,
    "type" : "INTERNAL_REQUEST"
  },
{code}

{noformat}
created: 7/15/2016, 1:04:16 PM
started: 7/15/2016, 1:13:45 PM
finished: 7/15/2016, 1:13:56 PM
{noformat}

The root cause of this seems to be how an upgrade is paused/resumed. The {{UpgradeResourceProvider}} loads the entire request in memory to iterate over it. In this case, that contains about 11,000 {{HostRoleCommandEntity}} where each one is between 2 and 3MB. That means, that we're trying to load about 33GB of data into memory.

This causes threads to die slowly, including scheduler threads, until the JVM can recover and start scheduling things again. 

The real question is _why_ each HRCEntity is so large. In many cases, the output includes information from HDFS, such as the state of SafeMode. These messages include the entire state of the system which is being captured to the stdout. I see two workarounds here:

- Change how {{UpgradeResourceProvider}} handles the pause/resume feature to avoid loading the entire request. Maybe pagination, maybe a custom query ... but not the entire request all at once.

- Configure the {{HostRoleCommandEntity}} to lazy-load the {{stdout}} and {{stderr}} data.




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)