[jira] [Commented] (YARN-3143) RM Apps REST API can return NPE or entries missing id and other fields

["Jason Lowe (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/YARN-3143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14306169#comment-14306169 ] 

Jason Lowe commented on YARN-3143:
----------------------------------

I think this is caused by this code in the RMWebServices.java:

{code}
    final ConcurrentMap<ApplicationId, RMApp> apps =
        rm.getRMContext().getRMApps();
    AppsInfo allApps = new AppsInfo();
    for (ApplicationReport report : appReports) {
      RMApp rmapp = apps.get(report.getApplicationId());

      if (finalStatusQuery != null && !finalStatusQuery.isEmpty()) {
        FinalApplicationStatus.valueOf(finalStatusQuery);
        if (!rmapp.getFinalApplicationStatus().toString()
            .equalsIgnoreCase(finalStatusQuery)) {
          continue;
        }
      }

      AppInfo app = new AppInfo(rmapp, hasAccess(rmapp, hsr),
          WebAppUtils.getHttpSchemePrefix(conf));
      allApps.add(app);
{code}

The problem here is that we're trying to look at a pool of applications asynchronously, therefore it is possible that we could get an app report for an application that is gone by the time we call apps.get(report.getApplicationId).  That means rmapp can be null, and note that AppInfo's constructor effectively does nothing if the RMApp argument is null.  That explains the lack of all sorts of fields by the time it gets to JSON.

So there are a couple of problems here:

# We should not be adding AppInfos if rmapp is null
# We can blindly dereference rmapp and generate an NPE if there's a final status query

> RM Apps REST API can return NPE or entries missing id and other fields
> ----------------------------------------------------------------------
>
>                 Key: YARN-3143
>                 URL: https://issues.apache.org/jira/browse/YARN-3143
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: webapp
>    Affects Versions: 2.5.2
>            Reporter: Kendall Thrapp
>            Assignee: Jason Lowe
>
> I'm seeing intermittent null pointer exceptions being returned by
> the YARN Apps REST API.
> For example:
> {code}
> http://{cluster}:{port}/ws/v1/cluster/apps?finalStatus=UNDEFINED
> {code}
> JSON Response was:
> {code}
> {"RemoteException":{"exception":"NullPointerException","javaClassName":"java.lang.NullPointerException"}}
> {code}
> At a glance appears to be only when we query for unfinished apps (i.e. finalStatus=UNDEFINED).  
> Possibly related, when I do get back a list of apps, sometimes one or more of the apps will be missing most of the fields, like id, name, user, etc., and the fields that are present all have zero for the value.  
> For example:
> {code}
> {"progress":0.0,"clusterId":0,"applicationTags":"","startedTime":0,"finishedTime":0,"elapsedTime":0,"allocatedMB":0,"allocatedVCores":0,"runningContainers":0,"preemptedResourceMB":0,"preemptedResourceVCores":0,"numNonAMContainerPreempted":0,"numAMContainerPreempted":0}
> {code}
> Let me know if there's any other information I can provide to help debug.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)