[jira] [Created] (YARN-12031) [Federation] Router GET scheduler-conf always fails: forwarded RM response entity is cast to ConfInfo

["Jose Luis López (Jira)" <ji...@apache.org>]

Jose Luis López created YARN-12031:
--------------------------------------

             Summary: [Federation] Router GET scheduler-conf always fails: forwarded RM response entity is cast to ConfInfo
                 Key: YARN-12031
                 URL: https://issues.apache.org/jira/browse/YARN-12031
             Project: Hadoop YARN
          Issue Type: Bug
          Components: federation, router, webapp
    Affects Versions: 3.6.0, 3.4.0
            Reporter: Jose Luis López


{{FederationInterceptorREST#getSchedulerConfiguration}} fans out to every active sub-cluster through {{DefaultRequestInterceptorREST#getSchedulerConfiguration}}, which forwards with {{RouterWebServiceUtil.genericForward(..., Response.class, ...)}} and returns the Jersey client's inbound {{Response}} unread. For an inbound response {{getEntity()}} is the raw entity stream ({{org.glassfish.jersey.client.internal.HttpUrlConnector$2}}), never a {{ConfInfo}}:

{code:java}
} else if (response.getStatus() == Status.BAD_REQUEST.getStatusCode()) {
  String errorMsg = String.valueOf(response.getEntity());
  federationConfInfo.getErrorMsgs().add(errorMsg);
} else if (response.getStatus() == Status.OK.getStatusCode()) {
  ConfInfo fedConfInfo = (ConfInfo) response.getEntity();
{code}

* A sub-cluster that answers 200 (mutable-conf CapacityScheduler) throws {{ClassCastException}}. The {{catch (Exception e)}} turns it into {{400 getSchedulerConfiguration error.}} for the whole federation and does not log the cause.
* A sub-cluster that answers 400 (no mutable store) adds an object-identity string such as {{org.glassfish.jersey.client.internal.HttpUrlConnector$2@74b97416}} to {{errorMsgs}} instead of the RM's message.

{{TestFederationInterceptorREST#testGetSchedulerConfiguration}} passes only because {{MockDefaultRequestInterceptorREST#getSchedulerConfiguration}} returns an outbound {{Response.status(OK).entity(new ConfInfo(...))}}, whose {{getEntity()}} is the {{ConfInfo}}. The cast has been wrong since the API was added (YARN-11446, 3.4.0): before HADOOP-15984, {{clientResponseToResponse}} also set {{getEntityInputStream()}} as the entity.

{{updateSchedulerConfiguration}} is not affected: it passes {{response.getEntity()}} through as the Router's own entity, which Jersey streams to the caller unchanged.

*Repro*

A real federation from the {{TestFederationSubCluster}} harness: ZooKeeper, two MiniYARNCluster RMs and the Router, each in its own JVM. SC-1 runs CapacityScheduler with {{yarn.scheduler.configuration.store.class=memory}}, SC-2 a plain CapacityScheduler. The attached {{TestSchedConfRepro.java}} drives it; its javadoc has the setup. trunk 9ede2e95c6f, Ubuntu 24.04, OpenJDK 17:

{noformat}
router GET accept=application/xml  status=400 body=getSchedulerConfiguration error.
router GET accept=application/json status=400 body=getSchedulerConfiguration error.
{noformat}

Forwarding to each RM with {{genericForward(..., Response.class, ...)}} and {{Accept: application/xml}}, as {{DefaultRequestInterceptorREST}} does:

{noformat}
SC-1 status=200 getEntity().getClass()=org.glassfish.jersey.client.internal.HttpUrlConnector$2
SC-1 status=200 cast -> java.lang.ClassCastException: class org.glassfish.jersey.client.internal.HttpUrlConnector$2 cannot be cast to class org.apache.hadoop.yarn.webapp.dao.ConfInfo
SC-1 status=200 readEntity(ConfInfo) items=17
SC-2 status=400 String.valueOf(getEntity())=org.glassfish.jersey.client.internal.HttpUrlConnector$2@74b97416
SC-2 status=400 readEntity(String)=This API only supports to retrieve scheduler configuration from a mutable-conf scheduler, underneath scheduler CapacityScheduler is not an instance of MutableConfScheduler
{noformat}

*Proposed fix*

Read the entity instead of casting it ({{readEntity}} also closes the response):

{code:java}
String errorMsg = response.readEntity(String.class);
...
ConfInfo fedConfInfo = response.readEntity(ConfInfo.class);
{code}

and make {{MockDefaultRequestInterceptorREST#getSchedulerConfiguration}} return a response that can only be read with {{readEntity}}, so the existing test fails without the fix. With that change the Router returns 200 with SC-1's configuration and SC-2's error message, for both XML and JSON callers. The fan-out always asks the RMs for XML ({{clone(hsr)}} pins {{Accept}}), so this does not depend on YARN-12025.

Statuses other than 200 and 400 (for example 403 for a non-admin caller) are still dropped silently and their response is not closed.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-dev-help@hadoop.apache.org