[jira] [Comment Edited] (YARN-11514) Extend SchedulerResponse with capacityVector

["Benjamin Teke (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/YARN-11514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17759092#comment-17759092 ] 

Benjamin Teke edited comment on YARN-11514 at 8/25/23 3:53 PM:
---------------------------------------------------------------

After some trials I suggest moving forward with the current jsonProvider. The ideal solution would be to replace Jettison with Jackson, as it would make adding new/custom fields way easier, but achieving exactly the same result would mean a lot of code change and even custom solutions, which has more risk than benefits especially because we're adding a simple map containing 2-4 elements. And since this is a public API we risk breaking other dependent components. So the current implementation looks like this:

 
{code:java}
"queueCapacityVectorInfo" : {
                    "configuredCapacityVector" : "[memory-mb=12.5%,vcores=12.5%]",
                    "capacityVectorEntries" : [ {
                      "resourceName" : "memory-mb",
                      "resourceValue" : "12.5%"
                    }, {
                      "resourceName" : "vcores",
                      "resourceValue" : "12.5%"
                    } ]
                  },
{code}



was (Author: bteke):
After some trials I suggest moving forward with the current jsonProvider. The ideal solution would be to replace Jettison with Jackson, as it would make adding new/custom fields way easier, but achieving exactly the same result would mean a lot of code change and even custom solutions, which isn't necessarily worth it. And since this is a public API we risk breaking other dependent components. So the current implementation looks like this:

 
{code:java}
"queueCapacityVectorInfo" : {
                    "configuredCapacityVector" : "[memory-mb=12.5%,vcores=12.5%]",
                    "capacityVectorEntries" : [ {
                      "resourceName" : "memory-mb",
                      "resourceValue" : "12.5%"
                    }, {
                      "resourceName" : "vcores",
                      "resourceValue" : "12.5%"
                    } ]
                  },
{code}


> Extend SchedulerResponse with capacityVector
> --------------------------------------------
>
>                 Key: YARN-11514
>                 URL: https://issues.apache.org/jira/browse/YARN-11514
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>            Reporter: Tamas Domok
>            Assignee: Benjamin Teke
>            Priority: Major
>              Labels: pull-request-available
>
> The goal is to add the *capacityVector* to the Scheduler response (XML/JSON).
>  - CapacitySchedulerQueueInfo.java
>  - PartitionQueueCapacitiesInfo.java
> The proposed format in the design doc (YARN-10888):
> {code:json}
> {
>    "capacityVector": {
>       "memory-mb": "30%",
>       "vcores": "16"
>    }
> }
> {code}
> {code:xml}
> <?xml version="1.0" encoding="UTF-8" ?>
> <capacityVector>
>   <memory-mb>30%</memory-mb>
>   <vcores>16</vcores>
> </capacityVector>
> {code}
> Unfortunately the current jsonProvider (MoxyJsonFeature or JettisonFeature not sure) serialise map structures in the following way:
> {code:json}
> {
>   "capacityVector":{
>     "entry":[
>       {
>         "key":"memory-mb",
>         "value":"12288"
>       },
>       {
>         "key":"vcores",
>         "value":"86%"
>       }
>     ]
>   }
> }
> {code}
> {code:xml}
> <?xml version="1.0" encoding="UTF-8"?>
> <capacityVector>
>    <entry>
>       <key>memory-mb</key>
>       <value>1288</value>
>    </entry>
>    <entry>
>       <key>vcores</key>
>       <value>12</value>
>    </entry>
> </capacityVector>
> {code}
> Based on some research with the following two dependencies we could achieve the proposed format:
>  - jersey-media-json-jackson (this one is used in the apps catalog already)
>  - jackson-dataformat-xml
> Some concerns:
>  - 2 more dependencies
>  - for the XML when the content depends on the runtime content of the map is not XSD friendly
>  - name is capacityVector but it's represented in a map
> An alternative could be to just store the capacityVector as a string, but then clients needs to parse it, and it's not particularly nice either:
> {code:json}
> {
>    "capacityVector": "[\"memory-mb\":  12288, \"vcores\":  86%]"
> }
> {code}
> {code:xml}
> <?xml version="1.0" encoding="UTF-8" ?>
> <capacityVector>[&quot;memory-mb&quot;:  12288, &quot;vcores&quot;:  86%]</capacityVector>
> {code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org