Hive LLAP low Vcore allocation

[Salih Şen <sl...@gmail.com>]

Hi
We are trying to improve our LLAP performance on our cluster but we've
noticed that event though LLAP daemon containers get configured memory,
they get only 1 vcore per container.
We are running 10 LLAP deamons using Slider. There are no other containers
running on the nodes that run LLAP daemons and there are 0 memory available
but 43 vcores running idle.

I can see the following lines on Slider logs so I suspect SliderAppMaster
doesn't request vcores from Yarn:

2018-10-16 18:38:42,503 [AmExecutor-006] INFO  appmaster.SliderAppMaster -
Registered service under /users/hive/services/org-apache-slider/llap0;
absolute path /registry/users/hive/services/org-apache-slider/llap0
2018-10-16 18:38:42,510 [AmExecutor-006] INFO  state.AppState - Reviewing
RoleStatus{name='LLAP', group=LLAP, key=1, desired=10, actual=0,
requested=0, releasing=0, failed=0, startFailed=0, started=0, completed=0,
totalRequested=0, preempted=0, nodeFailed=0, failedRecently=0,
limitsExceeded=0, resourceRequirements=<memory:445440, vCores:1>,
isAntiAffinePlacement=false, failureMessage='',
providerRole=ProviderRole{name='LLAP', group=LLAP, id=1, placementPolicy=0,
nodeFailureThreshold=3, placementTimeoutSeconds=30,
labelExpression='null'}, failedContainers=[],
healthThresholdMonitorEnabled=true} :
2018-10-16 18:38:42,510 [AmExecutor-006] INFO  state.AppState - LLAP:
Asking for 10 more nodes(s) for a total of 10
2018-10-16 18:38:42,512 [AmExecutor-006] INFO  state.AppState - Container
ask is Capability[<memory:445440, vCores:1>]Priority[1073741825] and label
= null
2018-10-16 18:38:42,513 [AmExecutor-006] INFO  state.AppState - Container
ask is Capability[<memory:445440, vCores:1>]Priority[1073741825] and label
= null
2018-10-16 18:38:42,513 [AmExecutor-006] INFO  state.AppState - Container
ask is Capability[<memory:445440, vCores:1>]Priority[1073741825] and label
= null
2018-10-16 18:38:42,513 [AmExecutor-006] INFO  state.AppState - Container
ask is Capability[<memory:445440, vCores:1>]Priority[1073741825] and label
= null
2018-10-16 18:38:42,513 [AmExecutor-006] INFO  state.AppState - Container
ask is Capability[<memory:445440, vCores:1>]Priority[1073741825] and label
= null
2018-10-16 18:38:42,513 [AmExecutor-006] INFO  state.AppState - Container
ask is Capability[<memory:445440, vCores:1>]Priority[1073741825] and label
= null
2018-10-16 18:38:42,513 [AmExecutor-006] INFO  state.AppState - Container
ask is Capability[<memory:445440, vCores:1>]Priority[1073741825] and label
= null
2018-10-16 18:38:42,513 [AmExecutor-006] INFO  state.AppState - Container
ask is Capability[<memory:445440, vCores:1>]Priority[1073741825] and label
= null
2018-10-16 18:38:42,514 [AmExecutor-006] INFO  state.AppState - Container
ask is Capability[<memory:445440, vCores:1>]Priority[1073741825] and label
= null
2018-10-16 18:38:42,514 [AmExecutor-006] INFO  state.AppState - Container
ask is Capability[<memory:445440, vCores:1>]Priority[1073741825] and label
= null

And here is the configuration output from same log file that might be
relevant:

 "credentials" : { },
  "components" : {
    "LLAP" : {
      "yarn.container.health.threshold.init.delay.secs" : "400",
      "yarn.role.priority" : "1",
      "yarn.component.instances" : "10",
      "yarn.memory" : "445440",
      "yarn.resource.normalization.enabled" : "false",
      "yarn.container.health.threshold.window.secs" : "300",
      "yarn.component.placement.policy" : "0",
      "yarn.container.health.threshold.percent" : "80"
    },
    "slider-appmaster" : {
      "yarn.vcores" : "1",
      "yarn.component.instances" : "1",
      "yarn.memory" : "1024"
    }
  }
},

yarn.nodemanager.resource.cpu-vcores,
yarn.scheduler.maximum-allocation-vcores,
hive.llap.daemon.vcpus.per.instance,
hive.llap.daemon.num.executors are all set to 44.

We can confirm 44 executors running per instance on LLAP Daemon web UI.

We are using HDP 2.7.3.2.6.4.0-91 with YARN 2.7.3, Hive 1.2.1000, Slider
0.92.0.

Any ideas how to utilize more CPU with LLAP daemons?

Thanks.