[REVIEW] slider placement documentation

[Steve Loughran <st...@hortonworks.com>]

Hi everyone

the incubator vote for the next release seems to be going well: we've enough positive votes, so unless there's a veto tomorrow, it'll be good.


I've now documented the placement work, though I still want to add some full examples.

Can anyone with time/interest have a look at the new document

http://slider.incubator.apache.org/docs/placement.html

thanks,

steve

Re: [REVIEW] slider placement documentation

[Steve Loughran <st...@hortonworks.com>]

> On 10 Jan 2016, at 11:21, Josh Elser <jo...@gmail.com> wrote:
> 
> Taking a glance at it now. I'll take the liberty of fixing nit-picky things since you made the effort to write all of this nice stuff down :)
> 
> * Are there official YARN node label docs from Apache? Google is only showing me vendor-hosted docs.
> 
> * Under "Placement policies", you have a dangling sentence "Here are the currently supported placement policies, with their". I'll remove the "with their" for now under the assumption that it wasn't something important you wanted to tack on :)
> 

+1

> * Is this set in yarn-site.xml or elsewhere? <quote>will be relaxed after the timeout specified for a component in "yarn.placement.escalate.seconds"; the default is 30 seconds.</quote>

resources.json, per component

> 
> * Unrelated to this doc, but how does that little HTML table in the SliderAM's webUI help convey this? Do we get context back from YARN about simple things like a timeout due to lack of resources? Or, do we simply just need to explicitly point users to something like "Container allocation is still outstanding, check your YARN capacity"? It seems like AA, node labels, etc all make it more likely for people run into the original question of "why didn't I get containers?".
> 

it does it a lot more now....I just didn't have time/setup to grab some screen shots. There's some explanation of why things are outstanding a bit now.

> <quote>One recurrent support problem people raise with Slider is that it appears to hang with one or more outstanding requests. When this situation arises, it is not a sign that there is anything wrong with Slider. It means there is nowhere for YARN to allocate the desired containers.<quote>
> 
> * Another unrelated question: I thought YARN had support for cgroups expressly for the purpose of restricting cpu/memory at the container level.
> 

yes

> <quote>CPU performance can be throttled</quote>
> 
> * This is really nice! Would it make sense to expose a sub-command off of `slider` which nicely prints out this information? I could see this being useful in helping our users (and enabling them to interact with persisted state we have without being explicitly aware of how we're storing it).
> 

comes automatically with the vcore allocation. You ask for 1 vcore, you get 1 vcore worth of CPU.

> <quote>Here are the contents of a test cluster, testagentecho, stored in the file ~/.slider/clusters/testagentecho/history/rolehistory-00000151b4b44461.json:</quote>
> 
> Looks really great, Steve. Awesome job. I'll go ahead and publish the small changes I've made.

+1

> 
> - Josh
> 
> Steve Loughran wrote:
>> 
>> Hi everyone
>> 
>> the incubator vote for the next release seems to be going well: we've enough positive votes, so unless there's a veto tomorrow, it'll be good.
>> 
>> 
>> I've now documented the placement work, though I still want to add some full examples.
>> 
>> Can anyone with time/interest have a look at the new document
>> 
>> http://slider.incubator.apache.org/docs/placement.html
>> 
>> thanks,
>> 
>> steve
>> 
> 

Re: [REVIEW] slider placement documentation

[Josh Elser <jo...@gmail.com>]

Taking a glance at it now. I'll take the liberty of fixing nit-picky 
things since you made the effort to write all of this nice stuff down :)

* Are there official YARN node label docs from Apache? Google is only 
showing me vendor-hosted docs.

* Under "Placement policies", you have a dangling sentence "Here are the 
currently supported placement policies, with their". I'll remove the 
"with their" for now under the assumption that it wasn't something 
important you wanted to tack on :)

* Is this set in yarn-site.xml or elsewhere? <quote>will be relaxed 
after the timeout specified for a component in 
"yarn.placement.escalate.seconds"; the default is 30 seconds.</quote>

* Unrelated to this doc, but how does that little HTML table in the 
SliderAM's webUI help convey this? Do we get context back from YARN 
about simple things like a timeout due to lack of resources? Or, do we 
simply just need to explicitly point users to something like "Container 
allocation is still outstanding, check your YARN capacity"? It seems 
like AA, node labels, etc all make it more likely for people run into 
the original question of "why didn't I get containers?".

<quote>One recurrent support problem people raise with Slider is that it 
appears to hang with one or more outstanding requests. When this 
situation arises, it is not a sign that there is anything wrong with 
Slider. It means there is nowhere for YARN to allocate the desired 
containers.<quote>

* Another unrelated question: I thought YARN had support for cgroups 
expressly for the purpose of restricting cpu/memory at the container level.

<quote>CPU performance can be throttled</quote>

* This is really nice! Would it make sense to expose a sub-command off 
of `slider` which nicely prints out this information? I could see this 
being useful in helping our users (and enabling them to interact with 
persisted state we have without being explicitly aware of how we're 
storing it).

<quote>Here are the contents of a test cluster, testagentecho, stored in 
the file 
~/.slider/clusters/testagentecho/history/rolehistory-00000151b4b44461.json:</quote>

Looks really great, Steve. Awesome job. I'll go ahead and publish the 
small changes I've made.

- Josh

Steve Loughran wrote:
>
> Hi everyone
>
> the incubator vote for the next release seems to be going well: we've enough positive votes, so unless there's a veto tomorrow, it'll be good.
>
>
> I've now documented the placement work, though I still want to add some full examples.
>
> Can anyone with time/interest have a look at the new document
>
> http://slider.incubator.apache.org/docs/placement.html
>
> thanks,
>
> steve
>