You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@slider.apache.org by sm...@apache.org on 2014/07/08 02:49:35 UTC
svn commit: r1608633 [1/7] - in /incubator/slider/site: content/design/
content/design/registry/ content/design/specification/ content/developing/
content/docs/ content/docs/configuration/ content/docs/slider_specs/
trunk/content/developing/
Author: smohanty
Date: Tue Jul 8 00:49:34 2014
New Revision: 1608633
URL: http://svn.apache.org/r1608633
Log:
SLIDER-121 manually generate slider site from CMS content for now
Modified:
incubator/slider/site/content/design/registry/initial_registry_design.html
incubator/slider/site/content/design/registry/service_registry_end_to_end_scenario.html
incubator/slider/site/content/design/rolehistory.html
incubator/slider/site/content/design/specification/cli-actions.html
incubator/slider/site/content/design/specification/slider-model.html
incubator/slider/site/content/developing/building.html
incubator/slider/site/content/developing/functional_tests.html
incubator/slider/site/content/developing/releasing.html
incubator/slider/site/content/developing/testing.html
incubator/slider/site/content/docs/client-configuration.html
incubator/slider/site/content/docs/configuration/core.html
incubator/slider/site/content/docs/configuration/redesign.html
incubator/slider/site/content/docs/configuration/specification.html
incubator/slider/site/content/docs/debugging.html
incubator/slider/site/content/docs/examples.html
incubator/slider/site/content/docs/exitcodes.html
incubator/slider/site/content/docs/getting_started.html
incubator/slider/site/content/docs/manpage.html
incubator/slider/site/content/docs/security.html
incubator/slider/site/content/docs/slider_specs/application_configuration.html
incubator/slider/site/content/docs/slider_specs/application_definition.html
incubator/slider/site/content/docs/slider_specs/application_instance_configuration.html
incubator/slider/site/content/docs/slider_specs/application_package.html
incubator/slider/site/content/docs/slider_specs/creating_app_definitions.html
incubator/slider/site/content/docs/slider_specs/resource_specification.html
incubator/slider/site/content/docs/slider_specs/writing_app_command_scripts.html
incubator/slider/site/content/docs/troubleshooting.html
incubator/slider/site/trunk/content/developing/building.md
Modified: incubator/slider/site/content/design/registry/initial_registry_design.html
URL: http://svn.apache.org/viewvc/incubator/slider/site/content/design/registry/initial_registry_design.html?rev=1608633&r1=1608632&r2=1608633&view=diff
==============================================================================
--- incubator/slider/site/content/design/registry/initial_registry_design.html (original)
+++ incubator/slider/site/content/design/registry/initial_registry_design.html Tue Jul 8 00:49:34 2014
@@ -225,49 +225,51 @@ can use the same datatype -or map to it-
<h2 id="example-curator-service-entry">Example Curator Service Entry</h2>
<p>This is the prototype's content</p>
<p>Toplevel</p>
-<pre class="codehilite"><code>service CuratorServiceInstance{name='slider', id='stevel.test_registry_am', address='192.168.1.101', port=62552, sslPort=null, payload=org.apache.slider.core.registry.info.ServiceInstanceData@4e9af21b, registrationTimeUTC=1397574073203, serviceType=DYNAMIC, uriSpec=org.apache.curator.x.discovery.UriSpec@ef8dacf0}</code></pre>
+<div class="codehilite"><pre><span class="n">service</span> <span class="n">CuratorServiceInstance</span><span class="p">{</span><span class="n">name</span><span class="o">=</span><span class="err">'</span><span class="n">slider</span><span class="err">'</span><span class="p">,</span> <span class="n">id</span><span class="o">=</span><span class="err">'</span><span class="n">stevel</span><span class="p">.</span><span class="n">test_registry_am</span><span class="err">'</span><span class="p">,</span> <span class="n">address</span><span class="o">=</span><span class="err">'</span><span class="mf">192.168.1.101</span><span class="err">'</span><span class="p">,</span> <span class="n">port</span><span class="o">=</span><span class="mi">62552</span><span class="p">,</span> <span class="n">sslPort</span><span class="o">=</span><span class="n">null</span><span class="p">,</span> <span class="n">payload</span><span class="o">=</span><span class="n">org</span><span clas
s="p">.</span><span class="n">apache</span><span class="p">.</span><span class="n">slider</span><span class="p">.</span><span class="n">core</span><span class="p">.</span><span class="n">registry</span><span class="p">.</span><span class="n">info</span><span class="p">.</span><span class="n">ServiceInstanceData</span><span class="err">@</span><span class="mf">4e9</span><span class="n">af21b</span><span class="p">,</span> <span class="n">registrationTimeUTC</span><span class="o">=</span><span class="mi">1397574073203</span><span class="p">,</span> <span class="n">serviceType</span><span class="o">=</span><span class="n">DYNAMIC</span><span class="p">,</span> <span class="n">uriSpec</span><span class="o">=</span><span class="n">org</span><span class="p">.</span><span class="n">apache</span><span class="p">.</span><span class="n">curator</span><span class="p">.</span><span class="n">x</span><span class="p">.</span><span class="n">discovery</span><span class="p">.</span><span class="n">
UriSpec</span><span class="err">@</span><span class="n">ef8dacf0</span><span class="p">}</span>
+</pre></div>
<p>Slider payload.</p>
-<pre class="codehilite"><code>payload=
-{
- "internalView" : {
- "endpoints" : {
- "/agents" : {
- "value" : "http://stevel-8.local:62552/ws/v1/slider/agents",
- "protocol" : "http",
- "type" : "url",
- "description" : "Agent API"
- }
- },
- "settings" : { }
- },
-
- "externalView" : {
- "endpoints" : {
- "/mgmt" : {
- "value" : "http://stevel-8.local:62552/ws/v1/slider/mgmt",
- "protocol" : "http",
- "type" : "url",
- "description" : "Management API"
- },
-
- "slider/IPC" : {
- "value" : "stevel-8.local/192.168.1.101:62550",
- "protocol" : "org.apache.hadoop.ipc.Protobuf",
- "type" : "address",
- "description" : "Slider AM RPC"
- },
- "registry" : {
- "value" : "http://stevel-8.local:62552/ws/registry",
- "protocol" : "http",
- "type" : "url",
- "description" : "Registry"
- }
- },
- "settings" : { }
- }
-}</code></pre>
+<div class="codehilite"><pre><span class="n">payload</span><span class="o">=</span>
+<span class="p">{</span>
+ <span class="s">"internalView"</span> <span class="o">:</span> <span class="p">{</span>
+ <span class="s">"endpoints"</span> <span class="o">:</span> <span class="p">{</span>
+ <span class="s">"/agents"</span> <span class="o">:</span> <span class="p">{</span>
+ <span class="s">"value"</span> <span class="o">:</span> <span class="s">"http://stevel-8.local:62552/ws/v1/slider/agents"</span><span class="p">,</span>
+ <span class="s">"protocol"</span> <span class="o">:</span> <span class="s">"http"</span><span class="p">,</span>
+ <span class="s">"type"</span> <span class="o">:</span> <span class="s">"url"</span><span class="p">,</span>
+ <span class="s">"description"</span> <span class="o">:</span> <span class="s">"Agent API"</span>
+ <span class="p">}</span>
+ <span class="p">},</span>
+ <span class="s">"settings"</span> <span class="o">:</span> <span class="p">{</span> <span class="p">}</span>
+ <span class="p">},</span>
+
+ <span class="s">"externalView"</span> <span class="o">:</span> <span class="p">{</span>
+ <span class="s">"endpoints"</span> <span class="o">:</span> <span class="p">{</span>
+ <span class="s">"/mgmt"</span> <span class="o">:</span> <span class="p">{</span>
+ <span class="s">"value"</span> <span class="o">:</span> <span class="s">"http://stevel-8.local:62552/ws/v1/slider/mgmt"</span><span class="p">,</span>
+ <span class="s">"protocol"</span> <span class="o">:</span> <span class="s">"http"</span><span class="p">,</span>
+ <span class="s">"type"</span> <span class="o">:</span> <span class="s">"url"</span><span class="p">,</span>
+ <span class="s">"description"</span> <span class="o">:</span> <span class="s">"Management API"</span>
+ <span class="p">},</span>
+
+ <span class="s">"slider/IPC"</span> <span class="o">:</span> <span class="p">{</span>
+ <span class="s">"value"</span> <span class="o">:</span> <span class="s">"stevel-8.local/192.168.1.101:62550"</span><span class="p">,</span>
+ <span class="s">"protocol"</span> <span class="o">:</span> <span class="s">"org.apache.hadoop.ipc.Protobuf"</span><span class="p">,</span>
+ <span class="s">"type"</span> <span class="o">:</span> <span class="s">"address"</span><span class="p">,</span>
+ <span class="s">"description"</span> <span class="o">:</span> <span class="s">"Slider AM RPC"</span>
+ <span class="p">},</span>
+ <span class="s">"registry"</span> <span class="o">:</span> <span class="p">{</span>
+ <span class="s">"value"</span> <span class="o">:</span> <span class="s">"http://stevel-8.local:62552/ws/registry"</span><span class="p">,</span>
+ <span class="s">"protocol"</span> <span class="o">:</span> <span class="s">"http"</span><span class="p">,</span>
+ <span class="s">"type"</span> <span class="o">:</span> <span class="s">"url"</span><span class="p">,</span>
+ <span class="s">"description"</span> <span class="o">:</span> <span class="s">"Registry"</span>
+ <span class="p">}</span>
+ <span class="p">},</span>
+ <span class="s">"settings"</span> <span class="o">:</span> <span class="p">{</span> <span class="p">}</span>
+ <span class="p">}</span>
+<span class="p">}</span>
+</pre></div>
</div>
<div id="footer">
Modified: incubator/slider/site/content/design/registry/service_registry_end_to_end_scenario.html
URL: http://svn.apache.org/viewvc/incubator/slider/site/content/design/registry/service_registry_end_to_end_scenario.html?rev=1608633&r1=1608632&r2=1608633&view=diff
==============================================================================
--- incubator/slider/site/content/design/registry/service_registry_end_to_end_scenario.html (original)
+++ incubator/slider/site/content/design/registry/service_registry_end_to_end_scenario.html Tue Jul 8 00:49:34 2014
@@ -274,7 +274,8 @@ Latest release: <strong>0.30-incubating<
<li>only retrieves configuration information from slider-deployed application instances. <em>We do not need to restrict ourselves here.</em></li>
</ul>
<h2 id="slider-client-lists-applications">Slider Client lists applications</h2>
-<pre class="codehilite"><code>slider registry --list [--servicetype <application-type>]</code></pre>
+<div class="codehilite"><pre><span class="n">slider</span> <span class="n">registry</span> <span class="o">--</span><span class="n">list</span> <span class="p">[</span><span class="o">--</span><span class="n">servicetype</span> <span class="o"><</span><span class="n">application</span><span class="o">-</span><span class="n">type</span><span class="o">></span><span class="p">]</span>
+</pre></div>
<ol>
@@ -289,7 +290,8 @@ Latest release: <strong>0.30-incubating<
</li>
</ol>
<h2 id="slider-client-lists-content-published-by-an-application-instance">Slider Client lists content published by an application instance</h2>
-<pre class="codehilite"><code>slider registry <instance> --listconf [--servicetype <application-type>]</code></pre>
+<div class="codehilite"><pre><span class="nx">slider</span> <span class="nx">registry</span> <span class="o"><</span><span class="nx">instance</span><span class="o">></span> <span class="o">--</span><span class="nx">listconf</span> <span class="err">[</span><span class="o">--</span><span class="nx">servicetype</span> <span class="o"><</span><span class="nx">application</span><span class="na">-type</span><span class="o">></span><span class="cp">]</span>
+</pre></div>
<ol>
@@ -310,7 +312,8 @@ Latest release: <strong>0.30-incubating<
</li>
</ol>
<h2 id="slider-client-retrieves-content-published-by-an-application-instance">Slider Client retrieves content published by an application instance</h2>
-<pre class="codehilite"><code>slider registry <instance> --getconf <document> [--format (xml|properties|text|html|csv|yaml|json,...) [--dest <file>] [--servicetype <application-type>]</code></pre>
+<div class="codehilite"><pre><span class="nx">slider</span> <span class="nx">registry</span> <span class="o"><</span><span class="nx">instance</span><span class="o">></span> <span class="o">--</span><span class="nx">getconf</span> <span class="o"><</span><span class="nb">document</span><span class="o">></span> <span class="err">[</span><span class="o">--</span><span class="nb">format</span> <span class="p">(</span><span class="kt">xml</span><span class="o">|</span><span class="nb">properties</span><span class="o">|</span><span class="nx">text</span><span class="o">|</span><span class="nx">html</span><span class="o">|</span><span class="nx">csv</span><span class="o">|</span><span class="nx">yaml</span><span class="o">|</span><span class="nx">json</span><span class="p">,</span><span class="nx">...</span><span class="p">)</span> <span class="err">[</span><span class="o">--</span><span class="nx">dest</span> <span class="o"><</span><span class="nb">file</span><span class=
"o">></span><span class="cp">]</span> <span class="cp">[</span><span class="o">--</span><span class="nx">servicetype</span> <span class="o"><</span><span class="nx">application</span><span class="na">-type</span><span class="o">></span><span class="cp">]</span>
+</pre></div>
<ol>
@@ -344,7 +347,8 @@ a file underneath.</p>
<p>Here a set of documents published is retrieved in the desired format of an application.</p>
<h2 id="slider-client-retrieves-document-and-applies-template-to-it">Slider Client retrieves document and applies template to it</h2>
<p>Here a set of documents published is retrieved in the desired format of an application.</p>
-<pre class="codehilite"><code>slider registry <instance> --source <document> [--template <path-to-template>] [--outfile <file>] [--servicetype <application-type>]</code></pre>
+<div class="codehilite"><pre><span class="nx">slider</span> <span class="nx">registry</span> <span class="o"><</span><span class="nx">instance</span><span class="o">></span> <span class="o">--</span><span class="nb">source</span> <span class="o"><</span><span class="nb">document</span><span class="o">></span> <span class="err">[</span><span class="o">--</span><span class="nx">template</span> <span class="o"><</span><span class="nb">path</span><span class="na">-to-template</span><span class="o">></span><span class="cp">]</span> <span class="cp">[</span><span class="o">--</span><span class="nx">outfile</span> <span class="o"><</span><span class="nb">file</span><span class="o">></span><span class="cp">]</span> <span class="cp">[</span><span class="o">--</span><span class="nx">servicetype</span> <span class="o"><</span><span class="nx">application</span><span class="na">-type</span><span class="o">></span><span class="cp">]</span>
+</pre></div>
<ol>
Modified: incubator/slider/site/content/design/rolehistory.html
URL: http://svn.apache.org/viewvc/incubator/slider/site/content/design/rolehistory.html?rev=1608633&r1=1608632&r2=1608633&view=diff
==============================================================================
--- incubator/slider/site/content/design/rolehistory.html (original)
+++ incubator/slider/site/content/design/rolehistory.html Tue Jul 8 00:49:34 2014
@@ -410,32 +410,35 @@ parts of the YARN cluster. </li>
</ul>
<h2 id="data-structures">Data Structures</h2>
<h3 id="rolehistory">RoleHistory</h3>
-<pre class="codehilite"><code>startTime: long
-saveTime: long
-dirty: boolean
-nodemap: NodeMap
-roles: RoleStatus[]
-outstandingRequests: transient OutstandingRequestTracker
-availableNodes: transient List<NodeInstance>[]</code></pre>
+<div class="codehilite"><pre><span class="n">startTime</span><span class="o">:</span> <span class="n">long</span>
+<span class="n">saveTime</span><span class="o">:</span> <span class="n">long</span>
+<span class="n">dirty</span><span class="o">:</span> <span class="n">boolean</span>
+<span class="n">nodemap</span><span class="o">:</span> <span class="n">NodeMap</span>
+<span class="n">roles</span><span class="o">:</span> <span class="n">RoleStatus</span><span class="o">[]</span>
+<span class="n">outstandingRequests</span><span class="o">:</span> <span class="n">transient</span> <span class="n">OutstandingRequestTracker</span>
+<span class="n">availableNodes</span><span class="o">:</span> <span class="n">transient</span> <span class="n">List</span><span class="o"><</span><span class="n">NodeInstance</span><span class="o">>[]</span>
+</pre></div>
<p>This is the aggregate data structure that is persisted to/from file</p>
<h3 id="nodemap">NodeMap</h3>
-<pre class="codehilite"><code>clusterNodes: Map: NodeId -> NodeInstance
-clusterNodes(): Iterable<NodeInstance>
-getOrCreate(NodeId): NodeInstance</code></pre>
+<div class="codehilite"><pre><span class="n">clusterNodes</span><span class="o">:</span> <span class="n">Map</span><span class="o">:</span> <span class="n">NodeId</span> <span class="o">-></span> <span class="n">NodeInstance</span>
+<span class="n">clusterNodes</span><span class="o">():</span> <span class="n">Iterable</span><span class="o"><</span><span class="n">NodeInstance</span><span class="o">></span>
+<span class="n">getOrCreate</span><span class="o">(</span><span class="n">NodeId</span><span class="o">):</span> <span class="n">NodeInstance</span>
+</pre></div>
<p>Maps a YARN NodeID record to a Slider <code>NodeInstance</code> structure</p>
<h3 id="nodeinstance">NodeInstance</h3>
<p>Every node in the cluster is modeled as an ragged array of <code>NodeEntry</code> instances, indexed
by role index -</p>
-<pre class="codehilite"><code>NodeEntry[roles]
-get(roleId): NodeEntry or null
-create(roleId): NodeEntry
-getNodeEntries(): NodeEntry[roles]
-getOrCreate(roleId): NodeEntry
-remove(roleId): NodeEntry</code></pre>
+<div class="codehilite"><pre><span class="n">NodeEntry</span><span class="p">[</span><span class="n">roles</span><span class="p">]</span>
+<span class="n">get</span><span class="p">(</span><span class="n">roleId</span><span class="p">)</span><span class="o">:</span> <span class="n">NodeEntry</span> <span class="n">or</span> <span class="n">null</span>
+<span class="n">create</span><span class="p">(</span><span class="n">roleId</span><span class="p">)</span><span class="o">:</span> <span class="n">NodeEntry</span>
+<span class="n">getNodeEntries</span><span class="p">()</span><span class="o">:</span> <span class="n">NodeEntry</span><span class="p">[</span><span class="n">roles</span><span class="p">]</span>
+<span class="n">getOrCreate</span><span class="p">(</span><span class="n">roleId</span><span class="p">)</span><span class="o">:</span> <span class="n">NodeEntry</span>
+<span class="n">remove</span><span class="p">(</span><span class="n">roleId</span><span class="p">)</span><span class="o">:</span> <span class="n">NodeEntry</span>
+</pre></div>
<p>This could be implemented in a map or an indexed array; the array is more
@@ -443,12 +446,13 @@ efficient but it does mandate that the n
<h3 id="nodeentry">NodeEntry</h3>
<p>Records the details about all of a roles containers on a node. The
<code>active</code> field records the number of containers currently active.</p>
-<pre class="codehilite"><code>active: int
-requested: transient int
-releasing: transient int
-last_used: long
+<div class="codehilite"><pre><span class="n">active</span><span class="o">:</span> <span class="n">int</span>
+<span class="n">requested</span><span class="o">:</span> <span class="n">transient</span> <span class="n">int</span>
+<span class="n">releasing</span><span class="o">:</span> <span class="n">transient</span> <span class="n">int</span>
+<span class="n">last_used</span><span class="o">:</span> <span class="n">long</span>
-NodeEntry.available(): boolean = active - releasing == 0 && requested == 0</code></pre>
+<span class="n">NodeEntry</span><span class="o">.</span><span class="na">available</span><span class="o">():</span> <span class="n">boolean</span> <span class="o">=</span> <span class="n">active</span> <span class="o">-</span> <span class="n">releasing</span> <span class="o">==</span> <span class="mi">0</span> <span class="o">&&</span> <span class="n">requested</span> <span class="o">==</span> <span class="mi">0</span>
+</pre></div>
<p>The two fields <code>releasing</code> and <code>requested</code> are used to track the ongoing
@@ -492,11 +496,12 @@ for uniform access.</p>
<p>Tracks an outstanding request. This is used to correlate an allocation response
(whose Container Priority file is used to locate this request), with the
node and role used in the request.</p>
-<pre class="codehilite"><code> roleId: int
- requestID : int
- node: string (may be null)
- requestedTime: long
- priority: int = requestID << 24 | roleId</code></pre>
+<div class="codehilite"><pre> <span class="nl">roleId:</span> <span class="kt">int</span>
+ <span class="n">requestID</span> <span class="o">:</span> <span class="kt">int</span>
+ <span class="nl">node:</span> <span class="n">string</span> <span class="p">(</span><span class="n">may</span> <span class="n">be</span> <span class="n">null</span><span class="p">)</span>
+ <span class="nl">requestedTime:</span> <span class="kt">long</span>
+ <span class="nl">priority:</span> <span class="kt">int</span> <span class="o">=</span> <span class="n">requestID</span> <span class="o"><<</span> <span class="mi">24</span> <span class="o">|</span> <span class="n">roleId</span>
+</pre></div>
<p>The node identifier may be null -which indicates that a request was made without
@@ -504,23 +509,26 @@ a specific target node</p>
<h3 id="outstandingrequesttracker">OutstandingRequestTracker</h3>
<p>Contains a map from requestID to the specific <code>OutstandingRequest</code> made,
and generates the request ID</p>
-<pre class="codehilite"><code>nextRequestId: int
-requestMap(RequestID) -> OutstandingRequest</code></pre>
+<div class="codehilite"><pre><span class="n">nextRequestId</span><span class="o">:</span> <span class="n">int</span>
+<span class="n">requestMap</span><span class="o">(</span><span class="n">RequestID</span><span class="o">)</span> <span class="o">-></span> <span class="n">OutstandingRequest</span>
+</pre></div>
<p>Operations</p>
-<pre class="codehilite"><code>addRequest(NodeInstance, RoleId): OutstandingRequest
- (and an updated request Map with a new entry)
-lookup(RequestID): OutstandingRequest
-remove(RequestID): OutstandingRequest
-listRequestsForNode(ClusterID): [OutstandingRequest]</code></pre>
+<div class="codehilite"><pre><span class="n">addRequest</span><span class="p">(</span><span class="n">NodeInstance</span><span class="p">,</span> <span class="n">RoleId</span><span class="p">)</span><span class="o">:</span> <span class="n">OutstandingRequest</span>
+ <span class="p">(</span><span class="n">and</span> <span class="n">an</span> <span class="n">updated</span> <span class="n">request</span> <span class="n">Map</span> <span class="n">with</span> <span class="n">a</span> <span class="n">new</span> <span class="n">entry</span><span class="p">)</span>
+<span class="n">lookup</span><span class="p">(</span><span class="n">RequestID</span><span class="p">)</span><span class="o">:</span> <span class="n">OutstandingRequest</span>
+<span class="n">remove</span><span class="p">(</span><span class="n">RequestID</span><span class="p">)</span><span class="o">:</span> <span class="n">OutstandingRequest</span>
+<span class="n">listRequestsForNode</span><span class="p">(</span><span class="n">ClusterID</span><span class="p">)</span><span class="o">:</span> <span class="p">[</span><span class="n">OutstandingRequest</span><span class="p">]</span>
+</pre></div>
<p>The list operation can be implemented inefficiently unless it is found
to be important -if so a more complex structure will be needed.</p>
<h3 id="availablenodes">AvailableNodes</h3>
<p>This is a field in <code>RoleHistory</code></p>
-<pre class="codehilite"><code>availableNodes: List<NodeInstance>[]</code></pre>
+<div class="codehilite"><pre><span class="n">availableNodes</span><span class="o">:</span> <span class="n">List</span><span class="o"><</span><span class="n">NodeInstance</span><span class="o">>[]</span>
+</pre></div>
<p>For each role, lists nodes that are available for data-local allocation,
@@ -588,23 +596,24 @@ after the last save is unknown.</p>
3: Mark all nodes as active at save time to that of the</p>
<p>//define a threshold
threshold = rolehistory.saveTime - 7<em>24</em>60<em>60</em> 1000</p>
-<pre class="codehilite"><code>for (clusterId, clusternode) in rolehistory.clusterNodes().entries() :
- for (role, nodeEntry) in clusterNode.getNodeEntries():
- nodeEntry.requested = 0
- nodeEntry.releasing = 0
- if nodeEntry.active > 0 :
- nodeEntry.last_used = rolehistory.saveTime;
- nodeEntry.n.active = 0
- if nodeEntry.last_used < threshold :
- clusterNode.remove(role)
- else:
- availableNodes[role].add(clusterId)
- if clusterNode.getNodeEntries() isEmpty :
- rolehistory.clusterNodes.remove(clusterId)
-
-
-for availableNode in availableNodes:
- sort(availableNode,new last_used_comparator())</code></pre>
+<div class="codehilite"><pre><span class="k">for</span> <span class="p">(</span><span class="n">clusterId</span><span class="p">,</span> <span class="n">clusternode</span><span class="p">)</span> <span class="n">in</span> <span class="n">rolehistory</span><span class="p">.</span><span class="n">clusterNodes</span><span class="p">().</span><span class="n">entries</span><span class="p">()</span> <span class="o">:</span>
+ <span class="k">for</span> <span class="p">(</span><span class="n">role</span><span class="p">,</span> <span class="n">nodeEntry</span><span class="p">)</span> <span class="n">in</span> <span class="n">clusterNode</span><span class="p">.</span><span class="n">getNodeEntries</span><span class="p">()</span><span class="o">:</span>
+ <span class="n">nodeEntry</span><span class="p">.</span><span class="n">requested</span> <span class="o">=</span> <span class="mi">0</span>
+ <span class="n">nodeEntry</span><span class="p">.</span><span class="n">releasing</span> <span class="o">=</span> <span class="mi">0</span>
+ <span class="k">if</span> <span class="n">nodeEntry</span><span class="p">.</span><span class="n">active</span> <span class="o">></span> <span class="mi">0</span> <span class="o">:</span>
+ <span class="n">nodeEntry</span><span class="p">.</span><span class="n">last_used</span> <span class="o">=</span> <span class="n">rolehistory</span><span class="p">.</span><span class="n">saveTime</span><span class="p">;</span>
+ <span class="n">nodeEntry</span><span class="p">.</span><span class="n">n</span><span class="p">.</span><span class="n">active</span> <span class="o">=</span> <span class="mi">0</span>
+ <span class="k">if</span> <span class="n">nodeEntry</span><span class="p">.</span><span class="n">last_used</span> <span class="o"><</span> <span class="n">threshold</span> <span class="o">:</span>
+ <span class="n">clusterNode</span><span class="p">.</span><span class="n">remove</span><span class="p">(</span><span class="n">role</span><span class="p">)</span>
+ <span class="nl">else:</span>
+ <span class="n">availableNodes</span><span class="p">[</span><span class="n">role</span><span class="p">].</span><span class="n">add</span><span class="p">(</span><span class="n">clusterId</span><span class="p">)</span>
+ <span class="k">if</span> <span class="n">clusterNode</span><span class="p">.</span><span class="n">getNodeEntries</span><span class="p">()</span> <span class="n">isEmpty</span> <span class="o">:</span>
+ <span class="n">rolehistory</span><span class="p">.</span><span class="n">clusterNodes</span><span class="p">.</span><span class="n">remove</span><span class="p">(</span><span class="n">clusterId</span><span class="p">)</span>
+
+
+<span class="k">for</span> <span class="n">availableNode</span> <span class="n">in</span> <span class="n">availableNodes</span><span class="o">:</span>
+ <span class="n">sort</span><span class="p">(</span><span class="n">availableNode</span><span class="p">,</span><span class="n">new</span> <span class="n">last_used_comparator</span><span class="p">())</span>
+</pre></div>
<p>After this operation, the structures are purged with all out of date entries,
@@ -613,17 +622,18 @@ and the available node list contains a s
<p>1: Create the initial data structures as the thaw operation
2: update the structure with the list of live nodes, removing those nodes
from the list of available nodes</p>
-<pre class="codehilite"><code>now = time()
-activeContainers = RM.getActiveContainers()
+<div class="codehilite"><pre><span class="n">now</span> <span class="o">=</span> <span class="n">time</span><span class="p">()</span>
+<span class="n">activeContainers</span> <span class="o">=</span> <span class="n">RM</span><span class="p">.</span><span class="n">getActiveContainers</span><span class="p">()</span>
-for container in activeContainers:
- nodeId = container.nodeId
- clusterNode = roleHistory.nodemap.getOrCreate(nodeId)
- role = extractRoleId(container.getPriority)
- nodeEntry = clusterNode.getOrCreate(role)
- nodeEntry.active++
- nodeEntry.last_used = now
- availableNodes[role].remove(nodeId)</code></pre>
+<span class="k">for</span> <span class="n">container</span> <span class="n">in</span> <span class="n">activeContainers</span><span class="o">:</span>
+ <span class="n">nodeId</span> <span class="o">=</span> <span class="n">container</span><span class="p">.</span><span class="n">nodeId</span>
+ <span class="n">clusterNode</span> <span class="o">=</span> <span class="n">roleHistory</span><span class="p">.</span><span class="n">nodemap</span><span class="p">.</span><span class="n">getOrCreate</span><span class="p">(</span><span class="n">nodeId</span><span class="p">)</span>
+ <span class="n">role</span> <span class="o">=</span> <span class="n">extractRoleId</span><span class="p">(</span><span class="n">container</span><span class="p">.</span><span class="n">getPriority</span><span class="p">)</span>
+ <span class="n">nodeEntry</span> <span class="o">=</span> <span class="n">clusterNode</span><span class="p">.</span><span class="n">getOrCreate</span><span class="p">(</span><span class="n">role</span><span class="p">)</span>
+ <span class="n">nodeEntry</span><span class="p">.</span><span class="n">active</span><span class="o">++</span>
+ <span class="n">nodeEntry</span><span class="p">.</span><span class="n">last_used</span> <span class="o">=</span> <span class="n">now</span>
+ <span class="n">availableNodes</span><span class="p">[</span><span class="n">role</span><span class="p">].</span><span class="n">remove</span><span class="p">(</span><span class="n">nodeId</span><span class="p">)</span>
+</pre></div>
<p>There's no need to resort the available node list -all that has happened
@@ -647,15 +657,16 @@ should assume that the history was saved
of the Slider Cluster -ideally after the most recent change, and that the information
in it is only an approximate about what the previous state of the cluster was.</p>
<h3 id="flex-requesting-a-container-in-role-role">Flex: Requesting a container in role <code>role</code></h3>
-<pre class="codehilite"><code>node = availableNodes[roleId].pop()
-if node != null :
- node.nodeEntry[roleId].requested++;
-outstanding = outstandingRequestTracker.addRequest(node, roleId)
-request.node = node
-request.priority = outstanding.priority
-
-//update existing Slider role status
-roleStatus[roleId].incRequested();</code></pre>
+<div class="codehilite"><pre><span class="n">node</span> <span class="o">=</span> <span class="n">availableNodes</span><span class="p">[</span><span class="n">roleId</span><span class="p">].</span><span class="n">pop</span><span class="p">()</span>
+<span class="k">if</span> <span class="n">node</span> <span class="o">!=</span> <span class="n">null</span> <span class="o">:</span>
+ <span class="n">node</span><span class="p">.</span><span class="n">nodeEntry</span><span class="p">[</span><span class="n">roleId</span><span class="p">].</span><span class="n">requested</span><span class="o">++</span><span class="p">;</span>
+<span class="n">outstanding</span> <span class="o">=</span> <span class="n">outstandingRequestTracker</span><span class="p">.</span><span class="n">addRequest</span><span class="p">(</span><span class="n">node</span><span class="p">,</span> <span class="n">roleId</span><span class="p">)</span>
+<span class="n">request</span><span class="p">.</span><span class="n">node</span> <span class="o">=</span> <span class="n">node</span>
+<span class="n">request</span><span class="p">.</span><span class="n">priority</span> <span class="o">=</span> <span class="n">outstanding</span><span class="p">.</span><span class="n">priority</span>
+
+<span class="c1">//update existing Slider role status</span>
+<span class="n">roleStatus</span><span class="p">[</span><span class="n">roleId</span><span class="p">].</span><span class="n">incRequested</span><span class="p">();</span>
+</pre></div>
<p>There is a bias here towards previous nodes, even if the number of nodes
@@ -711,7 +722,8 @@ be rebuilt at thaw time.</p>
another data structure to maintain and rebuild at cluster thaw time
from the last-used fields in the node entries.</p>
<h3 id="am-callback-oncontainersallocated">AM Callback : onContainersAllocated</h3>
-<pre class="codehilite"><code>void onContainersAllocated(List<Container> allocatedContainers)</code></pre>
+<div class="codehilite"><pre><span class="bp">void</span> <span class="nx">onContainersAllocated</span><span class="p">(</span><span class="nb">List</span><span class="o"><</span><span class="nx">Container</span><span class="o">></span> <span class="nx">allocatedContainers</span><span class="p">)</span>
+</pre></div>
<p>This is the callback received when containers have been allocated.
@@ -723,44 +735,45 @@ will need to correlate allocations with
the node-specific request count. Decrementing the request count
on the allocated node will not work, as the allocation may not be
to the node originally requested.</p>
-<pre class="codehilite"><code>assignments = []
-operations = []
-for container in allocatedContainers:
- cid = container.getId();
- roleId = container.priority & 0xff
- nodeId = container.nodeId
- outstanding = outstandingRequestTracker.remove(C.priority)
- roleStatus = lookupRoleStatus(container);
- roleStatus.decRequested();
- allocated = roleStatus.incActual();
- if outstanding == null || allocated > desired :
- operations.add(new ContainerReleaseOperation(cid))
- surplusNodes.add(cid);
- surplusContainers++
- roleStatus.decActual();
- else:
- assignments.add(new ContainerAssignment(container, role))
- node = nodemap.getOrCreate(nodeId)
- nodeentry = node.get(roleId)
- if nodeentry == null :
- nodeentry = new NodeEntry()
- node[roleId] = nodeentry
- nodeentry.active = 1
- else:
- if nodeentry.requested > 0 :
- nodeentry.requested--
- nodeentry.active++
- nodemap.dirty = true
-
- // work back from request ID to node where the
- // request was outstanding
- requestID = outstanding != null? outstanding.nodeId : null
- if requestID != null:
- reqNode = nodeMap.get(requestID)
- reqNodeEntry = reqNode.get(roleId)
- reqNodeEntry.requested--
- if reqNodeEntry.available() :
- availableNodeList.insert(reqNodeEntry)</code></pre>
+<div class="codehilite"><pre><span class="n">assignments</span> <span class="o">=</span> <span class="p">[]</span>
+<span class="n">operations</span> <span class="o">=</span> <span class="p">[]</span>
+<span class="k">for</span> <span class="n">container</span> <span class="n">in</span> <span class="n">allocatedContainers</span><span class="o">:</span>
+ <span class="n">cid</span> <span class="o">=</span> <span class="n">container</span><span class="p">.</span><span class="n">getId</span><span class="p">();</span>
+ <span class="n">roleId</span> <span class="o">=</span> <span class="n">container</span><span class="p">.</span><span class="n">priority</span> <span class="o">&</span> <span class="mh">0xff</span>
+ <span class="n">nodeId</span> <span class="o">=</span> <span class="n">container</span><span class="p">.</span><span class="n">nodeId</span>
+ <span class="n">outstanding</span> <span class="o">=</span> <span class="n">outstandingRequestTracker</span><span class="p">.</span><span class="n">remove</span><span class="p">(</span><span class="n">C</span><span class="p">.</span><span class="n">priority</span><span class="p">)</span>
+ <span class="n">roleStatus</span> <span class="o">=</span> <span class="n">lookupRoleStatus</span><span class="p">(</span><span class="n">container</span><span class="p">);</span>
+ <span class="n">roleStatus</span><span class="p">.</span><span class="n">decRequested</span><span class="p">();</span>
+ <span class="n">allocated</span> <span class="o">=</span> <span class="n">roleStatus</span><span class="p">.</span><span class="n">incActual</span><span class="p">();</span>
+ <span class="k">if</span> <span class="n">outstanding</span> <span class="o">==</span> <span class="n">null</span> <span class="o">||</span> <span class="n">allocated</span> <span class="o">></span> <span class="n">desired</span> <span class="o">:</span>
+ <span class="n">operations</span><span class="p">.</span><span class="n">add</span><span class="p">(</span><span class="n">new</span> <span class="n">ContainerReleaseOperation</span><span class="p">(</span><span class="n">cid</span><span class="p">))</span>
+ <span class="n">surplusNodes</span><span class="p">.</span><span class="n">add</span><span class="p">(</span><span class="n">cid</span><span class="p">);</span>
+ <span class="n">surplusContainers</span><span class="o">++</span>
+ <span class="n">roleStatus</span><span class="p">.</span><span class="n">decActual</span><span class="p">();</span>
+ <span class="nl">else:</span>
+ <span class="n">assignments</span><span class="p">.</span><span class="n">add</span><span class="p">(</span><span class="n">new</span> <span class="n">ContainerAssignment</span><span class="p">(</span><span class="n">container</span><span class="p">,</span> <span class="n">role</span><span class="p">))</span>
+ <span class="n">node</span> <span class="o">=</span> <span class="n">nodemap</span><span class="p">.</span><span class="n">getOrCreate</span><span class="p">(</span><span class="n">nodeId</span><span class="p">)</span>
+ <span class="n">nodeentry</span> <span class="o">=</span> <span class="n">node</span><span class="p">.</span><span class="n">get</span><span class="p">(</span><span class="n">roleId</span><span class="p">)</span>
+ <span class="k">if</span> <span class="n">nodeentry</span> <span class="o">==</span> <span class="n">null</span> <span class="o">:</span>
+ <span class="n">nodeentry</span> <span class="o">=</span> <span class="n">new</span> <span class="n">NodeEntry</span><span class="p">()</span>
+ <span class="n">node</span><span class="p">[</span><span class="n">roleId</span><span class="p">]</span> <span class="o">=</span> <span class="n">nodeentry</span>
+ <span class="n">nodeentry</span><span class="p">.</span><span class="n">active</span> <span class="o">=</span> <span class="mi">1</span>
+ <span class="nl">else:</span>
+ <span class="k">if</span> <span class="n">nodeentry</span><span class="p">.</span><span class="n">requested</span> <span class="o">></span> <span class="mi">0</span> <span class="o">:</span>
+ <span class="n">nodeentry</span><span class="p">.</span><span class="n">requested</span><span class="o">--</span>
+ <span class="n">nodeentry</span><span class="p">.</span><span class="n">active</span><span class="o">++</span>
+ <span class="n">nodemap</span><span class="p">.</span><span class="n">dirty</span> <span class="o">=</span> <span class="nb">true</span>
+
+ <span class="c1">// work back from request ID to node where the </span>
+ <span class="c1">// request was outstanding</span>
+ <span class="n">requestID</span> <span class="o">=</span> <span class="n">outstanding</span> <span class="o">!=</span> <span class="n">null</span><span class="o">?</span> <span class="n">outstanding</span><span class="p">.</span><span class="n">nodeId</span> <span class="o">:</span> <span class="n">null</span>
+ <span class="k">if</span> <span class="n">requestID</span> <span class="o">!=</span> <span class="n">null</span><span class="o">:</span>
+ <span class="n">reqNode</span> <span class="o">=</span> <span class="n">nodeMap</span><span class="p">.</span><span class="n">get</span><span class="p">(</span><span class="n">requestID</span><span class="p">)</span>
+ <span class="n">reqNodeEntry</span> <span class="o">=</span> <span class="n">reqNode</span><span class="p">.</span><span class="n">get</span><span class="p">(</span><span class="n">roleId</span><span class="p">)</span>
+ <span class="n">reqNodeEntry</span><span class="p">.</span><span class="n">requested</span><span class="o">--</span>
+ <span class="k">if</span> <span class="n">reqNodeEntry</span><span class="p">.</span><span class="n">available</span><span class="p">()</span> <span class="o">:</span>
+ <span class="n">availableNodeList</span><span class="p">.</span><span class="n">insert</span><span class="p">(</span><span class="n">reqNodeEntry</span><span class="p">)</span>
+</pre></div>
<ol>
@@ -782,7 +795,8 @@ in the total ordering of the list.</p>
</li>
</ol>
<h3 id="nmclientasync-callback-oncontainerstarted">NMClientAsync Callback: onContainerStarted()</h3>
-<pre class="codehilite"><code>onContainerStarted(ContainerId containerId)</code></pre>
+<div class="codehilite"><pre><span class="n">onContainerStarted</span><span class="p">(</span><span class="n">ContainerId</span> <span class="n">containerId</span><span class="p">)</span>
+</pre></div>
<p>The AM uses this as a signal to remove the container from the list
@@ -796,14 +810,16 @@ NodeEntry is decremented. If the node is
container, it is returned to the queue of available nodes.</p>
<h3 id="flex-releasing-a-role-instance-from-the-cluster">Flex: Releasing a role instance from the cluster</h3>
<p>Simple strategy: find a node with at least one active container</p>
-<pre class="codehilite"><code>select a node N in nodemap where for NodeEntry[roleId]: active > releasing;
-nodeentry = node.get(roleId)
-nodeentry.active--;</code></pre>
+<div class="codehilite"><pre><span class="n">select</span> <span class="n">a</span> <span class="n">node</span> <span class="n">N</span> <span class="n">in</span> <span class="n">nodemap</span> <span class="n">where</span> <span class="k">for</span> <span class="n">NodeEntry</span><span class="p">[</span><span class="n">roleId</span><span class="p">]</span><span class="o">:</span> <span class="n">active</span> <span class="o">></span> <span class="n">releasing</span><span class="p">;</span>
+<span class="n">nodeentry</span> <span class="o">=</span> <span class="n">node</span><span class="p">.</span><span class="n">get</span><span class="p">(</span><span class="n">roleId</span><span class="p">)</span>
+<span class="n">nodeentry</span><span class="p">.</span><span class="n">active</span><span class="o">--</span><span class="p">;</span>
+</pre></div>
<p>Advanced Strategy:</p>
-<pre class="codehilite"><code>Scan through the map looking for a node where active >1 && active > releasing.
-If none are found, fall back to the previous strategy</code></pre>
+<div class="codehilite"><pre><span class="n">Scan</span> <span class="n">through</span> <span class="n">the</span> <span class="n">map</span> <span class="n">looking</span> <span class="k">for</span> <span class="n">a</span> <span class="n">node</span> <span class="n">where</span> <span class="n">active</span> <span class="o">></span><span class="mi">1</span> <span class="o">&&</span> <span class="n">active</span> <span class="o">></span> <span class="n">releasing</span><span class="p">.</span>
+<span class="n">If</span> <span class="n">none</span> <span class="n">are</span> <span class="n">found</span><span class="p">,</span> <span class="n">fall</span> <span class="n">back</span> <span class="n">to</span> <span class="n">the</span> <span class="n">previous</span> <span class="n">strategy</span>
+</pre></div>
<p>This is guaranteed to release a container on any node with >1 container in use,
@@ -817,7 +833,8 @@ must: be of the target role, and not alr
</ol>
<p>After the AM processes the request, it triggers a callback</p>
<h3 id="am-callback-oncontainerscompleted">AM callback onContainersCompleted:</h3>
-<pre class="codehilite"><code>void onContainersCompleted(List<ContainerStatus> completedContainers)</code></pre>
+<div class="codehilite"><pre><span class="bp">void</span> <span class="nx">onContainersCompleted</span><span class="p">(</span><span class="nb">List</span><span class="o"><</span><span class="nx">ContainerStatus</span><span class="o">></span> <span class="nx">completedContainers</span><span class="p">)</span>
+</pre></div>
<p>This callback returns a list of containers that have completed.</p>
@@ -828,36 +845,37 @@ for release, as well as which were rejec
any role allocated onto them.</p>
<p>A container is considered to have failed if it was an active container which
has completed although it wasn't on the list of containers to release</p>
-<pre class="codehilite"><code>shouldReview = false
-for container in completedContainers:
- containerId = container.containerId
- nodeId = container.nodeId
- node = nodemap.get(nodeId)
- if node == null :
- // unknown node
- continue
- roleId = node.roleId
- nodeentry = node.get(roleId)
- nodeentry.active--
- nodemap.dirty = true
- if getContainersBeingReleased().containsKey(containerId) :
- // handle container completion
- nodeentry.releasing --
-
- // update existing Slider role status
- roleStatus[roleId].decReleasing();
- containersBeingReleased.remove(containerId)
- else:
- //failure of a live node
- roleStatus[roleId].decActual();
- shouldReview = true
-
- if nodeentry.available():
- nodentry.last_used = now()
- availableNodes[roleId].insert(node)
- //trigger a comparison of actual vs desired
-if shouldReview :
- reviewRequestAndReleaseNodes()</code></pre>
+<div class="codehilite"><pre><span class="n">shouldReview</span> <span class="o">=</span> <span class="nb">false</span>
+<span class="k">for</span> <span class="n">container</span> <span class="n">in</span> <span class="n">completedContainers</span><span class="o">:</span>
+ <span class="n">containerId</span> <span class="o">=</span> <span class="n">container</span><span class="p">.</span><span class="n">containerId</span>
+ <span class="n">nodeId</span> <span class="o">=</span> <span class="n">container</span><span class="p">.</span><span class="n">nodeId</span>
+ <span class="n">node</span> <span class="o">=</span> <span class="n">nodemap</span><span class="p">.</span><span class="n">get</span><span class="p">(</span><span class="n">nodeId</span><span class="p">)</span>
+ <span class="k">if</span> <span class="n">node</span> <span class="o">==</span> <span class="n">null</span> <span class="o">:</span>
+ <span class="c1">// unknown node</span>
+ <span class="k">continue</span>
+ <span class="n">roleId</span> <span class="o">=</span> <span class="n">node</span><span class="p">.</span><span class="n">roleId</span>
+ <span class="n">nodeentry</span> <span class="o">=</span> <span class="n">node</span><span class="p">.</span><span class="n">get</span><span class="p">(</span><span class="n">roleId</span><span class="p">)</span>
+ <span class="n">nodeentry</span><span class="p">.</span><span class="n">active</span><span class="o">--</span>
+ <span class="n">nodemap</span><span class="p">.</span><span class="n">dirty</span> <span class="o">=</span> <span class="nb">true</span>
+ <span class="k">if</span> <span class="n">getContainersBeingReleased</span><span class="p">().</span><span class="n">containsKey</span><span class="p">(</span><span class="n">containerId</span><span class="p">)</span> <span class="o">:</span>
+ <span class="c1">// handle container completion</span>
+ <span class="n">nodeentry</span><span class="p">.</span><span class="n">releasing</span> <span class="o">--</span>
+
+ <span class="c1">// update existing Slider role status</span>
+ <span class="n">roleStatus</span><span class="p">[</span><span class="n">roleId</span><span class="p">].</span><span class="n">decReleasing</span><span class="p">();</span>
+ <span class="n">containersBeingReleased</span><span class="p">.</span><span class="n">remove</span><span class="p">(</span><span class="n">containerId</span><span class="p">)</span>
+ <span class="nl">else:</span>
+ <span class="c1">//failure of a live node</span>
+ <span class="n">roleStatus</span><span class="p">[</span><span class="n">roleId</span><span class="p">].</span><span class="n">decActual</span><span class="p">();</span>
+ <span class="n">shouldReview</span> <span class="o">=</span> <span class="nb">true</span>
+
+ <span class="k">if</span> <span class="n">nodeentry</span><span class="p">.</span><span class="n">available</span><span class="p">()</span><span class="o">:</span>
+ <span class="n">nodentry</span><span class="p">.</span><span class="n">last_used</span> <span class="o">=</span> <span class="n">now</span><span class="p">()</span>
+ <span class="n">availableNodes</span><span class="p">[</span><span class="n">roleId</span><span class="p">].</span><span class="n">insert</span><span class="p">(</span><span class="n">node</span><span class="p">)</span>
+ <span class="c1">//trigger a comparison of actual vs desired</span>
+<span class="k">if</span> <span class="n">shouldReview</span> <span class="o">:</span>
+ <span class="n">reviewRequestAndReleaseNodes</span><span class="p">()</span>
+</pre></div>
<p>By calling <code>reviewRequestAndReleaseNodes()</code> the AM triggers
@@ -981,21 +999,24 @@ less efficient than a binary format, it
<p>Here are sequence of entries from a test run on a single node cluster; running 1 HBase Master
and two region servers.</p>
<p>Initial save; the instance of Role 1 (HBase master) is live, Role 2 (RS) is not.</p>
-<pre class="codehilite"><code>{"entry":{"org.apache.hoya.avro.RoleHistoryHeader":{"version":1,"saved":1384183475949,"savedx":"14247c3aeed","roles":3}}}
-{"entry":{"org.apache.hoya.avro.NodeEntryRecord":{"host":"192.168.1.85","role":1,"active":true,"last_used":0}}}
-{"entry":{"org.apache.hoya.avro.NodeEntryRecord":{"host":"192.168.1.85","role":2,"active":false,"last_used":0}}}</code></pre>
+<div class="codehilite"><pre><span class="p">{</span><span class="s">"entry"</span><span class="o">:</span><span class="p">{</span><span class="s">"org.apache.hoya.avro.RoleHistoryHeader"</span><span class="o">:</span><span class="p">{</span><span class="s">"version"</span><span class="o">:</span><span class="mi">1</span><span class="p">,</span><span class="s">"saved"</span><span class="o">:</span><span class="mi">1384183475949</span><span class="p">,</span><span class="s">"savedx"</span><span class="o">:</span><span class="s">"14247c3aeed"</span><span class="p">,</span><span class="s">"roles"</span><span class="o">:</span><span class="mi">3</span><span class="p">}}}</span>
+<span class="p">{</span><span class="s">"entry"</span><span class="o">:</span><span class="p">{</span><span class="s">"org.apache.hoya.avro.NodeEntryRecord"</span><span class="o">:</span><span class="p">{</span><span class="s">"host"</span><span class="o">:</span><span class="s">"192.168.1.85"</span><span class="p">,</span><span class="s">"role"</span><span class="o">:</span><span class="mi">1</span><span class="p">,</span><span class="s">"active"</span><span class="o">:</span><span class="nb">true</span><span class="p">,</span><span class="s">"last_used"</span><span class="o">:</span><span class="mi">0</span><span class="p">}}}</span>
+<span class="p">{</span><span class="s">"entry"</span><span class="o">:</span><span class="p">{</span><span class="s">"org.apache.hoya.avro.NodeEntryRecord"</span><span class="o">:</span><span class="p">{</span><span class="s">"host"</span><span class="o">:</span><span class="s">"192.168.1.85"</span><span class="p">,</span><span class="s">"role"</span><span class="o">:</span><span class="mi">2</span><span class="p">,</span><span class="s">"active"</span><span class="o">:</span><span class="nb">false</span><span class="p">,</span><span class="s">"last_used"</span><span class="o">:</span><span class="mi">0</span><span class="p">}}}</span>
+</pre></div>
<p>At least one RS is live: </p>
-<pre class="codehilite"><code>{"entry":{"org.apache.hoya.avro.RoleHistoryFooter":{"count":2}}}{"entry":{"org.apache.hoya.avro.RoleHistoryHeader":{"version":1,"saved":1384183476010,"savedx":"14247c3af2a","roles":3}}}
-{"entry":{"org.apache.hoya.avro.NodeEntryRecord":{"host":"192.168.1.85","role":1,"active":true,"last_used":0}}}
-{"entry":{"org.apache.hoya.avro.NodeEntryRecord":{"host":"192.168.1.85","role":2,"active":true,"last_used":0}}}</code></pre>
+<div class="codehilite"><pre><span class="p">{</span><span class="s">"entry"</span><span class="o">:</span><span class="p">{</span><span class="s">"org.apache.hoya.avro.RoleHistoryFooter"</span><span class="o">:</span><span class="p">{</span><span class="s">"count"</span><span class="o">:</span><span class="mi">2</span><span class="p">}}}{</span><span class="s">"entry"</span><span class="o">:</span><span class="p">{</span><span class="s">"org.apache.hoya.avro.RoleHistoryHeader"</span><span class="o">:</span><span class="p">{</span><span class="s">"version"</span><span class="o">:</span><span class="mi">1</span><span class="p">,</span><span class="s">"saved"</span><span class="o">:</span><span class="mi">1384183476010</span><span class="p">,</span><span class="s">"savedx"</span><span class="o">:</span><span class="s">"14247c3af2a"</span><span class="p">,</span><span class="s">"roles"</
span><span class="o">:</span><span class="mi">3</span><span class="p">}}}</span>
+<span class="p">{</span><span class="s">"entry"</span><span class="o">:</span><span class="p">{</span><span class="s">"org.apache.hoya.avro.NodeEntryRecord"</span><span class="o">:</span><span class="p">{</span><span class="s">"host"</span><span class="o">:</span><span class="s">"192.168.1.85"</span><span class="p">,</span><span class="s">"role"</span><span class="o">:</span><span class="mi">1</span><span class="p">,</span><span class="s">"active"</span><span class="o">:</span><span class="nb">true</span><span class="p">,</span><span class="s">"last_used"</span><span class="o">:</span><span class="mi">0</span><span class="p">}}}</span>
+<span class="p">{</span><span class="s">"entry"</span><span class="o">:</span><span class="p">{</span><span class="s">"org.apache.hoya.avro.NodeEntryRecord"</span><span class="o">:</span><span class="p">{</span><span class="s">"host"</span><span class="o">:</span><span class="s">"192.168.1.85"</span><span class="p">,</span><span class="s">"role"</span><span class="o">:</span><span class="mi">2</span><span class="p">,</span><span class="s">"active"</span><span class="o">:</span><span class="nb">true</span><span class="p">,</span><span class="s">"last_used"</span><span class="o">:</span><span class="mi">0</span><span class="p">}}}</span>
+</pre></div>
<p>Another entry is saved -presumably the second RS is now live, which triggered another write</p>
-<pre class="codehilite"><code>{"entry":{"org.apache.hoya.avro.RoleHistoryFooter":{"count":2}}}{"entry":{"org.apache.hoya.avro.RoleHistoryHeader":{"version":1,"saved":1384183476028,"savedx":"14247c3af3c","roles":3}}}
-{"entry":{"org.apache.hoya.avro.NodeEntryRecord":{"host":"192.168.1.85","role":1,"active":true,"last_used":0}}}
-{"entry":{"org.apache.hoya.avro.NodeEntryRecord":{"host":"192.168.1.85","role":2,"active":true,"last_used":0}}}</code></pre>
+<div class="codehilite"><pre><span class="p">{</span><span class="s">"entry"</span><span class="o">:</span><span class="p">{</span><span class="s">"org.apache.hoya.avro.RoleHistoryFooter"</span><span class="o">:</span><span class="p">{</span><span class="s">"count"</span><span class="o">:</span><span class="mi">2</span><span class="p">}}}{</span><span class="s">"entry"</span><span class="o">:</span><span class="p">{</span><span class="s">"org.apache.hoya.avro.RoleHistoryHeader"</span><span class="o">:</span><span class="p">{</span><span class="s">"version"</span><span class="o">:</span><span class="mi">1</span><span class="p">,</span><span class="s">"saved"</span><span class="o">:</span><span class="mi">1384183476028</span><span class="p">,</span><span class="s">"savedx"</span><span class="o">:</span><span class="s">"14247c3af3c"</span><span class="p">,</span><span class="s">"roles"</
span><span class="o">:</span><span class="mi">3</span><span class="p">}}}</span>
+<span class="p">{</span><span class="s">"entry"</span><span class="o">:</span><span class="p">{</span><span class="s">"org.apache.hoya.avro.NodeEntryRecord"</span><span class="o">:</span><span class="p">{</span><span class="s">"host"</span><span class="o">:</span><span class="s">"192.168.1.85"</span><span class="p">,</span><span class="s">"role"</span><span class="o">:</span><span class="mi">1</span><span class="p">,</span><span class="s">"active"</span><span class="o">:</span><span class="nb">true</span><span class="p">,</span><span class="s">"last_used"</span><span class="o">:</span><span class="mi">0</span><span class="p">}}}</span>
+<span class="p">{</span><span class="s">"entry"</span><span class="o">:</span><span class="p">{</span><span class="s">"org.apache.hoya.avro.NodeEntryRecord"</span><span class="o">:</span><span class="p">{</span><span class="s">"host"</span><span class="o">:</span><span class="s">"192.168.1.85"</span><span class="p">,</span><span class="s">"role"</span><span class="o">:</span><span class="mi">2</span><span class="p">,</span><span class="s">"active"</span><span class="o">:</span><span class="nb">true</span><span class="p">,</span><span class="s">"last_used"</span><span class="o">:</span><span class="mi">0</span><span class="p">}}}</span>
+</pre></div>
<p>At this point the cluster was frozen and thawed. Slider does not save the cluster state
@@ -1005,21 +1026,24 @@ is given a last_used timestamp of that t
<p>When the history is next saved, the master has come back onto the (single) node,
it is active while its <code>last_used</code> timestamp is the previous file's timestamp.
No region servers are yet live.</p>
-<pre class="codehilite"><code>{"entry":{"org.apache.hoya.avro.RoleHistoryFooter":{"count":2}}}{"entry":{"org.apache.hoya.avro.RoleHistoryHeader":{"version":1,"saved":1384183512173,"savedx":"14247c43c6d","roles":3}}}
-{"entry":{"org.apache.hoya.avro.NodeEntryRecord":{"host":"192.168.1.85","role":1,"active":true,"last_used":1384183476028}}}
-{"entry":{"org.apache.hoya.avro.NodeEntryRecord":{"host":"192.168.1.85","role":2,"active":false,"last_used":1384183476028}}}</code></pre>
+<div class="codehilite"><pre><span class="p">{</span><span class="s">"entry"</span><span class="o">:</span><span class="p">{</span><span class="s">"org.apache.hoya.avro.RoleHistoryFooter"</span><span class="o">:</span><span class="p">{</span><span class="s">"count"</span><span class="o">:</span><span class="mi">2</span><span class="p">}}}{</span><span class="s">"entry"</span><span class="o">:</span><span class="p">{</span><span class="s">"org.apache.hoya.avro.RoleHistoryHeader"</span><span class="o">:</span><span class="p">{</span><span class="s">"version"</span><span class="o">:</span><span class="mi">1</span><span class="p">,</span><span class="s">"saved"</span><span class="o">:</span><span class="mi">1384183512173</span><span class="p">,</span><span class="s">"savedx"</span><span class="o">:</span><span class="s">"14247c43c6d"</span><span class="p">,</span><span class="s">"roles"</
span><span class="o">:</span><span class="mi">3</span><span class="p">}}}</span>
+<span class="p">{</span><span class="s">"entry"</span><span class="o">:</span><span class="p">{</span><span class="s">"org.apache.hoya.avro.NodeEntryRecord"</span><span class="o">:</span><span class="p">{</span><span class="s">"host"</span><span class="o">:</span><span class="s">"192.168.1.85"</span><span class="p">,</span><span class="s">"role"</span><span class="o">:</span><span class="mi">1</span><span class="p">,</span><span class="s">"active"</span><span class="o">:</span><span class="nb">true</span><span class="p">,</span><span class="s">"last_used"</span><span class="o">:</span><span class="mi">1384183476028</span><span class="p">}}}</span>
+<span class="p">{</span><span class="s">"entry"</span><span class="o">:</span><span class="p">{</span><span class="s">"org.apache.hoya.avro.NodeEntryRecord"</span><span class="o">:</span><span class="p">{</span><span class="s">"host"</span><span class="o">:</span><span class="s">"192.168.1.85"</span><span class="p">,</span><span class="s">"role"</span><span class="o">:</span><span class="mi">2</span><span class="p">,</span><span class="s">"active"</span><span class="o">:</span><span class="nb">false</span><span class="p">,</span><span class="s">"last_used"</span><span class="o">:</span><span class="mi">1384183476028</span><span class="p">}}}</span>
+</pre></div>
<p>Here a region server is live</p>
-<pre class="codehilite"><code>{"entry":{"org.apache.hoya.avro.RoleHistoryFooter":{"count":2}}}{"entry":{"org.apache.hoya.avro.RoleHistoryHeader":{"version":1,"saved":1384183512199,"savedx":"14247c43c87","roles":3}}}
-{"entry":{"org.apache.hoya.avro.NodeEntryRecord":{"host":"192.168.1.85","role":1,"active":true,"last_used":1384183476028}}}
-{"entry":{"org.apache.hoya.avro.NodeEntryRecord":{"host":"192.168.1.85","role":2,"active":true,"last_used":1384183476028}}}</code></pre>
+<div class="codehilite"><pre><span class="p">{</span><span class="s">"entry"</span><span class="o">:</span><span class="p">{</span><span class="s">"org.apache.hoya.avro.RoleHistoryFooter"</span><span class="o">:</span><span class="p">{</span><span class="s">"count"</span><span class="o">:</span><span class="mi">2</span><span class="p">}}}{</span><span class="s">"entry"</span><span class="o">:</span><span class="p">{</span><span class="s">"org.apache.hoya.avro.RoleHistoryHeader"</span><span class="o">:</span><span class="p">{</span><span class="s">"version"</span><span class="o">:</span><span class="mi">1</span><span class="p">,</span><span class="s">"saved"</span><span class="o">:</span><span class="mi">1384183512199</span><span class="p">,</span><span class="s">"savedx"</span><span class="o">:</span><span class="s">"14247c43c87"</span><span class="p">,</span><span class="s">"roles"</
span><span class="o">:</span><span class="mi">3</span><span class="p">}}}</span>
+<span class="p">{</span><span class="s">"entry"</span><span class="o">:</span><span class="p">{</span><span class="s">"org.apache.hoya.avro.NodeEntryRecord"</span><span class="o">:</span><span class="p">{</span><span class="s">"host"</span><span class="o">:</span><span class="s">"192.168.1.85"</span><span class="p">,</span><span class="s">"role"</span><span class="o">:</span><span class="mi">1</span><span class="p">,</span><span class="s">"active"</span><span class="o">:</span><span class="nb">true</span><span class="p">,</span><span class="s">"last_used"</span><span class="o">:</span><span class="mi">1384183476028</span><span class="p">}}}</span>
+<span class="p">{</span><span class="s">"entry"</span><span class="o">:</span><span class="p">{</span><span class="s">"org.apache.hoya.avro.NodeEntryRecord"</span><span class="o">:</span><span class="p">{</span><span class="s">"host"</span><span class="o">:</span><span class="s">"192.168.1.85"</span><span class="p">,</span><span class="s">"role"</span><span class="o">:</span><span class="mi">2</span><span class="p">,</span><span class="s">"active"</span><span class="o">:</span><span class="nb">true</span><span class="p">,</span><span class="s">"last_used"</span><span class="o">:</span><span class="mi">1384183476028</span><span class="p">}}}</span>
+</pre></div>
<p>And here, another region server has started. This does not actually change the contents of the file</p>
-<pre class="codehilite"><code>{"entry":{"org.apache.hoya.avro.RoleHistoryFooter":{"count":2}}}{"entry":{"org.apache.hoya.avro.RoleHistoryHeader":{"version":1,"saved":1384183512217,"savedx":"14247c43c99","roles":3}}}
-{"entry":{"org.apache.hoya.avro.NodeEntryRecord":{"host":"192.168.1.85","role":1,"active":true,"last_used":1384183476028}}}
-{"entry":{"org.apache.hoya.avro.NodeEntryRecord":{"host":"192.168.1.85","role":2,"active":true,"last_used":1384183476028}}}</code></pre>
+<div class="codehilite"><pre><span class="p">{</span><span class="s">"entry"</span><span class="o">:</span><span class="p">{</span><span class="s">"org.apache.hoya.avro.RoleHistoryFooter"</span><span class="o">:</span><span class="p">{</span><span class="s">"count"</span><span class="o">:</span><span class="mi">2</span><span class="p">}}}{</span><span class="s">"entry"</span><span class="o">:</span><span class="p">{</span><span class="s">"org.apache.hoya.avro.RoleHistoryHeader"</span><span class="o">:</span><span class="p">{</span><span class="s">"version"</span><span class="o">:</span><span class="mi">1</span><span class="p">,</span><span class="s">"saved"</span><span class="o">:</span><span class="mi">1384183512217</span><span class="p">,</span><span class="s">"savedx"</span><span class="o">:</span><span class="s">"14247c43c99"</span><span class="p">,</span><span class="s">"roles"</
span><span class="o">:</span><span class="mi">3</span><span class="p">}}}</span>
+<span class="p">{</span><span class="s">"entry"</span><span class="o">:</span><span class="p">{</span><span class="s">"org.apache.hoya.avro.NodeEntryRecord"</span><span class="o">:</span><span class="p">{</span><span class="s">"host"</span><span class="o">:</span><span class="s">"192.168.1.85"</span><span class="p">,</span><span class="s">"role"</span><span class="o">:</span><span class="mi">1</span><span class="p">,</span><span class="s">"active"</span><span class="o">:</span><span class="nb">true</span><span class="p">,</span><span class="s">"last_used"</span><span class="o">:</span><span class="mi">1384183476028</span><span class="p">}}}</span>
+<span class="p">{</span><span class="s">"entry"</span><span class="o">:</span><span class="p">{</span><span class="s">"org.apache.hoya.avro.NodeEntryRecord"</span><span class="o">:</span><span class="p">{</span><span class="s">"host"</span><span class="o">:</span><span class="s">"192.168.1.85"</span><span class="p">,</span><span class="s">"role"</span><span class="o">:</span><span class="mi">2</span><span class="p">,</span><span class="s">"active"</span><span class="o">:</span><span class="nb">true</span><span class="p">,</span><span class="s">"last_used"</span><span class="o">:</span><span class="mi">1384183476028</span><span class="p">}}}</span>
+</pre></div>
<p>The <code>last_used</code> timestamps will not be changed until the cluster is shrunk or thawed, as the <code>active</code> flag being set