You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@slider.apache.org by bu...@apache.org on 2015/11/02 18:24:48 UTC
svn commit: r971096 [1/8] - in /websites/staging/slider/trunk/content: ./
design/ design/registry/ design/specification/ developing/ docs/ docs/api/
docs/configuration/ docs/configuration/revision-1/ docs/slider_specs/
downloads/ release_notes/
Author: buildbot
Date: Mon Nov 2 17:24:47 2015
New Revision: 971096
Log:
Staging update by buildbot for slider
Modified:
websites/staging/slider/trunk/content/ (props changed)
websites/staging/slider/trunk/content/design/architecture.html
websites/staging/slider/trunk/content/design/index.html
websites/staging/slider/trunk/content/design/registry/a_YARN_service_registry.html
websites/staging/slider/trunk/content/design/registry/index.html
websites/staging/slider/trunk/content/design/registry/initial_registry_design.html
websites/staging/slider/trunk/content/design/registry/p2p_service_registries.html
websites/staging/slider/trunk/content/design/registry/references.html
websites/staging/slider/trunk/content/design/registry/registry-model.html
websites/staging/slider/trunk/content/design/registry/service_registry_end_to_end_scenario.html
websites/staging/slider/trunk/content/design/registry/the_YARN_application_registration_and_binding_problem.html
websites/staging/slider/trunk/content/design/rolehistory.html
websites/staging/slider/trunk/content/design/specification/cli-actions.html
websites/staging/slider/trunk/content/design/specification/index.html
websites/staging/slider/trunk/content/design/specification/slider-model.html
websites/staging/slider/trunk/content/design/ssl_implementation.html
websites/staging/slider/trunk/content/developing/building.html
websites/staging/slider/trunk/content/developing/chaosmonkey.html
websites/staging/slider/trunk/content/developing/debugging.html
websites/staging/slider/trunk/content/developing/demo.html
websites/staging/slider/trunk/content/developing/functional_tests.html
websites/staging/slider/trunk/content/developing/index.html
websites/staging/slider/trunk/content/developing/manual_testing.html
websites/staging/slider/trunk/content/developing/releasing.html
websites/staging/slider/trunk/content/developing/releasing_process_v1.html
websites/staging/slider/trunk/content/developing/style.html
websites/staging/slider/trunk/content/developing/submitting_patches.html
websites/staging/slider/trunk/content/developing/testing.html
websites/staging/slider/trunk/content/developing/windows.html
websites/staging/slider/trunk/content/docs/api/index.html
websites/staging/slider/trunk/content/docs/api/slider_REST_api_v2.html
websites/staging/slider/trunk/content/docs/api/slider_REST_v1.html
websites/staging/slider/trunk/content/docs/client-configuration.html
websites/staging/slider/trunk/content/docs/configuration/appconfig.html.html
websites/staging/slider/trunk/content/docs/configuration/core.html
websites/staging/slider/trunk/content/docs/configuration/index.html
websites/staging/slider/trunk/content/docs/configuration/internal.html
websites/staging/slider/trunk/content/docs/configuration/resources.html
websites/staging/slider/trunk/content/docs/configuration/revision-1/index.html
websites/staging/slider/trunk/content/docs/configuration/revision-1/redesign.html
websites/staging/slider/trunk/content/docs/configuration/revision-1/specification.html
websites/staging/slider/trunk/content/docs/examples.html
websites/staging/slider/trunk/content/docs/exitcodes.html
websites/staging/slider/trunk/content/docs/getting_started.html
websites/staging/slider/trunk/content/docs/high_availability.html
websites/staging/slider/trunk/content/docs/index.html
websites/staging/slider/trunk/content/docs/manpage.html
websites/staging/slider/trunk/content/docs/security.html
websites/staging/slider/trunk/content/docs/slider_specs/application_configuration.html
websites/staging/slider/trunk/content/docs/slider_specs/application_definition.html
websites/staging/slider/trunk/content/docs/slider_specs/application_instance_configuration.html
websites/staging/slider/trunk/content/docs/slider_specs/application_needs.html
websites/staging/slider/trunk/content/docs/slider_specs/application_package.html
websites/staging/slider/trunk/content/docs/slider_specs/application_pkg_docker.html
websites/staging/slider/trunk/content/docs/slider_specs/application_pkg_upgrade.html
websites/staging/slider/trunk/content/docs/slider_specs/canonical_scenarios.html
websites/staging/slider/trunk/content/docs/slider_specs/creating_app_definitions.html
websites/staging/slider/trunk/content/docs/slider_specs/hello_world_slider_app.html
websites/staging/slider/trunk/content/docs/slider_specs/index.html
websites/staging/slider/trunk/content/docs/slider_specs/simple_pkg.html
websites/staging/slider/trunk/content/docs/slider_specs/specifying_exports.html
websites/staging/slider/trunk/content/docs/slider_specs/writing_app_command_scripts.html
websites/staging/slider/trunk/content/docs/ssl.html
websites/staging/slider/trunk/content/docs/troubleshooting.html
websites/staging/slider/trunk/content/downloads/index.html
websites/staging/slider/trunk/content/index.html
websites/staging/slider/trunk/content/mailing_lists.html
websites/staging/slider/trunk/content/release_notes/release-0.30.0.html
websites/staging/slider/trunk/content/release_notes/release-0.40.0.html
websites/staging/slider/trunk/content/release_notes/release-0.50.2.html
websites/staging/slider/trunk/content/release_notes/release-0.60.0.html
websites/staging/slider/trunk/content/release_notes/release-0.61.0.html
websites/staging/slider/trunk/content/release_notes/release-0.70.1.html
websites/staging/slider/trunk/content/release_notes/release-0.80.0.html
websites/staging/slider/trunk/content/team.html
Propchange: websites/staging/slider/trunk/content/
------------------------------------------------------------------------------
--- cms:source-revision (original)
+++ cms:source-revision Mon Nov 2 17:24:47 2015
@@ -1 +1 @@
-1708722
+1712094
Modified: websites/staging/slider/trunk/content/design/architecture.html
==============================================================================
--- websites/staging/slider/trunk/content/design/architecture.html (original)
+++ websites/staging/slider/trunk/content/design/architecture.html Mon Nov 2 17:24:47 2015
@@ -155,7 +155,7 @@
<div style="text-align: center">
<h1><a href="/index.html">Apache Slider (incubating)</a></h1>
<hr>
-Latest release: <strong>0.80.0-incubating</strong><br>
+Latest release: <strong>0.81.1-incubating</strong><br>
<br>
<a id="download-button-sidebar" class="btn btn-success btn-block" href="/downloads/" role="button">Download</a>
</div>
@@ -168,7 +168,18 @@ Latest release: <strong>0.80.0-incubatin
<h1 class="title"></h1>
- <!---
+ <style type="text/css">
+/* The following code is added by mdx_elementid.py
+ It was originally lifted from http://subversion.apache.org/style/site.css */
+/*
+ * Hide class="elementid-permalink", except when an enclosing heading
+ * has the :hover property.
+ */
+.headerlink, .elementid-permalink {
+ visibility: hidden;
+}
+h2:hover > .headerlink, h3:hover > .headerlink, h1:hover > .headerlink, h6:hover > .headerlink, h4:hover > .headerlink, h5:hover > .headerlink, dt:hover > .elementid-permalink { visibility: visible }</style>
+<!---
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
@@ -185,8 +196,8 @@ Latest release: <strong>0.80.0-incubatin
limitations under the License.
-->
-<h1 id="apache-slider-architecture">Apache Slider Architecture</h1>
-<h2 id="summary">Summary</h2>
+<h1 id="apache-slider-architecture">Apache Slider Architecture<a class="headerlink" href="#apache-slider-architecture" title="Permanent link">¶</a></h1>
+<h2 id="summary">Summary<a class="headerlink" href="#summary" title="Permanent link">¶</a></h2>
<p>Slider is a YARN application to deploy non-YARN-enabled applications in a YARN cluster</p>
<p>Slider consists of a YARN application master, the "Slider AM", and a client
application which communicates with YARN and the Slider AM via remote procedure
@@ -204,10 +215,10 @@ options and parameters.</p>
for each component; It then monitors the state of the application instance, receiving messages
from YARN when a remotely executed process finishes. It then deploys another instance of
that component.</p>
-<h2 id="slider-packaging">Slider Packaging</h2>
+<h2 id="slider-packaging">Slider Packaging<a class="headerlink" href="#slider-packaging" title="Permanent link">¶</a></h2>
<p>A key goal of Slider is to support the deployment of existing applications into
a YARN application instance, without having to extend Slider itself. </p>
-<h2 id="am-architecture">AM Architecture</h2>
+<h2 id="am-architecture">AM Architecture<a class="headerlink" href="#am-architecture" title="Permanent link">¶</a></h2>
<p>The application master consists of</p>
<ol>
<li>The AM engine which handles all integration with external services, specifically YARN and any Slider clients</li>
@@ -231,14 +242,14 @@ actual configuration and execution of th
<p>To summarize: Slider is not an classic YARN analysis application, which allocates and schedules work across the cluster in short-to-medium life containers with the lifespan of a query or an analytics session, but instead for an application with a lifespan of days to months. Slider works to keep the actual state of its application cluster to match the desired state, while the application has the tasks of recovering from node failure, locating peer nodes and working with data in an HDFS filesystem. </p>
<p>As such it is one of the first applications designed to use YARN as a platform for long-lived services -Samza being the other key example. These application's needs of YARN are different, and their application manager design is focused around maintaining the distributed application in its desired state rather than the ongoing progress of submitted work.</p>
<p>The clean model-view-controller split was implemented to isolate the model and aid mock testing of large clusters with simulated scale, and hence increase confidence that Slider can scale to work in large YARN clusters and with larger application instances. </p>
-<h3 id="failure-model">Failure Model</h3>
+<h3 id="failure-model">Failure Model<a class="headerlink" href="#failure-model" title="Permanent link">¶</a></h3>
<p>The application master is designed to be a <a href="https://www.usenix.org/legacy/events/hotos03/tech/full_papers/candea/candea.pdf">crash-only application</a>, clients are free to terminate
the application instance by asking YARN directly. </p>
<p>There is an RPC call to stop the application instance - this is a nicety which includes a message in the termination log, and
could, in future, perhaps warn the provider that the application instance is being torn down. That is a potentially dangerous feature
to add -as provider implementors may start to expect the method to be called reliably. Slider is designed to fail without
warning, to rebuild its state on a YARN-initiated restart, and to be manually terminated without any advance notice.</p>
-<h3 id="rpc-interface">RPC Interface</h3>
+<h3 id="rpc-interface">RPC Interface<a class="headerlink" href="#rpc-interface" title="Permanent link">¶</a></h3>
<p>The RPC interface allows the client to query the current application state, and to update it by pushing out a new JSON specification. </p>
<p>The core operations are</p>
<ul>
@@ -251,9 +262,9 @@ warning, to rebuild its state on a YARN-
it triggers a re-evaluation of the application state, building up the list of container add and release requests to make of
the YARN resource manager.</p>
<p>The final operation, <code>stopCluster()</code>, stops the application instance. </p>
-<h3 id="security-and-identity">Security and Identity</h3>
+<h3 id="security-and-identity">Security and Identity<a class="headerlink" href="#security-and-identity" title="Permanent link">¶</a></h3>
<p>Slider's security model is described in detail in <a href="/docs/security.html">an accompanying document</a></p>
-<h3 id="agent-to-application-master-secure-communication">Agent to Application Master Secure Communication</h3>
+<h3 id="agent-to-application-master-secure-communication">Agent to Application Master Secure Communication<a class="headerlink" href="#agent-to-application-master-secure-communication" title="Permanent link">¶</a></h3>
<p>By default, one-way SSL is leveraged to secure the communication between Slider agents and the Application Master. However, two-way SSL can be enabled. A more detailed discussion of the SSL implementation in Slider can be found <a href="/design/ssl_implementation.html">in the SSL documentation</a>.</p>
</div>
Modified: websites/staging/slider/trunk/content/design/index.html
==============================================================================
--- websites/staging/slider/trunk/content/design/index.html (original)
+++ websites/staging/slider/trunk/content/design/index.html Mon Nov 2 17:24:47 2015
@@ -155,7 +155,7 @@
<div style="text-align: center">
<h1><a href="/index.html">Apache Slider (incubating)</a></h1>
<hr>
-Latest release: <strong>0.80.0-incubating</strong><br>
+Latest release: <strong>0.81.1-incubating</strong><br>
<br>
<a id="download-button-sidebar" class="btn btn-success btn-block" href="/downloads/" role="button">Download</a>
</div>
@@ -168,7 +168,18 @@ Latest release: <strong>0.80.0-incubatin
<h1 class="title"></h1>
- <!---
+ <style type="text/css">
+/* The following code is added by mdx_elementid.py
+ It was originally lifted from http://subversion.apache.org/style/site.css */
+/*
+ * Hide class="elementid-permalink", except when an enclosing heading
+ * has the :hover property.
+ */
+.headerlink, .elementid-permalink {
+ visibility: hidden;
+}
+h2:hover > .headerlink, h3:hover > .headerlink, h1:hover > .headerlink, h6:hover > .headerlink, h4:hover > .headerlink, h5:hover > .headerlink, dt:hover > .elementid-permalink { visibility: visible }</style>
+<!---
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
@@ -185,7 +196,7 @@ Latest release: <strong>0.80.0-incubatin
limitations under the License.
-->
-<h1 id="apache-slider-architecture">Apache Slider Architecture</h1>
+<h1 id="apache-slider-architecture">Apache Slider Architecture<a class="headerlink" href="#apache-slider-architecture" title="Permanent link">¶</a></h1>
<ul>
<li><a href="/design/architecture.html">Architecture</a></li>
<li><a href="/docs/slider_specs/application_needs.html">Application Needs</a></li>
Modified: websites/staging/slider/trunk/content/design/registry/a_YARN_service_registry.html
==============================================================================
--- websites/staging/slider/trunk/content/design/registry/a_YARN_service_registry.html (original)
+++ websites/staging/slider/trunk/content/design/registry/a_YARN_service_registry.html Mon Nov 2 17:24:47 2015
@@ -155,7 +155,7 @@
<div style="text-align: center">
<h1><a href="/index.html">Apache Slider (incubating)</a></h1>
<hr>
-Latest release: <strong>0.80.0-incubating</strong><br>
+Latest release: <strong>0.81.1-incubating</strong><br>
<br>
<a id="download-button-sidebar" class="btn btn-success btn-block" href="/downloads/" role="button">Download</a>
</div>
@@ -168,7 +168,18 @@ Latest release: <strong>0.80.0-incubatin
<h1 class="title"></h1>
- <!---
+ <style type="text/css">
+/* The following code is added by mdx_elementid.py
+ It was originally lifted from http://subversion.apache.org/style/site.css */
+/*
+ * Hide class="elementid-permalink", except when an enclosing heading
+ * has the :hover property.
+ */
+.headerlink, .elementid-permalink {
+ visibility: hidden;
+}
+h2:hover > .headerlink, h3:hover > .headerlink, h1:hover > .headerlink, h6:hover > .headerlink, h4:hover > .headerlink, h5:hover > .headerlink, dt:hover > .elementid-permalink { visibility: visible }</style>
+<!---
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
@@ -185,9 +196,9 @@ Latest release: <strong>0.80.0-incubatin
limitations under the License.
-->
-<h1 id="a-yarn-service-registry-for-apache-slider">A YARN Service Registry for Apache Slider</h1>
-<h2 id="april-2014">April 2014</h2>
-<h1 id="introduction">Introduction</h1>
+<h1 id="a-yarn-service-registry-for-apache-slider">A YARN Service Registry for Apache Slider<a class="headerlink" href="#a-yarn-service-registry-for-apache-slider" title="Permanent link">¶</a></h1>
+<h2 id="april-2014">April 2014<a class="headerlink" href="#april-2014" title="Permanent link">¶</a></h2>
+<h1 id="introduction">Introduction<a class="headerlink" href="#introduction" title="Permanent link">¶</a></h1>
<p>This document looks at the needs and options of a service registry.</p>
<p>The core issue is that as the location(s) of a dynamically deployed application are unknown, the standard Hadoop and Java configuration model of some form of text files containing hostnames, ports and URLS no longer works. You cannot define up-front where a service will be.</p>
<p>Some Hadoop applications -HBase and Accumulo -have solved this with custom ZK bindings. This works for the specific clients, but requires hbase and accumulo client JARs in order to be able to work with the content. (or a re-implementation with knowledge of the non-standard contents of the ZK nodes)</p>
@@ -195,10 +206,10 @@ Latest release: <strong>0.80.0-incubatin
<p>YARN provides some minimal publishing of AM remote endpoints: a URL to what is assumed to be a Web UI (not a REST API), and an IPC port. The URL is displayed in the YARN UI -in which case it is accessed via a proxy which (currently) only support HTTP GET operations. The YARN API call to list all applications can be used to locate a named instance of an application by (user, application-type, name), and then obtain the raw URL and IPC endpoints. This enumeration process is an O(apps) operation on the YARN RM and only provides access to those two endpoints. Even with the RAW URL, REST operations have proven "troublesome", due to a web filter which redirects all direct requests to the proxy -unless it comes from the same host as the proxy.</p>
<p>Hadoop client applications tend to retrieve all their configuration information from files in the local filesystem, hadoop-site.xml, hdfs-site.xml, hbase-site.xml, etc. This requires the configuration files to be present on all systems. Tools such as Ambari can keep the files in the server up to date -assuming a low rate of change- ---but these tools do nothing for the client applications themselves. It is up to the cluster clients to (somehow) retrieve these files, and to keep their copies up to date. <em>This is a problem that exists with today's applications</em>. </p>
<p>As an example, if a YARN client does not know the value of "yarn.application.classpath", it cannot successfully deploy any application in the YARN cluster which needs the cluster-side Hadoop and YARN JARs on its application master's classpath. This is not a theoretical problem, as some clusters have a different classpath from the default: without a correct value the Slider AM does not start. And, as it is designed to be run remotely, it cannot rely on a local installation of YARN to provide the correct values (<a href="https://issues.apache.org/jira/browse/YARN-973">YARN-973</a>).</p>
-<h1 id="what-do-we-need">What do we need?</h1>
+<h1 id="what-do-we-need">What do we need?<a class="headerlink" href="#what-do-we-need" title="Permanent link">¶</a></h1>
<p><strong>Discovery</strong>: An IPC and URL discovery system for service-aware applications to use to look up a service to which it wishes to talk to. This is not an ORB -it's not doing redirection -, but it is something that needs to be used before starting IPC or REST communications. </p>
<p><strong>Configuration</strong>: A way for clients of a service to retrieve more configuration data than simply the service endpoints. For example: everything needed to create a site.xml document.</p>
-<h2 id="client-side">Client-side</h2>
+<h2 id="client-side">Client-side<a class="headerlink" href="#client-side" title="Permanent link">¶</a></h2>
<ul>
<li>
<p>Allow clients of a YARN application to locate the service instance and its service ports (web, IPC, REST...) efficiently even on a large YARN cluster. </p>
@@ -216,7 +227,7 @@ Latest release: <strong>0.80.0-incubatin
<p>cross-language support.</p>
</li>
</ul>
-<h2 id="for-all-services">For all Services</h2>
+<h2 id="for-all-services">For all Services<a class="headerlink" href="#for-all-services" title="Permanent link">¶</a></h2>
<ul>
<li>
<p>Allow services to publish their binding details for the AM and of code running in the containers (which may be published by the containers)</p>
@@ -234,7 +245,7 @@ Latest release: <strong>0.80.0-incubatin
<p>Services to be able to act as clients to other services</p>
</li>
</ul>
-<h2 id="for-slider-services-and-presumably-others">For Slider Services (and presumably others)</h2>
+<h2 id="for-slider-services-and-presumably-others">For Slider Services (and presumably others)<a class="headerlink" href="#for-slider-services-and-presumably-others" title="Permanent link">¶</a></h2>
<ul>
<li>
<p>Ability to publish information about configuration documents that can be retrieved -and URLs</p>
@@ -246,7 +257,7 @@ Latest release: <strong>0.80.0-incubatin
<p>Ability to use service paths as a way to ensure a single instance of a named service can be deployed by a user</p>
</li>
</ul>
-<h2 id="management-and-paas-uis">Management and PaaS UIs</h2>
+<h2 id="management-and-paas-uis">Management and PaaS UIs<a class="headerlink" href="#management-and-paas-uis" title="Permanent link">¶</a></h2>
<ul>
<li>
<p>Retrieve lists of web UI URLs of AM and of deployed components</p>
@@ -267,7 +278,7 @@ Latest release: <strong>0.80.0-incubatin
<p>Listen to changes in the service mix -the arrival and departure of service instances, as well as changes in their contents.</p>
</li>
</ul>
-<h2 id="other-needs">Other Needs</h2>
+<h2 id="other-needs">Other Needs<a class="headerlink" href="#other-needs" title="Permanent link">¶</a></h2>
<ul>
<li>
<p>Registry-configured applications. In-cluster applications should be able to subscribe to part of the registry
@@ -281,9 +292,9 @@ applications on which they depend themse
<p>Cloud-based deployments: work on virtual infrastructures where hostnames are unpredictable.</p>
</li>
</ul>
-<h1 id="open-source-registry-code">Open Source Registry code</h1>
+<h1 id="open-source-registry-code">Open Source Registry code<a class="headerlink" href="#open-source-registry-code" title="Permanent link">¶</a></h1>
<p>What can we use to implement this from ASF and ASF-compatible code? </p>
-<h2 id="zookeeper">Zookeeper</h2>
+<h2 id="zookeeper">Zookeeper<a class="headerlink" href="#zookeeper" title="Permanent link">¶</a></h2>
<p>We'd need a good reason not to use this. There are still some issues</p>
<ol>
<li>
@@ -299,7 +310,7 @@ applications on which they depend themse
<p>Impact of other ZK load on the behaviour of the service registry -will it cause problems if overloaded -and are they recoverable?</p>
</li>
</ol>
-<h2 id="apache-curator">Apache Curator</h2>
+<h2 id="apache-curator">Apache Curator<a class="headerlink" href="#apache-curator" title="Permanent link">¶</a></h2>
<p>Netflix's core curator -now <a href="http://curator.apache.org/">Apache Curator</a>- framework adds a lot to make working with ZK easier, including pluggable retry policies, binding tools and other things.</p>
<p>There is also its "experimental" <a href="http://curator.apache.org/curator-x-discovery-server/index.html">service discovery framework</a>, which</p>
<ol>
@@ -350,7 +361,7 @@ Jackson (1.8.8) in Hadoop 2.4. The upgra
}</p>
</li>
</ul>
-<h2 id="helix-service-registry"><a href="http://helix.apache.org/0.7.0-incubating-docs/recipes/service_discovery.html">Helix Service Registry</a></h2>
+<h2 id="helix-service-registry"><a href="http://helix.apache.org/0.7.0-incubating-docs/recipes/service_discovery.html">Helix Service Registry</a><a class="headerlink" href="#helix-service-registry" title="Permanent link">¶</a></h2>
<p>This is inside Helix somewhere, used in LI in production at scale -and worth looking at. LI separate their Helix Zookeeper Quorum from their application-layer quorum, to isolate load.</p>
<p>Notable features</p>
<ol>
@@ -358,24 +369,24 @@ Jackson (1.8.8) in Hadoop 2.4. The upgra
<li>The deployed application can pick up changes to its state the same way, allowing for live application manipulation.</li>
<li>Tracks nodes that continually join/leave the group and drops them as unreliable.</li>
</ol>
-<h2 id="twill-service-registry">Twill Service Registry</h2>
+<h2 id="twill-service-registry">Twill Service Registry<a class="headerlink" href="#twill-service-registry" title="Permanent link">¶</a></h2>
<p>Twill's <a href="http://twill.incubator.apache.org/apidocs/index.html">service registry code</a>, lets applications register a <a href="http://twill.incubator.apache.org/apidocs/org/apache/twill/discovery/Discoverable.html">(hostname, port)</a> pair in the registry by a name, a name by which clients can look up and enumerate all services with a specific name.</p>
<p>Clients can subscribe to changes in the list of services with a specific name -so picking up the arrival and departure of instances, and probe to see if a previously discovered entity is still registered.</p>
<p>Zookeeper- and in-memory registry implementations are provided.</p>
<p>One nice feature about this architecture -and Twill in general- is that its general single-method callback model means that it segues nicely into Java-8 lambda-expressions. This is something to retain in a YARN-wide service registry.</p>
<p>Comparing it to curator, it offers a proper subset of curator's registered services <a href="http://curator.apache.org/apidocs/org/apache/curator/x/discovery/ServiceInstance.html">ServiceInstance</a> -implying that you could publish and retrieve Curator-registered services via a new implementation of Twill's DiscoveryService. This would require extensions to the curator service discovery client allow ZK nodes to be watched for changes. This is a feature that would be useful in many use cases -such as watching service availability across a cluster, or simply blocking until a dependent service was launched.</p>
<p>As with curator, the amount of information that can be published isn't enough for management tools to make effective use of the service registration, while for slider there's no way to publish configuration data. However a YARN registry will inevitably be a superset of the Twill client's enumerated and retrieved data -so if its registration API were sufficient to register a minimal service, supporting the YARN registry via Twill's existing API should be straightforward.</p>
-<h2 id="twitter-commons-service-registration">Twitter Commons Service Registration</h2>
+<h2 id="twitter-commons-service-registration">Twitter Commons Service Registration<a class="headerlink" href="#twitter-commons-service-registration" title="Permanent link">¶</a></h2>
<p><a href="https://github.com/twitter/commons">Twitter Commons</a> has a service registration library, which allows for registration of sets of servers, <a href="http://twitter.github.io/commons/apidocs/com/twitter/common/service/registration/package-tree.html">publishing the hostname and port of each</a>., along with a map of string properties.</p>
<p>Zookeeper based, it suffices if all servers are identical and only publishing single (hostname, port) pairs for callers.</p>
-<h2 id="airbnb-smartstack">AirBnB Smartstack</h2>
+<h2 id="airbnb-smartstack">AirBnB Smartstack<a class="headerlink" href="#airbnb-smartstack" title="Permanent link">¶</a></h2>
<p>SmartStack is <a href="http://nerds.airbnb.com/smartstack-service-discovery-cloud/">Air BnB's cloud-based service discovery system</a>.</p>
<p>It has two parts, <em>Nerve</em> and <em>Synapse</em>:</p>
<p><a href="https://github.com/airbnb/nerve"><strong>Nerve</strong></a> is a ruby agent designed to monitor processes and register healthy instances in ZK (or to a mock reporter). It includes <a href="https://github.com/airbnb/nerve/tree/master/lib/nerve/service_watcher">probes for TCP ports, HTTP and rabbitMQ</a>. It's <a href="https://github.com/airbnb/nerve/blob/master/lib/nerve/service_watcher.rb">a fairly simple liveness monitor</a>.</p>
<p><a href="https://github.com/airbnb/synapse"><strong>Synapse</strong></a> takes the data and uses it to configure <a href="http://haproxy.1wt.eu/">HAProxy instances</a>. HAProxy handles the load balancing, queuing and integrating liveness probes into the queues. Synapse generates all the configuration files for an instance -but also tries to reconfigure the live instances via their socket APIs, </p>
<p>Alongside these, AirBnB have another published project on Github, <a href="https://github.com/airbnb/optica">Optica</a>, which is a web application for nodes to register themselves with (POST) and for others to query. It publishes events to RabbitMQ, and again uses ZK to store state.</p>
<p>AirBnB do complain a bit about ZK and its brittleness. They do mention that they suspect it is due to bugs in the Ruby ZK client library. This may be exacerbated by in-cloud deployments. Hard-coding the list of ZK nodes may work for a physical cluster, but in a virtualized cluster, the hostnames/IP Addresses of those nodes may change -leading to a meta-discovery problem: how to find the ZK quorum -especially if you can't control the DNS servers.</p>
-<h2 id="apache-directory"><a href="http://directory.apache.org/apacheds/">Apache Directory</a></h2>
+<h2 id="apache-directory"><a href="http://directory.apache.org/apacheds/">Apache Directory</a><a class="headerlink" href="#apache-directory" title="Permanent link">¶</a></h2>
<p>This is an embeddable LDAP server</p>
<ul>
<li>
Modified: websites/staging/slider/trunk/content/design/registry/index.html
==============================================================================
--- websites/staging/slider/trunk/content/design/registry/index.html (original)
+++ websites/staging/slider/trunk/content/design/registry/index.html Mon Nov 2 17:24:47 2015
@@ -155,7 +155,7 @@
<div style="text-align: center">
<h1><a href="/index.html">Apache Slider (incubating)</a></h1>
<hr>
-Latest release: <strong>0.80.0-incubating</strong><br>
+Latest release: <strong>0.81.1-incubating</strong><br>
<br>
<a id="download-button-sidebar" class="btn btn-success btn-block" href="/downloads/" role="button">Download</a>
</div>
@@ -168,7 +168,18 @@ Latest release: <strong>0.80.0-incubatin
<h1 class="title"></h1>
- <!---
+ <style type="text/css">
+/* The following code is added by mdx_elementid.py
+ It was originally lifted from http://subversion.apache.org/style/site.css */
+/*
+ * Hide class="elementid-permalink", except when an enclosing heading
+ * has the :hover property.
+ */
+.headerlink, .elementid-permalink {
+ visibility: hidden;
+}
+h2:hover > .headerlink, h3:hover > .headerlink, h1:hover > .headerlink, h6:hover > .headerlink, h4:hover > .headerlink, h5:hover > .headerlink, dt:hover > .elementid-permalink { visibility: visible }</style>
+<!---
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
@@ -185,8 +196,8 @@ Latest release: <strong>0.80.0-incubatin
limitations under the License.
-->
-<h1 id="apache-slider-service-registry">Apache Slider Service Registry</h1>
-<h2 id="important-this-document-is-obsolete">Important: This document is obsolete</h2>
+<h1 id="apache-slider-service-registry">Apache Slider Service Registry<a class="headerlink" href="#apache-slider-service-registry" title="Permanent link">¶</a></h1>
+<h2 id="important-this-document-is-obsolete">Important: This document is obsolete<a class="headerlink" href="#important-this-document-is-obsolete" title="Permanent link">¶</a></h2>
<p>This secion has been supplanted by the YARN Service Registry; a cross-application
service registry built into Apache Hadoop, with special integration with
the YARN Resource Manager, security in a secure cluster, and other features.</p>
@@ -194,7 +205,7 @@ the YARN Resource Manager, security in a
so we are confident that it works for Slider as well as other applications.
Indeed, for the Hadoop 2.6 release, Slider was the functional test suite for
the registry.</p>
-<h2 id="introduction">Introduction</h2>
+<h2 id="introduction">Introduction<a class="headerlink" href="#introduction" title="Permanent link">¶</a></h2>
<p>The service registry is a core part of the Slider Architecture -it is how
dynamically generated configurations are published for clients to pick up.</p>
<p>The need for a service registry goes beyond Slider, however. We effectively
@@ -209,7 +220,7 @@ to be on all YARN clusters, so code coul
including HBase, Accumulo, Oozie, -applications deployed by management tools.
Unless/until these applications self-published their binding data, it would
be the duty of the management tools to do the registration.</p>
-<h2 id="contents">Contents</h2>
+<h2 id="contents">Contents<a class="headerlink" href="#contents" title="Permanent link">¶</a></h2>
<ol>
<li><a href="the_YARN_application_registration_and_binding_problem.html">YARN Application Registration and Binding: the Problem</a></li>
<li><a href="a_YARN_service_registry.html">A YARN Service Registry</a></li>
Modified: websites/staging/slider/trunk/content/design/registry/initial_registry_design.html
==============================================================================
--- websites/staging/slider/trunk/content/design/registry/initial_registry_design.html (original)
+++ websites/staging/slider/trunk/content/design/registry/initial_registry_design.html Mon Nov 2 17:24:47 2015
@@ -155,7 +155,7 @@
<div style="text-align: center">
<h1><a href="/index.html">Apache Slider (incubating)</a></h1>
<hr>
-Latest release: <strong>0.80.0-incubating</strong><br>
+Latest release: <strong>0.81.1-incubating</strong><br>
<br>
<a id="download-button-sidebar" class="btn btn-success btn-block" href="/downloads/" role="button">Download</a>
</div>
@@ -168,7 +168,18 @@ Latest release: <strong>0.80.0-incubatin
<h1 class="title"></h1>
- <!---
+ <style type="text/css">
+/* The following code is added by mdx_elementid.py
+ It was originally lifted from http://subversion.apache.org/style/site.css */
+/*
+ * Hide class="elementid-permalink", except when an enclosing heading
+ * has the :hover property.
+ */
+.headerlink, .elementid-permalink {
+ visibility: hidden;
+}
+h2:hover > .headerlink, h3:hover > .headerlink, h1:hover > .headerlink, h6:hover > .headerlink, h4:hover > .headerlink, h5:hover > .headerlink, dt:hover > .elementid-permalink { visibility: visible }</style>
+<!---
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
@@ -185,7 +196,7 @@ Latest release: <strong>0.80.0-incubatin
limitations under the License.
-->
-<h1 id="april-2014-initial-registry-design-for-apache-slider">April 2014 Initial Registry Design for Apache Slider</h1>
+<h1 id="april-2014-initial-registry-design-for-apache-slider">April 2014 Initial Registry Design for Apache Slider<a class="headerlink" href="#april-2014-initial-registry-design-for-apache-slider" title="Permanent link">¶</a></h1>
<p>This is the plan for the initial registry design.</p>
<ol>
<li>
@@ -213,7 +224,7 @@ Latest release: <strong>0.80.0-incubatin
<p>AM to serve up curator service (later -host in RM? elsewhere?)</p>
</li>
</ol>
-<h3 id="forwards-compatilibity">forwards-compatilibity</h3>
+<h3 id="forwards-compatilibity">forwards-compatilibity<a class="headerlink" href="#forwards-compatilibity" title="Permanent link">¶</a></h3>
<ol>
<li>
<p>This initial design will hide the fact that Apache Curator is being used to discover services,
@@ -225,7 +236,7 @@ can use the same datatype -or map to it-
<p>The first implementation will not support watching for changes.</p>
</li>
</ol>
-<h3 id="initial-templates">Initial templates</h3>
+<h3 id="initial-templates">Initial templates<a class="headerlink" href="#initial-templates" title="Permanent link">¶</a></h3>
<ul>
<li>
<p>hadoop XML conf files</p>
@@ -237,7 +248,7 @@ can use the same datatype -or map to it-
<p>HTML listing of services</p>
</li>
</ul>
-<h2 id="example-curator-service-entry">Example Curator Service Entry</h2>
+<h2 id="example-curator-service-entry">Example Curator Service Entry<a class="headerlink" href="#example-curator-service-entry" title="Permanent link">¶</a></h2>
<p>This is the prototype's content</p>
<p>Toplevel</p>
<div class="codehilite"><pre><span class="n">service</span> <span class="n">CuratorServiceInstance</span><span class="p">{</span><span class="n">name</span><span class="p">=</span><span class="s">'slider'</span><span class="p">,</span> <span class="n">id</span><span class="p">=</span><span class="s">'stevel.test_registry_am'</span><span class="p">,</span> <span class="n">address</span><span class="p">=</span><span class="s">'192.168.1.101'</span><span class="p">,</span> <span class="n">port</span><span class="p">=</span>62552<span class="p">,</span> <span class="n">sslPort</span><span class="p">=</span><span class="n">null</span><span class="p">,</span> <span class="n">payload</span><span class="p">=</span><span class="n">org</span><span class="p">.</span><span class="n">apache</span><span class="p">.</span><span class="n">slider</span><span class="p">.</span><span class="n">core</span><span class="p">.</span><span class="n">registry</span><span class="p">.</
span><span class="n">info</span><span class="p">.</span><span class="n">ServiceInstanceData</span><span class="p">@</span>4<span class="n">e9af21b</span><span class="p">,</span> <span class="n">registrationTimeUTC</span><span class="p">=</span>1397574073203<span class="p">,</span> <span class="n">serviceType</span><span class="p">=</span><span class="n">DYNAMIC</span><span class="p">,</span> <span class="n">uriSpec</span><span class="p">=</span><span class="n">org</span><span class="p">.</span><span class="n">apache</span><span class="p">.</span><span class="n">curator</span><span class="p">.</span><span class="n">x</span><span class="p">.</span><span class="n">discovery</span><span class="p">.</span><span class="n">UriSpec</span><span class="p">@</span><span class="n">ef8dacf0</span><span class="p">}</span>
Modified: websites/staging/slider/trunk/content/design/registry/p2p_service_registries.html
==============================================================================
--- websites/staging/slider/trunk/content/design/registry/p2p_service_registries.html (original)
+++ websites/staging/slider/trunk/content/design/registry/p2p_service_registries.html Mon Nov 2 17:24:47 2015
@@ -155,7 +155,7 @@
<div style="text-align: center">
<h1><a href="/index.html">Apache Slider (incubating)</a></h1>
<hr>
-Latest release: <strong>0.80.0-incubating</strong><br>
+Latest release: <strong>0.81.1-incubating</strong><br>
<br>
<a id="download-button-sidebar" class="btn btn-success btn-block" href="/downloads/" role="button">Download</a>
</div>
@@ -168,7 +168,18 @@ Latest release: <strong>0.80.0-incubatin
<h1 class="title"></h1>
- <!---
+ <style type="text/css">
+/* The following code is added by mdx_elementid.py
+ It was originally lifted from http://subversion.apache.org/style/site.css */
+/*
+ * Hide class="elementid-permalink", except when an enclosing heading
+ * has the :hover property.
+ */
+.headerlink, .elementid-permalink {
+ visibility: hidden;
+}
+h2:hover > .headerlink, h3:hover > .headerlink, h1:hover > .headerlink, h6:hover > .headerlink, h4:hover > .headerlink, h5:hover > .headerlink, dt:hover > .elementid-permalink { visibility: visible }</style>
+<!---
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
@@ -185,7 +196,7 @@ Latest release: <strong>0.80.0-incubatin
limitations under the License.
-->
-<h1 id="p2p-service-registries-for-apache-slider">P2P Service Registries for Apache Slider</h1>
+<h1 id="p2p-service-registries-for-apache-slider">P2P Service Registries for Apache Slider<a class="headerlink" href="#p2p-service-registries-for-apache-slider" title="Permanent link">¶</a></h1>
<p>Alongside the centralized service registries, there's much prior work on
P2P discovery systems, especially for mobile and consumer devices.</p>
<p>They perform some multicast- or distributed hash table-based lookup,
@@ -212,7 +223,7 @@ clouds. There's multiple anecdotes of
registry for in-EC2 applications. At the very least, this service and its
equivalents in other cloud providers could be used to bootstrap ZK client
bindings in cloud environments. </p>
-<h2 id="service-location-protocol">Service Location Protocol</h2>
+<h2 id="service-location-protocol">Service Location Protocol<a class="headerlink" href="#service-location-protocol" title="Permanent link">¶</a></h2>
<p>Service Location Protocol is a protocol for discovery services that came out
of Sun, Novell and others -it is still available for printer discovery and
suchlike</p>
@@ -234,7 +245,7 @@ mentions in Windows Server technical not
<p>[<a href="http://www.ietf.org/rfc/rfc2610.txt">RFC2610</a>] <em>DHCP Options for Service Location Protocol, IETF, 1999</em></p>
</li>
</ul>
-<h2 id="zeroconf"><a href="http://www.zeroconf.org/">Zeroconf</a></h2>
+<h2 id="zeroconf"><a href="http://www.zeroconf.org/">Zeroconf</a><a class="headerlink" href="#zeroconf" title="Permanent link">¶</a></h2>
<p>The multicast discovery service implemented in Apple's Bonjour system
--multicasting DNS lookups to all peers in the subnet.</p>
<p>This allows for URLs and hostnames to be dynamically positioned, with
@@ -243,13 +254,13 @@ DNS domain searches allowing for enumera
subnet is is O(DNS-queries-across-subnet), hence implicitly <code>O(devices)*O(device-activity)</code>. </p>
<p>The special domains <code>_tcp.</code>, <code>_udp.</code> and their subdomains can also be
served up via a normal DNS server.</p>
-<h2 id="jiniapache-river"><a href="http://river.apache.org/doc/specs/html/lookup-spec.html">Jini/Apache River</a></h2>
+<h2 id="jiniapache-river"><a href="http://river.apache.org/doc/specs/html/lookup-spec.html">Jini/Apache River</a><a class="headerlink" href="#jiniapache-river" title="Permanent link">¶</a></h2>
<p>Attribute-driven service enumeration, which drives the, Java-client-only
model of downloading client-side code. There's no requirement for the remote
services to be in Java, only that drivers are.</p>
-<h2 id="serf"><a href="http://www.serfdom.io/">Serf</a></h2>
+<h2 id="serf"><a href="http://www.serfdom.io/">Serf</a><a class="headerlink" href="#serf" title="Permanent link">¶</a></h2>
<p>This is a library that implements the <a href="http://www.cs.cornell.edu/~asdas/research/dsn02-swim.pdf">SWIM protocol</a> to propagate information around a cluster. Apparently works in virtualized clusters too. It's already been used in a Flume-on-Hoya provider.</p>
-<h2 id="anubis"><a href="http://sourceforge.net/p/smartfrog/svn/HEAD/tree/trunk/core/components/anubis/">Anubis</a></h2>
+<h2 id="anubis"><a href="http://sourceforge.net/p/smartfrog/svn/HEAD/tree/trunk/core/components/anubis/">Anubis</a><a class="headerlink" href="#anubis" title="Permanent link">¶</a></h2>
<p>An HP Labs-built <a href="http://sourceforge.net/p/smartfrog/svn/HEAD/tree/trunk/core/components/anubis/doc/HPL-2005-72.pdf?format=raw">High Availability tuple-space</a> in SmartFrog; used in production in some of HP's telco products. An agent publishes facts into the T-Space, and within one heartbeat all other agents have it. One heart-beat later, unless there's been a change in the membership, the publisher knows the others have it. One heartbeat later the agents know the publisher knows it, etc.</p>
<p>Strengths: </p>
<ul>
@@ -285,7 +296,7 @@ mutual trust of HTTPS certificates to jo
<p>It proved good for workload sharing -tasks can be published to it, any
agent can say "I'm working on it" and take up the work. If the process
fails, the task becomes available again. We used this for distributed scheduling in a rendering farm.</p>
-<h2 id="carmen"><a href="http://www.hpl.hp.com/techreports/2002/HPL-2002-257">Carmen</a></h2>
+<h2 id="carmen"><a href="http://www.hpl.hp.com/techreports/2002/HPL-2002-257">Carmen</a><a class="headerlink" href="#carmen" title="Permanent link">¶</a></h2>
<p>This was another HP Labs project, related to the Cooltown "ubiquitous
computing" work, which was a decade too early to be relevant. It was
also positioned by management as a B2B platform, so ended up competing
Modified: websites/staging/slider/trunk/content/design/registry/references.html
==============================================================================
--- websites/staging/slider/trunk/content/design/registry/references.html (original)
+++ websites/staging/slider/trunk/content/design/registry/references.html Mon Nov 2 17:24:47 2015
@@ -155,7 +155,7 @@
<div style="text-align: center">
<h1><a href="/index.html">Apache Slider (incubating)</a></h1>
<hr>
-Latest release: <strong>0.80.0-incubating</strong><br>
+Latest release: <strong>0.81.1-incubating</strong><br>
<br>
<a id="download-button-sidebar" class="btn btn-success btn-block" href="/downloads/" role="button">Download</a>
</div>
@@ -168,7 +168,18 @@ Latest release: <strong>0.80.0-incubatin
<h1 class="title"></h1>
- <!---
+ <style type="text/css">
+/* The following code is added by mdx_elementid.py
+ It was originally lifted from http://subversion.apache.org/style/site.css */
+/*
+ * Hide class="elementid-permalink", except when an enclosing heading
+ * has the :hover property.
+ */
+.headerlink, .elementid-permalink {
+ visibility: hidden;
+}
+h2:hover > .headerlink, h3:hover > .headerlink, h1:hover > .headerlink, h6:hover > .headerlink, h4:hover > .headerlink, h5:hover > .headerlink, dt:hover > .elementid-permalink { visibility: visible }</style>
+<!---
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
@@ -185,10 +196,10 @@ Latest release: <strong>0.80.0-incubatin
limitations under the License.
-->
-<h1 id="service-registry-references">Service Registry References</h1>
+<h1 id="service-registry-references">Service Registry References<a class="headerlink" href="#service-registry-references" title="Permanent link">¶</a></h1>
<p>Service registration and discovery is a problem in distributed computing that has been explored for over thirty years, with
[Birrell81]'s <em>Grapevine</em> system the first known implementation -though of </p>
-<h1 id="papers">Papers</h1>
+<h1 id="papers">Papers<a class="headerlink" href="#papers" title="Permanent link">¶</a></h1>
<ul>
<li>
<p><strong>[Birrell81]</strong> Birrell, A. et al, <a href="http://research.microsoft.com/apps/pubs/default.aspx?id=63661"><em>Grapevine: An exercise in distributed computing</em></a>. Comm. ACM 25, 4 (Apr 1982), pp260-274.
Modified: websites/staging/slider/trunk/content/design/registry/registry-model.html
==============================================================================
--- websites/staging/slider/trunk/content/design/registry/registry-model.html (original)
+++ websites/staging/slider/trunk/content/design/registry/registry-model.html Mon Nov 2 17:24:47 2015
@@ -155,7 +155,7 @@
<div style="text-align: center">
<h1><a href="/index.html">Apache Slider (incubating)</a></h1>
<hr>
-Latest release: <strong>0.80.0-incubating</strong><br>
+Latest release: <strong>0.81.1-incubating</strong><br>
<br>
<a id="download-button-sidebar" class="btn btn-success btn-block" href="/downloads/" role="button">Download</a>
</div>
@@ -168,7 +168,18 @@ Latest release: <strong>0.80.0-incubatin
<h1 class="title"></h1>
- <!---
+ <style type="text/css">
+/* The following code is added by mdx_elementid.py
+ It was originally lifted from http://subversion.apache.org/style/site.css */
+/*
+ * Hide class="elementid-permalink", except when an enclosing heading
+ * has the :hover property.
+ */
+.headerlink, .elementid-permalink {
+ visibility: hidden;
+}
+h2:hover > .headerlink, h3:hover > .headerlink, h1:hover > .headerlink, h6:hover > .headerlink, h4:hover > .headerlink, h5:hover > .headerlink, dt:hover > .elementid-permalink { visibility: visible }</style>
+<!---
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
@@ -185,7 +196,7 @@ Latest release: <strong>0.80.0-incubatin
limitations under the License.
-->
-<h1 id="apache-slider-service-registry">Apache Slider: Service Registry</h1>
+<h1 id="apache-slider-service-registry">Apache Slider: Service Registry<a class="headerlink" href="#apache-slider-service-registry" title="Permanent link">¶</a></h1>
<p>The service registry model is designed to support dynamically
deployed Slider applications, <em>and</em> statically deployed versions
of the same application -provided the latter also registers itself,
Modified: websites/staging/slider/trunk/content/design/registry/service_registry_end_to_end_scenario.html
==============================================================================
--- websites/staging/slider/trunk/content/design/registry/service_registry_end_to_end_scenario.html (original)
+++ websites/staging/slider/trunk/content/design/registry/service_registry_end_to_end_scenario.html Mon Nov 2 17:24:47 2015
@@ -155,7 +155,7 @@
<div style="text-align: center">
<h1><a href="/index.html">Apache Slider (incubating)</a></h1>
<hr>
-Latest release: <strong>0.80.0-incubating</strong><br>
+Latest release: <strong>0.81.1-incubating</strong><br>
<br>
<a id="download-button-sidebar" class="btn btn-success btn-block" href="/downloads/" role="button">Download</a>
</div>
@@ -168,7 +168,18 @@ Latest release: <strong>0.80.0-incubatin
<h1 class="title"></h1>
- <!---
+ <style type="text/css">
+/* The following code is added by mdx_elementid.py
+ It was originally lifted from http://subversion.apache.org/style/site.css */
+/*
+ * Hide class="elementid-permalink", except when an enclosing heading
+ * has the :hover property.
+ */
+.headerlink, .elementid-permalink {
+ visibility: hidden;
+}
+h2:hover > .headerlink, h3:hover > .headerlink, h1:hover > .headerlink, h6:hover > .headerlink, h4:hover > .headerlink, h5:hover > .headerlink, dt:hover > .elementid-permalink { visibility: visible }</style>
+<!---
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
@@ -185,8 +196,8 @@ Latest release: <strong>0.80.0-incubatin
limitations under the License.
-->
-<h1 id="apache-slider-service-registry-end-to-end-scenarios">Apache Slider: Service Registry End-to-End Scenarios</h1>
-<h2 id="am-startup">AM startup</h2>
+<h1 id="apache-slider-service-registry-end-to-end-scenarios">Apache Slider: Service Registry End-to-End Scenarios<a class="headerlink" href="#apache-slider-service-registry-end-to-end-scenarios" title="Permanent link">¶</a></h1>
+<h2 id="am-startup">AM startup<a class="headerlink" href="#am-startup" title="Permanent link">¶</a></h2>
<ol>
<li>
<p>AM starts, reads in configuration, creates provider</p>
@@ -216,7 +227,7 @@ Latest release: <strong>0.80.0-incubatin
<p>AM triggers review of application state, requesting/releasing nodes as appropriate</p>
</li>
</ol>
-<h2 id="agent-startup-standalone">Agent Startup: standalone</h2>
+<h2 id="agent-startup-standalone">Agent Startup: standalone<a class="headerlink" href="#agent-startup-standalone" title="Permanent link">¶</a></h2>
<ol>
<li>
<p>Container is issued to AM</p>
@@ -237,7 +248,7 @@ Latest release: <strong>0.80.0-incubatin
<p>AM gives agent next state command.</p>
</li>
</ol>
-<h2 id="am-gets-state-from-agent">AM gets state from agent:</h2>
+<h2 id="am-gets-state-from-agent">AM gets state from agent:<a class="headerlink" href="#am-gets-state-from-agent" title="Permanent link">¶</a></h2>
<ol>
<li>
<p>Agent heartbeats in</p>
@@ -255,7 +266,7 @@ Latest release: <strong>0.80.0-incubatin
<p>Agent returns all config state, including: hostnames, allocated ports, generated values (e.g. database connection strings, URLs) - as two-level (allows agent to define which config options are relevant to which document)</p>
</li>
</ol>
-<h2 id="am-saves-state-for-serving">AM saves state for serving</h2>
+<h2 id="am-saves-state-for-serving">AM saves state for serving<a class="headerlink" href="#am-saves-state-for-serving" title="Permanent link">¶</a></h2>
<ol>
<li>
<p>AM saves state in RAM (assumptions: small, will rebuild on restart)</p>
@@ -267,7 +278,7 @@ Latest release: <strong>0.80.0-incubatin
<p>AM fields HTTP GET requests on content</p>
</li>
</ol>
-<h2 id="am-serves-content">AM Serves content</h2>
+<h2 id="am-serves-content">AM Serves content<a class="headerlink" href="#am-serves-content" title="Permanent link">¶</a></h2>
<p>A simple REST service serves up content on paths published to the service registry. It is also possible to enumerate documents published by GET operations on parent paths.</p>
<ol>
<li>
@@ -280,7 +291,7 @@ Latest release: <strong>0.80.0-incubatin
<p>response is streamed with headers of : <code>content-type</code>, <code>content-length</code>, do not cache in proxy, expires,<em> (with expiry date chosen as ??)</em></p>
</li>
</ol>
-<h1 id="slider-client">Slider Client</h1>
+<h1 id="slider-client">Slider Client<a class="headerlink" href="#slider-client" title="Permanent link">¶</a></h1>
<p>Currently slider client enumerates the YARN registry looking for slider instances -including any instances of the same application running before launching a cluster. </p>
<p>This </p>
<ul>
@@ -288,7 +299,7 @@ Latest release: <strong>0.80.0-incubatin
<li>has scale limitations <code>O(apps-in-YARN-cluster)</code> + <code>O(completed-apps-in-RM-memory)</code></li>
<li>only retrieves configuration information from slider-deployed application instances. <em>We do not need to restrict ourselves here.</em></li>
</ul>
-<h2 id="slider-client-lists-applications">Slider Client lists applications</h2>
+<h2 id="slider-client-lists-applications">Slider Client lists applications<a class="headerlink" href="#slider-client-lists-applications" title="Permanent link">¶</a></h2>
<div class="codehilite"><pre><span class="n">slider</span> <span class="n">registry</span> <span class="o">--</span><span class="n">list</span> <span class="p">[</span><span class="o">--</span><span class="n">servicetype</span> <span class="o"><</span><span class="n">application</span><span class="o">-</span><span class="n">type</span><span class="o">></span><span class="p">]</span>
</pre></div>
@@ -304,7 +315,7 @@ Latest release: <strong>0.80.0-incubatin
<p>Client enumerates registered services and lists them</p>
</li>
</ol>
-<h2 id="slider-client-lists-content-published-by-an-application-instance">Slider Client lists content published by an application instance</h2>
+<h2 id="slider-client-lists-content-published-by-an-application-instance">Slider Client lists content published by an application instance<a class="headerlink" href="#slider-client-lists-content-published-by-an-application-instance" title="Permanent link">¶</a></h2>
<div class="codehilite"><pre><span class="n">slider</span> <span class="n">registry</span> <span class="o"><</span><span class="n">instance</span><span class="o">></span> <span class="o">--</span><span class="n">listconf</span> <span class="p">[</span><span class="o">--</span><span class="n">servicetype</span> <span class="o"><</span><span class="n">application</span><span class="o">-</span><span class="n">type</span><span class="o">></span><span class="p">]</span>
</pre></div>
@@ -326,7 +337,7 @@ Latest release: <strong>0.80.0-incubatin
<p>Client displays list of content</p>
</li>
</ol>
-<h2 id="slider-client-retrieves-content-published-by-an-application-instance">Slider Client retrieves content published by an application instance</h2>
+<h2 id="slider-client-retrieves-content-published-by-an-application-instance">Slider Client retrieves content published by an application instance<a class="headerlink" href="#slider-client-retrieves-content-published-by-an-application-instance" title="Permanent link">¶</a></h2>
<div class="codehilite"><pre><span class="n">slider</span> <span class="n">registry</span> <span class="o"><</span><span class="n">instance</span><span class="o">></span> <span class="o">--</span><span class="n">getconf</span> <span class="o"><</span><span class="n">document</span><span class="o">></span> <span class="p">[</span><span class="o">--</span><span class="n">format</span> <span class="p">(</span><span class="n">xml</span><span class="o">|</span><span class="k">properties</span><span class="o">|</span><span class="n">text</span><span class="o">|</span><span class="n">html</span><span class="o">|</span><span class="n">csv</span><span class="o">|</span><span class="n">yaml</span><span class="o">|</span><span class="n">json</span><span class="p">,...)</span> <span class="p">[</span><span class="o">--</span><span class="n">dest</span> <span class="o"><</span><span class="n">file</span><span class="o">></span><span class="p">]</span> <span class="p">[</span><
span class="o">--</span><span class="n">servicetype</span> <span class="o"><</span><span class="n">application</span><span class="o">-</span><span class="n">type</span><span class="o">></span><span class="p">]</span>
</pre></div>
@@ -358,9 +369,9 @@ Latest release: <strong>0.80.0-incubatin
a file underneath.</p>
</li>
</ol>
-<h2 id="slider-client-retrieves-content-set-published-by-an-application-instance">Slider Client retrieves content set published by an application instance</h2>
+<h2 id="slider-client-retrieves-content-set-published-by-an-application-instance">Slider Client retrieves content set published by an application instance<a class="headerlink" href="#slider-client-retrieves-content-set-published-by-an-application-instance" title="Permanent link">¶</a></h2>
<p>Here a set of documents published is retrieved in the desired format of an application.</p>
-<h2 id="slider-client-retrieves-document-and-applies-template-to-it">Slider Client retrieves document and applies template to it</h2>
+<h2 id="slider-client-retrieves-document-and-applies-template-to-it">Slider Client retrieves document and applies template to it<a class="headerlink" href="#slider-client-retrieves-document-and-applies-template-to-it" title="Permanent link">¶</a></h2>
<p>Here a set of documents published is retrieved in the desired format of an application.</p>
<div class="codehilite"><pre><span class="n">slider</span> <span class="n">registry</span> <span class="o"><</span><span class="n">instance</span><span class="o">></span> <span class="o">--</span><span class="n">source</span> <span class="o"><</span><span class="n">document</span><span class="o">></span> <span class="p">[</span><span class="o">--</span><span class="n">template</span> <span class="o"><</span><span class="n">path</span><span class="o">-</span><span class="n">to</span><span class="o">-</span><span class="n">template</span><span class="o">></span><span class="p">]</span> <span class="p">[</span><span class="o">--</span><span class="n">outfile</span> <span class="o"><</span><span class="n">file</span><span class="o">></span><span class="p">]</span> <span class="p">[</span><span class="o">--</span><span class="n">servicetype</span> <span class="o"><</span><span class="n">application</span><span class="o">-</span><span class="n">type</span><span
class="o">></span><span class="p">]</span>
</pre></div>
Modified: websites/staging/slider/trunk/content/design/registry/the_YARN_application_registration_and_binding_problem.html
==============================================================================
--- websites/staging/slider/trunk/content/design/registry/the_YARN_application_registration_and_binding_problem.html (original)
+++ websites/staging/slider/trunk/content/design/registry/the_YARN_application_registration_and_binding_problem.html Mon Nov 2 17:24:47 2015
@@ -155,7 +155,7 @@
<div style="text-align: center">
<h1><a href="/index.html">Apache Slider (incubating)</a></h1>
<hr>
-Latest release: <strong>0.80.0-incubating</strong><br>
+Latest release: <strong>0.81.1-incubating</strong><br>
<br>
<a id="download-button-sidebar" class="btn btn-success btn-block" href="/downloads/" role="button">Download</a>
</div>
@@ -168,7 +168,18 @@ Latest release: <strong>0.80.0-incubatin
<h1 class="title"></h1>
- <!---
+ <style type="text/css">
+/* The following code is added by mdx_elementid.py
+ It was originally lifted from http://subversion.apache.org/style/site.css */
+/*
+ * Hide class="elementid-permalink", except when an enclosing heading
+ * has the :hover property.
+ */
+.headerlink, .elementid-permalink {
+ visibility: hidden;
+}
+h2:hover > .headerlink, h3:hover > .headerlink, h1:hover > .headerlink, h6:hover > .headerlink, h4:hover > .headerlink, h5:hover > .headerlink, dt:hover > .elementid-permalink { visibility: visible }</style>
+<!---
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
@@ -185,9 +196,9 @@ Latest release: <strong>0.80.0-incubatin
limitations under the License.
-->
-<h1 id="apache-slider-yarn-application-registration-and-binding-the-problem">Apache Slider YARN Application Registration and Binding: the Problem</h1>
-<h2 id="march-2014">March 2014</h2>
-<h1 id="how-to-bind-client-applications-to-services-dynamically-placed-applications">How to bind client applications to services dynamically placed applications?</h1>
+<h1 id="apache-slider-yarn-application-registration-and-binding-the-problem">Apache Slider YARN Application Registration and Binding: the Problem<a class="headerlink" href="#apache-slider-yarn-application-registration-and-binding-the-problem" title="Permanent link">¶</a></h1>
+<h2 id="march-2014">March 2014<a class="headerlink" href="#march-2014" title="Permanent link">¶</a></h2>
+<h1 id="how-to-bind-client-applications-to-services-dynamically-placed-applications">How to bind client applications to services dynamically placed applications?<a class="headerlink" href="#how-to-bind-client-applications-to-services-dynamically-placed-applications" title="Permanent link">¶</a></h1>
<p>There are some constraints here</p>
<ol>
<li>
@@ -210,7 +221,7 @@ applications? This is an issue which has
for Hadoop clusters -but if we are looking at registration and binding
of YARN applications, there should be no reason why
static applications cannot be discovered and bonded to using the same mechanisms. </p>
-<h1 id="other-constraints">Other constraints:</h1>
+<h1 id="other-constraints">Other constraints:<a class="headerlink" href="#other-constraints" title="Permanent link">¶</a></h1>
<ol>
<li>
<p>Reduce the amount of change needed in existing applications to a minimum
@@ -227,12 +238,12 @@ static applications cannot be discovered
outside of Slider. Rationale: want a mechanism that works with pure-YARN apps</p>
</li>
</ol>
-<h2 id="possible-solutions">Possible Solutions:</h2>
-<h3 id="zk">ZK</h3>
+<h2 id="possible-solutions">Possible Solutions:<a class="headerlink" href="#possible-solutions" title="Permanent link">¶</a></h2>
+<h3 id="zk">ZK<a class="headerlink" href="#zk" title="Permanent link">¶</a></h3>
<p>Client applications use ZK to find services (addresses #1, #2 and #3).
Requires location code in the client.</p>
<p>HBase and Accumulo do this as part of a failover-ready design.</p>
-<h3 id="dns">DNS</h3>
+<h3 id="dns">DNS<a class="headerlink" href="#dns" title="Permanent link">¶</a></h3>
<p>Client apps use DNS to find services, with custom DNS server for a
subdomain representing YARN services. Addresses #1; with a shortened TTL and
no DNS address caching, #3. #2 addressed only if other DNS entries are used to
@@ -243,7 +254,7 @@ over time. It does require the clients t
irrespective of the published TTL). It generates a load on the DNS servers
that is <code>O(clients/TTL)</code></p>
<p>Google Chubby offers a DNS service to handle this. ZK does not -yet.</p>
-<h3 id="floating-ip-addresses">Floating IP Addresses</h3>
+<h3 id="floating-ip-addresses">Floating IP Addresses<a class="headerlink" href="#floating-ip-addresses" title="Permanent link">¶</a></h3>
<p>If the clients know/cache IP addresses of services, these addresses could be
floated across service instances. Linux HA has floating IP address support,
while Docker containers can make use of them, especially if an integrated DHCP
@@ -251,7 +262,7 @@ server handles the assignment of IP addr
<p>ARP caching is the inevitable problem here, but it is still less brittle than
relying on applications to know not to cache IP addresses -and nor does it
place so much on DNS servers as short-TTL DNS entries.</p>
-<h3 id="ldap">LDAP</h3>
+<h3 id="ldap">LDAP<a class="headerlink" href="#ldap" title="Permanent link">¶</a></h3>
<p>Enterprise Directory services are used to publish/locate services. Requires
lookup into the directory on binding (#1, #2), re-lookup on failure (#3).
LDAP permissions can prevent untrusted applications registering.</p>
@@ -265,7 +276,7 @@ libraries. Spring-LDAP is focused on con
-not LDAP-driven application config.</p>
</li>
</ul>
-<h3 id="registration-web-service">Registration Web Service</h3>
+<h3 id="registration-web-service">Registration Web Service<a class="headerlink" href="#registration-web-service" title="Permanent link">¶</a></h3>
<p>Custom web service registration services used. </p>
<ul>
<li>
@@ -276,7 +287,7 @@ libraries. Spring-LDAP is focused on con
<p>Requires new client-side code anyway.</p>
</li>
</ul>
-<h3 id="zookeeper-url-schema">Zookeeper URL Schema</h3>
+<h3 id="zookeeper-url-schema">Zookeeper URL Schema<a class="headerlink" href="#zookeeper-url-schema" title="Permanent link">¶</a></h3>
<p>Offer our own <code>zk://</code> URL; java & .NET implementations (others?) to resolve, browser plugins. </p>
<ul>
<li>
@@ -287,7 +298,7 @@ libraries. Spring-LDAP is focused on con
will not work with tools such as CURL or web browsers</p>
</li>
</ul>
-<h3 id="am-side-config-generation">AM-side config generation</h3>
+<h3 id="am-side-config-generation">AM-side config generation<a class="headerlink" href="#am-side-config-generation" title="Permanent link">¶</a></h3>
<p>App-side config generation-YARN applications to generate client-side
configuration files for launch-time information (#1, #2).
The AM can dynamically create these, and as the storage load is all in
@@ -311,7 +322,7 @@ the entire cluster, and push them back.<
<p>Needs something similar for statically deployed applications.</p>
</li>
</ul>
-<h3 id="client-side-config-generation">Client-side config generation</h3>
+<h3 id="client-side-config-generation">Client-side config generation<a class="headerlink" href="#client-side-config-generation" title="Permanent link">¶</a></h3>
<p>YARN app to publish attributes as key-val pairs, client-side code to read and
generate configs from (#1, #2). Example configuration generators could
create: Hadoop-client XML, Spring, tomcat, guice configs, something for .NET.</p>
@@ -356,7 +367,7 @@ to log all slf/log4j to stderr, so that
<p>client side code per app to generate specific binding information</p>
</li>
</ul>
-<h3 id="load-balancer-app-yarn-app">Load-balancer app Yarn App</h3>
+<h3 id="load-balancer-app-yarn-app">Load-balancer app Yarn App<a class="headerlink" href="#load-balancer-app-yarn-app" title="Permanent link">¶</a></h3>
<p>Spread requests round a set of registered handlers, e.g web servers. Support
plugins for session binding/sharding. </p>
<p>Some web servers can do this already; a custom YARN app could use grizzy