You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@slider.apache.org by go...@apache.org on 2015/05/19 20:17:16 UTC
svn commit: r1680364 - in /incubator/slider/site/trunk:
content/docs/slider_specs/application_pkg_docker.md templates/nav.html
Author: gourksaha
Date: Tue May 19 18:17:16 2015
New Revision: 1680364
URL: http://svn.apache.org/r1680364
Log:
SLIDER-780 Support for Docker based application packaging in Slider - site documentation (Thomas Liu via gourksaha)
Added:
incubator/slider/site/trunk/content/docs/slider_specs/application_pkg_docker.md
Modified:
incubator/slider/site/trunk/templates/nav.html
Added: incubator/slider/site/trunk/content/docs/slider_specs/application_pkg_docker.md
URL: http://svn.apache.org/viewvc/incubator/slider/site/trunk/content/docs/slider_specs/application_pkg_docker.md?rev=1680364&view=auto
==============================================================================
--- incubator/slider/site/trunk/content/docs/slider_specs/application_pkg_docker.md (added)
+++ incubator/slider/site/trunk/content/docs/slider_specs/application_pkg_docker.md Tue May 19 18:17:16 2015
@@ -0,0 +1,192 @@
+<!---
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+# Apache Slider Docker Based Application Packaging Support
+
+##Introduction
+
+Docker is an open platform for developers and sysadmins to build, ship, and run distributed applications.
+
+- Users can use their existing Docker images without any modifications.
+- They need to create configuration files according to Slider's template for Docker based applications
+- Slider can deploy their Docker containers within YARN cluster, with resource management capability from YARN
+- They can use Slider to monitor and manage their Docker containers/applications
+
+## Terminology
+
+### Docker Image
+- Applications definition file
+- Is composed of one or multiple layers
+- Users create Docker containers out of Docker images
+
+### Docker Container
+- A running instance of a Docker image
+- Comprises the application and its dependencies
+- Enjoys the resource isolation and allocation benefits of VMs but is much more portable and efficient.
+
+##Specifications
+
+Refer to [*What is Docker*](https://www.docker.com/whatisdocker/) for a quick overview of Docker and why it excels VM.
+
+**Goal**
+
+- Slider shall be able to deploy an application defined as Docker image, monitor its running status, fetch exported configs, and maintain its lifecycle
+- Users can use their existing Docker images without any modifications
+- Slider shall be able to pull Docker images from Docker Hubs
+- Slider shall be able to aggregate application logs from and feed in input files into Docker containers
+- Slider shall be able to launch applications containing multiple Docker images, as well as maintaining multiple Docker containers on the same physical host
+- Users shall be able to specify the Docker images, configurations and if necessary, the special docker command to start/query/stop the container in Slider configuration files
+- Slider shall be able to deploy security credentials into the container if the application is running in secure mode
+
+**The scope of this JIRA doesnât yet include**
+
+- Application Docker containers integrated with Ambari client
+- Applications deployed in secure mode
+- Running application Docker containers in OS other than Linux
+- Docker images stored in private Docker hub - we would leave this work to administrators to configure Docker daemon to be able to pull from those hubs
+- Application Docker containers started with `docker run -i` option which requires interaction with the user
+- Docker images that require multiple initialization steps, more than just one `docker run`
+- YARN can only monitor resources consumed by SliderAgent, instead of that of the Docker containers
+
+**Commands to run**
+
+ slider create [app-name] --template appConfig.json --metainfo metainfo.json --resources resources.json
+
+Below is an example of how we can use Slider to deploy a multi-component Dockerized application
+
+**appConfig.json**
+```
+ {
+ "schema": "http://example.org/specification/v2.0.0",
+ "metadata": {
+ },
+ "global": {
+ },
+ "components": {
+ "REDIS": {
+ "memcached.commandPath":"/user/local/bin/docker",
+ "memcached.options":"-d -e REDIS_PASS=\"**None**\"",
+ "memcached.statusCommand":"/usr/bin/docker ps",
+ "memcached.inputFiles": [
+ {
+ "containerPath": "/tmp/input",
+ "fileLocalPath": "/Users/peter/config.json"
+ }
+ ]
+ }
+ }
+ }
+```
+Please note that in this example, we are specifying a different docker command path than the default `/usr/bin/docker` in appConfig.json. We also specify the options that we need to include in the `docker run` command (`-d` is by default included). This is just a demo as how to include docker run command options. We also need to expose a port to listen on for the application. Thus, we are specifying `site.global.listen_port` in appConfig.json and the export config in metainfo.json below
+
+ The config structure in appConfig.json(map) -> components(list) -> component(map) -> containers(list)-> container(map) have to match the one in metainfo.json.
+
+We are also adding a few configurations for the component *memcached* in appConfig.json here. These configurations are runtime parameters of the component, as opposed to being part of the Docker image definition in metainfo.json.
+
+**metainfo.json**
+```
+ {
+ "schemaVersion": "2.1",
+ "application": {
+ "name": "FAKEAPP",
+ "components": [
+ {
+ "name": "MEMCACHED",
+ "type": "docker",
+ "dockerContainers": [
+ {
+ "name": "memcached",
+ "image": "borja/memcached",
+ "additionalParam": "--appendonly yes",
+ "commandPath": "/usr/bin/docker",
+ "statusCommand": "docker inspect ${container_id}"
+ }
+ ]
+ },
+ {
+ "name": "REDIS",
+ "type": "docker",
+ "dockerContainers": [
+ {
+ "name": "redis",
+ "image": "dockerhub/redis",
+ "additionalParam": "--appendonly yes",
+ "commandPath": "/usr/bin/docker",
+ "ports": [
+ {
+ "containerPort" : "11211",
+ "hostPort": "${conf:configuration/global/port1}"
+ }
+ ],
+ "mounts": [
+ {
+ "containerMount": "/tmp/confg",
+ "hostMount": "${conf:configuration/global/app_root}/conf"
+ }
+ ]
+ }
+ ]
+ }
+ ]
+ }
+ }
+```
+Please note, in metainfo.json (we are trying to migrate from metainfo.xml to metainfo.json as an improvement to the packaging approach) we are adding some new fields in the component section to support Docker based applications:
+
+- **type**: if specified as `docker`, Slider will start Docker containers for the application; by default it is `process`, which means Slider will instantiate the application as normal process as today
+- **containers/container**: the Docker image of the application, the name of which will be specified in `image` field
+- **port, containerPort and hostPort**: the port of the container that will be binded to the hostPort field, which will be translated into `-p hostPort:containerPort` when starting the container with `docker run`
+- **mount, containerMount and hostMount**: the directories from the host that will be mounted into the container, which will be translated into â-v hostMount:containerMountâ when starting the container
+- **options**: allow users to specify any additional docker run command options to use
+- **status_command**: the command that Slider can use to query the status of the application component running in the container
+- Properties specified in metainfo.json can be overridden in appConfig.json: **commandPath, options, statusCommand, inputFiles, mounts, ports**
+
+**resources.json**
+```
+{
+ "schema" : "http://example.org/specification/v2.0.0",
+ "metadata" : {
+ },
+ "global" : {
+ },
+ "components": {
+ "slider-appmaster": {
+ },
+ "REDIS": {
+ "yarn.role.priority": "1",
+ "yarn.component.instances": "2",
+ "yarn.memory": "512"
+ },
+ "MEMCACHED": {
+ "yarn.role.priority": "1",
+ "yarn.component.instances": "2",
+ "yarn.memory": "512"
+ }
+
+ }
+}
+```
+With the configuration files above, Slider will ask for the required number of containers from YARN, as specified in resources.json. In each of those YARN containers, Slider will run `docker pull` to download the Docker image specified in metainfo.json. In the scope of this JIRA ticket, we are not supporting Docker images stored in private Docker hubs that require credentials to run docker pull. After downloading completes, Slider will start the containers by running, in this case, `/usr/bin/docker run -d borja/memcached`
+
+- For redis components, it will run:
+
+`docker run -d -p hostPort:containerPort -v hostMount:containerMount -net=bridge dockerhub/redis`
+
+- For memcached components, it will run:
+
+`docker run -d borja/memcached memcached_server.sh --appendonly`
+
Modified: incubator/slider/site/trunk/templates/nav.html
URL: http://svn.apache.org/viewvc/incubator/slider/site/trunk/templates/nav.html?rev=1680364&r1=1680363&r2=1680364&view=diff
==============================================================================
--- incubator/slider/site/trunk/templates/nav.html (original)
+++ incubator/slider/site/trunk/templates/nav.html Tue May 19 18:17:16 2015
@@ -72,6 +72,7 @@
<li role="presentation" class="divider"></li>
<li id="nav_use_app_package"><a href="/docs/slider_specs/index.html">Creating App Packages</a></li>
<li id="nav_use_app_package_upgrade"><a href="/docs/slider_specs/application_pkg_upgrade.html">Upgrading App Packages</a></li>
+<li id="nav_use_app_package_docker"><a href="/docs/slider_specs/application_pkg_docker.html">Docker App Packages</a></li>
<li role="presentation" class="divider"></li>
<li id="nav_use_client_config"><a href="/docs/client-configuration.html">Client Configuration</a></li>
<li id="nav_use_exitcodes"><a href="/docs/exitcodes.html">Client Exit Codes</a></li>