You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@slider.apache.org by go...@apache.org on 2016/03/16 21:58:07 UTC
[16/50] incubator-slider git commit: SLIDER-977 Kafka on YARN (KOYA)
app-package.
SLIDER-977 Kafka on YARN (KOYA) app-package.
Contribution from Thomas Weise, Siyuan Hua, and Jean-Baptiste Note.
Closes apache/incubator-slider#3
Project: http://git-wip-us.apache.org/repos/asf/incubator-slider/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-slider/commit/65e229e9
Tree: http://git-wip-us.apache.org/repos/asf/incubator-slider/tree/65e229e9
Diff: http://git-wip-us.apache.org/repos/asf/incubator-slider/diff/65e229e9
Branch: refs/heads/feature/SLIDER-906_docker_support
Commit: 65e229e943255f8c16c8676ad6a9f04487150c38
Parents: e1c802a
Author: Josh Elser <el...@apache.org>
Authored: Tue Jan 19 22:26:01 2016 -0500
Committer: Josh Elser <el...@apache.org>
Committed: Wed Jan 20 12:08:44 2016 -0500
----------------------------------------------------------------------
app-packages/kafka/README.md | 110 +++++++++++++++++++
app-packages/kafka/appConfig-default.json | 30 +++++
app-packages/kafka/build.sh | 5 +
app-packages/kafka/configuration/server.xml | 16 +++
app-packages/kafka/deploy.sh | 11 ++
app-packages/kafka/flex.sh | 5 +
app-packages/kafka/metainfo-default.xml | 56 ++++++++++
app-packages/kafka/package/scripts/kafka.py | 96 ++++++++++++++++
app-packages/kafka/package/scripts/params.py | 18 +++
.../kafka/package/scripts/status_params.py | 9 ++
app-packages/kafka/package/scripts/util.py | 16 +++
app-packages/kafka/pom.xml | 69 ++++++++++++
app-packages/kafka/resources-default.json | 20 ++++
.../kafka/src/assembly/koya-app-package.xml | 80 ++++++++++++++
.../kafka/src/assembly/koya-with-slider.xml | 64 +++++++++++
15 files changed, 605 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/65e229e9/app-packages/kafka/README.md
----------------------------------------------------------------------
diff --git a/app-packages/kafka/README.md b/app-packages/kafka/README.md
new file mode 100644
index 0000000..97ddef4
--- /dev/null
+++ b/app-packages/kafka/README.md
@@ -0,0 +1,110 @@
+Kafka On YARN (KOYA)
+====================
+
+[](https://gitter.im/DataTorrent/koya?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge)
+
+### Goals
+
+ * Use capabilities of YARN for Kafka broker management
+ * Automate broker recovery
+ * Make it easy to deploy, configure and monitor Kafka clusters
+ * Simplify management tasks (alternative to Kafka command line utilities)
+
+JIRA: https://issues.apache.org/jira/browse/KAFKA-1754
+
+Kafka as YARN application using Slider
+-----------------------------------------------
+
+### Build
+
+Checkout Slider code (https://github.com/apache/incubator-slider)
+```sh
+git clone git@github.com:apache/incubator-slider.git
+git checkout -b slider-0.80.0-incubating remotes/origin/releases/slider-0.80.0-incubating
+```
+The Slider version you checked out needs to match ${slider.version} in pom.xml
+
+Create symbolic link to Slider source code within the KOYA repository:
+```sh
+ln -s /path/to/repo/incubator-slider/ slider
+```
+Download Kafka binary package (http://kafka.apache.org/downloads.html)
+
+```sh
+mvn clean install -DskipTests -Dkafka.src=path/to/kafka_2.10-0.8.1.1.tgz -Dkafka.version=kafka_2.10-0.8.1.1
+```
+Artifacts:
+
+ - Archive with embedded Slider: __`target/koya-with-slider.zip`__
+ - Separate Slider application package: __`target/koya-slider-package-0.1.zip`__
+
+###Installation
+
+####Install Slider
+
+To use the archive with embedded Slider, copy it to the machine from which you launch YARN applications (Hadoop client, gateway or edge node). Extract the file and configure Slider:
+
+If the environment variables `HADOOP_CONF_DIR` or `JAVA_HOME` are not already defined through your Hadoop installation, you can export them in `slider-0.80.0-incubating/conf/slider-env.sh`
+
+Example for CDH 5.4:
+```
+export HADOOP_CONF_DIR=/etc/hadoop/conf
+export JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera
+```
+If the registry ZooKeeper quorum was not already configured through Hadoop, modify `slider-0.80.0-incubating/conf/slider-client.xml`:
+```
+ <property>
+ <name>hadoop.registry.zk.quorum</name>
+ <value>node26:2181,node27:2181,node28:2181</value>
+ </property>
+```
+Above steps are not required with HDP 2.2
+
+More information regarding Slider client configuration refer to http://slider.incubator.apache.org/docs/client-configuration.html
+
+### Configure KOYA application package
+
+Before the Kafka cluster can be launched, the brokers need to be defined. Currently Slider does not support [configuration properties at instance level](https://issues.apache.org/jira/browse/SLIDER-851), therefore each broker has to be configured as a component.
+
+If you use the full archive, the configuration file templates are already in your working directory. Otherwise extract them from the Slider package.
+
+####appConfig.json
+
+Extract the packaged configuration files you are going to customize:
+```
+unzip koya-slider-package-0.1.zip appConfig.json resources.json
+```
+Adjust following properties in the global section:
+```
+ "application.def": "koya-slider-package-0.1.zip",
+ "site.global.xmx_val": "256m",
+ "site.global.xms_val": "128m",
+ "site.broker.zookeeper.connect": "${ZK_HOST}"
+```
+Above will be used to configure server.properties and launch the Kafka server(s). All properties prefixed with `site.broker.` will be set in the server.properties file supplied with the Kafka archive. Only non-default settings need to be defined here.
+
+####resources.json
+
+Configure the number of servers and other resource requirements:
+```
+ "components" : {
+ "broker" : {
+ "yarn.role.priority" : "1",
+ "yarn.component.instances" : "5",
+ "yarn.memory" : "768",
+ "yarn.vcores" : "1",
+ "yarn.component.placement.policy":"1"
+ }
+```
+More information about the application configuration can be found [here](http://slider.incubator.apache.org/docs/configuration/core.html).
+
+### Deploy KOYA Cluster
+
+The Slider application package needs to be copied to the HDFS location that was specified as application.def in appConfig.json:
+```
+hdfs dfs -copyFromLocal koya-slider-package-0.1.zip /path/in/appConfig
+```
+Now the KOYA cluster can be deployed and launched:
+```
+slider-0.80.0-incubating/bin/slider create koya --template ~/koya/appConfig.json --resources ~/koya/resources.json
+```
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/65e229e9/app-packages/kafka/appConfig-default.json
----------------------------------------------------------------------
diff --git a/app-packages/kafka/appConfig-default.json b/app-packages/kafka/appConfig-default.json
new file mode 100644
index 0000000..9a0be4c
--- /dev/null
+++ b/app-packages/kafka/appConfig-default.json
@@ -0,0 +1,30 @@
+{
+ "schema": "http://example.org/specification/v2.0.0",
+ "metadata": {
+ },
+ "global": {
+ "application.def": ".slider/package/KOYA/koya-slider-package-${project.version}.zip",
+ "java_home": "${JAVA_HOME}",
+ "system_configs": "broker",
+
+ "site.global.app_user": "${USER_NAME}",
+ "site.global.app_root": "${AGENT_WORK_ROOT}/app/install/${kafka.version}",
+ "site.global.app_install_dir": "${AGENT_WORK_ROOT}/app/install",
+ "site.global.pid_file": "${AGENT_WORK_ROOT}/app/run/koya.pid",
+
+ "site.global.kafka_version": "${kafka.version}",
+ "site.broker.xmx_val": "512m",
+ "site.broker.xms_val": "128m",
+ "site.broker.instance.name": "${USER}/${CLUSTER_NAME}",
+ "site.broker.zookeeper": "${ZK_HOST}",
+
+ "site.server.port": "${KAFKA_BROKER.ALLOCATED_PORT}{PER_CONTAINER}"
+ },
+ "components": {
+ "broker": {
+ },
+ "slider-appmaster": {
+ "jvm.heapsize": "256M"
+ }
+ }
+}
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/65e229e9/app-packages/kafka/build.sh
----------------------------------------------------------------------
diff --git a/app-packages/kafka/build.sh b/app-packages/kafka/build.sh
new file mode 100755
index 0000000..7951f9e
--- /dev/null
+++ b/app-packages/kafka/build.sh
@@ -0,0 +1,5 @@
+#! /bin/bash
+
+wget http://apache.websitebeheerjd.nl/kafka/0.8.2.1/kafka_2.10-0.8.2.1.tgz -O $HOME/kafka_2.10-0.8.2.1.tgz
+mvn clean install -DskipTests -Dkafka.src=$HOME/kafka_2.10-0.8.2.1.tgz -Dkafka.version=kafka_2.10-0.8.2.1
+unzip -o target/koya-slider-package-0.1.zip appConfig.json
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/65e229e9/app-packages/kafka/configuration/server.xml
----------------------------------------------------------------------
diff --git a/app-packages/kafka/configuration/server.xml b/app-packages/kafka/configuration/server.xml
new file mode 100644
index 0000000..68b36c9
--- /dev/null
+++ b/app-packages/kafka/configuration/server.xml
@@ -0,0 +1,16 @@
+<?xml version="1.0"?>
+<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
+<configuration>
+ <property>
+ <name>zookeeper.connect</name>
+ <value>${@//site/broker/zookeeper}/kafka/${@//site/broker/instance.name}</value>
+ </property>
+ <property>
+ <name>broker.id</name>
+ <value>${@//site/global/app_container_tag}</value>
+ </property>
+ <property>
+ <name>log.dirs</name>
+ <value>${@//site/global/app_pid_dir}</value>
+ </property>
+</configuration>
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/65e229e9/app-packages/kafka/deploy.sh
----------------------------------------------------------------------
diff --git a/app-packages/kafka/deploy.sh b/app-packages/kafka/deploy.sh
new file mode 100755
index 0000000..1f8f473
--- /dev/null
+++ b/app-packages/kafka/deploy.sh
@@ -0,0 +1,11 @@
+#! /bin/bash
+
+APPNAME=${1:-koya}
+slider install-package --replacepkg --name KOYA --package target/koya-slider-package-0.1.zip
+slider stop $APPNAME
+slider destroy $APPNAME
+slider create $APPNAME --filesystem hdfs://root --queue dev --template appConfig.json --resources resources-default.json
+
+
+
+
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/65e229e9/app-packages/kafka/flex.sh
----------------------------------------------------------------------
diff --git a/app-packages/kafka/flex.sh b/app-packages/kafka/flex.sh
new file mode 100755
index 0000000..20a1003
--- /dev/null
+++ b/app-packages/kafka/flex.sh
@@ -0,0 +1,5 @@
+#! /bin/bash
+
+APPNAME=${1:-koya}
+APPNUM=${2:-12}
+slider flex $APPNAME --component BROKER0 $APPNUM --filesystem hdfs://root
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/65e229e9/app-packages/kafka/metainfo-default.xml
----------------------------------------------------------------------
diff --git a/app-packages/kafka/metainfo-default.xml b/app-packages/kafka/metainfo-default.xml
new file mode 100644
index 0000000..f1eec0a
--- /dev/null
+++ b/app-packages/kafka/metainfo-default.xml
@@ -0,0 +1,56 @@
+<metainfo>
+ <schemaVersion>2.0</schemaVersion>
+ <application>
+ <name>KAFKA</name>
+ <comment>Scalable High-thoughput Distributed Message Queue</comment>
+ <version>${kafka.version}</version>
+ <exportedConfigs>None</exportedConfigs>
+
+ <exportGroups>
+ <exportGroup>
+ <name>servers</name>
+ <exports>
+ <export>
+ <name>org.apache.kafka.broker</name>
+ <value>${KAFKA_BROKER_HOST}:${site.server.port}</value>
+ </export>
+ </exports>
+ </exportGroup>
+ </exportGroups>
+
+ <components>
+ <component>
+ <name>KAFKA_BROKER</name>
+ <category>SLAVE</category>
+ <minInstanceCount>1</minInstanceCount>
+ <compExports>servers-org.apache.kafka.broker</compExports>
+ <exportedConfigs>broker</exportedConfigs>
+ <commandScript>
+ <script>scripts/kafka.py</script>
+ <scriptType>PYTHON</scriptType>
+ </commandScript>
+ </component>
+ </components>
+
+ <configFiles>
+ <configFile>
+ <type>xml</type>
+ <fileName>server.xml</fileName>
+ <dictionaryName>server</dictionaryName>
+ </configFile>
+ </configFiles>
+
+ <osSpecifics>
+ <osSpecific>
+ <osType>any</osType>
+ <packages>
+ <package>
+ <type>tarball</type>
+ <name>files/${kafka.version}.tgz</name>
+ </package>
+ </packages>
+ </osSpecific>
+ </osSpecifics>
+
+ </application>
+</metainfo>
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/65e229e9/app-packages/kafka/package/scripts/kafka.py
----------------------------------------------------------------------
diff --git a/app-packages/kafka/package/scripts/kafka.py b/app-packages/kafka/package/scripts/kafka.py
new file mode 100644
index 0000000..38b12a3
--- /dev/null
+++ b/app-packages/kafka/package/scripts/kafka.py
@@ -0,0 +1,96 @@
+import logging
+import sys
+import os
+import inspect
+import pprint
+import util
+from resource_management import *
+
+logger = logging.getLogger()
+
+class Kafka(Script):
+ def install(self, env):
+ self.install_packages(env)
+
+ def configure(self, env):
+ import params
+ env.set_params(params)
+
+ def start(self, env):
+ import params
+ env.set_params(params)
+ self.configure(env)
+
+ # log the component configuration
+ ppp = pprint.PrettyPrinter(indent=4)
+ logger.info("broker component config: " + ppp.pformat(params.broker_config))
+
+ # log the environment variables
+ logger.info("Env Variables:")
+ for key in os.environ.keys():
+ logger.info("%10s %s \n" % (key,os.environ[key]))
+ pass
+
+ # This updating thing is changing files in-place and it really
+ # should not (static cache)
+
+ # For kafka 0.8.1.1, there is no way to set the log dir to location other than params.app_root + "/logs"
+ if(params.kafka_version.find("0.8.1.1") != -1):
+ os.symlink(params.app_root + "/logs", params.app_log_dir + "/kafka")
+ else:
+ kafkaLogConfig = {"kafka.logs.dir" : params.app_log_dir + "/kafka"}
+ util.updating(params.app_root + "/config/log4j.properties", kafkaLogConfig)
+# File(format("{params.app_root}/conf/log4j.properties"),
+# owner=params.app_user,
+# content=InlineTemplate(param.log4j_prop))
+ pass
+
+ # update the broker properties for different brokers
+ server_conf=format("{params.conf_dir}/server.slider.properties")
+ PropertiesFile(server_conf, properties = params.broker_config, owner=params.app_user)
+
+ # execute the process
+ process_cmd = format("{app_root}/bin/kafka-server-start.sh {server_conf}")
+ os.environ['LOG_DIR'] = params.app_log_dir + "/kafka"
+ HEAP_OPT = ""
+ if params.xmx:
+ HEAP_OPT = HEAP_OPT + " -Xmx" + params.xmx
+ pass
+ if params.xms:
+ HEAP_OPT = HEAP_OPT + " -Xms" + params.xms
+ pass
+ if HEAP_OPT:
+ os.environ['KAFKA_HEAP_OPTS'] = HEAP_OPT
+ pass
+ Execute(process_cmd,
+ user=params.app_user,
+ logoutput=True,
+ wait_for_finish=False,
+ pid_file=params.pid_file
+ )
+
+ def stop(self, env):
+ import params
+ env.set_params(params)
+ pid = format("`cat {pid_file}` >/dev/null 2>&1")
+ Execute(format("kill {pid}"),
+ user=params.app_user
+ )
+ Execute(format("kill -9 {pid}"),
+ ignore_failures=True,
+ user=params.app_user
+ )
+ Execute(format("rm -f {pid_file}"),
+ user=params.app_user)
+
+ def status(self, env):
+ import status_params
+ env.set_params(status_params)
+## jps_cmd = format("{java64_home}/bin/jps")
+ no_op_test = format("ls {pid_file} >/dev/null 2>&1 && ps `cat {pid_file}` >/dev/null 2>&1")
+## cmd = format("echo `{jps_cmd} | grep Kafka | cut -d' ' -f1` > {pid_file}")
+## Execute(cmd, not_if=no_op_test)
+ check_process_status(status_params.pid_file)
+
+if __name__ == "__main__":
+ Kafka().execute()
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/65e229e9/app-packages/kafka/package/scripts/params.py
----------------------------------------------------------------------
diff --git a/app-packages/kafka/package/scripts/params.py b/app-packages/kafka/package/scripts/params.py
new file mode 100644
index 0000000..de3994e
--- /dev/null
+++ b/app-packages/kafka/package/scripts/params.py
@@ -0,0 +1,18 @@
+from resource_management import *
+
+config = Script.get_config()
+
+
+app_root = config['configurations']['global']['app_root']
+java64_home = config['hostLevelParams']['java_home']
+app_user = config['configurations']['global']['app_user']
+pid_file = config['configurations']['global']['pid_file']
+app_log_dir = config['configurations']['global']['app_log_dir']
+kafka_version = config['configurations']['global']['kafka_version']
+xmx = config['configurations']['broker']['xmx_val']
+xms = config['configurations']['broker']['xms_val']
+
+conf_dir = format("{app_root}/config")
+
+broker_config=dict(line.strip().split('=') for line in open(format("{conf_dir}/server.properties")) if not (line.startswith('#') or re.match(r'^\s*$', line)))
+broker_config.update(config['configurations']['server'])
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/65e229e9/app-packages/kafka/package/scripts/status_params.py
----------------------------------------------------------------------
diff --git a/app-packages/kafka/package/scripts/status_params.py b/app-packages/kafka/package/scripts/status_params.py
new file mode 100644
index 0000000..cc46d5b
--- /dev/null
+++ b/app-packages/kafka/package/scripts/status_params.py
@@ -0,0 +1,9 @@
+from resource_management import *
+
+config = Script.get_config()
+
+app_root = config['configurations']['global']['app_root']
+java64_home = config['hostLevelParams']['java_home']
+app_user = config['configurations']['global']['app_user']
+pid_file = config['configurations']['global']['pid_file']
+
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/65e229e9/app-packages/kafka/package/scripts/util.py
----------------------------------------------------------------------
diff --git a/app-packages/kafka/package/scripts/util.py b/app-packages/kafka/package/scripts/util.py
new file mode 100644
index 0000000..ca4cb41
--- /dev/null
+++ b/app-packages/kafka/package/scripts/util.py
@@ -0,0 +1,16 @@
+import re
+from os import fsync
+
+def updating(filename,dico):
+
+ RE = '(('+'|'.join(dico.keys())+')\s*=)[^\r\n]*?(\r?\n|\r)'
+ pat = re.compile(RE)
+
+ def jojo(mat,dic = dico):
+ return str(dic[mat.group(2)]).join(mat.group(1,3))
+
+ with open(filename,'rb') as f:
+ content = f.read()
+
+ with open(filename,'wb') as f:
+ f.write(pat.sub(jojo,content))
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/65e229e9/app-packages/kafka/pom.xml
----------------------------------------------------------------------
diff --git a/app-packages/kafka/pom.xml b/app-packages/kafka/pom.xml
new file mode 100644
index 0000000..267eb04
--- /dev/null
+++ b/app-packages/kafka/pom.xml
@@ -0,0 +1,69 @@
+<?xml version="1.0"?>
+<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
+ <!-- Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. See the NOTICE file distributed with this work for additional information regarding copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. -->
+
+
+ <modelVersion>4.0.0</modelVersion>
+ <groupId>com.datatorrent</groupId>
+ <version>0.1</version>
+ <artifactId>koya-slider-package</artifactId>
+ <packaging>jar</packaging>
+ <name>koya-slider-package</name>
+ <description>A package of koya on slider</description>
+
+ <properties>
+ <slider.version>0.90.0-incubating-SNAPSHOT</slider.version>
+ <kafka.src>${user.home}/kafka_2.10-0.8.2.1.tgz</kafka.src>
+ <kafka.version>kafka_2.10-0.8.2.1</kafka.version>
+ </properties>
+
+ <build>
+ <plugins>
+ <plugin>
+ <groupId>org.apache.maven.plugins</groupId>
+ <artifactId>maven-assembly-plugin</artifactId>
+ <version>2.5.4</version>
+ <executions>
+ <execution>
+ <id>app-package</id>
+ <phase>package</phase>
+ <goals>
+ <goal>single</goal>
+ </goals>
+ <configuration>
+ <descriptor>src/assembly/koya-app-package.xml</descriptor>
+ <appendAssemblyId>false</appendAssemblyId>
+ <finalName>${app.package.name}</finalName>
+ </configuration>
+ </execution>
+ <execution>
+ <id>package-with-slider</id>
+ <phase>package</phase>
+ <goals>
+ <goal>single</goal>
+ </goals>
+ <configuration>
+ <descriptor>src/assembly/koya-with-slider.xml</descriptor>
+ <appendAssemblyId>false</appendAssemblyId>
+ <finalName>koya-with-slider</finalName>
+ <attach>false</attach>
+ </configuration>
+ </execution>
+
+ </executions>
+ </plugin>
+ </plugins>
+
+ </build>
+
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.slider</groupId>
+ <artifactId>slider-core</artifactId>
+ <version>${slider.version}</version>
+ </dependency>
+ </dependencies>
+
+
+</project>
+
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/65e229e9/app-packages/kafka/resources-default.json
----------------------------------------------------------------------
diff --git a/app-packages/kafka/resources-default.json b/app-packages/kafka/resources-default.json
new file mode 100644
index 0000000..b320650
--- /dev/null
+++ b/app-packages/kafka/resources-default.json
@@ -0,0 +1,20 @@
+{
+ "schema" : "http://example.org/specification/v2.0.0",
+ "metadata" : {
+ },
+ "global" : {
+ "yarn.container.failure.threshold":"10",
+ "yarn.container.failure.window.hours":"1"
+ },
+ "components" : {
+ "KAFKA_BROKER" : {
+ "yarn.role.priority" : "1",
+ "yarn.component.instances" : "5",
+ "yarn.memory" : "768",
+ "yarn.vcores" : "1",
+ "yarn.component.placement.policy":"1"
+ },
+ "slider-appmaster" : {
+ }
+ }
+}
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/65e229e9/app-packages/kafka/src/assembly/koya-app-package.xml
----------------------------------------------------------------------
diff --git a/app-packages/kafka/src/assembly/koya-app-package.xml b/app-packages/kafka/src/assembly/koya-app-package.xml
new file mode 100644
index 0000000..dd8bd5c
--- /dev/null
+++ b/app-packages/kafka/src/assembly/koya-app-package.xml
@@ -0,0 +1,80 @@
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing, software
+ ~ distributed under the License is distributed on an "AS IS" BASIS,
+ ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ ~ See the License for the specific language governing permissions and
+ ~ limitations under the License.
+ -->
+
+
+<assembly
+ xmlns="http://maven.apache.org/plugins/maven-assembly-plugin/assembly/1.1.0"
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+ xsi:schemaLocation="http://maven.apache.org/plugins/maven-assembly-plugin/assembly/1.1.0 http://maven.apache.org/xsd/assembly-1.1.0.xsd">
+ <id>koya</id>
+ <formats>
+ <format>zip</format>
+ </formats>
+ <includeBaseDirectory>false</includeBaseDirectory>
+
+ <files>
+ <file>
+ <source>appConfig-default.json</source>
+ <destName>appConfig.json</destName>
+ <outputDirectory>/</outputDirectory>
+ <filtered>true</filtered>
+ <fileMode>0755</fileMode>
+ </file>
+ <file>
+ <source>resources-default.json</source>
+ <destName>resources.json</destName>
+ <outputDirectory>/</outputDirectory>
+ <filtered>true</filtered>
+ <fileMode>0755</fileMode>
+ </file>
+ <file>
+ <source>metainfo-default.xml</source>
+ <destName>metainfo.xml</destName>
+ <outputDirectory>/</outputDirectory>
+ <filtered>true</filtered>
+ <fileMode>0755</fileMode>
+ </file>
+ <file>
+ <source>${kafka.src}</source>
+ <destName>${kafka.version}.tgz</destName>
+ <outputDirectory>/package/files/</outputDirectory>
+ <fileMode>0755</fileMode>
+ </file>
+ </files>
+
+ <fileSets>
+ <fileSet>
+ <directory>${project.basedir}</directory>
+ <outputDirectory>/</outputDirectory>
+ <excludes>
+ <exclude>pom.xml</exclude>
+ <exclude>src/**</exclude>
+ <exclude>target/**</exclude>
+ <exclude>package/files/**</exclude>
+ <exclude>appConfig-default.json</exclude>
+ <exclude>resources-default.json</exclude>
+ <exclude>metainfo-default.xml</exclude>
+ <exclude>.*</exclude>
+ <exclude>.*/**</exclude>
+ </excludes>
+ <fileMode>0755</fileMode>
+ <directoryMode>0755</directoryMode>
+ </fileSet>
+
+ </fileSets>
+</assembly>
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/65e229e9/app-packages/kafka/src/assembly/koya-with-slider.xml
----------------------------------------------------------------------
diff --git a/app-packages/kafka/src/assembly/koya-with-slider.xml b/app-packages/kafka/src/assembly/koya-with-slider.xml
new file mode 100644
index 0000000..0f64927
--- /dev/null
+++ b/app-packages/kafka/src/assembly/koya-with-slider.xml
@@ -0,0 +1,64 @@
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing, software
+ ~ distributed under the License is distributed on an "AS IS" BASIS,
+ ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ ~ See the License for the specific language governing permissions and
+ ~ limitations under the License.
+ -->
+
+
+<assembly
+ xmlns="http://maven.apache.org/plugins/maven-assembly-plugin/assembly/1.1.0"
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+ xsi:schemaLocation="http://maven.apache.org/plugins/maven-assembly-plugin/assembly/1.1.0 http://maven.apache.org/xsd/assembly-1.1.0.xsd">
+ <id>koya</id>
+ <formats>
+ <format>zip</format>
+ </formats>
+ <includeBaseDirectory>false</includeBaseDirectory>
+
+ <files>
+ <file>
+ <source>target/koya-slider-package-${project.version}.zip</source>
+ <outputDirectory>/</outputDirectory>
+ <fileMode>0755</fileMode>
+ </file>
+ <file>
+ <source>appConfig-default.json</source>
+ <outputDirectory>/</outputDirectory>
+ <filtered>true</filtered>
+ <fileMode>0755</fileMode>
+ </file>
+ <file>
+ <source>resources-default.json</source>
+ <outputDirectory>/</outputDirectory>
+ <filtered>true</filtered>
+ <fileMode>0755</fileMode>
+ </file>
+ <file>
+ <source>metainfo-default.xml</source>
+ <outputDirectory>/</outputDirectory>
+ <filtered>true</filtered>
+ <fileMode>0755</fileMode>
+ </file>
+ </files>
+
+ <fileSets>
+ <fileSet>
+ <directory>../../../../slider-assembly/target/slider-${slider.version}-all</directory>
+ <outputDirectory>/</outputDirectory>
+ <fileMode>0755</fileMode>
+ <directoryMode>0755</directoryMode>
+ </fileSet>
+ </fileSets>
+</assembly>