You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@slider.apache.org by el...@apache.org on 2016/01/20 05:26:08 UTC
[1/5] incubator-slider git commit: SLIDER-977 Kafka on YARN (KOYA)
app-package.
Repository: incubator-slider
Updated Branches:
refs/heads/develop e1c802aab -> 6379d1b21
SLIDER-977 Kafka on YARN (KOYA) app-package.
Closes apache/incubator-slider#3
Project: http://git-wip-us.apache.org/repos/asf/incubator-slider/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-slider/commit/0ce5b417
Tree: http://git-wip-us.apache.org/repos/asf/incubator-slider/tree/0ce5b417
Diff: http://git-wip-us.apache.org/repos/asf/incubator-slider/diff/0ce5b417
Branch: refs/heads/develop
Commit: 0ce5b4178b3bc841e928eb35e016916ab3486d34
Parents: e1c802a
Author: Josh Elser <el...@apache.org>
Authored: Tue Jan 19 22:26:01 2016 -0500
Committer: Josh Elser <el...@apache.org>
Committed: Tue Jan 19 23:17:48 2016 -0500
----------------------------------------------------------------------
app-packages/kafka/README.md | 110 +++++++++++++++++++
app-packages/kafka/appConfig-default.json | 30 +++++
app-packages/kafka/build.sh | 5 +
app-packages/kafka/configuration/server.xml | 16 +++
app-packages/kafka/deploy.sh | 11 ++
app-packages/kafka/flex.sh | 5 +
app-packages/kafka/metainfo-default.xml | 56 ++++++++++
app-packages/kafka/package/scripts/kafka.py | 96 ++++++++++++++++
app-packages/kafka/package/scripts/params.py | 18 +++
.../kafka/package/scripts/status_params.py | 9 ++
app-packages/kafka/package/scripts/util.py | 16 +++
app-packages/kafka/pom.xml | 69 ++++++++++++
app-packages/kafka/resources-default.json | 20 ++++
.../kafka/src/assembly/koya-app-package.xml | 80 ++++++++++++++
.../kafka/src/assembly/koya-with-slider.xml | 64 +++++++++++
15 files changed, 605 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/0ce5b417/app-packages/kafka/README.md
----------------------------------------------------------------------
diff --git a/app-packages/kafka/README.md b/app-packages/kafka/README.md
new file mode 100644
index 0000000..97ddef4
--- /dev/null
+++ b/app-packages/kafka/README.md
@@ -0,0 +1,110 @@
+Kafka On YARN (KOYA)
+====================
+
+[](https://gitter.im/DataTorrent/koya?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge)
+
+### Goals
+
+ * Use capabilities of YARN for Kafka broker management
+ * Automate broker recovery
+ * Make it easy to deploy, configure and monitor Kafka clusters
+ * Simplify management tasks (alternative to Kafka command line utilities)
+
+JIRA: https://issues.apache.org/jira/browse/KAFKA-1754
+
+Kafka as YARN application using Slider
+-----------------------------------------------
+
+### Build
+
+Checkout Slider code (https://github.com/apache/incubator-slider)
+```sh
+git clone git@github.com:apache/incubator-slider.git
+git checkout -b slider-0.80.0-incubating remotes/origin/releases/slider-0.80.0-incubating
+```
+The Slider version you checked out needs to match ${slider.version} in pom.xml
+
+Create symbolic link to Slider source code within the KOYA repository:
+```sh
+ln -s /path/to/repo/incubator-slider/ slider
+```
+Download Kafka binary package (http://kafka.apache.org/downloads.html)
+
+```sh
+mvn clean install -DskipTests -Dkafka.src=path/to/kafka_2.10-0.8.1.1.tgz -Dkafka.version=kafka_2.10-0.8.1.1
+```
+Artifacts:
+
+ - Archive with embedded Slider: __`target/koya-with-slider.zip`__
+ - Separate Slider application package: __`target/koya-slider-package-0.1.zip`__
+
+###Installation
+
+####Install Slider
+
+To use the archive with embedded Slider, copy it to the machine from which you launch YARN applications (Hadoop client, gateway or edge node). Extract the file and configure Slider:
+
+If the environment variables `HADOOP_CONF_DIR` or `JAVA_HOME` are not already defined through your Hadoop installation, you can export them in `slider-0.80.0-incubating/conf/slider-env.sh`
+
+Example for CDH 5.4:
+```
+export HADOOP_CONF_DIR=/etc/hadoop/conf
+export JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera
+```
+If the registry ZooKeeper quorum was not already configured through Hadoop, modify `slider-0.80.0-incubating/conf/slider-client.xml`:
+```
+ <property>
+ <name>hadoop.registry.zk.quorum</name>
+ <value>node26:2181,node27:2181,node28:2181</value>
+ </property>
+```
+Above steps are not required with HDP 2.2
+
+More information regarding Slider client configuration refer to http://slider.incubator.apache.org/docs/client-configuration.html
+
+### Configure KOYA application package
+
+Before the Kafka cluster can be launched, the brokers need to be defined. Currently Slider does not support [configuration properties at instance level](https://issues.apache.org/jira/browse/SLIDER-851), therefore each broker has to be configured as a component.
+
+If you use the full archive, the configuration file templates are already in your working directory. Otherwise extract them from the Slider package.
+
+####appConfig.json
+
+Extract the packaged configuration files you are going to customize:
+```
+unzip koya-slider-package-0.1.zip appConfig.json resources.json
+```
+Adjust following properties in the global section:
+```
+ "application.def": "koya-slider-package-0.1.zip",
+ "site.global.xmx_val": "256m",
+ "site.global.xms_val": "128m",
+ "site.broker.zookeeper.connect": "${ZK_HOST}"
+```
+Above will be used to configure server.properties and launch the Kafka server(s). All properties prefixed with `site.broker.` will be set in the server.properties file supplied with the Kafka archive. Only non-default settings need to be defined here.
+
+####resources.json
+
+Configure the number of servers and other resource requirements:
+```
+ "components" : {
+ "broker" : {
+ "yarn.role.priority" : "1",
+ "yarn.component.instances" : "5",
+ "yarn.memory" : "768",
+ "yarn.vcores" : "1",
+ "yarn.component.placement.policy":"1"
+ }
+```
+More information about the application configuration can be found [here](http://slider.incubator.apache.org/docs/configuration/core.html).
+
+### Deploy KOYA Cluster
+
+The Slider application package needs to be copied to the HDFS location that was specified as application.def in appConfig.json:
+```
+hdfs dfs -copyFromLocal koya-slider-package-0.1.zip /path/in/appConfig
+```
+Now the KOYA cluster can be deployed and launched:
+```
+slider-0.80.0-incubating/bin/slider create koya --template ~/koya/appConfig.json --resources ~/koya/resources.json
+```
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/0ce5b417/app-packages/kafka/appConfig-default.json
----------------------------------------------------------------------
diff --git a/app-packages/kafka/appConfig-default.json b/app-packages/kafka/appConfig-default.json
new file mode 100644
index 0000000..9a0be4c
--- /dev/null
+++ b/app-packages/kafka/appConfig-default.json
@@ -0,0 +1,30 @@
+{
+ "schema": "http://example.org/specification/v2.0.0",
+ "metadata": {
+ },
+ "global": {
+ "application.def": ".slider/package/KOYA/koya-slider-package-${project.version}.zip",
+ "java_home": "${JAVA_HOME}",
+ "system_configs": "broker",
+
+ "site.global.app_user": "${USER_NAME}",
+ "site.global.app_root": "${AGENT_WORK_ROOT}/app/install/${kafka.version}",
+ "site.global.app_install_dir": "${AGENT_WORK_ROOT}/app/install",
+ "site.global.pid_file": "${AGENT_WORK_ROOT}/app/run/koya.pid",
+
+ "site.global.kafka_version": "${kafka.version}",
+ "site.broker.xmx_val": "512m",
+ "site.broker.xms_val": "128m",
+ "site.broker.instance.name": "${USER}/${CLUSTER_NAME}",
+ "site.broker.zookeeper": "${ZK_HOST}",
+
+ "site.server.port": "${KAFKA_BROKER.ALLOCATED_PORT}{PER_CONTAINER}"
+ },
+ "components": {
+ "broker": {
+ },
+ "slider-appmaster": {
+ "jvm.heapsize": "256M"
+ }
+ }
+}
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/0ce5b417/app-packages/kafka/build.sh
----------------------------------------------------------------------
diff --git a/app-packages/kafka/build.sh b/app-packages/kafka/build.sh
new file mode 100755
index 0000000..7951f9e
--- /dev/null
+++ b/app-packages/kafka/build.sh
@@ -0,0 +1,5 @@
+#! /bin/bash
+
+wget http://apache.websitebeheerjd.nl/kafka/0.8.2.1/kafka_2.10-0.8.2.1.tgz -O $HOME/kafka_2.10-0.8.2.1.tgz
+mvn clean install -DskipTests -Dkafka.src=$HOME/kafka_2.10-0.8.2.1.tgz -Dkafka.version=kafka_2.10-0.8.2.1
+unzip -o target/koya-slider-package-0.1.zip appConfig.json
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/0ce5b417/app-packages/kafka/configuration/server.xml
----------------------------------------------------------------------
diff --git a/app-packages/kafka/configuration/server.xml b/app-packages/kafka/configuration/server.xml
new file mode 100644
index 0000000..68b36c9
--- /dev/null
+++ b/app-packages/kafka/configuration/server.xml
@@ -0,0 +1,16 @@
+<?xml version="1.0"?>
+<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
+<configuration>
+ <property>
+ <name>zookeeper.connect</name>
+ <value>${@//site/broker/zookeeper}/kafka/${@//site/broker/instance.name}</value>
+ </property>
+ <property>
+ <name>broker.id</name>
+ <value>${@//site/global/app_container_tag}</value>
+ </property>
+ <property>
+ <name>log.dirs</name>
+ <value>${@//site/global/app_pid_dir}</value>
+ </property>
+</configuration>
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/0ce5b417/app-packages/kafka/deploy.sh
----------------------------------------------------------------------
diff --git a/app-packages/kafka/deploy.sh b/app-packages/kafka/deploy.sh
new file mode 100755
index 0000000..1f8f473
--- /dev/null
+++ b/app-packages/kafka/deploy.sh
@@ -0,0 +1,11 @@
+#! /bin/bash
+
+APPNAME=${1:-koya}
+slider install-package --replacepkg --name KOYA --package target/koya-slider-package-0.1.zip
+slider stop $APPNAME
+slider destroy $APPNAME
+slider create $APPNAME --filesystem hdfs://root --queue dev --template appConfig.json --resources resources-default.json
+
+
+
+
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/0ce5b417/app-packages/kafka/flex.sh
----------------------------------------------------------------------
diff --git a/app-packages/kafka/flex.sh b/app-packages/kafka/flex.sh
new file mode 100755
index 0000000..20a1003
--- /dev/null
+++ b/app-packages/kafka/flex.sh
@@ -0,0 +1,5 @@
+#! /bin/bash
+
+APPNAME=${1:-koya}
+APPNUM=${2:-12}
+slider flex $APPNAME --component BROKER0 $APPNUM --filesystem hdfs://root
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/0ce5b417/app-packages/kafka/metainfo-default.xml
----------------------------------------------------------------------
diff --git a/app-packages/kafka/metainfo-default.xml b/app-packages/kafka/metainfo-default.xml
new file mode 100644
index 0000000..f1eec0a
--- /dev/null
+++ b/app-packages/kafka/metainfo-default.xml
@@ -0,0 +1,56 @@
+<metainfo>
+ <schemaVersion>2.0</schemaVersion>
+ <application>
+ <name>KAFKA</name>
+ <comment>Scalable High-thoughput Distributed Message Queue</comment>
+ <version>${kafka.version}</version>
+ <exportedConfigs>None</exportedConfigs>
+
+ <exportGroups>
+ <exportGroup>
+ <name>servers</name>
+ <exports>
+ <export>
+ <name>org.apache.kafka.broker</name>
+ <value>${KAFKA_BROKER_HOST}:${site.server.port}</value>
+ </export>
+ </exports>
+ </exportGroup>
+ </exportGroups>
+
+ <components>
+ <component>
+ <name>KAFKA_BROKER</name>
+ <category>SLAVE</category>
+ <minInstanceCount>1</minInstanceCount>
+ <compExports>servers-org.apache.kafka.broker</compExports>
+ <exportedConfigs>broker</exportedConfigs>
+ <commandScript>
+ <script>scripts/kafka.py</script>
+ <scriptType>PYTHON</scriptType>
+ </commandScript>
+ </component>
+ </components>
+
+ <configFiles>
+ <configFile>
+ <type>xml</type>
+ <fileName>server.xml</fileName>
+ <dictionaryName>server</dictionaryName>
+ </configFile>
+ </configFiles>
+
+ <osSpecifics>
+ <osSpecific>
+ <osType>any</osType>
+ <packages>
+ <package>
+ <type>tarball</type>
+ <name>files/${kafka.version}.tgz</name>
+ </package>
+ </packages>
+ </osSpecific>
+ </osSpecifics>
+
+ </application>
+</metainfo>
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/0ce5b417/app-packages/kafka/package/scripts/kafka.py
----------------------------------------------------------------------
diff --git a/app-packages/kafka/package/scripts/kafka.py b/app-packages/kafka/package/scripts/kafka.py
new file mode 100644
index 0000000..38b12a3
--- /dev/null
+++ b/app-packages/kafka/package/scripts/kafka.py
@@ -0,0 +1,96 @@
+import logging
+import sys
+import os
+import inspect
+import pprint
+import util
+from resource_management import *
+
+logger = logging.getLogger()
+
+class Kafka(Script):
+ def install(self, env):
+ self.install_packages(env)
+
+ def configure(self, env):
+ import params
+ env.set_params(params)
+
+ def start(self, env):
+ import params
+ env.set_params(params)
+ self.configure(env)
+
+ # log the component configuration
+ ppp = pprint.PrettyPrinter(indent=4)
+ logger.info("broker component config: " + ppp.pformat(params.broker_config))
+
+ # log the environment variables
+ logger.info("Env Variables:")
+ for key in os.environ.keys():
+ logger.info("%10s %s \n" % (key,os.environ[key]))
+ pass
+
+ # This updating thing is changing files in-place and it really
+ # should not (static cache)
+
+ # For kafka 0.8.1.1, there is no way to set the log dir to location other than params.app_root + "/logs"
+ if(params.kafka_version.find("0.8.1.1") != -1):
+ os.symlink(params.app_root + "/logs", params.app_log_dir + "/kafka")
+ else:
+ kafkaLogConfig = {"kafka.logs.dir" : params.app_log_dir + "/kafka"}
+ util.updating(params.app_root + "/config/log4j.properties", kafkaLogConfig)
+# File(format("{params.app_root}/conf/log4j.properties"),
+# owner=params.app_user,
+# content=InlineTemplate(param.log4j_prop))
+ pass
+
+ # update the broker properties for different brokers
+ server_conf=format("{params.conf_dir}/server.slider.properties")
+ PropertiesFile(server_conf, properties = params.broker_config, owner=params.app_user)
+
+ # execute the process
+ process_cmd = format("{app_root}/bin/kafka-server-start.sh {server_conf}")
+ os.environ['LOG_DIR'] = params.app_log_dir + "/kafka"
+ HEAP_OPT = ""
+ if params.xmx:
+ HEAP_OPT = HEAP_OPT + " -Xmx" + params.xmx
+ pass
+ if params.xms:
+ HEAP_OPT = HEAP_OPT + " -Xms" + params.xms
+ pass
+ if HEAP_OPT:
+ os.environ['KAFKA_HEAP_OPTS'] = HEAP_OPT
+ pass
+ Execute(process_cmd,
+ user=params.app_user,
+ logoutput=True,
+ wait_for_finish=False,
+ pid_file=params.pid_file
+ )
+
+ def stop(self, env):
+ import params
+ env.set_params(params)
+ pid = format("`cat {pid_file}` >/dev/null 2>&1")
+ Execute(format("kill {pid}"),
+ user=params.app_user
+ )
+ Execute(format("kill -9 {pid}"),
+ ignore_failures=True,
+ user=params.app_user
+ )
+ Execute(format("rm -f {pid_file}"),
+ user=params.app_user)
+
+ def status(self, env):
+ import status_params
+ env.set_params(status_params)
+## jps_cmd = format("{java64_home}/bin/jps")
+ no_op_test = format("ls {pid_file} >/dev/null 2>&1 && ps `cat {pid_file}` >/dev/null 2>&1")
+## cmd = format("echo `{jps_cmd} | grep Kafka | cut -d' ' -f1` > {pid_file}")
+## Execute(cmd, not_if=no_op_test)
+ check_process_status(status_params.pid_file)
+
+if __name__ == "__main__":
+ Kafka().execute()
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/0ce5b417/app-packages/kafka/package/scripts/params.py
----------------------------------------------------------------------
diff --git a/app-packages/kafka/package/scripts/params.py b/app-packages/kafka/package/scripts/params.py
new file mode 100644
index 0000000..de3994e
--- /dev/null
+++ b/app-packages/kafka/package/scripts/params.py
@@ -0,0 +1,18 @@
+from resource_management import *
+
+config = Script.get_config()
+
+
+app_root = config['configurations']['global']['app_root']
+java64_home = config['hostLevelParams']['java_home']
+app_user = config['configurations']['global']['app_user']
+pid_file = config['configurations']['global']['pid_file']
+app_log_dir = config['configurations']['global']['app_log_dir']
+kafka_version = config['configurations']['global']['kafka_version']
+xmx = config['configurations']['broker']['xmx_val']
+xms = config['configurations']['broker']['xms_val']
+
+conf_dir = format("{app_root}/config")
+
+broker_config=dict(line.strip().split('=') for line in open(format("{conf_dir}/server.properties")) if not (line.startswith('#') or re.match(r'^\s*$', line)))
+broker_config.update(config['configurations']['server'])
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/0ce5b417/app-packages/kafka/package/scripts/status_params.py
----------------------------------------------------------------------
diff --git a/app-packages/kafka/package/scripts/status_params.py b/app-packages/kafka/package/scripts/status_params.py
new file mode 100644
index 0000000..cc46d5b
--- /dev/null
+++ b/app-packages/kafka/package/scripts/status_params.py
@@ -0,0 +1,9 @@
+from resource_management import *
+
+config = Script.get_config()
+
+app_root = config['configurations']['global']['app_root']
+java64_home = config['hostLevelParams']['java_home']
+app_user = config['configurations']['global']['app_user']
+pid_file = config['configurations']['global']['pid_file']
+
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/0ce5b417/app-packages/kafka/package/scripts/util.py
----------------------------------------------------------------------
diff --git a/app-packages/kafka/package/scripts/util.py b/app-packages/kafka/package/scripts/util.py
new file mode 100644
index 0000000..ca4cb41
--- /dev/null
+++ b/app-packages/kafka/package/scripts/util.py
@@ -0,0 +1,16 @@
+import re
+from os import fsync
+
+def updating(filename,dico):
+
+ RE = '(('+'|'.join(dico.keys())+')\s*=)[^\r\n]*?(\r?\n|\r)'
+ pat = re.compile(RE)
+
+ def jojo(mat,dic = dico):
+ return str(dic[mat.group(2)]).join(mat.group(1,3))
+
+ with open(filename,'rb') as f:
+ content = f.read()
+
+ with open(filename,'wb') as f:
+ f.write(pat.sub(jojo,content))
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/0ce5b417/app-packages/kafka/pom.xml
----------------------------------------------------------------------
diff --git a/app-packages/kafka/pom.xml b/app-packages/kafka/pom.xml
new file mode 100644
index 0000000..267eb04
--- /dev/null
+++ b/app-packages/kafka/pom.xml
@@ -0,0 +1,69 @@
+<?xml version="1.0"?>
+<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
+ <!-- Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. See the NOTICE file distributed with this work for additional information regarding copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. -->
+
+
+ <modelVersion>4.0.0</modelVersion>
+ <groupId>com.datatorrent</groupId>
+ <version>0.1</version>
+ <artifactId>koya-slider-package</artifactId>
+ <packaging>jar</packaging>
+ <name>koya-slider-package</name>
+ <description>A package of koya on slider</description>
+
+ <properties>
+ <slider.version>0.90.0-incubating-SNAPSHOT</slider.version>
+ <kafka.src>${user.home}/kafka_2.10-0.8.2.1.tgz</kafka.src>
+ <kafka.version>kafka_2.10-0.8.2.1</kafka.version>
+ </properties>
+
+ <build>
+ <plugins>
+ <plugin>
+ <groupId>org.apache.maven.plugins</groupId>
+ <artifactId>maven-assembly-plugin</artifactId>
+ <version>2.5.4</version>
+ <executions>
+ <execution>
+ <id>app-package</id>
+ <phase>package</phase>
+ <goals>
+ <goal>single</goal>
+ </goals>
+ <configuration>
+ <descriptor>src/assembly/koya-app-package.xml</descriptor>
+ <appendAssemblyId>false</appendAssemblyId>
+ <finalName>${app.package.name}</finalName>
+ </configuration>
+ </execution>
+ <execution>
+ <id>package-with-slider</id>
+ <phase>package</phase>
+ <goals>
+ <goal>single</goal>
+ </goals>
+ <configuration>
+ <descriptor>src/assembly/koya-with-slider.xml</descriptor>
+ <appendAssemblyId>false</appendAssemblyId>
+ <finalName>koya-with-slider</finalName>
+ <attach>false</attach>
+ </configuration>
+ </execution>
+
+ </executions>
+ </plugin>
+ </plugins>
+
+ </build>
+
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.slider</groupId>
+ <artifactId>slider-core</artifactId>
+ <version>${slider.version}</version>
+ </dependency>
+ </dependencies>
+
+
+</project>
+
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/0ce5b417/app-packages/kafka/resources-default.json
----------------------------------------------------------------------
diff --git a/app-packages/kafka/resources-default.json b/app-packages/kafka/resources-default.json
new file mode 100644
index 0000000..b320650
--- /dev/null
+++ b/app-packages/kafka/resources-default.json
@@ -0,0 +1,20 @@
+{
+ "schema" : "http://example.org/specification/v2.0.0",
+ "metadata" : {
+ },
+ "global" : {
+ "yarn.container.failure.threshold":"10",
+ "yarn.container.failure.window.hours":"1"
+ },
+ "components" : {
+ "KAFKA_BROKER" : {
+ "yarn.role.priority" : "1",
+ "yarn.component.instances" : "5",
+ "yarn.memory" : "768",
+ "yarn.vcores" : "1",
+ "yarn.component.placement.policy":"1"
+ },
+ "slider-appmaster" : {
+ }
+ }
+}
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/0ce5b417/app-packages/kafka/src/assembly/koya-app-package.xml
----------------------------------------------------------------------
diff --git a/app-packages/kafka/src/assembly/koya-app-package.xml b/app-packages/kafka/src/assembly/koya-app-package.xml
new file mode 100644
index 0000000..dd8bd5c
--- /dev/null
+++ b/app-packages/kafka/src/assembly/koya-app-package.xml
@@ -0,0 +1,80 @@
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing, software
+ ~ distributed under the License is distributed on an "AS IS" BASIS,
+ ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ ~ See the License for the specific language governing permissions and
+ ~ limitations under the License.
+ -->
+
+
+<assembly
+ xmlns="http://maven.apache.org/plugins/maven-assembly-plugin/assembly/1.1.0"
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+ xsi:schemaLocation="http://maven.apache.org/plugins/maven-assembly-plugin/assembly/1.1.0 http://maven.apache.org/xsd/assembly-1.1.0.xsd">
+ <id>koya</id>
+ <formats>
+ <format>zip</format>
+ </formats>
+ <includeBaseDirectory>false</includeBaseDirectory>
+
+ <files>
+ <file>
+ <source>appConfig-default.json</source>
+ <destName>appConfig.json</destName>
+ <outputDirectory>/</outputDirectory>
+ <filtered>true</filtered>
+ <fileMode>0755</fileMode>
+ </file>
+ <file>
+ <source>resources-default.json</source>
+ <destName>resources.json</destName>
+ <outputDirectory>/</outputDirectory>
+ <filtered>true</filtered>
+ <fileMode>0755</fileMode>
+ </file>
+ <file>
+ <source>metainfo-default.xml</source>
+ <destName>metainfo.xml</destName>
+ <outputDirectory>/</outputDirectory>
+ <filtered>true</filtered>
+ <fileMode>0755</fileMode>
+ </file>
+ <file>
+ <source>${kafka.src}</source>
+ <destName>${kafka.version}.tgz</destName>
+ <outputDirectory>/package/files/</outputDirectory>
+ <fileMode>0755</fileMode>
+ </file>
+ </files>
+
+ <fileSets>
+ <fileSet>
+ <directory>${project.basedir}</directory>
+ <outputDirectory>/</outputDirectory>
+ <excludes>
+ <exclude>pom.xml</exclude>
+ <exclude>src/**</exclude>
+ <exclude>target/**</exclude>
+ <exclude>package/files/**</exclude>
+ <exclude>appConfig-default.json</exclude>
+ <exclude>resources-default.json</exclude>
+ <exclude>metainfo-default.xml</exclude>
+ <exclude>.*</exclude>
+ <exclude>.*/**</exclude>
+ </excludes>
+ <fileMode>0755</fileMode>
+ <directoryMode>0755</directoryMode>
+ </fileSet>
+
+ </fileSets>
+</assembly>
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/0ce5b417/app-packages/kafka/src/assembly/koya-with-slider.xml
----------------------------------------------------------------------
diff --git a/app-packages/kafka/src/assembly/koya-with-slider.xml b/app-packages/kafka/src/assembly/koya-with-slider.xml
new file mode 100644
index 0000000..0f64927
--- /dev/null
+++ b/app-packages/kafka/src/assembly/koya-with-slider.xml
@@ -0,0 +1,64 @@
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing, software
+ ~ distributed under the License is distributed on an "AS IS" BASIS,
+ ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ ~ See the License for the specific language governing permissions and
+ ~ limitations under the License.
+ -->
+
+
+<assembly
+ xmlns="http://maven.apache.org/plugins/maven-assembly-plugin/assembly/1.1.0"
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+ xsi:schemaLocation="http://maven.apache.org/plugins/maven-assembly-plugin/assembly/1.1.0 http://maven.apache.org/xsd/assembly-1.1.0.xsd">
+ <id>koya</id>
+ <formats>
+ <format>zip</format>
+ </formats>
+ <includeBaseDirectory>false</includeBaseDirectory>
+
+ <files>
+ <file>
+ <source>target/koya-slider-package-${project.version}.zip</source>
+ <outputDirectory>/</outputDirectory>
+ <fileMode>0755</fileMode>
+ </file>
+ <file>
+ <source>appConfig-default.json</source>
+ <outputDirectory>/</outputDirectory>
+ <filtered>true</filtered>
+ <fileMode>0755</fileMode>
+ </file>
+ <file>
+ <source>resources-default.json</source>
+ <outputDirectory>/</outputDirectory>
+ <filtered>true</filtered>
+ <fileMode>0755</fileMode>
+ </file>
+ <file>
+ <source>metainfo-default.xml</source>
+ <outputDirectory>/</outputDirectory>
+ <filtered>true</filtered>
+ <fileMode>0755</fileMode>
+ </file>
+ </files>
+
+ <fileSets>
+ <fileSet>
+ <directory>../../../../slider-assembly/target/slider-${slider.version}-all</directory>
+ <outputDirectory>/</outputDirectory>
+ <fileMode>0755</fileMode>
+ <directoryMode>0755</directoryMode>
+ </fileSet>
+ </fileSets>
+</assembly>
[2/5] incubator-slider git commit: SLIDER-977 Add ASL headers to all
files.
Posted by el...@apache.org.
SLIDER-977 Add ASL headers to all files.
Of extreme importance, the license headers on app-packages/kafka/src/assembly/koya-app-package.xml
and app-packages/kafka/src/assembly/koya-with-slider.xml were modified, not in content, but only
formatting. These text of the license headers on these files was not changed.
Project: http://git-wip-us.apache.org/repos/asf/incubator-slider/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-slider/commit/835bac85
Tree: http://git-wip-us.apache.org/repos/asf/incubator-slider/tree/835bac85
Diff: http://git-wip-us.apache.org/repos/asf/incubator-slider/diff/835bac85
Branch: refs/heads/develop
Commit: 835bac8568999dba4a8e15bef8613f34bbabf6e8
Parents: 0ce5b41
Author: Josh Elser <el...@apache.org>
Authored: Tue Jan 19 22:41:27 2016 -0500
Committer: Josh Elser <el...@apache.org>
Committed: Tue Jan 19 23:21:26 2016 -0500
----------------------------------------------------------------------
app-packages/kafka/README.md | 29 +++++++++++----
app-packages/kafka/build.sh | 16 +++++++++
app-packages/kafka/configuration/server.xml | 16 +++++++++
app-packages/kafka/deploy.sh | 19 +++++++---
app-packages/kafka/flex.sh | 15 ++++++++
app-packages/kafka/metainfo-default.xml | 17 +++++++++
app-packages/kafka/package/scripts/kafka.py | 15 ++++++++
app-packages/kafka/package/scripts/params.py | 15 ++++++++
.../kafka/package/scripts/status_params.py | 16 ++++++++-
app-packages/kafka/package/scripts/util.py | 18 +++++++++-
app-packages/kafka/pom.xml | 38 ++++++++++++++------
.../kafka/src/assembly/koya-app-package.xml | 30 ++++++++--------
.../kafka/src/assembly/koya-with-slider.xml | 30 ++++++++--------
13 files changed, 219 insertions(+), 55 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/835bac85/app-packages/kafka/README.md
----------------------------------------------------------------------
diff --git a/app-packages/kafka/README.md b/app-packages/kafka/README.md
index 97ddef4..34f0d04 100644
--- a/app-packages/kafka/README.md
+++ b/app-packages/kafka/README.md
@@ -1,3 +1,20 @@
+<!---
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
Kafka On YARN (KOYA)
====================
@@ -17,7 +34,7 @@ Kafka as YARN application using Slider
### Build
-Checkout Slider code (https://github.com/apache/incubator-slider)
+Checkout Slider code (https://github.com/apache/incubator-slider)
```sh
git clone git@github.com:apache/incubator-slider.git
git checkout -b slider-0.80.0-incubating remotes/origin/releases/slider-0.80.0-incubating
@@ -44,14 +61,14 @@ Artifacts:
To use the archive with embedded Slider, copy it to the machine from which you launch YARN applications (Hadoop client, gateway or edge node). Extract the file and configure Slider:
-If the environment variables `HADOOP_CONF_DIR` or `JAVA_HOME` are not already defined through your Hadoop installation, you can export them in `slider-0.80.0-incubating/conf/slider-env.sh`
+If the environment variables `HADOOP_CONF_DIR` or `JAVA_HOME` are not already defined through your Hadoop installation, you can export them in `slider-0.80.0-incubating/conf/slider-env.sh`
Example for CDH 5.4:
-```
+```
export HADOOP_CONF_DIR=/etc/hadoop/conf
export JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera
```
-If the registry ZooKeeper quorum was not already configured through Hadoop, modify `slider-0.80.0-incubating/conf/slider-client.xml`:
+If the registry ZooKeeper quorum was not already configured through Hadoop, modify `slider-0.80.0-incubating/conf/slider-client.xml`:
```
<property>
<name>hadoop.registry.zk.quorum</name>
@@ -72,9 +89,9 @@ If you use the full archive, the configuration file templates are already in you
Extract the packaged configuration files you are going to customize:
```
-unzip koya-slider-package-0.1.zip appConfig.json resources.json
+unzip koya-slider-package-0.1.zip appConfig.json resources.json
```
-Adjust following properties in the global section:
+Adjust following properties in the global section:
```
"application.def": "koya-slider-package-0.1.zip",
"site.global.xmx_val": "256m",
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/835bac85/app-packages/kafka/build.sh
----------------------------------------------------------------------
diff --git a/app-packages/kafka/build.sh b/app-packages/kafka/build.sh
index 7951f9e..35eeeba 100755
--- a/app-packages/kafka/build.sh
+++ b/app-packages/kafka/build.sh
@@ -1,5 +1,21 @@
#! /bin/bash
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
wget http://apache.websitebeheerjd.nl/kafka/0.8.2.1/kafka_2.10-0.8.2.1.tgz -O $HOME/kafka_2.10-0.8.2.1.tgz
mvn clean install -DskipTests -Dkafka.src=$HOME/kafka_2.10-0.8.2.1.tgz -Dkafka.version=kafka_2.10-0.8.2.1
unzip -o target/koya-slider-package-0.1.zip appConfig.json
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/835bac85/app-packages/kafka/configuration/server.xml
----------------------------------------------------------------------
diff --git a/app-packages/kafka/configuration/server.xml b/app-packages/kafka/configuration/server.xml
index 68b36c9..b22db45 100644
--- a/app-packages/kafka/configuration/server.xml
+++ b/app-packages/kafka/configuration/server.xml
@@ -1,5 +1,21 @@
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
<configuration>
<property>
<name>zookeeper.connect</name>
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/835bac85/app-packages/kafka/deploy.sh
----------------------------------------------------------------------
diff --git a/app-packages/kafka/deploy.sh b/app-packages/kafka/deploy.sh
index 1f8f473..9b69c90 100755
--- a/app-packages/kafka/deploy.sh
+++ b/app-packages/kafka/deploy.sh
@@ -1,11 +1,22 @@
#! /bin/bash
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
APPNAME=${1:-koya}
slider install-package --replacepkg --name KOYA --package target/koya-slider-package-0.1.zip
slider stop $APPNAME
slider destroy $APPNAME
slider create $APPNAME --filesystem hdfs://root --queue dev --template appConfig.json --resources resources-default.json
-
-
-
-
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/835bac85/app-packages/kafka/flex.sh
----------------------------------------------------------------------
diff --git a/app-packages/kafka/flex.sh b/app-packages/kafka/flex.sh
index 20a1003..36c769b 100755
--- a/app-packages/kafka/flex.sh
+++ b/app-packages/kafka/flex.sh
@@ -1,5 +1,20 @@
#! /bin/bash
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
APPNAME=${1:-koya}
APPNUM=${2:-12}
slider flex $APPNAME --component BROKER0 $APPNUM --filesystem hdfs://root
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/835bac85/app-packages/kafka/metainfo-default.xml
----------------------------------------------------------------------
diff --git a/app-packages/kafka/metainfo-default.xml b/app-packages/kafka/metainfo-default.xml
index f1eec0a..05dd1f6 100644
--- a/app-packages/kafka/metainfo-default.xml
+++ b/app-packages/kafka/metainfo-default.xml
@@ -1,3 +1,20 @@
+<?xml version="1.0"?>
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
<metainfo>
<schemaVersion>2.0</schemaVersion>
<application>
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/835bac85/app-packages/kafka/package/scripts/kafka.py
----------------------------------------------------------------------
diff --git a/app-packages/kafka/package/scripts/kafka.py b/app-packages/kafka/package/scripts/kafka.py
index 38b12a3..325c65c 100644
--- a/app-packages/kafka/package/scripts/kafka.py
+++ b/app-packages/kafka/package/scripts/kafka.py
@@ -1,3 +1,18 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
import logging
import sys
import os
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/835bac85/app-packages/kafka/package/scripts/params.py
----------------------------------------------------------------------
diff --git a/app-packages/kafka/package/scripts/params.py b/app-packages/kafka/package/scripts/params.py
index de3994e..1bc3f2f 100644
--- a/app-packages/kafka/package/scripts/params.py
+++ b/app-packages/kafka/package/scripts/params.py
@@ -1,3 +1,18 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
from resource_management import *
config = Script.get_config()
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/835bac85/app-packages/kafka/package/scripts/status_params.py
----------------------------------------------------------------------
diff --git a/app-packages/kafka/package/scripts/status_params.py b/app-packages/kafka/package/scripts/status_params.py
index cc46d5b..7ae91bd 100644
--- a/app-packages/kafka/package/scripts/status_params.py
+++ b/app-packages/kafka/package/scripts/status_params.py
@@ -1,3 +1,18 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
from resource_management import *
config = Script.get_config()
@@ -6,4 +21,3 @@ app_root = config['configurations']['global']['app_root']
java64_home = config['hostLevelParams']['java_home']
app_user = config['configurations']['global']['app_user']
pid_file = config['configurations']['global']['pid_file']
-
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/835bac85/app-packages/kafka/package/scripts/util.py
----------------------------------------------------------------------
diff --git a/app-packages/kafka/package/scripts/util.py b/app-packages/kafka/package/scripts/util.py
index ca4cb41..996a897 100644
--- a/app-packages/kafka/package/scripts/util.py
+++ b/app-packages/kafka/package/scripts/util.py
@@ -1,3 +1,19 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
import re
from os import fsync
@@ -10,7 +26,7 @@ def updating(filename,dico):
return str(dic[mat.group(2)]).join(mat.group(1,3))
with open(filename,'rb') as f:
- content = f.read()
+ content = f.read()
with open(filename,'wb') as f:
f.write(pat.sub(jojo,content))
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/835bac85/app-packages/kafka/pom.xml
----------------------------------------------------------------------
diff --git a/app-packages/kafka/pom.xml b/app-packages/kafka/pom.xml
index 267eb04..fbb9db8 100644
--- a/app-packages/kafka/pom.xml
+++ b/app-packages/kafka/pom.xml
@@ -1,15 +1,32 @@
<?xml version="1.0"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
- <!-- Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. See the NOTICE file distributed with this work for additional information regarding copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. -->
+ <!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+ <parent>
+ <groupId>org.apache.slider.packages</groupId>
+ <artifactId>slider-app-packages</artifactId>
+ <version>0.90.0-incubating-SNAPSHOT</version>
+ <relativePath>../pom.xml</relativePath>
+ </parent>
<modelVersion>4.0.0</modelVersion>
- <groupId>com.datatorrent</groupId>
- <version>0.1</version>
- <artifactId>koya-slider-package</artifactId>
+ <artifactId>kafka-app-package</artifactId>
<packaging>jar</packaging>
- <name>koya-slider-package</name>
- <description>A package of koya on slider</description>
+ <name>Slider Kafka App Package</name>
+ <description>Slider Kafka App Package</description>
<properties>
<slider.version>0.90.0-incubating-SNAPSHOT</slider.version>
@@ -22,7 +39,7 @@
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-assembly-plugin</artifactId>
- <version>2.5.4</version>
+ <version>>${maven-assembly-plugin.version}</version>
<executions>
<execution>
<id>app-package</id>
@@ -35,12 +52,12 @@
<appendAssemblyId>false</appendAssemblyId>
<finalName>${app.package.name}</finalName>
</configuration>
- </execution>
+ </execution>
<execution>
<id>package-with-slider</id>
- <phase>package</phase>
+ <phase>package</phase>
<goals>
- <goal>single</goal>
+ <goal>single</goal>
</goals>
<configuration>
<descriptor>src/assembly/koya-with-slider.xml</descriptor>
@@ -66,4 +83,3 @@
</project>
-
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/835bac85/app-packages/kafka/src/assembly/koya-app-package.xml
----------------------------------------------------------------------
diff --git a/app-packages/kafka/src/assembly/koya-app-package.xml b/app-packages/kafka/src/assembly/koya-app-package.xml
index dd8bd5c..9d984c1 100644
--- a/app-packages/kafka/src/assembly/koya-app-package.xml
+++ b/app-packages/kafka/src/assembly/koya-app-package.xml
@@ -1,21 +1,19 @@
<!--
- ~ Licensed to the Apache Software Foundation (ASF) under one
- ~ or more contributor license agreements. See the NOTICE file
- ~ distributed with this work for additional information
- ~ regarding copyright ownership. The ASF licenses this file
- ~ to you under the Apache License, Version 2.0 (the
- ~ "License"); you may not use this file except in compliance
- ~ with the License. You may obtain a copy of the License at
- ~
- ~ http://www.apache.org/licenses/LICENSE-2.0
- ~
- ~ Unless required by applicable law or agreed to in writing, software
- ~ distributed under the License is distributed on an "AS IS" BASIS,
- ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- ~ See the License for the specific language governing permissions and
- ~ limitations under the License.
- -->
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
<assembly
xmlns="http://maven.apache.org/plugins/maven-assembly-plugin/assembly/1.1.0"
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/835bac85/app-packages/kafka/src/assembly/koya-with-slider.xml
----------------------------------------------------------------------
diff --git a/app-packages/kafka/src/assembly/koya-with-slider.xml b/app-packages/kafka/src/assembly/koya-with-slider.xml
index 0f64927..98d730c 100644
--- a/app-packages/kafka/src/assembly/koya-with-slider.xml
+++ b/app-packages/kafka/src/assembly/koya-with-slider.xml
@@ -1,21 +1,19 @@
<!--
- ~ Licensed to the Apache Software Foundation (ASF) under one
- ~ or more contributor license agreements. See the NOTICE file
- ~ distributed with this work for additional information
- ~ regarding copyright ownership. The ASF licenses this file
- ~ to you under the Apache License, Version 2.0 (the
- ~ "License"); you may not use this file except in compliance
- ~ with the License. You may obtain a copy of the License at
- ~
- ~ http://www.apache.org/licenses/LICENSE-2.0
- ~
- ~ Unless required by applicable law or agreed to in writing, software
- ~ distributed under the License is distributed on an "AS IS" BASIS,
- ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- ~ See the License for the specific language governing permissions and
- ~ limitations under the License.
- -->
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
<assembly
xmlns="http://maven.apache.org/plugins/maven-assembly-plugin/assembly/1.1.0"
[4/5] incubator-slider git commit: SLIDER-977 Remove unnecessary
properties in pom/assembly.
Posted by el...@apache.org.
SLIDER-977 Remove unnecessary properties in pom/assembly.
Project: http://git-wip-us.apache.org/repos/asf/incubator-slider/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-slider/commit/2c3c7bdf
Tree: http://git-wip-us.apache.org/repos/asf/incubator-slider/tree/2c3c7bdf
Diff: http://git-wip-us.apache.org/repos/asf/incubator-slider/diff/2c3c7bdf
Branch: refs/heads/develop
Commit: 2c3c7bdfdd8d3302e98845fd45be48376196511d
Parents: fffbd6f
Author: Josh Elser <el...@apache.org>
Authored: Tue Jan 19 23:20:46 2016 -0500
Committer: Josh Elser <el...@apache.org>
Committed: Tue Jan 19 23:21:32 2016 -0500
----------------------------------------------------------------------
app-packages/kafka/pom.xml | 4 +---
app-packages/kafka/src/assembly/koya-with-slider.xml | 2 +-
2 files changed, 2 insertions(+), 4 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/2c3c7bdf/app-packages/kafka/pom.xml
----------------------------------------------------------------------
diff --git a/app-packages/kafka/pom.xml b/app-packages/kafka/pom.xml
index 0a5f1e1..743b622 100644
--- a/app-packages/kafka/pom.xml
+++ b/app-packages/kafka/pom.xml
@@ -19,7 +19,7 @@
<parent>
<groupId>org.apache.slider.packages</groupId>
<artifactId>slider-app-packages</artifactId>
- <version>0.90.0-incubating-SNAPSHOT</version>
+ <version>0.91.0-incubating-SNAPSHOT</version>
<relativePath>../pom.xml</relativePath>
</parent>
<modelVersion>4.0.0</modelVersion>
@@ -29,7 +29,6 @@
<description>Slider Kafka App Package</description>
<properties>
- <slider.version>0.90.0-incubating-SNAPSHOT</slider.version>
<kafka.src>${project.build.directory}/kafka_2.10-0.8.2.1.tgz</kafka.src>
<kafka.version>kafka_2.10-0.8.2.1</kafka.version>
</properties>
@@ -112,7 +111,6 @@
<dependency>
<groupId>org.apache.slider</groupId>
<artifactId>slider-core</artifactId>
- <version>${slider.version}</version>
</dependency>
</dependencies>
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/2c3c7bdf/app-packages/kafka/src/assembly/koya-with-slider.xml
----------------------------------------------------------------------
diff --git a/app-packages/kafka/src/assembly/koya-with-slider.xml b/app-packages/kafka/src/assembly/koya-with-slider.xml
index a5c6384..76c590f 100644
--- a/app-packages/kafka/src/assembly/koya-with-slider.xml
+++ b/app-packages/kafka/src/assembly/koya-with-slider.xml
@@ -53,7 +53,7 @@
<fileSets>
<fileSet>
- <directory>../../../../slider-assembly/target/slider-${slider.version}-all</directory>
+ <directory>../../../../slider-assembly/target/slider-${project.version}-all</directory>
<outputDirectory>/</outputDirectory>
<fileMode>0755</fileMode>
<directoryMode>0755</directoryMode>
[3/5] incubator-slider git commit: SLIDER-977 Address a bunch of
consistency issues in the kafka-app-package
Posted by el...@apache.org.
SLIDER-977 Address a bunch of consistency issues in the kafka-app-package
Match the "naming" (KOYA can be the short name, but slider-kafka-app-package
would be the proper name). Updated docs where I saw necessary. Make sure we
include the version string in all artifacts. Dynamically pull down a the kafka
tarball in the build.
Project: http://git-wip-us.apache.org/repos/asf/incubator-slider/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-slider/commit/fffbd6f6
Tree: http://git-wip-us.apache.org/repos/asf/incubator-slider/tree/fffbd6f6
Diff: http://git-wip-us.apache.org/repos/asf/incubator-slider/diff/fffbd6f6
Branch: refs/heads/develop
Commit: fffbd6f607bbeb75f8c1725f1ce9c7f775cf08a3
Parents: 835bac8
Author: Josh Elser <el...@apache.org>
Authored: Tue Jan 19 23:14:49 2016 -0500
Committer: Josh Elser <el...@apache.org>
Committed: Tue Jan 19 23:21:30 2016 -0500
----------------------------------------------------------------------
app-packages/kafka/README.md | 10 ++---
app-packages/kafka/appConfig-default.json | 2 +-
app-packages/kafka/build.sh | 2 +-
app-packages/kafka/deploy.sh | 2 +-
app-packages/kafka/pom.xml | 43 ++++++++++++++++++--
.../kafka/src/assembly/koya-with-slider.xml | 2 +-
app-packages/pom.xml | 2 +
pom.xml | 1 +
8 files changed, 51 insertions(+), 13 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/fffbd6f6/app-packages/kafka/README.md
----------------------------------------------------------------------
diff --git a/app-packages/kafka/README.md b/app-packages/kafka/README.md
index 34f0d04..0a3de5a 100644
--- a/app-packages/kafka/README.md
+++ b/app-packages/kafka/README.md
@@ -53,7 +53,7 @@ mvn clean install -DskipTests -Dkafka.src=path/to/kafka_2.10-0.8.1.1.tgz -Dkafka
Artifacts:
- Archive with embedded Slider: __`target/koya-with-slider.zip`__
- - Separate Slider application package: __`target/koya-slider-package-0.1.zip`__
+ - Separate Slider application package: __`target/slider-kafka-app-package-0.90.0-incubating-SNAPSHOT.zip`__
###Installation
@@ -89,11 +89,11 @@ If you use the full archive, the configuration file templates are already in you
Extract the packaged configuration files you are going to customize:
```
-unzip koya-slider-package-0.1.zip appConfig.json resources.json
+unzip slider-kafka-app-package-0.90.0-incubating-SNAPSHOT.zip appConfig.json resources.json
```
Adjust following properties in the global section:
```
- "application.def": "koya-slider-package-0.1.zip",
+ "application.def": "slider-kafka-app-package-0.90.0-incubating-SNAPSHOT.zip",
"site.global.xmx_val": "256m",
"site.global.xms_val": "128m",
"site.broker.zookeeper.connect": "${ZK_HOST}"
@@ -119,9 +119,9 @@ More information about the application configuration can be found [here](http://
The Slider application package needs to be copied to the HDFS location that was specified as application.def in appConfig.json:
```
-hdfs dfs -copyFromLocal koya-slider-package-0.1.zip /path/in/appConfig
+hdfs dfs -copyFromLocal slider-kafka-app-package-0.90.0-incubating-SNAPSHOT.zip /path/in/appConfig
```
Now the KOYA cluster can be deployed and launched:
```
-slider-0.80.0-incubating/bin/slider create koya --template ~/koya/appConfig.json --resources ~/koya/resources.json
+slider-0.90.0-incubating/bin/slider create koya --template ~/koya/appConfig.json --resources ~/koya/resources.json
```
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/fffbd6f6/app-packages/kafka/appConfig-default.json
----------------------------------------------------------------------
diff --git a/app-packages/kafka/appConfig-default.json b/app-packages/kafka/appConfig-default.json
index 9a0be4c..d7b6d6f 100644
--- a/app-packages/kafka/appConfig-default.json
+++ b/app-packages/kafka/appConfig-default.json
@@ -3,7 +3,7 @@
"metadata": {
},
"global": {
- "application.def": ".slider/package/KOYA/koya-slider-package-${project.version}.zip",
+ "application.def": ".slider/package/KOYA/slider-kafka-app-package-${project.version}.zip",
"java_home": "${JAVA_HOME}",
"system_configs": "broker",
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/fffbd6f6/app-packages/kafka/build.sh
----------------------------------------------------------------------
diff --git a/app-packages/kafka/build.sh b/app-packages/kafka/build.sh
index 35eeeba..cc5207b 100755
--- a/app-packages/kafka/build.sh
+++ b/app-packages/kafka/build.sh
@@ -18,4 +18,4 @@
wget http://apache.websitebeheerjd.nl/kafka/0.8.2.1/kafka_2.10-0.8.2.1.tgz -O $HOME/kafka_2.10-0.8.2.1.tgz
mvn clean install -DskipTests -Dkafka.src=$HOME/kafka_2.10-0.8.2.1.tgz -Dkafka.version=kafka_2.10-0.8.2.1
-unzip -o target/koya-slider-package-0.1.zip appConfig.json
+unzip -o target/slider-kafka-app-package-0.90.0-incubating-SNAPSHOT.zip appConfig.json
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/fffbd6f6/app-packages/kafka/deploy.sh
----------------------------------------------------------------------
diff --git a/app-packages/kafka/deploy.sh b/app-packages/kafka/deploy.sh
index 9b69c90..11e32f3 100755
--- a/app-packages/kafka/deploy.sh
+++ b/app-packages/kafka/deploy.sh
@@ -16,7 +16,7 @@
# limitations under the License.
APPNAME=${1:-koya}
-slider install-package --replacepkg --name KOYA --package target/koya-slider-package-0.1.zip
+slider install-package --replacepkg --name KOYA --package target/slider-kafka-app-package-0.90.0-incubating-SNAPSHOT.zip
slider stop $APPNAME
slider destroy $APPNAME
slider create $APPNAME --filesystem hdfs://root --queue dev --template appConfig.json --resources resources-default.json
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/fffbd6f6/app-packages/kafka/pom.xml
----------------------------------------------------------------------
diff --git a/app-packages/kafka/pom.xml b/app-packages/kafka/pom.xml
index fbb9db8..0a5f1e1 100644
--- a/app-packages/kafka/pom.xml
+++ b/app-packages/kafka/pom.xml
@@ -23,14 +23,14 @@
<relativePath>../pom.xml</relativePath>
</parent>
<modelVersion>4.0.0</modelVersion>
- <artifactId>kafka-app-package</artifactId>
+ <artifactId>slider-kafka-app-package</artifactId>
<packaging>jar</packaging>
<name>Slider Kafka App Package</name>
<description>Slider Kafka App Package</description>
<properties>
<slider.version>0.90.0-incubating-SNAPSHOT</slider.version>
- <kafka.src>${user.home}/kafka_2.10-0.8.2.1.tgz</kafka.src>
+ <kafka.src>${project.build.directory}/kafka_2.10-0.8.2.1.tgz</kafka.src>
<kafka.version>kafka_2.10-0.8.2.1</kafka.version>
</properties>
@@ -39,7 +39,7 @@
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-assembly-plugin</artifactId>
- <version>>${maven-assembly-plugin.version}</version>
+ <version>${maven-assembly-plugin.version}</version>
<executions>
<execution>
<id>app-package</id>
@@ -62,13 +62,48 @@
<configuration>
<descriptor>src/assembly/koya-with-slider.xml</descriptor>
<appendAssemblyId>false</appendAssemblyId>
- <finalName>koya-with-slider</finalName>
+ <finalName>slider-kafka-app-package-with-slider-${project.version}</finalName>
<attach>false</attach>
</configuration>
</execution>
</executions>
</plugin>
+ <plugin>
+ <artifactId>maven-compiler-plugin</artifactId>
+ <version>${maven-compiler-plugin.version}</version>
+ <dependencies>
+ <dependency>
+ <groupId>org.codehaus.groovy</groupId>
+ <artifactId>groovy-eclipse-compiler</artifactId>
+ <version>${groovy-eclipse-compiler.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>org.codehaus.groovy</groupId>
+ <artifactId>groovy-eclipse-batch</artifactId>
+ <version>${groovy-eclipse-batch.version}</version>
+ </dependency>
+ </dependencies>
+ </plugin>
+ <plugin>
+ <groupId>org.codehaus.mojo</groupId>
+ <artifactId>wagon-maven-plugin</artifactId>
+ <version>${wagon-maven-plugin.version}</version>
+ <executions>
+ <execution>
+ <id>download-kafka</id>
+ <phase>prepare-package</phase>
+ <goals>
+ <goal>download-single</goal>
+ </goals>
+ <configuration>
+ <url>https://dist.apache.org</url>
+ <fromFile>repos/dist/release/kafka/0.8.2.1/kafka_2.10-0.8.2.1.tgz</fromFile>
+ <toDir>${project.build.directory}</toDir>
+ </configuration>
+ </execution>
+ </executions>
+ </plugin>
</plugins>
</build>
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/fffbd6f6/app-packages/kafka/src/assembly/koya-with-slider.xml
----------------------------------------------------------------------
diff --git a/app-packages/kafka/src/assembly/koya-with-slider.xml b/app-packages/kafka/src/assembly/koya-with-slider.xml
index 98d730c..a5c6384 100644
--- a/app-packages/kafka/src/assembly/koya-with-slider.xml
+++ b/app-packages/kafka/src/assembly/koya-with-slider.xml
@@ -27,7 +27,7 @@
<files>
<file>
- <source>target/koya-slider-package-${project.version}.zip</source>
+ <source>target/slider-kafka-app-package-${project.version}.zip</source>
<outputDirectory>/</outputDirectory>
<fileMode>0755</fileMode>
</file>
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/fffbd6f6/app-packages/pom.xml
----------------------------------------------------------------------
diff --git a/app-packages/pom.xml b/app-packages/pom.xml
index a286d0c..aa9632b 100644
--- a/app-packages/pom.xml
+++ b/app-packages/pom.xml
@@ -52,6 +52,7 @@
<modules>
<module>accumulo</module>
<module>hbase</module>
+ <module>kafka</module>
<module>storm</module>
<module>tomcat</module>
</modules>
@@ -83,6 +84,7 @@
<modules>
<module>accumulo</module>
<module>hbase</module>
+ <module>kafka</module>
<module>storm</module>
<module>hbase-win</module>
<module>storm-win</module>
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/fffbd6f6/pom.xml
----------------------------------------------------------------------
diff --git a/pom.xml b/pom.xml
index d39a4e4..5e8cbc2 100644
--- a/pom.xml
+++ b/pom.xml
@@ -223,6 +223,7 @@
<maven-failsafe-plugin.version>${maven-surefire-plugin.version}</maven-failsafe-plugin.version>
<maven-verifier-plugin.version>1.0</maven-verifier-plugin.version>
<apache-rat-plugin.version>0.11</apache-rat-plugin.version>
+ <wagon-maven-plugin.version>1.0</wagon-maven-plugin.version>
<!-- build options-->
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
[5/5] incubator-slider git commit: SLIDER-977 Add the expected RAT
exclusions for an app-package.
Posted by el...@apache.org.
SLIDER-977 Add the expected RAT exclusions for an app-package.
Project: http://git-wip-us.apache.org/repos/asf/incubator-slider/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-slider/commit/6379d1b2
Tree: http://git-wip-us.apache.org/repos/asf/incubator-slider/tree/6379d1b2
Diff: http://git-wip-us.apache.org/repos/asf/incubator-slider/diff/6379d1b2
Branch: refs/heads/develop
Commit: 6379d1b2121ea5936d4763b80fb4f249629333df
Parents: 2c3c7bd
Author: Josh Elser <el...@apache.org>
Authored: Tue Jan 19 23:24:18 2016 -0500
Committer: Josh Elser <el...@apache.org>
Committed: Tue Jan 19 23:24:18 2016 -0500
----------------------------------------------------------------------
app-packages/kafka/pom.xml | 28 ++++++++++++++++++++++++++++
1 file changed, 28 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/6379d1b2/app-packages/kafka/pom.xml
----------------------------------------------------------------------
diff --git a/app-packages/kafka/pom.xml b/app-packages/kafka/pom.xml
index 743b622..d491109 100644
--- a/app-packages/kafka/pom.xml
+++ b/app-packages/kafka/pom.xml
@@ -114,5 +114,33 @@
</dependency>
</dependencies>
+ <profiles>
+ <profile>
+ <id>rat</id>
+ <build>
+ <plugins>
+ <plugin>
+ <groupId>org.apache.rat</groupId>
+ <artifactId>apache-rat-plugin</artifactId>
+ <version>${apache-rat-plugin.version}</version>
+ <executions>
+ <execution>
+ <id>check-licenses</id>
+ <goals>
+ <goal>check</goal>
+ </goals>
+ </execution>
+ </executions>
+ <configuration>
+ <excludes>
+ <exclude>appConfig-default.json</exclude>
+ <exclude>resources-default.json</exclude>
+ </excludes>
+ </configuration>
+ </plugin>
+ </plugins>
+ </build>
+ </profile>
+ </profiles>
</project>