You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@bigtop.apache.org by rv...@apache.org on 2011/10/31 01:30:06 UTC
svn commit: r1195290 [1/6] - in /incubator/bigtop/branches/hadoop-0.22: ./
bigtop-deploy/ bigtop-deploy/puppet/ bigtop-deploy/puppet/manifests/
bigtop-deploy/puppet/modules/ bigtop-deploy/puppet/modules/hadoop-flume/
bigtop-deploy/puppet/modules/hadoop...
Author: rvs
Date: Mon Oct 31 00:30:01 2011
New Revision: 1195290
URL: http://svn.apache.org/viewvc?rev=1195290&view=rev
Log:
Merging the Bigtop 0.2.0 RC code
Added:
incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/
incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/
incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/manifests/
incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/manifests/cluster.pp
incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/manifests/site.pp
- copied, changed from r1185814, incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/common/pig/do-component-build
incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/
incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop/
incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-flume/
incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-flume/manifests/
incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-flume/manifests/init.pp
- copied, changed from r1185814, incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/deb/sqoop/control
incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-flume/tests/
incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-flume/tests/init.pp
- copied, changed from r1185814, incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/common/whirr/do-component-build
incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-hbase/
incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-hbase/manifests/
incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-hbase/manifests/init.pp
incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-hbase/templates/
incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-hbase/templates/hbase-env.sh
incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-hbase/templates/hbase-site.xml
incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-hbase/tests/
incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-hbase/tests/init.pp
- copied, changed from r1185814, incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/common/hbase/do-component-build
incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-hive/
incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-hive/manifests/
incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-hive/manifests/init.pp
- copied, changed from r1185814, incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/common/hbase/do-component-build
incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-hive/templates/
incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-hive/templates/hive-site.xml
incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-hive/tests/
incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-hive/tests/init.pp
- copied, changed from r1185814, incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/common/hive/do-component-build
incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-oozie/
incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-oozie/manifests/
incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-oozie/manifests/init.pp
- copied, changed from r1185814, incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/common/pig/do-component-build
incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-oozie/templates/
incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-oozie/templates/oozie-site.xml
incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-oozie/tests/
incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-oozie/tests/init.pp
- copied, changed from r1185814, incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/common/hive/do-component-build
incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-pig/
incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-pig/manifests/
incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-pig/manifests/init.pp
- copied, changed from r1185814, incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/common/hbase/do-component-build
incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-pig/templates/
incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-pig/templates/pig.properties
incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-pig/tests/
incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-pig/tests/init.pp
- copied, changed from r1185814, incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/common/hive/do-component-build
incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-sqoop/
incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-sqoop/manifests/
incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-sqoop/manifests/init.pp
- copied, changed from r1185814, incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/common/pig/do-component-build
incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-sqoop/tests/
incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-sqoop/tests/init.pp
- copied, changed from r1185814, incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/common/hive/do-component-build
incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-zookeeper/
incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-zookeeper/manifests/
incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-zookeeper/manifests/init.pp
incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-zookeeper/templates/
incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-zookeeper/templates/zoo.cfg
- copied, changed from r1185814, incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/common/hbase/do-component-build
incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-zookeeper/tests/
incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-zookeeper/tests/init.pp
- copied, changed from r1185814, incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/common/hbase/do-component-build
incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop/lib/
incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop/lib/facter/
incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop/lib/facter/hadoop_storage_locations.rb
- copied, changed from r1185814, incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/deb/sqoop/control
incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop/manifests/
incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop/manifests/.init.pp.swp
incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop/manifests/init.pp
incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop/templates/
incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop/templates/core-site.xml
incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop/templates/hadoop
- copied, changed from r1185814, incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/common/hadoop/hadoop.default
incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop/templates/hadoop-env.sh
incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop/templates/hdfs-site.xml
incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop/templates/mapred-site.xml
incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop/templates/taskcontroller.cfg
incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/kerberos/
incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/kerberos/lib/
incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/kerberos/lib/facter/
incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/kerberos/lib/facter/kadm_keytab.rb
- copied, changed from r1185814, incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/common/hbase/do-component-build
incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/kerberos/manifests/
incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/kerberos/manifests/init.pp
incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/kerberos/templates/
incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/kerberos/templates/kadm5.acl
- copied, changed from r1185814, incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/common/hbase/do-component-build
incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/kerberos/templates/kdc.conf
incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/kerberos/templates/krb5.conf
incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/kerberos/tests/
incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/kerberos/tests/init.pp
- copied, changed from r1185814, incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/common/hadoop/hadoop.default
incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/common/bigtop-utils/
incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/common/bigtop-utils/LICENSE
incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/common/bigtop-utils/bigtop-detect-javahome
- copied, changed from r1185814, incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/deb/whirr/control
incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/common/hadoop/hadoop.nofiles.conf
incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/common/hbase/hbase.nofiles.conf
incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/common/zookeeper/zoo.cfg
- copied, changed from r1185814, incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/common/hbase/do-component-build
incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/deb/bigtop-utils/
incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/deb/bigtop-utils/compat
incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/deb/bigtop-utils/control
- copied, changed from r1185814, incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/common/hbase/do-component-build
incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/deb/bigtop-utils/copyright
incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/deb/bigtop-utils/rules (contents, props changed)
- copied, changed from r1185814, incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/deb/zookeeper/rules
incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/deb/hadoop/hadoop-sbin.postinst
- copied, changed from r1185814, incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/common/pig/do-component-build
incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/deb/hadoop/hadoop.daemon.postinst.tpl
incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/rpm/bigtop-utils/
incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/rpm/bigtop-utils/RPMS/
incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/rpm/bigtop-utils/RPMS/.gitignore
incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/rpm/bigtop-utils/SPECS/
incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/rpm/bigtop-utils/SPECS/.gitignore
incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/rpm/bigtop-utils/SPECS/bigtop-utils.spec
incubator/bigtop/branches/hadoop-0.22/bigtop-test-framework/src/main/groovy/org/apache/bigtop/itest/pmanager/UrpmiCmdLinePackageManager.groovy
incubator/bigtop/branches/hadoop-0.22/bigtop-tests/test-artifacts/package/src/main/resources/package_data_urpmi.xml
Modified:
incubator/bigtop/branches/hadoop-0.22/README
incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/common/flume/install_flume.sh
incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/common/hadoop/hadoop.default
incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/common/hadoop/install_hadoop.sh
incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/common/hbase/do-component-build
incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/common/hbase/install_hbase.sh
incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/common/hive/do-component-build
incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/common/hive/hadoop-hive.sh
incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/common/hive/hadoop-hive.sh.suse
incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/common/hive/install_hive.sh
incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/common/mahout/install_mahout.sh
incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/common/oozie/create-package-layout
incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/common/oozie/do-component-build
incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/common/oozie/oozie.init
incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/common/pig/do-component-build
incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/common/pig/install_pig.sh
incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/common/sqoop/install_sqoop.sh
incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/common/sqoop/sqoop-metastore.sh
incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/common/sqoop/sqoop-metastore.sh.suse
incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/common/whirr/do-component-build
incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/common/whirr/install_whirr.sh
incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/common/zookeeper/hadoop-zookeeper.sh
incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/common/zookeeper/hadoop-zookeeper.sh.suse
incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/common/zookeeper/install_zookeeper.sh
incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/deb/flume/control
incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/deb/flume/service-init.d.tpl
incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/deb/hadoop/control
incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/deb/hadoop/hadoop-sbin.install
incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/deb/hadoop/hadoop.install
incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/deb/hadoop/rules
incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/deb/hadoop/service-init.d.tpl
incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/deb/hbase/control
incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/deb/hbase/hadoop-hbase.install
incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/deb/hbase/rules
incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/deb/hbase/service-init.d.tpl
incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/deb/hive/changelog
incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/deb/hive/control
incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/deb/mahout/control
incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/deb/oozie/control
incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/deb/oozie/oozie.postinst
incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/deb/pig/control
incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/deb/pig/rules
incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/deb/sqoop/control
incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/deb/whirr/control
incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/deb/whirr/rules
incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/deb/zookeeper/control
incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/deb/zookeeper/hadoop-zookeeper-server.init
incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/deb/zookeeper/rules
incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/rpm/flume/SOURCES/init.d
incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/rpm/flume/SOURCES/init.d.suse
incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/rpm/flume/SPECS/flume.spec
incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/rpm/hadoop/SOURCES/hadoop-init.tmpl
incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/rpm/hadoop/SOURCES/hadoop-init.tmpl.suse
incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/rpm/hadoop/SPECS/hadoop.spec
incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/rpm/hbase/SOURCES/hadoop-hbase.sh
incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/rpm/hbase/SOURCES/hadoop-hbase.sh.suse
incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/rpm/hbase/SPECS/hbase.spec
incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/rpm/hive/SOURCES/install_hive.sh
incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/rpm/hive/SPECS/hive.spec
incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/rpm/mahout/SPECS/mahout.spec
incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/rpm/oozie/SPECS/oozie.spec
incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/rpm/pig/SPECS/pig.spec
incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/rpm/sqoop/SPECS/sqoop.spec
incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/rpm/whirr/SPECS/whirr.spec
incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/rpm/zookeeper/SPECS/zookeeper.spec
incubator/bigtop/branches/hadoop-0.22/bigtop-test-framework/src/main/groovy/org/apache/bigtop/itest/pmanager/PackageManager.groovy
incubator/bigtop/branches/hadoop-0.22/bigtop-test-framework/src/main/groovy/org/apache/bigtop/itest/pmanager/RPMPackage.groovy
incubator/bigtop/branches/hadoop-0.22/bigtop-tests/test-artifacts/flume/pom.xml
incubator/bigtop/branches/hadoop-0.22/bigtop-tests/test-artifacts/hadoop/pom.xml
incubator/bigtop/branches/hadoop-0.22/bigtop-tests/test-artifacts/hbase/pom.xml
incubator/bigtop/branches/hadoop-0.22/bigtop-tests/test-artifacts/oozie/pom.xml
incubator/bigtop/branches/hadoop-0.22/bigtop-tests/test-artifacts/pom.xml
incubator/bigtop/branches/hadoop-0.22/bigtop-tests/test-execution/smokes/hbase/pom.xml
incubator/bigtop/branches/hadoop-0.22/bigtop-tests/test-execution/smokes/pom.xml
incubator/bigtop/branches/hadoop-0.22/bigtop.mk
incubator/bigtop/branches/hadoop-0.22/package.mk
Modified: incubator/bigtop/branches/hadoop-0.22/README
URL: http://svn.apache.org/viewvc/incubator/bigtop/branches/hadoop-0.22/README?rev=1195290&r1=1195289&r2=1195290&view=diff
==============================================================================
--- incubator/bigtop/branches/hadoop-0.22/README (original)
+++ incubator/bigtop/branches/hadoop-0.22/README Mon Oct 31 00:30:01 2011
@@ -49,6 +49,21 @@ On Debian-based systems one also needs
* build-essential dh-make debhelper devscripts
* reprepro
+On openSUSE 11.4+, in addition, one also needs to ensure the following is installed:
+
+
+relaxngDatatype
+docbook-utils docbook-simple
+asciidoc
+fuse-devel
+docbook5
+docbook5-xsl-stylesheets
+libxml2-devel
+xmlformat
+xmlto
+libxslt
+libopenssl-devel
+
## Building packages
$ make [component-name]-[rpm|deb]
Added: incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/manifests/cluster.pp
URL: http://svn.apache.org/viewvc/incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/manifests/cluster.pp?rev=1195290&view=auto
==============================================================================
--- incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/manifests/cluster.pp (added)
+++ incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/manifests/cluster.pp Mon Oct 31 00:30:01 2011
@@ -0,0 +1,161 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+class hadoop_cluster_node {
+ $hadoop_namenode_host="$hadoop_head_node"
+ $hadoop_namenode_port="17020"
+ $hadoop_namenode_thrift_port="10090"
+ # $hadoop_dfs_namenode_plugins="org.apache.hadoop.thriftfs.NamenodePlugin"
+ # $hadoop_dfs_datanode_plugins="org.apache.hadoop.thriftfs.DatanodePlugin"
+
+ $hadoop_jobtracker_host="$hadoop_head_node"
+ $hadoop_jobtracker_port="8021"
+ $hadoop_jobtracker_thrift_port="9290"
+ # $hadoop_mapred_jobtracker_plugins="org.apache.hadoop.thriftfs.ThriftJobTrackerPlugin"
+ # $hadoop_mapred_tasktracker_plugins="org.apache.hadoop.mapred.TaskTrackerCmonInst"
+
+ $hadoop_core_proxyusers = { oozie => { groups => 'root,hadoop,jenkins,oozie,users', hosts => "${hadoop_head_node},localhost,127.0.0.1" } }
+
+ $hadoop_hbase_rootdir = "hdfs://$hadoop_namenode_host:$hadoop_namenode_port/hbase"
+ $hadoop_hbase_zookeeper_quorum = "$hadoop_head_node"
+
+ $hadoop_zookeeper_ensemble = ["$hadoop_head_node:2888:3888"]
+
+ $namenode_data_dirs = ["/mnt/namenode"]
+ $hdfs_data_dirs = ["/mnt/hdfs"]
+ $mapred_data_dirs = ["/mnt/scratch"]
+
+ if ($hadoop_security_authentication == "kerberos") {
+ $kerberos_domain = "compute-1.internal"
+ $kerberos_realm = "EXAMPLE.COM"
+ $kerberos_kdc_server = "$hadoop_head_node"
+
+ include kerberos::client
+ kerberos::client::host_keytab { ["hdfs", "mapred", "hbase", "oozie"]:
+ princs_map => { hdfs => [ "host", "hdfs" ],
+ mapred => [ "mapred" ],
+ hbase => [ "hbase" ],
+ oozie => [ "oozie" ], },
+ }
+ }
+}
+
+class hadoop_worker_node inherits hadoop_cluster_node {
+ hadoop::datanode { "datanode":
+ namenode_host => $hadoop_namenode_host,
+ namenode_port => $hadoop_namenode_port,
+ dirs => $hdfs_data_dirs,
+ auth => $hadoop_security_authentication,
+ }
+
+ hadoop::tasktracker { "tasktracker":
+ namenode_host => $hadoop_namenode_host,
+ namenode_port => $hadoop_namenode_port,
+ jobtracker_host => $hadoop_jobtracker_host,
+ jobtracker_port => $hadoop_jobtracker_port,
+ dirs => $mapred_data_dirs,
+ auth => $hadoop_security_authentication,
+ }
+
+ hadoop-hbase::server { "hbase region server":
+ rootdir => $hadoop_hbase_rootdir,
+ zookeeper_quorum => $hadoop_hbase_zookeeper_quorum,
+ kerberos_realm => $kerberos_realm,
+ }
+}
+
+class hadoop_head_node inherits hadoop_cluster_node {
+
+ if ($hadoop_security_authentication == "kerberos") {
+ include kerberos::kdc, kerberos::kdc::admin_server
+ }
+
+ hadoop::namenode { "namenode":
+ port => $hadoop_namenode_port,
+ jobtracker_host => $hadoop_jobtracker_host,
+ jobtracker_port => $hadoop_jobtracker_port,
+ dirs => $namenode_data_dirs,
+ # thrift_port => $hadoop_namenode_thrift_port,
+ auth => $hadoop_security_authentication,
+ }
+
+ hadoop::secondarynamenode { "secondary namenode":
+ namenode_host => $hadoop_namenode_host,
+ namenode_port => $hadoop_namenode_port,
+ auth => $hadoop_security_authentication,
+ }
+
+ hadoop::jobtracker { "jobtracker":
+ namenode_host => $hadoop_namenode_host,
+ namenode_port => $hadoop_namenode_port,
+ host => $hadoop_jobtracker_host,
+ port => $hadoop_jobtracker_port,
+ dirs => $mapred_data_dirs,
+ # thrift_port => $hadoop_jobtracker_thrift_port,
+ auth => $hadoop_security_authentication,
+ }
+
+ hadoop-hbase::master { "hbase master":
+ rootdir => $hadoop_hbase_rootdir,
+ zookeeper_quorum => $hadoop_hbase_zookeeper_quorum,
+ kerberos_realm => $kerberos_realm,
+ }
+
+ hadoop-oozie::server { "oozie server":
+ kerberos_realm => $kerberos_realm,
+ }
+
+ hadoop-zookeeper::server { "zookeeper":
+ myid => "0",
+ ensemble => $hadoop_zookeeper_ensemble,
+ }
+
+ hadoop::create_hdfs_dirs { [ "/mapred", "/tmp", "/system", "/user", "/hbase", "/benchmarks", "/user/jenkins", "/user/hive" ]:
+ hdfs_dirs_meta => { "/tmp" => { perm => "777", user => "hdfs" },
+ "/mapred" => { perm => "755", user => "mapred" },
+ "/system" => { perm => "755", user => "hdfs" },
+ "/user" => { perm => "755", user => "hdfs" },
+ "/hbase" => { perm => "755", user => "hbase" },
+ "/benchmarks" => { perm => "777", user => "hdfs" },
+ "/user/jenkins" => { perm => "777", user => "jenkins"},
+ "/user/hive" => { perm => "777", user => "hive" } },
+ }
+}
+
+class hadoop_gateway_node inherits hadoop_head_node {
+ # hadoop::client { "gateway":
+ # namenode_host => $hadoop_namenode_host,
+ # namenode_port => $hadoop_namenode_port,
+ # jobtracker_host => $hadoop_jobtracker_host,
+ # jobtracker_port => $hadoop_jobtracker_port,
+ # # auth => $hadoop_security_authentication,
+ # }
+
+ hadoop-pig::client { "pig client":
+ }
+ hadoop-hive::client { "hive client":
+ hbase_zookeeper_quorum => $hadoop_hbase_zookeeper_quorum,
+ }
+ hadoop-sqoop::client { "sqoop client":
+ }
+ hadoop-oozie::client { "oozie client":
+ }
+ hadoop-hbase::client { "hbase client":
+ }
+ hadoop-zookeeper::client { "zookeeper client":
+ }
+ hadoop-flume::client { "flume client":
+ }
+}
Copied: incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/manifests/site.pp (from r1185814, incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/common/pig/do-component-build)
URL: http://svn.apache.org/viewvc/incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/manifests/site.pp?p2=incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/manifests/site.pp&p1=incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/common/pig/do-component-build&r1=1185814&r2=1195290&rev=1195290&view=diff
==============================================================================
--- incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/common/pig/do-component-build (original)
+++ incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/manifests/site.pp Mon Oct 31 00:30:01 2011
@@ -1,4 +1,3 @@
-#!/bin/sh
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
@@ -14,8 +13,28 @@
# See the License for the specific language governing permissions and
# limitations under the License.
-set -ex
+stage {"pre": before => Stage["main"]}
-ant -Djavac.version=1.6 -Djava5.home=${JAVA5_HOME} -Dforrest.home=${FORREST_HOME} -Ddist.dir=debian/tmp -Dversion=${PIG_BASE_VERSION} package "$@"
+yumrepo { "Bigtop":
+ baseurl => "http://bigtop01.cloudera.org:8080/job/Bigtop-trunk-centos5/lastSuccessfulBuild/artifact/output/",
+ descr => "Bigtop packages",
+ enabled => 1,
+ gpgcheck => 0,
+}
+package { "jdk":
+ ensure => "installed",
+}
+# $hadoop_head_node="beefy.my.org"
+# $hadoop_security_authentication="kerberos"
+
+import "cluster.pp"
+
+node default {
+ if ($hadoop_head_node == $fqdn) {
+ include hadoop_head_node
+ } else {
+ include hadoop_worker_node
+ }
+}
Copied: incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-flume/manifests/init.pp (from r1185814, incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/deb/sqoop/control)
URL: http://svn.apache.org/viewvc/incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-flume/manifests/init.pp?p2=incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-flume/manifests/init.pp&p1=incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/deb/sqoop/control&r1=1185814&r2=1195290&rev=1195290&view=diff
==============================================================================
--- incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/deb/sqoop/control (original)
+++ incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-flume/manifests/init.pp Mon Oct 31 00:30:01 2011
@@ -12,24 +12,40 @@
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
-Source: sqoop
-Section: misc
-Priority: extra
-Maintainer: Alex Newman <ne...@cloudera.com>
-Build-Depends: debhelper (>= 6), ant, ant-optional, git-core, sed, asciidoc, xmlto
-Standards-Version: 3.8.0
-Homepage: http://www.cloudera.com
-Package: sqoop
-Architecture: all
-Depends: hadoop
-Description: Tool for easy imports and exports of data sets between databases and HDFS
- Sqoop is a tool that provides the ability to import and export data sets between
- the Hadoop Distributed File System (HDFS) and relational databases.
+class hadoop-flume {
+ define client {
+ package { "flume":
+ ensure => latest,
+ }
+ }
-Package: sqoop-metastore
-Architecture: all
-Depends: sqoop (= ${source:Version}), adduser
-Description: Shared metadata repository for Sqoop.
- This optional package hosts a metadata server for Sqoop clients across a network to use.
+ # It really is flume node, but node is a reserved keyword for puppet
+ define agent {
+ package { "flume-node":
+ ensure => latest,
+ }
+ service { "flume-node":
+ ensure => running,
+ require => Package["flume-node"],
+ # FIXME: this need to be fixed in upstream flume
+ hasstatus => false,
+ hasrestart => true,
+ }
+ }
+
+ define master {
+ package { "flume-master":
+ ensure => latest,
+ }
+
+ service { "flume-master":
+ ensure => running,
+ require => Package["flume-node"],
+ # FIXME: this need to be fixed in upstream flume
+ hasstatus => true,
+ hasrestart => true,
+ }
+ }
+}
Copied: incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-flume/tests/init.pp (from r1185814, incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/common/whirr/do-component-build)
URL: http://svn.apache.org/viewvc/incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-flume/tests/init.pp?p2=incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-flume/tests/init.pp&p1=incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/common/whirr/do-component-build&r1=1185814&r2=1195290&rev=1195290&view=diff
==============================================================================
--- incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/common/whirr/do-component-build (original)
+++ incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-flume/tests/init.pp Mon Oct 31 00:30:01 2011
@@ -1,4 +1,3 @@
-#!/bin/sh
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
@@ -14,7 +13,8 @@
# See the License for the specific language governing permissions and
# limitations under the License.
-set -ex
+hadoop-flume::client { "test-flume": }
+hadoop-flume::agent { "test-flume-node": }
+hadoop-flume::master { "test-flume-master": }
-mvn clean source:jar install assembly:assembly -Pjavadoc site "$@"
Added: incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-hbase/manifests/init.pp
URL: http://svn.apache.org/viewvc/incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-hbase/manifests/init.pp?rev=1195290&view=auto
==============================================================================
--- incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-hbase/manifests/init.pp (added)
+++ incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-hbase/manifests/init.pp Mon Oct 31 00:30:01 2011
@@ -0,0 +1,71 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+class hadoop-hbase {
+ class client-package {
+ package { "hadoop-hbase":
+ ensure => latest,
+ }
+ }
+
+ class common-server-config {
+ include client-package
+
+ file { "/etc/hbase/conf/hbase-site.xml":
+ content => template("hadoop-hbase/hbase-site.xml"),
+ require => Package["hadoop-hbase"],
+ }
+ file { "/etc/hbase/conf/hbase-env.sh":
+ content => template("hadoop-hbase/hbase-env.sh"),
+ require => Package["hadoop-hbase"],
+ }
+ }
+
+ define client {
+ include client-package
+ }
+
+ define server($rootdir, $zookeeper_quorum, $kerberos_realm = "") {
+ include common-server-config
+
+ package { "hadoop-hbase-regionserver":
+ ensure => latest,
+ }
+
+ service { "hadoop-hbase-regionserver":
+ ensure => running,
+ require => Package["hadoop-hbase-regionserver"],
+ subscribe => File["/etc/hbase/conf/hbase-site.xml", "/etc/hbase/conf/hbase-env.sh"],
+ hasrestart => true,
+ hasstatus => true,
+ }
+ }
+
+ define master($rootdir, $zookeeper_quorum, $kerberos_realm = "") {
+ include common-server-config
+
+ package { "hadoop-hbase-master":
+ ensure => latest,
+ }
+
+ service { "hadoop-hbase-master":
+ ensure => running,
+ require => Package["hadoop-hbase-master"],
+ subscribe => File["/etc/hbase/conf/hbase-site.xml", "/etc/hbase/conf/hbase-env.sh"],
+ hasrestart => true,
+ hasstatus => true,
+ }
+ }
+}
Added: incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-hbase/templates/hbase-env.sh
URL: http://svn.apache.org/viewvc/incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-hbase/templates/hbase-env.sh?rev=1195290&view=auto
==============================================================================
--- incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-hbase/templates/hbase-env.sh (added)
+++ incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-hbase/templates/hbase-env.sh Mon Oct 31 00:30:01 2011
@@ -0,0 +1,76 @@
+#
+#/**
+# * Copyright 2007 The Apache Software Foundation
+# *
+# * Licensed to the Apache Software Foundation (ASF) under one
+# * or more contributor license agreements. See the NOTICE file
+# * distributed with this work for additional information
+# * regarding copyright ownership. The ASF licenses this file
+# * to you under the Apache License, Version 2.0 (the
+# * "License"); you may not use this file except in compliance
+# * with the License. You may obtain a copy of the License at
+# *
+# * http://www.apache.org/licenses/LICENSE-2.0
+# *
+# * Unless required by applicable law or agreed to in writing, software
+# * distributed under the License is distributed on an "AS IS" BASIS,
+# * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# * See the License for the specific language governing permissions and
+# * limitations under the License.
+# */
+
+# Set environment variables here.
+
+# The java implementation to use. Java 1.6 required.
+# export JAVA_HOME=/usr/java/jdk1.6.0/
+
+# Extra Java CLASSPATH elements. Optional.
+export HBASE_CLASSPATH=/etc/hadoop/conf
+
+# The maximum amount of heap to use, in MB. Default is 1000.
+# export HBASE_HEAPSIZE=1000
+
+# Extra Java runtime options.
+# Below are what we set by default. May only work with SUN JVM.
+# For more on why as well as other possible settings,
+# see http://wiki.apache.org/hadoop/PerformanceTuning
+export HBASE_OPTS="$HBASE_OPTS -ea -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode"
+
+# Uncomment below to enable java garbage collection logging.
+# export HBASE_OPTS="$HBASE_OPTS -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -Xloggc:$HBASE_HOME/logs/gc-hbase.log"
+
+# Uncomment and adjust to enable JMX exporting
+# See jmxremote.password and jmxremote.access in $JRE_HOME/lib/management to configure remote password access.
+# More details at: http://java.sun.com/javase/6/docs/technotes/guides/management/agent.html
+#
+# export HBASE_JMX_BASE="-Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false"
+# export HBASE_MASTER_OPTS="$HBASE_MASTER_OPTS $HBASE_JMX_BASE -Dcom.sun.management.jmxremote.port=10101"
+# export HBASE_REGIONSERVER_OPTS="$HBASE_REGIONSERVER_OPTS $HBASE_JMX_BASE -Dcom.sun.management.jmxremote.port=10102"
+# export HBASE_THRIFT_OPTS="$HBASE_THRIFT_OPTS $HBASE_JMX_BASE -Dcom.sun.management.jmxremote.port=10103"
+# export HBASE_ZOOKEEPER_OPTS="$HBASE_ZOOKEEPER_OPTS $HBASE_JMX_BASE -Dcom.sun.management.jmxremote.port=10104"
+
+# File naming hosts on which HRegionServers will run. $HBASE_HOME/conf/regionservers by default.
+# export HBASE_REGIONSERVERS=${HBASE_HOME}/conf/regionservers
+
+# Extra ssh options. Empty by default.
+# export HBASE_SSH_OPTS="-o ConnectTimeout=1 -o SendEnv=HBASE_CONF_DIR"
+
+# Where log files are stored. $HBASE_HOME/logs by default.
+# export HBASE_LOG_DIR=${HBASE_HOME}/logs
+
+# A string representing this instance of hbase. $USER by default.
+# export HBASE_IDENT_STRING=$USER
+
+# The scheduling priority for daemon processes. See 'man nice'.
+# export HBASE_NICENESS=10
+
+# The directory where pid files are stored. /tmp by default.
+# export HBASE_PID_DIR=/var/hadoop/pids
+
+# Seconds to sleep between slave commands. Unset by default. This
+# can be useful in large clusters, where, e.g., slave rsyncs can
+# otherwise arrive faster than the master can service them.
+# export HBASE_SLAVE_SLEEP=0.1
+
+# Tell HBase whether it should manage it's own instance of Zookeeper or not.
+# export HBASE_MANAGES_ZK=true
Added: incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-hbase/templates/hbase-site.xml
URL: http://svn.apache.org/viewvc/incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-hbase/templates/hbase-site.xml?rev=1195290&view=auto
==============================================================================
--- incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-hbase/templates/hbase-site.xml (added)
+++ incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-hbase/templates/hbase-site.xml Mon Oct 31 00:30:01 2011
@@ -0,0 +1,58 @@
+<?xml version="1.0"?>
+<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
+<!--
+/**
+ * Copyright 2010 The Apache Software Foundation
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+-->
+<configuration>
+ <property>
+ <name>hbase.cluster.distributed</name>
+ <value>true</value>
+ </property>
+
+ <property>
+ <name>hbase.zookeeper.quorum</name>
+ <value><%= zookeeper_quorum %></value>
+ </property>
+
+ <property>
+ <name>hbase.rootdir</name>
+ <value><%= rootdir %></value>
+ </property>
+
+<% if kerberos_realm != "" %>
+ <property>
+ <name>hbase.regionserver.kerberos.principal</name>
+ <value>hbase/_HOST@<%= kerberos_realm %></value>
+ </property>
+ <property>
+ <name>hbase.regionserver.keytab.file</name>
+ <value>/etc/hbase.keytab</value>
+ </property>
+ <property>
+ <name>hbase.master.kerberos.principal</name>
+ <value>hbase/_HOST@<%= kerberos_realm %></value>
+ </property>
+ <property>
+ <name>hbase.master.keytab.file</name>
+ <value>/etc/hbase.keytab</value>
+ </property>
+<% end %>
+</configuration>
Copied: incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-hbase/tests/init.pp (from r1185814, incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/common/hbase/do-component-build)
URL: http://svn.apache.org/viewvc/incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-hbase/tests/init.pp?p2=incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-hbase/tests/init.pp&p1=incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/common/hbase/do-component-build&r1=1185814&r2=1195290&rev=1195290&view=diff
==============================================================================
--- incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/common/hbase/do-component-build (original)
+++ incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-hbase/tests/init.pp Mon Oct 31 00:30:01 2011
@@ -1,4 +1,3 @@
-#!/bin/sh
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
@@ -14,6 +13,14 @@
# See the License for the specific language governing permissions and
# limitations under the License.
-set -ex
-
-mvn clean -DskipTests -Dhbase.version=${HBASE_VERSION} -Dhadoop.profile=22 install site assembly:assembly "$@"
+hadoop-hbase::client { "hbase shell": }
+hadoop-hbase::master { "test-hbase-master":
+ rootdir => "hdfs://localhost:17020/hbase",
+ zookeeper_quorum => "localhost",
+ kerberos_realm => "KRB.YOU.ORG",
+}
+hadoop-hbase::server { "test-hbase-server":
+ rootdir => "hdfs://localhost:17020/hbase",
+ zookeeper_quorum => "localhost",
+ kerberos_realm => "KRB.YOU.ORG",
+}
Copied: incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-hive/manifests/init.pp (from r1185814, incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/common/hbase/do-component-build)
URL: http://svn.apache.org/viewvc/incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-hive/manifests/init.pp?p2=incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-hive/manifests/init.pp&p1=incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/common/hbase/do-component-build&r1=1185814&r2=1195290&rev=1195290&view=diff
==============================================================================
--- incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/common/hbase/do-component-build (original)
+++ incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-hive/manifests/init.pp Mon Oct 31 00:30:01 2011
@@ -1,4 +1,3 @@
-#!/bin/sh
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
@@ -14,6 +13,15 @@
# See the License for the specific language governing permissions and
# limitations under the License.
-set -ex
+class hadoop-hive {
+ define client($hbase_master = "", $hbase_zookeeper_quorum = "") {
+ package { "hadoop-hive":
+ ensure => latest,
+ }
-mvn clean -DskipTests -Dhbase.version=${HBASE_VERSION} -Dhadoop.profile=22 install site assembly:assembly "$@"
+ file { "/etc/hive/conf/hive-site.xml":
+ content => template('hadoop-hive/hive-site.xml'),
+ require => Package["hadoop-hive"],
+ }
+ }
+}
Added: incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-hive/templates/hive-site.xml
URL: http://svn.apache.org/viewvc/incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-hive/templates/hive-site.xml?rev=1195290&view=auto
==============================================================================
--- incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-hive/templates/hive-site.xml (added)
+++ incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-hive/templates/hive-site.xml Mon Oct 31 00:30:01 2011
@@ -0,0 +1,63 @@
+<?xml version="1.0"?>
+<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
+
+<!-- Licensed to the Apache Software Foundation (ASF) under one or more -->
+<!-- contributor license agreements. See the NOTICE file distributed with -->
+<!-- this work for additional information regarding copyright ownership. -->
+<!-- The ASF licenses this file to You under the Apache License, Version 2.0 -->
+<!-- (the "License"); you may not use this file except in compliance with -->
+<!-- the License. You may obtain a copy of the License at -->
+<!-- -->
+<!-- http://www.apache.org/licenses/LICENSE-2.0 -->
+<!-- -->
+<!-- Unless required by applicable law or agreed to in writing, software -->
+<!-- distributed under the License is distributed on an "AS IS" BASIS, -->
+<!-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -->
+<!-- See the License for the specific language governing permissions and -->
+<!-- limitations under the License. -->
+
+<configuration>
+
+<!-- Hive Configuration can either be stored in this file or in the hadoop configuration files -->
+<!-- that are implied by Hadoop setup variables. -->
+<!-- Aside from Hadoop setup variables - this file is provided as a convenience so that Hive -->
+<!-- users do not have to edit hadoop configuration files (that may be managed as a centralized -->
+<!-- resource). -->
+
+<!-- Hive Execution Parameters -->
+
+<% if hbase_master != "" %>
+<property>
+ <name>hbase.master</name>
+ <value><%= hbase_master %></value>
+ <description>http://wiki.apache.org/hadoop/Hive/HBaseIntegration</description>
+</property>
+<% end %>
+
+<% if hbase_zookeeper_quorum != "" %>
+<property>
+ <name>hbase.zookeeper.quorum</name>
+ <value><%= hbase_zookeeper_quorum %></value>
+ <description>http://wiki.apache.org/hadoop/Hive/HBaseIntegration</description>
+</property>
+<% end %>
+
+<property>
+ <name>javax.jdo.option.ConnectionURL</name>
+ <value>jdbc:derby:;databaseName=/var/lib/hive/metastore/metastore_db;create=true</value>
+ <description>JDBC connect string for a JDBC metastore</description>
+</property>
+
+<property>
+ <name>javax.jdo.option.ConnectionDriverName</name>
+ <value>org.apache.derby.jdbc.EmbeddedDriver</value>
+ <description>Driver class name for a JDBC metastore</description>
+</property>
+
+<property>
+ <name>hive.hwi.war.file</name>
+ <value>/usr/lib/hive/lib/hive-hwi-0.7.0-cdh3u1-SNAPSHOT.war</value>
+ <description>This is the WAR file with the jsp content for Hive Web Interface</description>
+</property>
+
+</configuration>
Copied: incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-hive/tests/init.pp (from r1185814, incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/common/hive/do-component-build)
URL: http://svn.apache.org/viewvc/incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-hive/tests/init.pp?p2=incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-hive/tests/init.pp&p1=incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/common/hive/do-component-build&r1=1185814&r2=1195290&rev=1195290&view=diff
==============================================================================
--- incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/common/hive/do-component-build (original)
+++ incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-hive/tests/init.pp Mon Oct 31 00:30:01 2011
@@ -1,4 +1,3 @@
-#!/bin/sh
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
@@ -14,6 +13,4 @@
# See the License for the specific language governing permissions and
# limitations under the License.
-set -ex
-
-ant -f src/build.xml package "$@"
+hadoop-hive::client { "test-hive": }
Copied: incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-oozie/manifests/init.pp (from r1185814, incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/common/pig/do-component-build)
URL: http://svn.apache.org/viewvc/incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-oozie/manifests/init.pp?p2=incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-oozie/manifests/init.pp&p1=incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/common/pig/do-component-build&r1=1185814&r2=1195290&rev=1195290&view=diff
==============================================================================
--- incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/common/pig/do-component-build (original)
+++ incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-oozie/manifests/init.pp Mon Oct 31 00:30:01 2011
@@ -1,4 +1,3 @@
-#!/bin/sh
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
@@ -14,8 +13,29 @@
# See the License for the specific language governing permissions and
# limitations under the License.
-set -ex
+class hadoop-oozie {
+ define client($kerberos_realm = "") {
+ package { "oozie-client":
+ ensure => latest,
+ }
+ }
-ant -Djavac.version=1.6 -Djava5.home=${JAVA5_HOME} -Dforrest.home=${FORREST_HOME} -Ddist.dir=debian/tmp -Dversion=${PIG_BASE_VERSION} package "$@"
+ define server($kerberos_realm = "") {
+ package { "oozie":
+ ensure => latest,
+ }
+ file { "/etc/oozie/oozie-site.xml":
+ content => template("hadoop-oozie/oozie-site.xml"),
+ require => Package["oozie"],
+ }
+ service { "oozie":
+ ensure => running,
+ require => Package["oozie"],
+ hasrestart => true,
+ hasstatus => true,
+ }
+
+ }
+}
Added: incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-oozie/templates/oozie-site.xml
URL: http://svn.apache.org/viewvc/incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-oozie/templates/oozie-site.xml?rev=1195290&view=auto
==============================================================================
--- incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-oozie/templates/oozie-site.xml (added)
+++ incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-oozie/templates/oozie-site.xml Mon Oct 31 00:30:01 2011
@@ -0,0 +1,298 @@
+<?xml version="1.0"?>
+<!--
+ Copyright (c) 2010 Yahoo! Inc. All rights reserved.
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+<configuration>
+
+<!--
+ Refer to the oozie-default.xml file for the complete list of
+ Oozie configuration properties and their default values.
+-->
+
+ <property>
+ <name>oozie.system.id</name>
+ <value>oozie-${user.name}</value>
+ <description>
+ The Oozie system ID.
+ </description>
+ </property>
+
+ <property>
+ <name>oozie.systemmode</name>
+ <value>NORMAL</value>
+ <description>
+ System mode for Oozie at startup.
+ </description>
+ </property>
+
+ <property>
+ <name>oozie.service.AuthorizationService.security.enabled</name>
+ <value>false</value>
+ <description>
+ Specifies whether security (user name/admin role) is enabled or not.
+ If disabled any user can manage Oozie system and manage any job.
+ </description>
+ </property>
+
+ <property>
+ <name>oozie.service.PurgeService.older.than</name>
+ <value>30</value>
+ <description>
+ Jobs older than this value, in days, will be purged by the PurgeService.
+ </description>
+ </property>
+
+ <property>
+ <name>oozie.service.PurgeService.purge.interval</name>
+ <value>3600</value>
+ <description>
+ Interval at which the purge service will run, in seconds.
+ </description>
+ </property>
+
+ <property>
+ <name>oozie.service.CallableQueueService.queue.size</name>
+ <value>10000</value>
+ <description>Max callable queue size</description>
+ </property>
+
+ <property>
+ <name>oozie.service.CallableQueueService.threads</name>
+ <value>10</value>
+ <description>Number of threads used for executing callables</description>
+ </property>
+
+ <property>
+ <name>oozie.service.CallableQueueService.callable.concurrency</name>
+ <value>3</value>
+ <description>
+ Maximum concurrency for a given callable type.
+ Each command is a callable type (submit, start, run, signal, job, jobs, suspend,resume, etc).
+ Each action type is a callable type (Map-Reduce, Pig, SSH, FS, sub-workflow, etc).
+ All commands that use action executors (action-start, action-end, action-kill and action-check) use
+ the action type as the callable type.
+ </description>
+ </property>
+
+ <property>
+ <name>oozie.service.coord.normal.default.timeout
+ </name>
+ <value>120</value>
+ <description>Default timeout for a coordinator action input check (in minutes) for normal job.
+ -1 means infinite timeout</description>
+ </property>
+
+ <property>
+ <name>oozie.db.schema.name</name>
+ <value>oozie</value>
+ <description>
+ Oozie DataBase Name
+ </description>
+ </property>
+
+ <property>
+ <name>oozie.service.StoreService.create.db.schema</name>
+ <value>true</value>
+ <description>
+ Creates Oozie DB.
+
+ If set to true, it creates the DB schema if it does not exist. If the DB schema exists is a NOP.
+ If set to false, it does not create the DB schema. If the DB schema does not exist it fails start up.
+ </description>
+ </property>
+
+ <property>
+ <name>oozie.service.StoreService.jdbc.driver</name>
+ <value>org.apache.derby.jdbc.EmbeddedDriver</value>
+ <description>
+ JDBC driver class.
+ </description>
+ </property>
+
+ <property>
+ <name>oozie.service.StoreService.jdbc.url</name>
+ <value>jdbc:derby:${oozie.data.dir}/${oozie.db.schema.name}-db;create=true</value>
+ <description>
+ JDBC URL.
+ </description>
+ </property>
+
+ <property>
+ <name>oozie.service.StoreService.jdbc.username</name>
+ <value>sa</value>
+ <description>
+ DB user name.
+ </description>
+ </property>
+
+ <property>
+ <name>oozie.service.StoreService.jdbc.password</name>
+ <value> </value>
+ <description>
+ DB user password.
+
+ IMPORTANT: if password is emtpy leave a 1 space string, the service trims the value,
+ if empty Configuration assumes it is NULL.
+
+ IMPORTANT: if the StoreServicePasswordService is active, it will reset this value with the value given in
+ the console.
+ </description>
+ </property>
+
+ <property>
+ <name>oozie.service.StoreService.pool.max.active.conn</name>
+ <value>10</value>
+ <description>
+ Max number of connections.
+ </description>
+ </property>
+
+ <property>
+ <name>oozie.service.HadoopAccessorService.kerberos.enabled</name>
+ <value><%= if (kerberos_realm != "") ; "true" else "false" end %></value>
+ <description>
+ Indicates if Oozie is configured to use Kerberos.
+ </description>
+ </property>
+
+ <property>
+ <name>local.realm</name>
+ <value><%= kerberos_realm %></value>
+ <description>
+ Kerberos Realm used by Oozie and Hadoop. Using 'local.realm' to be aligned with Hadoop configuration
+ </description>
+ </property>
+
+ <property>
+ <name>oozie.service.HadoopAccessorService.keytab.file</name>
+ <value>/etc/oozie.keytab</value>
+ <description>
+ Location of the Oozie user keytab file.
+ </description>
+ </property>
+
+ <property>
+ <name>oozie.service.HadoopAccessorService.kerberos.principal</name>
+ <value>${user.name}/<%= fqdn %>@${local.realm}</value>
+ <description>
+ Kerberos principal for Oozie service.
+ </description>
+ </property>
+
+ <property>
+ <name>oozie.service.HadoopAccessorService.jobTracker.whitelist</name>
+ <value> </value>
+ <description>
+ Whitelisted job tracker for Oozie service.
+ </description>
+ </property>
+
+ <property>
+ <name>oozie.service.HadoopAccessorService.nameNode.whitelist</name>
+ <value> </value>
+ <description>
+ Whitelisted job tracker for Oozie service.
+ </description>
+ </property>
+
+ <property>
+ <name>oozie.service.WorkflowAppService.system.libpath</name>
+ <value>/user/${user.name}/share/lib</value>
+ <description>
+ System library path to use for workflow applications.
+ This path is added to workflow application if their job properties sets
+ the property 'oozie.use.system.libpath' to true.
+ </description>
+ </property>
+
+ <property>
+ <name>use.system.libpath.for.mapreduce.and.pig.jobs</name>
+ <value>false</value>
+ <description>
+ If set to true, submissions of MapReduce and Pig jobs will include
+ automatically the system library path, thus not requiring users to
+ specify where the Pig JAR files are. Instead, the ones from the system
+ library path are used.
+ </description>
+ </property>
+
+ <property>
+ <name>oozie.authentication.type</name>
+ <value>simple</value>
+ <description>
+ Defines authentication used for Oozie HTTP endpoint.
+ Supported values are: simple | kerberos | #AUTHENTICATION_HANDLER_CLASSNAME#
+ </description>
+ </property>
+
+ <property>
+ <name>oozie.authentication.token.validity</name>
+ <value>36000</value>
+ <description>
+ Indicates how long (in seconds) an authentication token is valid before it has
+ to be renewed.
+ </description>
+ </property>
+
+ <property>
+ <name>oozie.authentication.signature.secret</name>
+ <value>oozie</value>
+ <description>
+ The signature secret for signing the authentication tokens.
+ If not set a random secret is generated at startup time.
+ In order to authentiation to work correctly across multiple hosts
+ the secret must be the same across al the hosts.
+ </description>
+ </property>
+
+ <property>
+ <name>oozie.authentication.cookie.domain</name>
+ <value></value>
+ <description>
+ The domain to use for the HTTP cookie that stores the authentication token.
+ In order to authentiation to work correctly across multiple hosts
+ the domain must be correctly set.
+ </description>
+ </property>
+
+ <property>
+ <name>oozie.authentication.simple.anonymous.allowed</name>
+ <value>true</value>
+ <description>
+ Indicates if anonymous requests are allowed.
+ This setting is meaningful only when using 'simple' authentication.
+ </description>
+ </property>
+
+ <property>
+ <name>oozie.authentication.kerberos.principal</name>
+ <value>HTTP/<%= fqdn %>@${local.realm}</value>
+ <description>
+ Indicates the Kerberos principal to be used for HTTP endpoint.
+ The principal MUST start with 'HTTP/' as per Kerberos HTTP SPNEGO specification.
+ </description>
+ </property>
+
+ <property>
+ <name>oozie.authentication.kerberos.keytab</name>
+ <value>${oozie.service.HadoopAccessorService.keytab.file}</value>
+ <description>
+ Location of the keytab file with the credentials for the principal.
+ Referring to the same keytab file Oozie uses for its Kerberos credentials for Hadoop.
+ </description>
+ </property>
+
+</configuration>
+
Copied: incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-oozie/tests/init.pp (from r1185814, incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/common/hive/do-component-build)
URL: http://svn.apache.org/viewvc/incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-oozie/tests/init.pp?p2=incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-oozie/tests/init.pp&p1=incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/common/hive/do-component-build&r1=1185814&r2=1195290&rev=1195290&view=diff
==============================================================================
--- incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/common/hive/do-component-build (original)
+++ incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-oozie/tests/init.pp Mon Oct 31 00:30:01 2011
@@ -1,4 +1,3 @@
-#!/bin/sh
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
@@ -14,6 +13,5 @@
# See the License for the specific language governing permissions and
# limitations under the License.
-set -ex
-
-ant -f src/build.xml package "$@"
+hadoop-oozie::server { "test-oozie-server": }
+hadoop-oozie::client { "test-oozie": }
Copied: incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-pig/manifests/init.pp (from r1185814, incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/common/hbase/do-component-build)
URL: http://svn.apache.org/viewvc/incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-pig/manifests/init.pp?p2=incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-pig/manifests/init.pp&p1=incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/common/hbase/do-component-build&r1=1185814&r2=1195290&rev=1195290&view=diff
==============================================================================
--- incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/common/hbase/do-component-build (original)
+++ incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-pig/manifests/init.pp Mon Oct 31 00:30:01 2011
@@ -1,4 +1,3 @@
-#!/bin/sh
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
@@ -14,6 +13,19 @@
# See the License for the specific language governing permissions and
# limitations under the License.
-set -ex
-
-mvn clean -DskipTests -Dhbase.version=${HBASE_VERSION} -Dhadoop.profile=22 install site assembly:assembly "$@"
+class hadoop-pig {
+ define client {
+ package { "hadoop-pig":
+ ensure => latest,
+ require => Package["hadoop"],
+ }
+
+ file { "/etc/pig/conf/pig.properties":
+ content => template('hadoop-pig/pig.properties'),
+ require => Package["hadoop-pig"],
+ owner => "root", /* FIXME: I'm really no sure about these -- we might end */
+ mode => "755", /* up deploying/testing a different thing compared */
+ /* to a straight rpm/deb deployment */
+ }
+ }
+}
Added: incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-pig/templates/pig.properties
URL: http://svn.apache.org/viewvc/incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-pig/templates/pig.properties?rev=1195290&view=auto
==============================================================================
--- incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-pig/templates/pig.properties (added)
+++ incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-pig/templates/pig.properties Mon Oct 31 00:30:01 2011
@@ -0,0 +1,70 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+# Pig configuration file. All values can be overwritten by command line arguments.
+# see bin/pig -help
+
+# log4jconf log4j configuration file
+# log4jconf=./conf/log4j.properties
+
+# brief logging (no timestamps)
+brief=false
+
+# clustername, name of the hadoop jobtracker. If no port is defined port 50020 will be used.
+#cluster
+
+#debug level, INFO is default
+debug=INFO
+
+# a file that contains pig script
+#file=
+
+# load jarfile, colon separated
+#jar=
+
+#verbose print all log messages to screen (default to print only INFO and above to screen)
+verbose=false
+
+#exectype local|mapreduce, mapreduce is default
+#exectype=mapreduce
+# hod realted properties
+#ssh.gateway
+#hod.expect.root
+#hod.expect.uselatest
+#hod.command
+#hod.config.dir
+#hod.param
+
+
+#Do not spill temp files smaller than this size (bytes)
+pig.spill.size.threshold=5000000
+#EXPERIMENT: Activate garbage collection when spilling a file bigger than this size (bytes)
+#This should help reduce the number of files being spilled.
+pig.spill.gc.activation.size=40000000
+
+
+######################
+# Everything below this line is Yahoo specific. Note that I've made
+# (almost) no changes to the lines above to make merging in from Apache
+# easier. Any values I don't want from above I override below.
+#
+# This file is configured for use with HOD on the production clusters. If you
+# want to run pig with a static cluster you will need to remove everything
+# below this line and set the cluster value (above) to the
+# hostname and port of your job tracker.
+
+exectype=mapreduce
+log.file=
Copied: incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-pig/tests/init.pp (from r1185814, incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/common/hive/do-component-build)
URL: http://svn.apache.org/viewvc/incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-pig/tests/init.pp?p2=incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-pig/tests/init.pp&p1=incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/common/hive/do-component-build&r1=1185814&r2=1195290&rev=1195290&view=diff
==============================================================================
--- incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/common/hive/do-component-build (original)
+++ incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-pig/tests/init.pp Mon Oct 31 00:30:01 2011
@@ -1,4 +1,3 @@
-#!/bin/sh
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
@@ -14,6 +13,5 @@
# See the License for the specific language governing permissions and
# limitations under the License.
-set -ex
-
-ant -f src/build.xml package "$@"
+include pig
+pig::client { "test-pig": }
Copied: incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-sqoop/manifests/init.pp (from r1185814, incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/common/pig/do-component-build)
URL: http://svn.apache.org/viewvc/incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-sqoop/manifests/init.pp?p2=incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-sqoop/manifests/init.pp&p1=incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/common/pig/do-component-build&r1=1185814&r2=1195290&rev=1195290&view=diff
==============================================================================
--- incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/common/pig/do-component-build (original)
+++ incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-sqoop/manifests/init.pp Mon Oct 31 00:30:01 2011
@@ -1,4 +1,3 @@
-#!/bin/sh
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
@@ -14,8 +13,30 @@
# See the License for the specific language governing permissions and
# limitations under the License.
-set -ex
+class hadoop-sqoop {
+ define client {
+ package { "sqoop":
+ ensure => latest,
+ }
-ant -Djavac.version=1.6 -Djava5.home=${JAVA5_HOME} -Dforrest.home=${FORREST_HOME} -Ddist.dir=debian/tmp -Dversion=${PIG_BASE_VERSION} package "$@"
+ # FIXME: this is NOT supposed to be needed
+ # but for now that's the easiest way to add
+ # extra jars to sqoop classpath
+ file { "/usr/lib/sqoop/lib/mysql-connector-java-5.1.12.jar":
+ ensure => "/usr/share/java/mysql-connector-java-5.1.12.jar",
+ }
+ }
+ define metastore {
+ package { "sqoop-metastore":
+ ensure => latest,
+ }
+ service { "sqoop-metastore":
+ ensure => running,
+ require => Package["sqoop-metastore"],
+ hasstatus => true,
+ hasrestart => true,
+ }
+ }
+}
Copied: incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-sqoop/tests/init.pp (from r1185814, incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/common/hive/do-component-build)
URL: http://svn.apache.org/viewvc/incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-sqoop/tests/init.pp?p2=incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-sqoop/tests/init.pp&p1=incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/common/hive/do-component-build&r1=1185814&r2=1195290&rev=1195290&view=diff
==============================================================================
--- incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/common/hive/do-component-build (original)
+++ incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-sqoop/tests/init.pp Mon Oct 31 00:30:01 2011
@@ -1,4 +1,3 @@
-#!/bin/sh
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
@@ -14,6 +13,5 @@
# See the License for the specific language governing permissions and
# limitations under the License.
-set -ex
-
-ant -f src/build.xml package "$@"
+hadoop-sqoop::client { "test-sqoop": }
+hadoop-sqoop::metastore { "test-sqoop-metastore": }
Added: incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-zookeeper/manifests/init.pp
URL: http://svn.apache.org/viewvc/incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-zookeeper/manifests/init.pp?rev=1195290&view=auto
==============================================================================
--- incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-zookeeper/manifests/init.pp (added)
+++ incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-zookeeper/manifests/init.pp Mon Oct 31 00:30:01 2011
@@ -0,0 +1,46 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+class hadoop-zookeeper {
+ define client {
+ package { "hadoop-zookeeper":
+ ensure => latest,
+ }
+ }
+
+ define server($myid, $ensemble = ["localhost:2888:3888"]) {
+ package { "hadoop-zookeeper-server":
+ ensure => latest,
+ }
+
+ service { "hadoop-zookeeper-server":
+ ensure => running,
+ require => Package["hadoop-zookeeper-server"],
+ subscribe => File["/etc/zookeeper/conf/zoo.cfg"],
+ hasrestart => true,
+ hasstatus => true,
+ }
+
+ file { "/etc/zookeeper/conf/zoo.cfg":
+ content => template("hadoop-zookeeper/zoo.cfg"),
+ require => Package["hadoop-zookeeper-server"],
+ }
+
+ file { "/var/lib/zookeeper/myid":
+ content => inline_template("<%= myid %>"),
+ require => Package["hadoop-zookeeper-server"],
+ }
+ }
+}
Copied: incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-zookeeper/templates/zoo.cfg (from r1185814, incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/common/hbase/do-component-build)
URL: http://svn.apache.org/viewvc/incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-zookeeper/templates/zoo.cfg?p2=incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-zookeeper/templates/zoo.cfg&p1=incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/common/hbase/do-component-build&r1=1185814&r2=1195290&rev=1195290&view=diff
==============================================================================
--- incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/common/hbase/do-component-build (original)
+++ incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-zookeeper/templates/zoo.cfg Mon Oct 31 00:30:01 2011
@@ -1,4 +1,3 @@
-#!/bin/sh
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
@@ -14,6 +13,19 @@
# See the License for the specific language governing permissions and
# limitations under the License.
-set -ex
-
-mvn clean -DskipTests -Dhbase.version=${HBASE_VERSION} -Dhadoop.profile=22 install site assembly:assembly "$@"
+maxClientCnxns=50
+# The number of milliseconds of each tick
+tickTime=2000
+# The number of ticks that the initial
+# synchronization phase can take
+initLimit=10
+# The number of ticks that can pass between
+# sending a request and getting an acknowledgement
+syncLimit=5
+# the directory where the snapshot is stored.
+dataDir=/var/lib/zookeeper
+# the port at which the clients will connect
+clientPort=2181
+<% ensemble.each_with_index do |server,idx| %>
+server.<%= idx %>=<%= server %>
+<% end %>
Copied: incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-zookeeper/tests/init.pp (from r1185814, incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/common/hbase/do-component-build)
URL: http://svn.apache.org/viewvc/incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-zookeeper/tests/init.pp?p2=incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-zookeeper/tests/init.pp&p1=incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/common/hbase/do-component-build&r1=1185814&r2=1195290&rev=1195290&view=diff
==============================================================================
--- incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/common/hbase/do-component-build (original)
+++ incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop-zookeeper/tests/init.pp Mon Oct 31 00:30:01 2011
@@ -1,4 +1,3 @@
-#!/bin/sh
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
@@ -14,6 +13,8 @@
# See the License for the specific language governing permissions and
# limitations under the License.
-set -ex
-
-mvn clean -DskipTests -Dhbase.version=${HBASE_VERSION} -Dhadoop.profile=22 install site assembly:assembly "$@"
+hadoop-zookeeper::client { "zoo visitor": }
+hadoop-zookeeper::server { "test-oozie-server":
+ myid => "0",
+ ensemble => ["foo:2888:3888", "bar:2888:3888", "baz:2888:3888"],
+}
Copied: incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop/lib/facter/hadoop_storage_locations.rb (from r1185814, incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/deb/sqoop/control)
URL: http://svn.apache.org/viewvc/incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop/lib/facter/hadoop_storage_locations.rb?p2=incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop/lib/facter/hadoop_storage_locations.rb&p1=incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/deb/sqoop/control&r1=1185814&r2=1195290&rev=1195290&view=diff
==============================================================================
--- incubator/bigtop/branches/hadoop-0.22/bigtop-packages/src/deb/sqoop/control (original)
+++ incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop/lib/facter/hadoop_storage_locations.rb Mon Oct 31 00:30:01 2011
@@ -12,24 +12,29 @@
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
-Source: sqoop
-Section: misc
-Priority: extra
-Maintainer: Alex Newman <ne...@cloudera.com>
-Build-Depends: debhelper (>= 6), ant, ant-optional, git-core, sed, asciidoc, xmlto
-Standards-Version: 3.8.0
-Homepage: http://www.cloudera.com
-
-Package: sqoop
-Architecture: all
-Depends: hadoop
-Description: Tool for easy imports and exports of data sets between databases and HDFS
- Sqoop is a tool that provides the ability to import and export data sets between
- the Hadoop Distributed File System (HDFS) and relational databases.
-
-Package: sqoop-metastore
-Architecture: all
-Depends: sqoop (= ${source:Version}), adduser
-Description: Shared metadata repository for Sqoop.
- This optional package hosts a metadata server for Sqoop clients across a network to use.
+
+# We make the assumption hadoop's data files will be located in /data/
+# Puppet needs to know where they are
+Facter.add("hadoop_storage_locations") do
+ setcode do
+
+ data_dir_path = "/data/"
+ storage_locations = ""
+
+ # We need to check /data/ exist
+ if File.directory?(data_dir_path)
+
+ # We assume all data directory will be a number
+ Dir.foreach(data_dir_path) { |directory|
+ storage_locations += (data_dir_path + directory + ';') if directory =~ /\d+/
+ }
+ end
+
+ # Return the list of storage locations for hadoop
+ if storage_locations == ""
+ storage_locations = "/mnt"
+ end
+ storage_locations
+ end
+end
Added: incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop/manifests/.init.pp.swp
URL: http://svn.apache.org/viewvc/incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop/manifests/.init.pp.swp?rev=1195290&view=auto
==============================================================================
Files incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop/manifests/.init.pp.swp (added) and incubator/bigtop/branches/hadoop-0.22/bigtop-deploy/puppet/modules/hadoop/manifests/.init.pp.swp Mon Oct 31 00:30:01 2011 differ