You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@bigtop.apache.org by ev...@apache.org on 2017/07/06 15:21:49 UTC
[1/3] bigtop git commit: BIGTOP-2832. Toolchain failed to install on
Debian 8
Repository: bigtop
Updated Branches:
refs/heads/branch-1.2 199e7b2ca -> b3bd55846
BIGTOP-2832. Toolchain failed to install on Debian 8
Project: http://git-wip-us.apache.org/repos/asf/bigtop/repo
Commit: http://git-wip-us.apache.org/repos/asf/bigtop/commit/b3bd5584
Tree: http://git-wip-us.apache.org/repos/asf/bigtop/tree/b3bd5584
Diff: http://git-wip-us.apache.org/repos/asf/bigtop/diff/b3bd5584
Branch: refs/heads/branch-1.2
Commit: b3bd55846914a6a5d4ceae02c1c7f51904f99246
Parents: acce540
Author: Evans Ye <ev...@apache.org>
Authored: Wed Jul 5 14:42:54 2017 +0800
Committer: Evans Ye <ev...@apache.org>
Committed: Thu Jul 6 23:21:33 2017 +0800
----------------------------------------------------------------------
bigtop_toolchain/manifests/jdk.pp | 17 ++++++++++++++++-
bigtop_toolchain/manifests/packages.pp | 2 +-
2 files changed, 17 insertions(+), 2 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/bigtop/blob/b3bd5584/bigtop_toolchain/manifests/jdk.pp
----------------------------------------------------------------------
diff --git a/bigtop_toolchain/manifests/jdk.pp b/bigtop_toolchain/manifests/jdk.pp
index 8c1518b..4e15eb1 100644
--- a/bigtop_toolchain/manifests/jdk.pp
+++ b/bigtop_toolchain/manifests/jdk.pp
@@ -17,8 +17,23 @@ class bigtop_toolchain::jdk {
case $::operatingsystem {
/Debian/: {
require apt
- unless $os[release][major] > "8" {
+ unless $operatingsystemmajrelease > "8" {
+ # we pin openjdk-8-* and ca-certificates-java to backports
require apt::backports
+
+ apt::pin { 'backports_jdk':
+ packages => 'openjdk-8-*',
+ priority => 500,
+ release => 'jessie-backports',
+ } ->
+ apt::pin { 'backports_ca':
+ packages => 'ca-certificates-java',
+ priority => 500,
+ release => 'jessie-backports',
+ } ->
+ exec {'own_update':
+ command => '/usr/bin/apt-get update'
+ } -> Package['openjdk-8-jdk']
}
package { 'openjdk-8-jdk' :
http://git-wip-us.apache.org/repos/asf/bigtop/blob/b3bd5584/bigtop_toolchain/manifests/packages.pp
----------------------------------------------------------------------
diff --git a/bigtop_toolchain/manifests/packages.pp b/bigtop_toolchain/manifests/packages.pp
index 6101e35..7a15d3b 100644
--- a/bigtop_toolchain/manifests/packages.pp
+++ b/bigtop_toolchain/manifests/packages.pp
@@ -159,7 +159,7 @@ class bigtop_toolchain::packages {
] }
/(Ubuntu|Debian)/: {
# Debian-9 is using mariadb instead of mysql
- if ($operatingsystem == "Debian") and ($os[release][major] > "8") {
+ if ($operatingsystem == "Debian") and ($operatingsystemmajrelease > "8") {
$mysql_dev="libmariadb-dev"
} else {
$mysql_dev="libmysqlclient-dev"
[2/3] bigtop git commit: BIGTOP-2829. [iTest] build failed during
Maven integration test phase
Posted by ev...@apache.org.
BIGTOP-2829. [iTest] build failed during Maven integration test phase
Project: http://git-wip-us.apache.org/repos/asf/bigtop/repo
Commit: http://git-wip-us.apache.org/repos/asf/bigtop/commit/acce5407
Tree: http://git-wip-us.apache.org/repos/asf/bigtop/tree/acce5407
Diff: http://git-wip-us.apache.org/repos/asf/bigtop/diff/acce5407
Branch: refs/heads/branch-1.2
Commit: acce5407bc5dc47fcfe0b7c7dd6981cd9d8e9a33
Parents: 26bb157
Author: Evans Ye <ev...@apache.org>
Authored: Sun Jul 2 22:06:40 2017 +0800
Committer: Evans Ye <ev...@apache.org>
Committed: Thu Jul 6 23:21:33 2017 +0800
----------------------------------------------------------------------
bigtop-test-framework/pom.xml | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/bigtop/blob/acce5407/bigtop-test-framework/pom.xml
----------------------------------------------------------------------
diff --git a/bigtop-test-framework/pom.xml b/bigtop-test-framework/pom.xml
index 62d1ba7..bb85def 100644
--- a/bigtop-test-framework/pom.xml
+++ b/bigtop-test-framework/pom.xml
@@ -79,7 +79,7 @@
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-failsafe-plugin</artifactId>
- <version>2.17</version>
+ <version>2.20</version>
<configuration>
<includes>
<include>**/*IntegrationTest*</include>
[3/3] bigtop git commit: BIGTOP-2819. Polish the README.md for 1.2.1
release
Posted by ev...@apache.org.
BIGTOP-2819. Polish the README.md for 1.2.1 release
Project: http://git-wip-us.apache.org/repos/asf/bigtop/repo
Commit: http://git-wip-us.apache.org/repos/asf/bigtop/commit/26bb157c
Tree: http://git-wip-us.apache.org/repos/asf/bigtop/tree/26bb157c
Diff: http://git-wip-us.apache.org/repos/asf/bigtop/diff/26bb157c
Branch: refs/heads/branch-1.2
Commit: 26bb157c3d0d09bf628f0d169130dabc2543c12e
Parents: 199e7b2
Author: Evans Ye <ev...@apache.org>
Authored: Sun Jul 2 12:05:47 2017 +0800
Committer: Evans Ye <ev...@apache.org>
Committed: Thu Jul 6 23:21:33 2017 +0800
----------------------------------------------------------------------
README.md | 32 ++++++++++++++++----------------
1 file changed, 16 insertions(+), 16 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/bigtop/blob/26bb157c/README.md
----------------------------------------------------------------------
diff --git a/README.md b/README.md
index 40da1c7..5715c96 100644
--- a/README.md
+++ b/README.md
@@ -24,18 +24,20 @@ The primary goal of Apache Bigtop is to build a community around the packaging a
Immediately Get Started with Deployment and Smoke Testing of BigTop
===================================================================
-The simplest way to get a feel for how bigtop works, is to just cd into `bigtop-deploy/vm` and try out the recipes under vagrant-puppet-vm, vagrant-puppet-docker, and so on. Each one rapidly spins up, and runs the bigtop smoke tests on, a local bigtop based big data distribution. Once you get the gist, you can hack around with the recipes to learn how the puppet/rpm/smoke-tests all work together, going deeper into the components you are interested in as described below.
+The simplest way to get a feel for how bigtop works, is to just cd into `provisioner` and try out the recipes under vagrant or docker. Each one rapidly spins up, and runs the bigtop smoke tests on, a local bigtop based big data distribution. Once you get the gist, you can hack around with the recipes to learn how the puppet/rpm/smoke-tests all work together, going deeper into the components you are interested in as described below.
Quick overview of source code directories
=========================================
-* __bigtop-deploy__ : deployment scripts, puppet stuff, VM utilities for Apache Bigtop.
-* __bigtop-packages__ : RPM/DEB specifications for Apache Bigtop subcomponents
+* __bigtop-deploy__ : deployment scripts and puppet stuff for Apache Bigtop.
+* __bigtop-packages__ : RPM/DEB specifications for Apache Bigtop subcomponents.
* __bigtop-test-framework__ : The source code for the iTest utilities (framework used by smoke tests).
* __bigtop-tests__ :
* __test-artifacts__ : source for tests.
* __test-execution__ : maven pom drivers for running the integration tests found in test-artifacts.
* __bigtop-toolchain__ : puppet scripts for setting up an instance which can build Apache Bigtop, sets up utils like jdk/maven/protobufs/...
+* __provisioner__ : Vagrant and Docker Provisioner that automatically spin up Hadoop environment with one click.
+* __docker__ : Dockerfiles and Docker Sandbox build scripts.
Also, there is a new project underway, Apache Bigtop blueprints, which aims to create templates/examples that demonstrate/compare various Apache Hadoop ecosystem components with one another.
@@ -91,17 +93,17 @@ You can go to the [Apache Bigtop website](http://bigtop.apache.org/) for notes o
Getting Started
===============
-Below are some recipes for getting started with using Apache Bigtop. As Apache Bigtop has different subprojects, these recipes will continue to evolve.
+Below are some recipes for getting started with using Apache Bigtop. As Apache Bigtop has different subprojects, these recipes will continue to evolve.
For specific questions it's always a good idea to ping the mailing list at dev-subscribe@bigtop.apache.org to get some immediate feedback, or [open a JIRA](https://issues.apache.org/jira/browse/BIGTOP).
-For Users: Running the smoke tests.
+For Users: Running the smoke tests
-----------------------------------
The simplest way to test bigtop is described in bigtop-tests/smoke-tests/README file
For integration (API level) testing with maven, read on.
-For Users: Running the integration tests.
+For Users: Running the integration tests
-----------------------------------------
WARNING: since testing packages requires installing them on a live system it is highly recommended to use VMs for that. Testing Apache Bigtop is done using iTest framework. The tests are organized in maven submodules, with one submodule per Apache Bigtop component. The bigtop-tests/test-execution/smokes/pom.xml defines all submodules to be tested, and each submodule is in its own directory under smokes/, for example:
@@ -159,15 +161,13 @@ For Users: Creating Your Own Apache Hadoop Environment
Another common use case for Apache Bigtop is creating / setting up your own Apache Hadoop distribution.
For details on this, check out the bigtop-deploy/README.md file, which describes how to use the puppet repos
-to create and setup your VMs.
-There is a current effort underway to create vagrant/docker recipes as well, which will be contained in the
-bigtop-deploy/ package.
-
+to create and setup your VMs.
+You can also try out provisioner to quickly get the idea how it works.
For Developers: Building the entire distribution from scratch
-------------------------------------------------------------
-Packages have been built for CentOS/RHEL 5 and 6, Fedora 18, SuSE Linux Enterprise 11, OpenSUSE12.2, Ubuntu LTS Lucid and Precise, and Ubuntu Quantal. They can probably be built for other platforms as well. Some of the binary artifacts might be compatible with other closely related distributions.
+Packages have been built for CentOS, Fedora, OpenSUSE, Ubuntu, and Debian. They can probably be built for other platforms as well. Some of the binary artifacts might be compatible with other closely related distributions.
__On all systems, Building Apache Bigtop requires certain set of tools__
@@ -175,14 +175,14 @@ __On all systems, Building Apache Bigtop requires certain set of tools__
./gradlew toolchain
- This build task expected Puppet 3.x to be installed; user has to have sudo permissions. The task will pull down and install
+ This build task expected Puppet to be installed; user has to have sudo permissions. The task will pull down and install
all development dependencies, frameworks and SDKs, required to build the stack on your platform.
To immediately set environment after running toolchain, run
. /etc/profile.d/bigtop.sh
-* __Building packages__ : `gradle [component-name]-[rpm|deb]`
+* __Building packages__ : `gradle [component-name]-pkg`
If -Dbuildwithdeps=true is set, the Gradle will follow the order of the build specified in
the "dependencies" section of bigtop.bom file. Otherwise just a single component will get build (original behavior).
@@ -191,7 +191,7 @@ __On all systems, Building Apache Bigtop requires certain set of tools__
name with -Dbomfile=<filename> system property in the build time.
You can visualize all tasks dependencies by running `gradle tasks --all`
-* __Building local YUM/APT repositories__ : `gradle [component-name]-[yum|apt]`
+* __Building local YUM/APT repositories__ : `gradle [yum|apt]`
* __Recommended build environments__
@@ -200,10 +200,10 @@ __On all systems, Building Apache Bigtop requires certain set of tools__
environment configured and cached. All currently supported OSes could be pulled
from official Bigtop repository at https://hub.docker.com/r/bigtop/slaves/tags/
- To build a component (bigtop-groovy) for a particular OS (ubuntu-14.04) you can
+ To build a component (bigtop-groovy) for a particular OS (ubuntu-16.04) you can
run the following from a clone of Bigtop workspace (assuming your system has
Docker engine setup and working)
- ```docker run --rm -u jenkins:jenkins -v `pwd`:/ws --workdir /ws bigtop/slaves:trunk-ubuntu-14.04
+ ```docker run --rm -u jenkins:jenkins -v `pwd`:/ws --workdir /ws bigtop/slaves:trunk-ubuntu-16.04
bash -l -c './gradlew allclean ; ./gradlew bigtop-groovy-pkg'```
For Developers: Building and modifying the web site