You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by sp...@apache.org on 2017/05/16 20:59:13 UTC
cassandra-builds git commit: Unify version handling for docker based
package building
Repository: cassandra-builds
Updated Branches:
refs/heads/master 81ab99ead -> f3fdb2bd9
Unify version handling for docker based package building
Build either snapshot or release packages based on specified git path.
Releases will only be build from tags. Provided changes will simplify
manual invocation of docker for developers, as well as making it
possible to automatically build packages in Jenkins at some point.
Closes #3
Project: http://git-wip-us.apache.org/repos/asf/cassandra-builds/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra-builds/commit/f3fdb2bd
Tree: http://git-wip-us.apache.org/repos/asf/cassandra-builds/tree/f3fdb2bd
Diff: http://git-wip-us.apache.org/repos/asf/cassandra-builds/diff/f3fdb2bd
Branch: refs/heads/master
Commit: f3fdb2bd94b0128a90fabec6809d7397fa9df8b5
Parents: 81ab99e
Author: Stefan Podkowinski <s....@gmail.com>
Authored: Tue May 16 20:20:11 2017 +0200
Committer: Stefan Podkowinski <s....@gmail.com>
Committed: Tue May 16 22:57:55 2017 +0200
----------------------------------------------------------------------
README.md | 11 +++-
build-scripts/cassandra-deb-packaging.sh | 16 ++++++
build-scripts/cassandra-rpm-packaging.sh | 17 ++++++
docker/build-debs.sh | 75 ++++++++++++++++++++++++---
docker/build-rpms.sh | 60 ++++++++++++++++++---
docker/centos7-image.docker | 3 +-
docker/jessie-image.docker | 3 +-
7 files changed, 169 insertions(+), 16 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/cassandra-builds/blob/f3fdb2bd/README.md
----------------------------------------------------------------------
diff --git a/README.md b/README.md
index f483b17..3b77fdd 100644
--- a/README.md
+++ b/README.md
@@ -15,14 +15,21 @@
```docker build -f docker/jessie-image.docker docker/```
* RPM:
```docker build -f docker/centos7-image.docker docker/```
+ The image will contain a clone of the Apache git repository by default. Using a different repository is possible by adding the `--build-arg CASSANDRA_GIT_URL=https://github.com/myuser/cassandra.git` parameter. All successive builds will be executed based on the repository cloned during docker image creation.
2. Run build script through docker (specify branch, e.g. cassandra-3.0 and version, e.g. 3.0.11):
* Debian:
- ```docker run -v `pwd`/dist:/dist `docker images -f label=org.cassandra.buildenv=jessie -q` /home/build/build-debs.sh <branch>```
+ ```docker run --rm -v `pwd`/dist:/dist `docker images -f label=org.cassandra.buildenv=jessie -q` /home/build/build-debs.sh <branch/tag>```
* RPM:
- ```docker run -v `pwd`/dist:/dist `docker images -f label=org.cassandra.buildenv=centos -q` /home/build/build-rpms.sh <branch> <version>```
+ ```docker run --rm -v `pwd`/dist:/dist `docker images -f label=org.cassandra.buildenv=centos -q` /home/build/build-rpms.sh <branch/tag>```
You should find newly created Debian and RPM packages in the `dist` directory.
+### Note about versioning
+
+Packages for official releases can only be build from tags. In this case, the tag must match the known versioning scheme. A number of sanity checks will be run to make sure the version matches any version defined in `build.xml` and `debian/changes`. But you'll have to manually keep these values in sync for every release.
+
+Builds based on any branch will use the version defined in either `build.xml` (RPM) or `debian/changes` (deb). Afterwards a snapshot indicator will be appended.
+
## Publishing packages
TODO
http://git-wip-us.apache.org/repos/asf/cassandra-builds/blob/f3fdb2bd/build-scripts/cassandra-deb-packaging.sh
----------------------------------------------------------------------
diff --git a/build-scripts/cassandra-deb-packaging.sh b/build-scripts/cassandra-deb-packaging.sh
new file mode 100755
index 0000000..4466ced
--- /dev/null
+++ b/build-scripts/cassandra-deb-packaging.sh
@@ -0,0 +1,16 @@
+#!/bin/bash -xe
+
+CASSANDRA_BUILDS_DIR="${WORKSPACE}/cassandra-builds"
+CASSANDRA_GIT_URL=$1
+CASSANDRA_BRANCH=$2
+
+# Create build images containing the build tool-chain, Java and an Apache Cassandra git working directory
+docker build --build-arg CASSANDRA_GIT_URL=$CASSANDRA_GIT_URL -f ${CASSANDRA_BUILDS_DIR}/docker/jessie-image.docker ${CASSANDRA_BUILDS_DIR}/docker/
+
+# Create target directory for packages generated in docker run below
+mkdir -p ${CASSANDRA_BUILDS_DIR}/dist
+chmod 777 ${CASSANDRA_BUILDS_DIR}/dist
+
+# Run build script through docker (specify branch, e.g. cassandra-3.0 and version, e.g. 3.0.11):
+docker run --rm -v ${CASSANDRA_BUILDS_DIR}/dist:/dist `docker images -f label=org.cassandra.buildenv=jessie -q | awk 'NR==1'` /home/build/build-debs.sh $CASSANDRA_BRANCH
+
http://git-wip-us.apache.org/repos/asf/cassandra-builds/blob/f3fdb2bd/build-scripts/cassandra-rpm-packaging.sh
----------------------------------------------------------------------
diff --git a/build-scripts/cassandra-rpm-packaging.sh b/build-scripts/cassandra-rpm-packaging.sh
new file mode 100755
index 0000000..747d781
--- /dev/null
+++ b/build-scripts/cassandra-rpm-packaging.sh
@@ -0,0 +1,17 @@
+#!/bin/bash -xe
+
+CASSANDRA_BUILDS_DIR="${WORKSPACE}/cassandra-builds"
+CASSANDRA_GIT_URL=$1
+CASSANDRA_BRANCH=$2
+CASSANDRA_VERSION=$3
+
+# Create build images containing the build tool-chain, Java and an Apache Cassandra git working directory
+docker build --build-arg CASSANDRA_GIT_URL=$CASSANDRA_GIT_URL -f ${CASSANDRA_BUILDS_DIR}/docker/centos7-image.docker ${CASSANDRA_BUILDS_DIR}/docker/
+
+# Create target directory for packages generated in docker run below
+mkdir -p ${CASSANDRA_BUILDS_DIR}/dist
+chmod 777 ${CASSANDRA_BUILDS_DIR}/dist
+
+# Run build script through docker (specify branch, e.g. cassandra-3.0 and version, e.g. 3.0.11):
+docker run --rm -v ${CASSANDRA_BUILDS_DIR}/dist:/dist `docker images -f label=org.cassandra.buildenv=centos -q | awk 'NR==1'` /home/build/build-rpms.sh $CASSANDRA_BRANCH $CASSANDRA_VERSION
+
http://git-wip-us.apache.org/repos/asf/cassandra-builds/blob/f3fdb2bd/docker/build-debs.sh
----------------------------------------------------------------------
diff --git a/docker/build-debs.sh b/docker/build-debs.sh
index 9d55deb..39bd688 100755
--- a/docker/build-debs.sh
+++ b/docker/build-debs.sh
@@ -1,8 +1,8 @@
#!/bin/bash -x
-set -eu
+set -e
if [ "$#" -ne 1 ]; then
- echo "build-debs.sh branch"
+ echo "build-debs.sh <branch>"
exit 1
fi
@@ -10,10 +10,73 @@ CASSANDRA_BRANCH=$1
cd $CASSANDRA_DIR
git fetch
-git checkout $CASSANDRA_BRANCH
-# javadoc target is broken in docker without this mkdir
-mkdir -p ./build/javadoc
-ant artifacts -Drelease=true
+git checkout $CASSANDRA_BRANCH || exit 1
+
+# Used version for build will always depend on the git referenced used for checkout above
+# Branches will always be created as snapshots, while tags are releases
+tag=`git describe --tags --exact-match` 2> /dev/null
+branch=`git symbolic-ref -q --short HEAD` 2> /dev/null
+
+is_tag=false
+is_branch=false
+git_version=''
+
+if [ "$tag" ]; then
+ # Official release
+ is_tag=true
+ regx_tag="cassandra-([0-9.]+)(-tentative)?$"
+ if [[ $tag =~ $regx_tag ]]; then
+ git_version=${BASH_REMATCH[1]}
+ else
+ echo "Error: could not recognize version from tag $tag">&2
+ exit 2
+ fi
+elif [ "$branch" ]; then
+ # Dev branch
+ is_branch=true
+ regx_branch="cassandra-([0-9.]+)$"
+ if [[ $branch =~ $regx_branch ]]; then
+ git_version=${BASH_REMATCH[1]}
+ else
+ # This could be either trunk or any dev branch, so we won't be able to get the version
+ # from the branch name. In this case, fall back to debian change log version.
+ git_version=$(dpkg-parsechangelog | sed -ne 's/^Version: \([^-|~|+]*\).*/\1/p')
+ if [ -z $git_version ]; then
+ echo "Error: could not recognize version from branch $branch">&2
+ exit 2
+ else
+ echo "Warning: could not recognize version from branch, using dpkg version: $git_version"
+ fi
+ fi
+else
+ echo "Error: invalid git reference; must either be branch or tag">&2
+ exit 1
+fi
+
+# The version used for the deb build process will the current version in the debian/changelog file.
+# See debian/rules for how the value is read. The only thing left for us to do here is to check if
+# the changes file contains the correct version for the checked out git revision. The version value
+# has to be updated manually by a committer and we only warn and abort on mismatches here.
+changelog_version=$(dpkg-parsechangelog | sed -ne 's/^Version: \([^~|+]*\).*/\1/p')
+chl_expected="$git_version"
+if $is_branch ; then
+ chl_expected="${git_version}[0-9.]*-SNAPSHOT"
+fi
+if [[ ! $changelog_version =~ $chl_expected ]]; then
+ echo "Error: changelog version seems to be missing -SNAPSHOT suffix for branch">&2
+ exit 3
+fi
+
+# Version (base.version) in build.xml must be set manually as well. Let's validate the set value.
+buildxml_version=`grep 'property\s*name="base.version"' build.xml |sed -ne 's/.*value="\([^"]*\)".*/\1/p'`
+if [ $buildxml_version != $git_version ]; then
+ echo "Error: build.xml version ($buildxml_version) not matching git tag derived version ($git_version)">&2
+ exit 4
+fi
+
+# Install build dependencies and build package
echo "y" | sudo mk-build-deps --install
dpkg-buildpackage -uc -us
+
+# Copy created artifacts to dist dir mapped to docker host directory (must have proper permissions)
cp ../cassandra[-_]* $DEB_DIST_DIR
http://git-wip-us.apache.org/repos/asf/cassandra-builds/blob/f3fdb2bd/docker/build-rpms.sh
----------------------------------------------------------------------
diff --git a/docker/build-rpms.sh b/docker/build-rpms.sh
index abd875c..3bfd69e 100755
--- a/docker/build-rpms.sh
+++ b/docker/build-rpms.sh
@@ -1,20 +1,68 @@
#!/bin/bash -x
-set -eu
+#set -eu
-if [ "$#" -ne 2 ]; then
- echo "build-rpms.sh branch version"
+if [ "$#" -ne 1 ]; then
+ echo "build-rpms.sh branch"
exit 1
fi
CASSANDRA_BRANCH=$1
-CASSANDRA_VERSION=$2
cd $CASSANDRA_DIR
git fetch
-git checkout $CASSANDRA_BRANCH
+git checkout $CASSANDRA_BRANCH || exit 1
+
+# Used version for build will always depend on the git referenced used for checkout above
+# Branches will always be created as snapshots, while tags are releases
+tag=`git describe --tags --exact-match` 2> /dev/null
+branch=`git symbolic-ref -q --short HEAD` 2> /dev/null
+
+is_tag=false
+is_branch=false
+git_version=''
+
+# Parse version from build.xml so we can verify version against release tags and use the build.xml version
+# for any branches. Truncate from snapshot suffix if needed.
+buildxml_version=`grep 'property\s*name="base.version"' build.xml |sed -ne 's/.*value="\([^"]*\)".*/\1/p'`
+regx_snapshot="([0-9.]+)-SNAPSHOT$"
+if [[ $buildxml_version =~ $regx_snapshot ]]; then
+ buildxml_version=${BASH_REMATCH[1]}
+fi
+
+if [ "$tag" ]; then
+ # Official release
+ is_tag=true
+ regx_tag="cassandra-([0-9.]+)(-tentative)?$"
+ if [[ $tag =~ $regx_tag ]]; then
+ git_version=${BASH_REMATCH[1]}
+ else
+ echo "Error: could not recognize version from tag $tag">&2
+ exit 2
+ fi
+ if [ $buildxml_version != $git_version ]; then
+ echo "Error: build.xml version ($buildxml_version) not matching git tag derived version ($git_version)">&2
+ exit 4
+ fi
+ CASSANDRA_VERSION=$git_version
+ CASSANDRA_REVISION='1%{?dist}'
+elif [ "$branch" ]; then
+ # Dev branch
+ is_branch=true
+ # This could be either trunk or any dev branch, so we won't be able to get the version
+ # from the branch name. In this case, fall back to version specified in build.xml.
+ CASSANDRA_VERSION="${buildxml_version}"
+ dt=`date +"%Y%m%d"`
+ ref=`git rev-parse --short HEAD`
+ CASSANDRA_REVISION="${dt}git${ref}%{?dist}"
+else
+ echo "Error: invalid git reference; must either be branch or tag">&2
+ exit 1
+fi
+
# javadoc target is broken in docker without this mkdir
mkdir -p ./build/javadoc
+# Artifact will only be used internally for build process and won't be found with snapshot suffix
ant artifacts -Drelease=true
cp ./build/apache-cassandra-*-src.tar.gz ${RPM_BUILD_DIR}/SOURCES/
-rpmbuild --define="version ${CASSANDRA_VERSION}" -ba ./redhat/cassandra.spec
+rpmbuild --define="version ${CASSANDRA_VERSION}" --define="revision ${CASSANDRA_REVISION}" -ba ./redhat/cassandra.spec
cp $RPM_BUILD_DIR/SRPMS/*.rpm $RPM_BUILD_DIR/RPMS/noarch/*.rpm $RPM_DIST_DIR
http://git-wip-us.apache.org/repos/asf/cassandra-builds/blob/f3fdb2bd/docker/centos7-image.docker
----------------------------------------------------------------------
diff --git a/docker/centos7-image.docker b/docker/centos7-image.docker
index fc82d4d..c082939 100644
--- a/docker/centos7-image.docker
+++ b/docker/centos7-image.docker
@@ -40,7 +40,8 @@ USER build
RUN mkdir -p $RPM_BUILD_DIR/{BUILD,BUILDROOT,RPMS,SOURCES,SPECS,SRPMS}
# Clone Cassandra and cache maven artifacts
-RUN git clone https://git.apache.org/cassandra.git ${CASSANDRA_DIR}
+ARG CASSANDRA_GIT_URL=https://git.apache.org/cassandra.git
+RUN git clone ${CASSANDRA_GIT_URL} ${CASSANDRA_DIR}
WORKDIR $CASSANDRA_DIR
RUN ant maven-ant-tasks-retrieve-build
http://git-wip-us.apache.org/repos/asf/cassandra-builds/blob/f3fdb2bd/docker/jessie-image.docker
----------------------------------------------------------------------
diff --git a/docker/jessie-image.docker b/docker/jessie-image.docker
index 80bce88..b21e671 100644
--- a/docker/jessie-image.docker
+++ b/docker/jessie-image.docker
@@ -35,7 +35,8 @@ RUN echo "build ALL=(root) NOPASSWD:ALL" > /etc/sudoers.d/build && \
USER build
# clone Cassandra and cache maven artifacts
-RUN git clone https://git.apache.org/cassandra.git ${CASSANDRA_DIR}
+ARG CASSANDRA_GIT_URL=https://git.apache.org/cassandra.git
+RUN git clone ${CASSANDRA_GIT_URL} ${CASSANDRA_DIR}
WORKDIR ${CASSANDRA_DIR}
RUN ant maven-ant-tasks-retrieve-build
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org