You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@cloudstack.apache.org by Ian Duffy <ia...@ianduffy.ie> on 2014/09/24 02:04:35 UTC
SystemVM file name changes
Hi All,
I noticed the filename of the generated systemvms changed on
http://jenkins.buildacloud.org/job/build-systemvm64-master/
They now include a [0-9]* before the hypervisor name representing the build
number.
Why do we do this? Its annoying for external resources linking to the last
successful build.
Re: SystemVM file name changes
Posted by Ian Duffy <ia...@ianduffy.ie>.
> Did this. Next builds _should_ be back to their old urls.
Great! Thanks Leo. :)
On 25 September 2014 00:04, Leo Simons <LS...@schubergphilis.com> wrote:
> On Sep 24, 2014, at 5:33 PM, Ian Duffy <ia...@ianduffy.ie> wrote:
> >> Aren’t they published to S3 somewhere? It’d probably be better if users
> > fetch systemvms from S3……
> >
> > Both not a clue what the S3 urls are though.
>
> Based on
>
> http://mail-archives.apache.org/mod_mbox/cloudstack-dev/201402.mbox/%3C3F6B7B36-32A2-4F11-AEFF-36A400D67B7D@GMAIL.com%3E
>
> It ought to be
>
> https://s3.amazonaws.com/systemvm-templates.cloudstack.org/master/32/systemvmtemplate-master-xen.vhd.bz2
>
> https://s3.amazonaws.com/systemvm-templates.cloudstack.org/master/64/systemvm64template-master-xen.vhd.bz2
>
> and so forth but I’m getting permission denied. Hugo, any idea?
>
> >>> Isn't this what we use /etc/cloudstack-release for?
> >>> You could include extra information in there if you wish, version,
> >> branch,
> >>> git sha1, etc. which would give great traceability.
> >>
> >> Sounds like a plan. I’ll see about submitting a patch tonight. The issue
> >> is that /etc/cloudstack-release is parsed in a location or two (I forget
> >> the details) so it kind-of has to match existing regexes.
>
> Grmbl is all I have to say.
>
> >> In the meantime...
> >> export BUILD_NUMBER=
> >> at the right place in the relevant build job should fix the immediate
> >> issue...
>
> Did this. Next builds _should_ be back to their old urls.
>
>
> g’night!
>
>
> Leo
>
>
Re: SystemVM file name changes
Posted by Leo Simons <LS...@schubergphilis.com>.
On Sep 24, 2014, at 5:33 PM, Ian Duffy <ia...@ianduffy.ie> wrote:
>> Aren’t they published to S3 somewhere? It’d probably be better if users
> fetch systemvms from S3……
>
> Both not a clue what the S3 urls are though.
Based on
http://mail-archives.apache.org/mod_mbox/cloudstack-dev/201402.mbox/%3C3F6B7B36-32A2-4F11-AEFF-36A400D67B7D@GMAIL.com%3E
It ought to be
https://s3.amazonaws.com/systemvm-templates.cloudstack.org/master/32/systemvmtemplate-master-xen.vhd.bz2
https://s3.amazonaws.com/systemvm-templates.cloudstack.org/master/64/systemvm64template-master-xen.vhd.bz2
and so forth but I’m getting permission denied. Hugo, any idea?
>>> Isn't this what we use /etc/cloudstack-release for?
>>> You could include extra information in there if you wish, version,
>> branch,
>>> git sha1, etc. which would give great traceability.
>>
>> Sounds like a plan. I’ll see about submitting a patch tonight. The issue
>> is that /etc/cloudstack-release is parsed in a location or two (I forget
>> the details) so it kind-of has to match existing regexes.
Grmbl is all I have to say.
>> In the meantime...
>> export BUILD_NUMBER=
>> at the right place in the relevant build job should fix the immediate
>> issue...
Did this. Next builds _should_ be back to their old urls.
g’night!
Leo
Re: SystemVM file name changes
Posted by Ian Duffy <ia...@ianduffy.ie>.
> Aren’t they published to S3 somewhere? It’d probably be better if users
fetch systemvms from S3……
Both not a clue what the S3 urls are though.
> though one wonders why users need to fetch
systemvms built from master at all…
Ease of access for running from the latest master branch I'd imagine.
On 24 September 2014 15:21, Leo Simons <LS...@schubergphilis.com> wrote:
> On Sep 24, 2014, at 11:26 AM, Ian Duffy <ia...@ianduffy.ie> wrote:
> >> The general concept is one of traceable and repeatable builds.
> >
> > In the context of jenkins.buildacloud.org I would disagree with this.
> Only
> > two build jobs are ever kept. When a time arises when you need to figure
> > out what commit your systemvm was based of I'd be willing to bet that job
> > has been cleared out of jenkins ages ago.
> >
> >> In RPM land, the jenkins $BUILD_NUMBER should go into the RPM Release
> > field and not into the RPM Version field. But in this case we don’t have
> > RPM, or spec files
> >
> > Isn't this what we use /etc/cloudstack-release for?
> > You could include extra information in there if you wish, version,
> branch,
> > git sha1, etc. which would give great traceability.
>
> Sounds like a plan. I’ll see about submitting a patch tonight. The issue
> is that /etc/cloudstack-release is parsed in a location or two (I forget
> the details) so it kind-of has to match existing regexes.
>
> > The supplied download-template script is dependant on having a local
> > nexus/artifactory that can do searching on published build artifacts. In
> > the context of jenkins.buildacloud.org the system vms are not uploaded
> to
> > nexus/artifactory and just kept on the jenkins box.
>
> Aren’t they published to S3 somewhere? It’d probably be better if users
> fetch systemvms from S3……though one wonders why users need to fetch
> systemvms built from master at all…
>
> > Changing this causes 404s for people example:
> >
> https://mail-archives.apache.org/mod_mbox/cloudstack-users/201409.mbox/browser
>
> Hmm. We can’t have too many people badgering jenkins like this I guess. So
> let’s improve on that!
>
> In the meantime...
> export BUILD_NUMBER=
> at the right place in the relevant build job should fix the immediate
> issue...
>
>
> cheers,
>
>
> Leo
>
>
Re: SystemVM file name changes
Posted by Leo Simons <LS...@schubergphilis.com>.
On Sep 24, 2014, at 11:26 AM, Ian Duffy <ia...@ianduffy.ie> wrote:
>> The general concept is one of traceable and repeatable builds.
>
> In the context of jenkins.buildacloud.org I would disagree with this. Only
> two build jobs are ever kept. When a time arises when you need to figure
> out what commit your systemvm was based of I'd be willing to bet that job
> has been cleared out of jenkins ages ago.
>
>> In RPM land, the jenkins $BUILD_NUMBER should go into the RPM Release
> field and not into the RPM Version field. But in this case we don’t have
> RPM, or spec files
>
> Isn't this what we use /etc/cloudstack-release for?
> You could include extra information in there if you wish, version, branch,
> git sha1, etc. which would give great traceability.
Sounds like a plan. I’ll see about submitting a patch tonight. The issue
is that /etc/cloudstack-release is parsed in a location or two (I forget
the details) so it kind-of has to match existing regexes.
> The supplied download-template script is dependant on having a local
> nexus/artifactory that can do searching on published build artifacts. In
> the context of jenkins.buildacloud.org the system vms are not uploaded to
> nexus/artifactory and just kept on the jenkins box.
Aren’t they published to S3 somewhere? It’d probably be better if users
fetch systemvms from S3……though one wonders why users need to fetch
systemvms built from master at all…
> Changing this causes 404s for people example:
> https://mail-archives.apache.org/mod_mbox/cloudstack-users/201409.mbox/browser
Hmm. We can’t have too many people badgering jenkins like this I guess. So
let’s improve on that!
In the meantime...
export BUILD_NUMBER=
at the right place in the relevant build job should fix the immediate issue...
cheers,
Leo
Re: SystemVM file name changes
Posted by Ian Duffy <ia...@ianduffy.ie>.
> The general concept is one of traceable and repeatable builds.
In the context of jenkins.buildacloud.org I would disagree with this. Only
two build jobs are ever kept. When a time arises when you need to figure
out what commit your systemvm was based of I'd be willing to bet that job
has been cleared out of jenkins ages ago.
> In RPM land, the jenkins $BUILD_NUMBER should go into the RPM Release
field and not into the RPM Version field. But in this case we don’t have
RPM, or spec files
Isn't this what we use /etc/cloudstack-release for?
You could include extra information in there if you wish, version, branch,
git sha1, etc. which would give great traceability.
The supplied download-template script is dependant on having a local
nexus/artifactory that can do searching on published build artifacts. In
the context of jenkins.buildacloud.org the system vms are not uploaded to
nexus/artifactory and just kept on the jenkins box.
The RPM one grabs a zip of all RPM is the last successful build. The zip
filename doesn't contain any version information.
Changing this causes 404s for people example:
https://mail-archives.apache.org/mod_mbox/cloudstack-users/201409.mbox/browser
On 24 September 2014 07:37, Leo Simons <LS...@schubergphilis.com> wrote:
> Hey Ian,
>
> On Sep 24, 2014, at 2:04 AM, Ian Duffy <ia...@ianduffy.ie> wrote:
> > I noticed the filename of the generated systemvms changed on
> > http://jenkins.buildacloud.org/job/build-systemvm64-master/
> >
> > They now include a [0-9]* before the hypervisor name representing the
> build
> > number.
>
> That’s one of the things coming out of the 1290e10 merge.
>
> > Why do we do this? Its annoying for external resources linking to the
> last
> > successful build.
>
> The general concept is one of traceable and repeatable builds.
>
> In RPM terms, given just a random binary RPM found on a random system
> * it should be possible to unambiguously determine the source RPM from
> which it was built
> * it should be possible to determine who or what built it, and when
> * it should be possible to trace the source RPM back to the exact version
> control revision
> * it should be possible to determine easily whether that binary is the
> same as
> another one on another machine / in another repository that has the same
> identifiers
>
> In RPM land, the jenkins $BUILD_NUMBER should go into the RPM Release
> field and not into the RPM Version field. But in this case we don’t have
> RPM, or spec files, so there is only the filename to accomplish the same
> (which is why the branch name is also in there), and it has to be in this
> spot or cloudstack gets a bit confused.
>
> What we do in our integration environment is to apply just a bit of logic
> to parse out the version number from the file name. I.e. to get an RPM from
> jenkins see fetch-rpms.sh included below, or to fetch the most recent
> systemvm from a directory listing see download-template.sh also included
> below. Would something like that work for you?
>
> If not, the logic for creating the version is in build.sh:
>
> version_tag=
> if [ ! -z "${version}" ]; then
> if [ ! -z "${BUILD_NUMBER}" ]; then
> version="${version}.${BUILD_NUMBER}"
> fi
> version_tag="-${version}"
> elif [ ! -z "${BUILD_NUMBER}" ]; then
> version="${BUILD_NUMBER}"
> version_tag="-${BUILD_NUMBER}”
> fi
>
> So we can change that, or we can unset BUILD_NUMBER in the jenkins build
> prior to invocation. What do you think?
>
> It would be much better to exclusively use yum/maven repositories and
> their logic to centralize this kind of logic server-side, or at least to be
> creating and updating symlinks to ‘latest’. That’s a work in progress...
>
>
> cheers,
>
>
> Leo
>
> download-templates.sh
> =====================
> #!/bin/bash
> #
> # Licensed to the Apache Software Foundation (ASF) under one
> # or more contributor license agreements. See the NOTICE file
> # distributed with this work for additional information
> # regarding copyright ownership. The ASF licenses this file
> # to you under the Apache License, Version 2.0 (the
> # "License"); you may not use this file except in compliance
> # with the License. You may obtain a copy of the License at
> #
> # http://www.apache.org/licenses/LICENSE-2.0
> #
> # Unless required by applicable law or agreed to in writing,
> # software distributed under the License is distributed on an
> # "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
> # KIND, either express or implied. See the License for the
> # specific language governing permissions and limitations
> # under the License.
>
> # Download systemvm templates onto secondary storage
>
> set -e
> echo $@ | grep debug >/dev/null && DEBUG="1"
> echo $@ | grep trace >/dev/null && TRACE="1"
>
> function source_common() {
> local SOURCE="${BASH_SOURCE[0]}"
> DIR="$( cd -P "$( dirname "${SOURCE}" )" && pwd )"
> source "${DIR}/common.sh"
> }
> source_common
>
> NFS_HOST="${1:-localhost}"
> NFS_SECONDARY_EXPORT="${2:-/storage/secondary}"
> MOUNT_PATH="${3:-/mnt/secondary}"
> BASE_URL="${4:-https://artifacts.schubergphilis.com/artifacts/cloudstack}"
> TEMPLATE="${5:-systemvmtemplate}"
> DB_HOST="${6:-localhost}"
> DB_USER="${7:-root}"
> DB_PASSWORD="${8:-}"
>
> function find_template() {
> if ! `echo "${TEMPLATE}" | egrep '^[a-zA-Z0-9]+-[0-9]+(\.[0-9]+)*$'`;
> then
> # (apache) directory listing
> # find only links
> # filter by template name
> # find only xen template
> # strip suffix
> # take the first (newest) one
> newest_first="?C=M;O=D"
> TEMPLATE=$(curl "${BASE_URL}/${newest_first}" | \
> egrep -o 'href="[^"]+"' | \
> egrep "${TEMPLATE}" | \
> egrep -o "[^/\"]+-xen\.vhd\.bz2" | \
> sed 's/-xen.vhd.bz2//' |\
> head -n 1)
> if [[ "${TEMPLATE}" == "" ]]; then
> error "No possible match found for ${TEMPLATE} at ${BASE_URL}"
> fi
> log INFO "Using template ${TEMPLATE}"
> fi
> }
>
> function install_xen_template() {
> XEN_URL="${BASE_URL}/${TEMPLATE}-xen.vhd.bz2"
> # KVM_URL="${BASE_URL}/${TEMPLATE}-kvm.vhd.bz2"
>
> [ -d "${MOUNT_PATH}" ] || mkdir "${MOUNT_PATH}"
>
> mount -t nfs "${NFS_HOST}:${NFS_SECONDARY_EXPORT}" "${MOUNT_PATH}" ||
> true
>
>
> /usr/share/cloudstack-common/scripts/storage/secondary/cloud-install-sys-tmplt
> \
> -F \
> -m "${MOUNT_PATH}" \
> -h "xenserver" \
> -u "${XEN_URL}" \
> -o "${DB_HOST}" \
> -r "${DB_USER}" \
> -d "${DB_PASSWORD}"
> }
>
> function main() {
> find_template
> install_xen_template
> }
>
> # we only run main() if not source-d
> return 2>/dev/null || main
>
>
>
> fetch-rpms.sh
> =============
> #!/bin/bash
> #
> # Licensed to the Apache Software Foundation (ASF) under one
> # or more contributor license agreements. See the NOTICE file
> # distributed with this work for additional information
> # regarding copyright ownership. The ASF licenses this file
> # to you under the Apache License, Version 2.0 (the
> # "License"); you may not use this file except in compliance
> # with the License. You may obtain a copy of the License at
> #
> # http://www.apache.org/licenses/LICENSE-2.0
> #
> # Unless required by applicable law or agreed to in writing,
> # software distributed under the License is distributed on an
> # "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
> # KIND, either express or implied. See the License for the
> # specific language governing permissions and limitations
> # under the License.
>
> # script to install http://jenkins.buildacloud.org/ RPMs to
> # the local repo
>
> function usage() {
> cat <<END
> Usage:
> ./fetch-rpms.sh [archiveUrl]
>
> * Pass [archiveURL] to link to the build artifacts
> (default is
> http://jenkins.buildacloud.org/job/
> package-rhel63-master/
> lastSuccessfulBuild/artifact/*zip*
> /archive.zip
> )
> * Set \$DEBUG=1 to enable debug logging
> * Set \$TRACE=1 to enable trace logging
> END
> exit 0
> }
>
> echo $@ | grep help >/dev/null && usage
> echo $@ | grep '\-h' >/dev/null && usage
> echo $@ | grep debug >/dev/null && DEBUG="1"
> echo $@ | grep trace >/dev/null && TRACE="1"
>
> function source_common() {
> # from http://stackoverflow.com/a/246128
> local SOURCE="${BASH_SOURCE[0]}"
> local DIR=""
>
> while [ -h "$SOURCE" ]; do
> # resolve $SOURCE until the file is no longer a symlink
> DIR="$( cd -P "$( dirname "${SOURCE}" )" && pwd )"
> SOURCE="$(readlink "${SOURCE}")"
> # if $SOURCE was a relative symlink, we need to resolve it relative
> # to the path where the symlink file was located
> [[ ${SOURCE} != /* ]] && SOURCE="${DIR}/${SOURCE}"
> done
> DIR="$( cd -P "$( dirname "${SOURCE}" )" && pwd )"
>
> source "${DIR}/common.sh"
> }
> source_common
>
> ###
> ### Configuration
> ###
>
> if [[ "$1" != "" ]]; then
> ARCHIVE_URL="${1}"
> else
> JENKINS="http://jenkins.buildacloud.org"
> JOB="package-rhel63-master"
>
> ARCHIVE_URL="${JENKINS}/job/${JOB}/lastSuccessfulBuild/artifact/*zip*/archive.zip"
> fi
> STORAGE_DIR="${BASEDIR}/../storage"
> ARCHIVE_FILE="${STORAGE_DIR}/archive.zip"
>
> ###
> ### Script logic
> ###
>
> function prepare() {
> cd "${TARGET_DIR}"
> init_settings
> }
>
> function run() {
> local do_download=1
> if [[ -f "${ARCHIVE_FILE}" ]]; then
> if [[ "${CI_RUN}" == "1" ]]; then
> log WARN "${ARCHIVE_FILE} already exists, removing"
> rm -f "${ARCHIVE_FILE}"
> else
> log INFO "${ARCHIVE_FILE} already exists, reusing"
> do_download=0
> fi
> fi
> if [[ ${do_download} -eq 1 ]]; then
> log INFO "Downloading ${ARCHIVE_URL}"
> curl -o "${ARCHIVE_FILE}" "${ARCHIVE_URL}"
> fi
>
> if [[ -d "${STORAGE_DIR}/archive" ]]; then
> rm -rf "${STORAGE_DIR}/archive"
> fi
> mkdir "${STORAGE_DIR}/archive"
> cd "${STORAGE_DIR}"
> unzip -o "${ARCHIVE_FILE}"
> mkdir -p "${STORAGE_DIR}/rpmrepo/cloudstack-${CLOUDSTACK_VERSION}"
> local numfiles=$(find \
> ${STORAGE_DIR}/archive \
> -name "cloudstack*${CLOUDSTACK_VERSION}*.rpm" | \
> wc -l | awk '{print $1}')
> if [[ ${numfiles} -lt 4 ]]; then
> error "Expecting at least 4 rpms matching
> cloudstack*${CLOUDSTACK_VERSION}*.rpm, but got only ${numfiles}"
> fi
> find ${STORAGE_DIR}/archive -name
> "cloudstack*${CLOUDSTACK_VERSION}*.rpm" -exec \
> mv \{\} "${STORAGE_DIR}/rpmrepo/cloudstack-${CLOUDSTACK_VERSION}/" \;
> rm -rf "${STORAGE_DIR}/archive"
> }
>
> # we only run main() if not source-d
> return 2>/dev/null || main
>
>
Re: SystemVM file name changes
Posted by Leo Simons <LS...@schubergphilis.com>.
Hey Ian,
On Sep 24, 2014, at 2:04 AM, Ian Duffy <ia...@ianduffy.ie> wrote:
> I noticed the filename of the generated systemvms changed on
> http://jenkins.buildacloud.org/job/build-systemvm64-master/
>
> They now include a [0-9]* before the hypervisor name representing the build
> number.
That’s one of the things coming out of the 1290e10 merge.
> Why do we do this? Its annoying for external resources linking to the last
> successful build.
The general concept is one of traceable and repeatable builds.
In RPM terms, given just a random binary RPM found on a random system
* it should be possible to unambiguously determine the source RPM from which it was built
* it should be possible to determine who or what built it, and when
* it should be possible to trace the source RPM back to the exact version control revision
* it should be possible to determine easily whether that binary is the same as
another one on another machine / in another repository that has the same
identifiers
In RPM land, the jenkins $BUILD_NUMBER should go into the RPM Release field and not into the RPM Version field. But in this case we don’t have RPM, or spec files, so there is only the filename to accomplish the same (which is why the branch name is also in there), and it has to be in this spot or cloudstack gets a bit confused.
What we do in our integration environment is to apply just a bit of logic to parse out the version number from the file name. I.e. to get an RPM from jenkins see fetch-rpms.sh included below, or to fetch the most recent systemvm from a directory listing see download-template.sh also included below. Would something like that work for you?
If not, the logic for creating the version is in build.sh:
version_tag=
if [ ! -z "${version}" ]; then
if [ ! -z "${BUILD_NUMBER}" ]; then
version="${version}.${BUILD_NUMBER}"
fi
version_tag="-${version}"
elif [ ! -z "${BUILD_NUMBER}" ]; then
version="${BUILD_NUMBER}"
version_tag="-${BUILD_NUMBER}”
fi
So we can change that, or we can unset BUILD_NUMBER in the jenkins build prior to invocation. What do you think?
It would be much better to exclusively use yum/maven repositories and their logic to centralize this kind of logic server-side, or at least to be creating and updating symlinks to ‘latest’. That’s a work in progress...
cheers,
Leo
download-templates.sh
=====================
#!/bin/bash
#
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
# Download systemvm templates onto secondary storage
set -e
echo $@ | grep debug >/dev/null && DEBUG="1"
echo $@ | grep trace >/dev/null && TRACE="1"
function source_common() {
local SOURCE="${BASH_SOURCE[0]}"
DIR="$( cd -P "$( dirname "${SOURCE}" )" && pwd )"
source "${DIR}/common.sh"
}
source_common
NFS_HOST="${1:-localhost}"
NFS_SECONDARY_EXPORT="${2:-/storage/secondary}"
MOUNT_PATH="${3:-/mnt/secondary}"
BASE_URL="${4:-https://artifacts.schubergphilis.com/artifacts/cloudstack}"
TEMPLATE="${5:-systemvmtemplate}"
DB_HOST="${6:-localhost}"
DB_USER="${7:-root}"
DB_PASSWORD="${8:-}"
function find_template() {
if ! `echo "${TEMPLATE}" | egrep '^[a-zA-Z0-9]+-[0-9]+(\.[0-9]+)*$'`; then
# (apache) directory listing
# find only links
# filter by template name
# find only xen template
# strip suffix
# take the first (newest) one
newest_first="?C=M;O=D"
TEMPLATE=$(curl "${BASE_URL}/${newest_first}" | \
egrep -o 'href="[^"]+"' | \
egrep "${TEMPLATE}" | \
egrep -o "[^/\"]+-xen\.vhd\.bz2" | \
sed 's/-xen.vhd.bz2//' |\
head -n 1)
if [[ "${TEMPLATE}" == "" ]]; then
error "No possible match found for ${TEMPLATE} at ${BASE_URL}"
fi
log INFO "Using template ${TEMPLATE}"
fi
}
function install_xen_template() {
XEN_URL="${BASE_URL}/${TEMPLATE}-xen.vhd.bz2"
# KVM_URL="${BASE_URL}/${TEMPLATE}-kvm.vhd.bz2"
[ -d "${MOUNT_PATH}" ] || mkdir "${MOUNT_PATH}"
mount -t nfs "${NFS_HOST}:${NFS_SECONDARY_EXPORT}" "${MOUNT_PATH}" || true
/usr/share/cloudstack-common/scripts/storage/secondary/cloud-install-sys-tmplt \
-F \
-m "${MOUNT_PATH}" \
-h "xenserver" \
-u "${XEN_URL}" \
-o "${DB_HOST}" \
-r "${DB_USER}" \
-d "${DB_PASSWORD}"
}
function main() {
find_template
install_xen_template
}
# we only run main() if not source-d
return 2>/dev/null || main
fetch-rpms.sh
=============
#!/bin/bash
#
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
# script to install http://jenkins.buildacloud.org/ RPMs to
# the local repo
function usage() {
cat <<END
Usage:
./fetch-rpms.sh [archiveUrl]
* Pass [archiveURL] to link to the build artifacts
(default is
http://jenkins.buildacloud.org/job/
package-rhel63-master/
lastSuccessfulBuild/artifact/*zip*
/archive.zip
)
* Set \$DEBUG=1 to enable debug logging
* Set \$TRACE=1 to enable trace logging
END
exit 0
}
echo $@ | grep help >/dev/null && usage
echo $@ | grep '\-h' >/dev/null && usage
echo $@ | grep debug >/dev/null && DEBUG="1"
echo $@ | grep trace >/dev/null && TRACE="1"
function source_common() {
# from http://stackoverflow.com/a/246128
local SOURCE="${BASH_SOURCE[0]}"
local DIR=""
while [ -h "$SOURCE" ]; do
# resolve $SOURCE until the file is no longer a symlink
DIR="$( cd -P "$( dirname "${SOURCE}" )" && pwd )"
SOURCE="$(readlink "${SOURCE}")"
# if $SOURCE was a relative symlink, we need to resolve it relative
# to the path where the symlink file was located
[[ ${SOURCE} != /* ]] && SOURCE="${DIR}/${SOURCE}"
done
DIR="$( cd -P "$( dirname "${SOURCE}" )" && pwd )"
source "${DIR}/common.sh"
}
source_common
###
### Configuration
###
if [[ "$1" != "" ]]; then
ARCHIVE_URL="${1}"
else
JENKINS="http://jenkins.buildacloud.org"
JOB="package-rhel63-master"
ARCHIVE_URL="${JENKINS}/job/${JOB}/lastSuccessfulBuild/artifact/*zip*/archive.zip"
fi
STORAGE_DIR="${BASEDIR}/../storage"
ARCHIVE_FILE="${STORAGE_DIR}/archive.zip"
###
### Script logic
###
function prepare() {
cd "${TARGET_DIR}"
init_settings
}
function run() {
local do_download=1
if [[ -f "${ARCHIVE_FILE}" ]]; then
if [[ "${CI_RUN}" == "1" ]]; then
log WARN "${ARCHIVE_FILE} already exists, removing"
rm -f "${ARCHIVE_FILE}"
else
log INFO "${ARCHIVE_FILE} already exists, reusing"
do_download=0
fi
fi
if [[ ${do_download} -eq 1 ]]; then
log INFO "Downloading ${ARCHIVE_URL}"
curl -o "${ARCHIVE_FILE}" "${ARCHIVE_URL}"
fi
if [[ -d "${STORAGE_DIR}/archive" ]]; then
rm -rf "${STORAGE_DIR}/archive"
fi
mkdir "${STORAGE_DIR}/archive"
cd "${STORAGE_DIR}"
unzip -o "${ARCHIVE_FILE}"
mkdir -p "${STORAGE_DIR}/rpmrepo/cloudstack-${CLOUDSTACK_VERSION}"
local numfiles=$(find \
${STORAGE_DIR}/archive \
-name "cloudstack*${CLOUDSTACK_VERSION}*.rpm" | \
wc -l | awk '{print $1}')
if [[ ${numfiles} -lt 4 ]]; then
error "Expecting at least 4 rpms matching cloudstack*${CLOUDSTACK_VERSION}*.rpm, but got only ${numfiles}"
fi
find ${STORAGE_DIR}/archive -name "cloudstack*${CLOUDSTACK_VERSION}*.rpm" -exec \
mv \{\} "${STORAGE_DIR}/rpmrepo/cloudstack-${CLOUDSTACK_VERSION}/" \;
rm -rf "${STORAGE_DIR}/archive"
}
# we only run main() if not source-d
return 2>/dev/null || main