You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@mesos.apache.org by vi...@apache.org on 2017/08/04 19:01:57 UTC
[1/3] mesos git commit: Updated local development workflow of mesos
website.
Repository: mesos
Updated Branches:
refs/heads/master cdb90b91c -> 8734511fd
Updated local development workflow of mesos website.
Made the layout and scripts consistent with CI based automatic
publishing of the website.
Review: https://reviews.apache.org/r/60440
Project: http://git-wip-us.apache.org/repos/asf/mesos/repo
Commit: http://git-wip-us.apache.org/repos/asf/mesos/commit/8734511f
Tree: http://git-wip-us.apache.org/repos/asf/mesos/tree/8734511f
Diff: http://git-wip-us.apache.org/repos/asf/mesos/diff/8734511f
Branch: refs/heads/master
Commit: 8734511fd4e21d6c50d6bb11832c86451322da7d
Parents: 23541ef
Author: Vinod Kone <vi...@gmail.com>
Authored: Thu Jun 22 11:05:51 2017 +0800
Committer: Vinod Kone <vi...@gmail.com>
Committed: Fri Aug 4 12:01:35 2017 -0700
----------------------------------------------------------------------
site/Dockerfile | 2 +-
site/README.md | 52 ++++++++----------------------------------
site/build.sh | 2 +-
site/entrypoint.sh | 33 +++++++++++++++++++++++++++
site/mesos-website-dev.sh | 45 ++++++++++++++++++++++++++++++++++++
5 files changed, 90 insertions(+), 44 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/mesos/blob/8734511f/site/Dockerfile
----------------------------------------------------------------------
diff --git a/site/Dockerfile b/site/Dockerfile
index 230cfc7..2dd9e28 100644
--- a/site/Dockerfile
+++ b/site/Dockerfile
@@ -13,4 +13,4 @@ EXPOSE 4567
EXPOSE 35729
WORKDIR /mesos/site
-CMD bash build.sh
+CMD bash entrypoint.sh
http://git-wip-us.apache.org/repos/asf/mesos/blob/8734511f/site/README.md
----------------------------------------------------------------------
diff --git a/site/README.md b/site/README.md
index ebd3e6a..0a5cd61 100644
--- a/site/README.md
+++ b/site/README.md
@@ -1,51 +1,22 @@
# Apache Mesos website generator
-This will generate the site available at http://mesos.apache.org. We use docker
+This will generate the Mesos website locally. We use docker
to simplify our generation and development workflows.
-
-## Build the docker image
-
-Under the root of your local Mesos git repository, run the following command
-to build the docker image.
-
-```
-docker build -t mesos/website site
-```
-
## Generating the site
-Before we start the docker container, we need to generate the help endpoint
-documentation. Running `make` is needed to generate the latest version of
-the master and agent binaries.
+Before we generate the website, please make sure to generate the help endpoint
+documentation. The endpoint generation script expects Mesos master and agent
+binaries to be available; so please run `make` if necessary.
```
make -jN
../support/generate-endpoint-help.py
```
-To build and run the website inside a docker container please substitute
-*<path-to-mesos-source>* with the actual location of Mesos source code tree
-and *<path-to-mesos-website-svn-source>* with the actual location of Mesos
-website svn source code tree in the command below and run it.
+To generate the website:
```
-sudo docker run -it --rm \
- -p 4567:4567 \
- -p 35729:35729 \
- -v <path-to-mesos-source>:/mesos \
- -v <path-to-mesos-website-svn-source>/publish:/mesos/site/publish \
- mesos/website
-```
-
-If you don't intend to update the svn source code of the Mesos website, you
-could run the command below under the root of Mesos source code instead.
-
-```
-sudo docker run -it --rm \
- -p 4567:4567 \
- -p 35729:35729 \
- -v `pwd`:/mesos \
- mesos/website
+./site/mesos-websiste-dev.sh
```
This will start a container, generate the website from your local Mesos git
@@ -75,10 +46,7 @@ destroy/remove the container.
## Publishing the Site
-Because we mount the `publish` folder under Mesos website svn source code into
-the docker container we launched above, all website contents would be ready
-under the `publish` folder when the generation finishes.
-
-The website uses svnpubsub. The `publish` folder contains the websites content
-and when committed to the svn repository it will be automatically deployed to
-the live site.
+Developers are not expected to publish the website. There is a CI job on ASF
+Jenkins (Mesos-Websitebot) that automatically publishes the website when there
+are changes detected in the Mesos repository. See
+`support/jenkins/websitebot.sh` for details.
http://git-wip-us.apache.org/repos/asf/mesos/blob/8734511f/site/build.sh
----------------------------------------------------------------------
diff --git a/site/build.sh b/site/build.sh
index 06b1b32..bfb9274 100755
--- a/site/build.sh
+++ b/site/build.sh
@@ -25,4 +25,4 @@ function exit_hook {
trap exit_hook EXIT
-bundle install && bundle exec rake && bundle exec rake dev
+bundle exec rake && bundle exec rake dev
http://git-wip-us.apache.org/repos/asf/mesos/blob/8734511f/site/entrypoint.sh
----------------------------------------------------------------------
diff --git a/site/entrypoint.sh b/site/entrypoint.sh
new file mode 100755
index 0000000..f30c911
--- /dev/null
+++ b/site/entrypoint.sh
@@ -0,0 +1,33 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# This is a wrapper script for building Mesos website as a non-root user.
+set -e
+set -o pipefail
+
+# This needs to be run under `root` user for `bundle exec rake` to
+# work properly. See MESOS-7859.
+bundle install
+
+# We do not create a temp user for MacOS because the host volumes are always
+# mapped to UID 1000 and GID 1000 inside the container. And any writes by the
+# root user inside the container to the mounted host volume will end up with
+# the original UID and GID of the host user.
+# TODO(vinod): On Linux, create a temp user and run the below script as that
+# user to avoid doing any writes as root user on host volumes.
+/mesos/site/build.sh
http://git-wip-us.apache.org/repos/asf/mesos/blob/8734511f/site/mesos-website-dev.sh
----------------------------------------------------------------------
diff --git a/site/mesos-website-dev.sh b/site/mesos-website-dev.sh
new file mode 100755
index 0000000..2be99db
--- /dev/null
+++ b/site/mesos-website-dev.sh
@@ -0,0 +1,45 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# This is a wrapper for building Mesos website locally.
+# TODO(vinod): Simplify the development workflow by getting rid of `docker`
+# or consolidating with the publish workflow in `support/mesos-website.sh`.
+# See MESOS-7860 for details.
+
+set -e
+set -o pipefail
+
+MESOS_DIR=$(git rev-parse --show-toplevel)
+
+pushd "$MESOS_DIR"
+
+TAG=mesos/website-`date +%s`-$RANDOM
+
+docker build -t $TAG site
+
+trap 'docker rmi $TAG' EXIT
+
+docker run \
+ -it \
+ --rm \
+ -p 4567:4567 \
+ -p 35729:35729 \
+ -v $MESOS_DIR:/mesos:Z \
+ $TAG
+
+popd # $MESOS_DIR
[2/3] mesos git commit: Added scripts to automate website publishing.
Posted by vi...@apache.org.
Added scripts to automate website publishing.
These scripts are expected to be run by ASF CI on any commit to
the mesos repo. The directory layout is similar to tidybot CI job.
Review: https://reviews.apache.org/r/60439
Project: http://git-wip-us.apache.org/repos/asf/mesos/repo
Commit: http://git-wip-us.apache.org/repos/asf/mesos/commit/23541eff
Tree: http://git-wip-us.apache.org/repos/asf/mesos/tree/23541eff
Diff: http://git-wip-us.apache.org/repos/asf/mesos/diff/23541eff
Branch: refs/heads/master
Commit: 23541effba3d24bf5b0eeb457601fe6148002134
Parents: c1fdaea
Author: Vinod Kone <vi...@gmail.com>
Authored: Thu Jun 22 10:43:33 2017 +0800
Committer: Vinod Kone <vi...@gmail.com>
Committed: Fri Aug 4 12:01:35 2017 -0700
----------------------------------------------------------------------
support/jenkins/websitebot.sh | 51 ++++++++++++++++++++++++++++++++
support/mesos-website.sh | 48 ++++++++++++++++++++++++++++++
support/mesos-website/Dockerfile | 26 ++++++++++++++++
support/mesos-website/build.sh | 44 +++++++++++++++++++++++++++
support/mesos-website/entrypoint.sh | 33 +++++++++++++++++++++
5 files changed, 202 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/mesos/blob/23541eff/support/jenkins/websitebot.sh
----------------------------------------------------------------------
diff --git a/support/jenkins/websitebot.sh b/support/jenkins/websitebot.sh
new file mode 100755
index 0000000..dccbbc3
--- /dev/null
+++ b/support/jenkins/websitebot.sh
@@ -0,0 +1,51 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# This script is executed by ASF CI to automatically publish the website
+# whenever there is a new commit in the mesos repo.
+set -e
+set -o pipefail
+
+: "${WORKSPACE:?"Environment variable 'WORKSPACE' must be set"}"
+
+export MESOS_DIR="$WORKSPACE/mesos"
+export MESOS_SITE_DIR="$WORKSPACE/mesos-site"
+
+pushd "$MESOS_DIR"
+
+if [ "$(git status --porcelain | wc -l)" -ne 0 ]; then
+ echo "This script is not intended to run on a live checkout, but the source
+ tree contains untracked files or uncommitted changes."
+
+ exit 1
+fi
+
+MESOS_HEAD_SHA=$(git rev-parse --verify --short HEAD)
+
+./support/mesos-website.sh
+
+popd # $MESOS_DIR
+
+# Commit the updated changes to the site repo for website update.
+pushd "$MESOS_SITE_DIR"
+
+git add .
+git commit -m "Updated the website built from mesos SHA: $MESOS_HEAD_SHA."
+git push origin HEAD:refs/heads/asf-site
+
+popd # $MESOS_SITE_DIR
http://git-wip-us.apache.org/repos/asf/mesos/blob/23541eff/support/mesos-website.sh
----------------------------------------------------------------------
diff --git a/support/mesos-website.sh b/support/mesos-website.sh
new file mode 100755
index 0000000..ce416df
--- /dev/null
+++ b/support/mesos-website.sh
@@ -0,0 +1,48 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# This is a wrapper for building Mesos website locally.
+set -e
+set -o pipefail
+
+MESOS_DIR=$(git rev-parse --show-toplevel)
+
+: "${MESOS_SITE_DIR:?"Environment variable 'MESOS_SITE_DIR' must be set"}"
+
+pushd "$MESOS_DIR"
+
+TAG=mesos/website-`date +%s`-$RANDOM
+
+docker build -t $TAG support/mesos-website
+
+trap 'docker rmi $TAG' EXIT
+
+# NOTE: We set `LOCAL_USER_ID` environment variable to enable running the
+# container process with the same UID as the user running this script. This
+# ensures that any writes to the mounted volumes will have the same permissions
+# as the user running the script, making it easy to do cleanups; otherwise
+# any writes will have permissions of UID 0 by default on Linux.
+
+docker run \
+ --rm \
+ -e LOCAL_USER_ID="$(id -u "$USER")" \
+ -v "$MESOS_DIR":/mesos:Z \
+ -v "$MESOS_SITE_DIR/content":/mesos/site/publish:Z \
+ $TAG
+
+popd # $MESOS_DIR
http://git-wip-us.apache.org/repos/asf/mesos/blob/23541eff/support/mesos-website/Dockerfile
----------------------------------------------------------------------
diff --git a/support/mesos-website/Dockerfile b/support/mesos-website/Dockerfile
new file mode 100644
index 0000000..4156eb4
--- /dev/null
+++ b/support/mesos-website/Dockerfile
@@ -0,0 +1,26 @@
+FROM centos:7
+MAINTAINER "dev@mesos.apache.org"
+
+LABEL Description="This image is used for generating Mesos web site."
+
+# Install Mesos build dependencies.
+RUN yum install -y which
+RUN yum groupinstall -y 'Development Tools'
+RUN yum install -y epel-release
+RUN yum install -y clang git maven cmake
+RUN yum install -y java-1.8.0-openjdk-devel python-devel zlib-devel libcurl-devel openssl-devel cyrus-sasl-devel cyrus-sasl-md5 apr-devel subversion-devel apr-utils-devel libevent-devel libev-devel
+
+# Install website dependencies.
+RUN yum install -y ruby ruby-devel doxygen
+RUN gem install bundler
+
+
+# Setup environment to build Mesos and website.
+ENV CC gcc
+ENV CXX g++
+ENV LANG en_US.UTF-8
+ENV LC_ALL en_US.UTF-8
+
+
+WORKDIR /mesos
+CMD bash support/mesos-website/entrypoint.sh
http://git-wip-us.apache.org/repos/asf/mesos/blob/23541eff/support/mesos-website/build.sh
----------------------------------------------------------------------
diff --git a/support/mesos-website/build.sh b/support/mesos-website/build.sh
new file mode 100755
index 0000000..afbec48
--- /dev/null
+++ b/support/mesos-website/build.sh
@@ -0,0 +1,44 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# This is a script for building Mesos website.
+set -e
+set -o pipefail
+
+function exit_hook {
+ # Remove generated documents when exit.
+ cd /mesos/site && bundle exec rake clean_docs
+}
+
+trap exit_hook EXIT
+
+# Build mesos to get the latest master and agent binaries.
+./bootstrap
+mkdir -p build
+pushd build
+../configure --disable-python
+make -j6
+popd # build
+
+# Generate the endpoint docs from the latest mesos and agent binaries.
+./support/generate-endpoint-help.py
+
+# Build the website.
+pushd site
+bundle exec rake
+popd # site
http://git-wip-us.apache.org/repos/asf/mesos/blob/23541eff/support/mesos-website/entrypoint.sh
----------------------------------------------------------------------
diff --git a/support/mesos-website/entrypoint.sh b/support/mesos-website/entrypoint.sh
new file mode 100755
index 0000000..72fd723
--- /dev/null
+++ b/support/mesos-website/entrypoint.sh
@@ -0,0 +1,33 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# This is a wrapper script for building Mesos website as a non-root user.
+set -e
+set -o pipefail
+
+# This needs to be run under `root` user for `bundle exec rake` to
+# work properly. See MESOS-7859.
+pushd site
+bundle install
+popd # site
+
+# Create a local user account.
+useradd -u $LOCAL_USER_ID -s /bin/bash -m tempuser
+
+# Build mesos and the website as the new user.
+su -c support/mesos-website/build.sh tempuser
[3/3] mesos git commit: Updated endpoint help generator script to
work inside Docker.
Posted by vi...@apache.org.
Updated endpoint help generator script to work inside Docker.
Changed agent flags to make this script work inside Docker container.
This is needed because this script will be run as part of website
publishing process which runs on ASF CI inside Docker.
Review: https://reviews.apache.org/r/60438
Project: http://git-wip-us.apache.org/repos/asf/mesos/repo
Commit: http://git-wip-us.apache.org/repos/asf/mesos/commit/c1fdaea2
Tree: http://git-wip-us.apache.org/repos/asf/mesos/tree/c1fdaea2
Diff: http://git-wip-us.apache.org/repos/asf/mesos/diff/c1fdaea2
Branch: refs/heads/master
Commit: c1fdaea22aec3c86017209b8352ff7d40589a8b0
Parents: cdb90b9
Author: Vinod Kone <vi...@gmail.com>
Authored: Thu Jun 22 10:33:27 2017 +0800
Committer: Vinod Kone <vi...@gmail.com>
Committed: Fri Aug 4 12:01:35 2017 -0700
----------------------------------------------------------------------
support/generate-endpoint-help.py | 5 ++++-
1 file changed, 4 insertions(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/mesos/blob/c1fdaea2/support/generate-endpoint-help.py
----------------------------------------------------------------------
diff --git a/support/generate-endpoint-help.py b/support/generate-endpoint-help.py
index 19cff57..7e59b35 100755
--- a/support/generate-endpoint-help.py
+++ b/support/generate-endpoint-help.py
@@ -52,10 +52,13 @@ MASTER_COMMAND = [
'--work_dir=/tmp/mesos'
]
+# NOTE: The agent flags here ensure that this script can run inside docker.
AGENT_COMMAND = [
'mesos-agent.sh',
'--master=%s:%s' % (HOST_IP, MASTER_PORT),
- '--work_dir=/tmp/mesos'
+ '--work_dir=/tmp/mesos',
+ '--systemd_enable_support=false',
+ '--launcher=posix'
]