You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@mesos.apache.org by bb...@apache.org on 2019/09/18 10:52:37 UTC
[mesos] branch master updated (9333717 -> fb467a0)
This is an automated email from the ASF dual-hosted git repository.
bbannier pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/mesos.git.
from 9333717 Included new Python CLI in distribution tarball.
new 2232d48 Moved cpplint configuration into dedicated file.
new 526043b Added check script to check for license headers.
new cbaca81 Added gitlint config.
new a138c2b Added separate script to install developer setup.
new 454661d Switched commit hooks to pre-commit.
new 37d76ff Removed old mesos-style and references.
new 3478e40 Revert "Updated cpplint.py to be less verbose when there is no linting issue."
new 2af3396 Revert "Updated cpplint to be compatible with Python 3."
new fb467a0 Enabled a number of additional pre-commit checks.
The 9 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
bootstrap | 70 ---
docs/advanced-contribution.md | 5 +-
docs/beginner-contribution.md | 2 +-
docs/c++-style-guide.md | 2 +-
docs/clang-format.md | 5 +-
support/CPPLINT.cfg | 2 +
support/build-virtualenv | 72 ---
.../cli/__init__.py => support/check-license.py | 20 +-
support/cpplint.py | 34 +-
support/gitignore | 11 +-
support/gitlint | 48 ++
support/hooks/commit-msg | 46 --
support/hooks/post-rewrite | 40 --
support/hooks/pre-commit | 41 --
support/llvm/README.md | 14 +-
support/mesos-split.py | 2 -
support/mesos-style.py | 581 +--------------------
support/pre-commit-config.yaml | 54 ++
bootstrap.bat => support/setup-dev.bat | 28 +-
support/setup-dev.sh | 68 +++
20 files changed, 251 insertions(+), 894 deletions(-)
create mode 100644 support/CPPLINT.cfg
delete mode 100755 support/build-virtualenv
copy src/python/cli_new/lib/cli/__init__.py => support/check-license.py (69%)
mode change 100644 => 100755
create mode 100644 support/gitlint
delete mode 100755 support/hooks/commit-msg
delete mode 100755 support/hooks/post-rewrite
delete mode 100755 support/hooks/pre-commit
create mode 100644 support/pre-commit-config.yaml
rename bootstrap.bat => support/setup-dev.bat (83%)
create mode 100755 support/setup-dev.sh
[mesos] 04/09: Added separate script to install developer setup.
Posted by bb...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
bbannier pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/mesos.git
commit a138c2bd7cb3749f1dceb0e520e1138536abb531
Author: Benjamin Bannier <bb...@apache.org>
AuthorDate: Wed Sep 18 11:37:13 2019 +0200
Added separate script to install developer setup.
This patch breaks the installation of developer tools (i.e., linter
configuration files and git hooks) out of `./bootstrap`. This not only
simplifies and streamlines the setup, but will allow us to add
developer-only features without breaking users who are just interested
in building a distribution tarball.
Review: https://reviews.apache.org/r/71299/
---
bootstrap | 78 ----------------------------------
docs/advanced-contribution.md | 2 +-
docs/beginner-contribution.md | 2 +-
docs/clang-format.md | 5 ++-
support/gitignore | 12 +++---
support/llvm/README.md | 14 +++---
bootstrap.bat => support/setup-dev.bat | 2 +-
bootstrap => support/setup-dev.sh | 58 ++++++++-----------------
8 files changed, 38 insertions(+), 135 deletions(-)
diff --git a/bootstrap b/bootstrap
index e88e8cb..5727550 100755
--- a/bootstrap
+++ b/bootstrap
@@ -10,84 +10,6 @@ __EOF__
exit 1
fi
-# Helper functions to get the absolute path of a directory as well as compute
-# the relative path between two directories.
-# Unfortunately these functions are not as readable as, say, python's
-# os.path.relpath method, but they allow us to avoid a hard dependence on
-# python or other tools during bootstrap.
-abspath() {
- cd "`dirname "${1}"`"
- echo "${PWD}"/"`basename "${1}"`"
- cd "${OLDPWD}"
-}
-relpath() {
- local from to up
- from="`abspath "${1%/}"`" to="`abspath "${2%/}"/`"
- while test "${to}" = "${to#"${from}"/}" \
- -a "${to}" != "${from}"; do
- from="${from%/*}" up="../${up}"
- done
- to="${up%/}${to#${from}}"
- echo "${to:-.}"
-}
-
-# Grab a reference to the repo's git directory. Usually this is simply .git in
-# the repo's top level directory. However, when submodules are used, it may
-# appear elsewhere. The most up-to-date way of finding this directory is to use
-# `git rev-parse --git-common-dir`. This is necessary to support things like
-# git worktree in addition to git submodules. However, as of January 2016,
-# support for the '--git-common-dir' flag is fairly new, forcing us to fall
-# back to the older '--git-dir' flag if '--git-common-dir' is not supported. We
-# do this by checking the output of `git rev-parse --git-common-dir` and seeing
-# if it gives us a valid directory back. If not, we set the git directory using
-# the '--git-dir' flag instead.
-_gitdir=`git rev-parse --git-common-dir`
-if test ! -d "${_gitdir}"; then
- _gitdir=`git rev-parse --git-dir`
-fi
-
-# Grab a reference to the git hooks directory as well as the relative path from
-# the git hooks directory to the current directory.
-_hooksdir=${_gitdir}/hooks
-_relpath=`relpath "${_hooksdir}" "${PWD}"`
-
-# Install mesos default hooks and gitignore template.
-if test ! -e "${_hooksdir}/pre-commit"; then
- ln -s "${_relpath}/support/hooks/pre-commit" "${_hooksdir}/pre-commit"
-fi
-
-if test ! -e "${_hooksdir}/post-rewrite"; then
- ln -s "${_relpath}/support/hooks/post-rewrite" "${_hooksdir}/post-rewrite"
-fi
-
-if test ! -e "${_hooksdir}/commit-msg"; then
- ln -s "${_relpath}/support/hooks/commit-msg" "${_hooksdir}/commit-msg"
-fi
-
-if test ! -e .gitignore; then
- ln -s support/gitignore .gitignore
-fi
-
-if test ! -e .reviewboardrc; then
- ln -s support/reviewboardrc .reviewboardrc
-fi
-
-if test ! -e .clang-format; then
- ln -s support/clang-format .clang-format
-fi
-
-if test ! -e .clang-tidy; then
- ln -s support/clang-tidy .clang-tidy
-fi
-
-if test ! -e CPPLINT.cfg; then
- ln -s support/CPPLINT.cfg CPPLINT.cfg
-fi
-
-if test ! -e .gitlint; then
- ln -s support/gitlint .gitlint
-fi
-
if [ -n "$AUTOMAKE" ] || [ -n "$ACLOCAL" ] ; then
if [ -z "$ACLOCAL" ] || [ -z "$AUTOMAKE" ] ; then
_present="AUTOMAKE"
diff --git a/docs/advanced-contribution.md b/docs/advanced-contribution.md
index 573138d..6582849 100644
--- a/docs/advanced-contribution.md
+++ b/docs/advanced-contribution.md
@@ -66,7 +66,7 @@ Here is the standard procedure for proposing and making changes to Mesos:
2. Make your changes to the code (using whatever IDE/editor you choose) to actually fix the bug or implement the feature.
1. Before beginning, please read the [Mesos C++ Style Guide](c++-style-guide.md). It is recommended to use the git pre-commit hook (`support/hooks/pre-commit`) to automatically check for style errors. See the hook script for instructions to enable it.
2. Most of your changes will probably be to files inside of `BASE_MESOS_DIR`
- 3. From inside of the root Mesos directory: `./bootstrap` (Only required if building from git repository).
+ 3. From inside of the root Mesos directory: `./bootstrap` and `./support/setup-dev.sh`.
4. To build, we recommend that you don't build inside of the src directory. We recommend you do the following:
1. From inside of the root Mesos directory: `mkdir build && cd build`
2. `../configure`
diff --git a/docs/beginner-contribution.md b/docs/beginner-contribution.md
index 471f5dd..3a49862 100644
--- a/docs/beginner-contribution.md
+++ b/docs/beginner-contribution.md
@@ -19,7 +19,7 @@ First, download the latest development version of the Mesos codebase. In order t
If you're proposing a documentation-only change, then you don't need to build Mesos to get started.
-If you're making a functional change to the code, then you should build Mesos first. Once you have the Mesos source code on your local machine, you can install the necessary dependencies and build it. Instructions for this process can be found in the [building](building.md) page. Note that the `bootstrap` script in the repository's root directory will install git hooks which will help you adhere to Mesos style when committing.
+If you're making a functional change to the code, then you should build Mesos first. Once you have the Mesos source code on your local machine, you can install the necessary dependencies and build it. Instructions for this process can be found in the [building](building.md) page. Note that the `./support/setup-dev.sh` script will install git hooks which will help you adhere to Mesos style when committing.
## Find a Problem to Solve
diff --git a/docs/clang-format.md b/docs/clang-format.md
index 4289813..ff00685 100644
--- a/docs/clang-format.md
+++ b/docs/clang-format.md
@@ -36,8 +36,9 @@ layout: documentation
By default, ClangFormat uses the configuration defined in a `.clang-format` or
`_clang-format` file located in the nearest parent directory of the input file.
-The `bootstrap` script creates a `.clang-format` symlink at the top-level
-directory which points to `support/clang-format` for ClangFormat to find.
+The `./support/setup-dev.sh` script creates a `.clang-format` symlink at the
+top-level directory which points to `support/clang-format` for ClangFormat to
+find.
### Editor Integration
diff --git a/support/gitignore b/support/gitignore
index 3d53411..703b14b 100644
--- a/support/gitignore
+++ b/support/gitignore
@@ -2,11 +2,6 @@
.gitignore
# Files created by ./bootstrap.
-.clang-format
-.clang-tidy
-.reviewboardrc
-.gitlint
-CPPLINT.cfg
Makefile.in
aclocal.m4
ar-lib
@@ -25,6 +20,13 @@ ltversion.m4
lt~obsolete.m4
missing
+# Files created by ./support/setup-dev.sh
+.clang-format
+.clang-tidy
+.reviewboardrc
+.gitlint
+CPPLINT.cfg
+
# Recommended build directory.
/build/
diff --git a/support/llvm/README.md b/support/llvm/README.md
index 1889302..7491c4e 100644
--- a/support/llvm/README.md
+++ b/support/llvm/README.md
@@ -28,9 +28,9 @@ and format selected regions of code.
### Configuration
By default, [ClangFormat] uses the configuration defined in a `.clang-format`
-file located in the nearest parent directory of the input file.
-The `bootstrap` script creates a `.clang-format` symlink at `<MESOS_DIR>`
-which points to `support/clang-format` for [ClangFormat] to find.
+file located in the nearest parent directory of the input file. The
+`support/setup-dev.sh` script creates a `.clang-format` symlink at
+`<MESOS_DIR>` which points to `support/clang-format` for [ClangFormat] to find.
### Integration
@@ -55,10 +55,10 @@ Refer to https://clang.llvm.org/docs/ClangFormat.html#vim-integration
### Configuration
-By default, [ClangTidy] uses the configuration defined in a `.clang-tidy`
-file located in the nearest parent directory of the input file.
-The `bootstrap` script creates a `.clang-tidy` symlink at `<MESOS_DIR>`
-which points to `support/clang-tidy` for [ClangTidy] to find.
+By default, [ClangTidy] uses the configuration defined in a `.clang-tidy` file
+located in the nearest parent directory of the input file. The
+`./support/setup-dev.sh` script creates a `.clang-tidy` symlink at
+`<MESOS_DIR>` which points to `support/clang-tidy` for [ClangTidy] to find.
### Invocation
diff --git a/bootstrap.bat b/support/setup-dev.bat
similarity index 96%
rename from bootstrap.bat
rename to support/setup-dev.bat
index 7ca6449..3f4154a 100644
--- a/bootstrap.bat
+++ b/support/setup-dev.bat
@@ -70,7 +70,7 @@ goto:eof
:: If we are not in the root directory, print error and exit.
:not_in_root
echo. 1>&2
-echo You must run bootstrap from the root of the distribution. 1>&2
+echo You must run support/setup-dev.bat from the root of the distribution. 1>&2
echo. 1>&2
exit /b 1
diff --git a/bootstrap b/support/setup-dev.sh
similarity index 70%
copy from bootstrap
copy to support/setup-dev.sh
index e88e8cb..183effa 100755
--- a/bootstrap
+++ b/support/setup-dev.sh
@@ -1,10 +1,26 @@
#!/bin/sh
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
# Make sure that we are in the right directory.
if test ! -f configure.ac; then
cat >&2 <<__EOF__
-You must run bootstrap from the root of the distribution.
+You must run 'support/setup-dev.sh' from the root of the distribution.
__EOF__
exit 1
@@ -64,6 +80,7 @@ if test ! -e "${_hooksdir}/commit-msg"; then
ln -s "${_relpath}/support/hooks/commit-msg" "${_hooksdir}/commit-msg"
fi
+
if test ! -e .gitignore; then
ln -s support/gitignore .gitignore
fi
@@ -87,42 +104,3 @@ fi
if test ! -e .gitlint; then
ln -s support/gitlint .gitlint
fi
-
-if [ -n "$AUTOMAKE" ] || [ -n "$ACLOCAL" ] ; then
- if [ -z "$ACLOCAL" ] || [ -z "$AUTOMAKE" ] ; then
- _present="AUTOMAKE"
- _missing="ACLOCAL"
-
- [ -n "$ACLOCAL" ] && _present="ACLOCAL" && _missing="AUTOMAKE"
-
- cat >&2 <<__EOF__
-
-[ERROR]: You are providing the path to ${_present}
-through your environment but no reference to ${_missing}.
-To fix this error please specify ${_missing} too.
-
-As an example, if you are using automake-1.12 and have
-available aclocal-1.12 you will want to do the following:
-
- AUTOMAKE="/usr/local/bin/automake-1.12" \\
- ACLOCAL="/usr/local/bin/aclocal-1.12" \\
- ./bootstrap
-
-Your current environment has:
- AUTOMAKE="$AUTOMAKE"
- ACLOCAL="$ACLOCAL"
-
-__EOF__
- exit 1
- fi
-else
- AUTOMAKE="$(which automake)"
-fi
-
-
-# Note that we don't use '--no-recursive' because older versions of
-# autoconf/autoreconf bail with that option. Unfortunately this means
-# that we'll modify a lot of files in 3rdparty/libprocess, but that
-# may change in the future.
-
-autoreconf --install -Wall --verbose "${@}"
[mesos] 07/09: Revert "Updated cpplint.py to be less verbose when
there is no linting issue."
Posted by bb...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
bbannier pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/mesos.git
commit 3478e40c656160b8f08e0ad8c154289417bb6aaa
Author: Benjamin Bannier <bb...@apache.org>
AuthorDate: Wed Sep 18 11:37:17 2019 +0200
Revert "Updated cpplint.py to be less verbose when there is no linting issue."
This reverts commit c0f8f56d5a93f3fb870e448fedfd22f1491356ca.
This patch was necessary when we were running cpplint via
`support/mesos-style.py` to prevent it from cluttering up the hook
output. When running under pre-commit linter output is not shown if no
errors occur so we can undo our change to stay closer to upstream.
Review: https://reviews.apache.org/r/71207/
---
support/cpplint.py | 3 +--
1 file changed, 1 insertion(+), 2 deletions(-)
diff --git a/support/cpplint.py b/support/cpplint.py
index 66ec8b3..5671873 100755
--- a/support/cpplint.py
+++ b/support/cpplint.py
@@ -978,8 +978,7 @@ class _CppLintState(object):
for category, count in iteritems(self.errors_by_category):
sys.stderr.write('Category \'%s\' errors found: %d\n' %
(category, count))
- if self.error_count > 0:
- sys.stdout.write('Total errors found: %d\n' % self.error_count)
+ sys.stdout.write('Total errors found: %d\n' % self.error_count)
_cpplint_state = _CppLintState()
[mesos] 06/09: Removed old mesos-style and references.
Posted by bb...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
bbannier pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/mesos.git
commit 37d76fff124d28a0281b9231058bb1b92fc65abe
Author: Benjamin Bannier <bb...@apache.org>
AuthorDate: Wed Sep 18 11:37:15 2019 +0200
Removed old mesos-style and references.
This patch removes references to `support/mesos-style.py` which was
replaced with a pre-commit setup in a previous commit. We also remove
the tool itself.
Review: https://reviews.apache.org/r/71206/
---
docs/c++-style-guide.md | 2 +-
support/build-virtualenv | 72 ------
support/hooks/commit-msg | 46 ----
support/hooks/post-rewrite | 40 ----
support/hooks/pre-commit | 41 ----
support/mesos-split.py | 2 -
support/mesos-style.py | 557 ++-------------------------------------------
7 files changed, 17 insertions(+), 743 deletions(-)
diff --git a/docs/c++-style-guide.md b/docs/c++-style-guide.md
index 8a48afe..8df99c8 100644
--- a/docs/c++-style-guide.md
+++ b/docs/c++-style-guide.md
@@ -7,7 +7,7 @@ layout: documentation
The Mesos codebase follows the [Google C++ Style Guide](https://google.github.io/styleguide/cppguide.html) with some notable differences, as described below.
-Note that the [clang-format](clang-format.md) tool can be helpful to ensure that some of the mechanical style rules are obeyed. Commits should be checked with the script `support/mesos-style.py` for high-level conformance, or with `support/mesos-tidy.sh` for conformance to low-level expectations.
+Note that the [clang-format](clang-format.md) tool can be helpful to ensure that some of the mechanical style rules are obeyed. Commits should be checked with the command `pre-commit run cpplint` for high-level conformance, or with `support/mesos-tidy.sh` for conformance to low-level expectations.
## Scoping
diff --git a/support/build-virtualenv b/support/build-virtualenv
deleted file mode 100755
index 7dc03b0..0000000
--- a/support/build-virtualenv
+++ /dev/null
@@ -1,72 +0,0 @@
-#!/usr/bin/env bash
-
-# This script sets up a Python virtualenv for the Web UI. This creates
-# a new virtualenv and installs nodeenv inside the virtualenv.
-
-set -e
-trap "exit 1" INT
-
-CURRDIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
-
-: ${VIRTUALENV_NAME:="linters"}
-: ${VIRTUALENV_DIRECTORY:="${CURRDIR}/.virtualenv"}
-
-: ${PYTHON:="$(which python)"}
-: ${VIRTUALENV:="$(which virtualenv)"}
-
-OLD_PYTHONPATH="${PYTHONPATH}"
-PYTHONPATH=""
-
-# If we already have a virtual environment activated,
-# bail out and advise the user to deactivate.
-OLD_VIRTUAL_ENV="${VIRTUAL_ENV}"
-if [ "${OLD_VIRTUAL_ENV}" != "" ]; then
- echo "Please deactivate your current virtual environment in order to continue."
- echo "source deactivate"
- exit 1
-fi
-
-# Verify that python is installed.
-if [ "${PYTHON}" = "" ]; then
- echo "You must have python installed in order to continue."
- exit 1
-fi
-
-# Old versions of virtualenv do not remove the bin directory in the
-# virtual environment even when using `--clear`. We thus remove the
-# entire directory in case the virtual environment already exists.
-# See https://github.com/pypa/virtualenv/issues/2 for more info.
-rm -rf ${VIRTUALENV_DIRECTORY}
-
-PYTHON_MAJOR=$(${PYTHON} -c 'import sys; print(sys.version_info[0])')
-PYTHON_MINOR=$(${PYTHON} -c 'import sys; print(sys.version_info[1])')
-
-if [ "${PYTHON_MAJOR}" = "3" ]; then
- if [ "${PYTHON_MINOR}" -lt "6" ]; then
- echo "You must be running python 3.6 or newer in order to continue."
- echo "Consider running as 'PYTHON=python3 ${0}' or similar."
- exit 1
- else
- # Set up a virtual environment for the linters.
- ${PYTHON} -m venv --prompt="${VIRTUALENV_NAME}" ${VIRTUALENV_DIRECTORY}
- fi
-else
- echo "You must be running python 3.6 or newer in order to continue."
- echo "Consider running as 'PYTHON=python3 ${0}' or similar."
- exit 1
-fi
-
-source ${VIRTUALENV_DIRECTORY}/bin/activate
-pip install --upgrade pip
-pip install -r ${CURRDIR}/pip-requirements.txt
-
-# Add Node.js virtual environment to the existing virtual environment.
-nodeenv -p
-
-# Restart the virtual environment to then have npm available.
-deactivate
-source ${VIRTUALENV_DIRECTORY}/bin/activate
-
-# Install the JavaScript linter in the virtual environment.
-npm install -g eslint@5.1.0
-deactivate
diff --git a/support/hooks/commit-msg b/support/hooks/commit-msg
deleted file mode 100755
index a0c218d..0000000
--- a/support/hooks/commit-msg
+++ /dev/null
@@ -1,46 +0,0 @@
-#!/usr/bin/env bash
-#
-# A hook script to verify commit message format. Called by "git commit"
-# with one argument, the name of the file that has the commit message.
-# The hook exits with non-zero status after issuing an appropriate
-# message if it wants to stop the commit. The hook is allowed to edit the
-# commit message file.
-#
-# To enable this hook, run `bootstrap` or do this from the root of the repo:
-#
-# $ ln -s ../../support/hooks/commit-msg .git/hooks/commit-msg
-
-COMMIT_MESSAGE=$(cat "$1")
-FIRST_LINE=$(head -n 1 "$1")
-LAST_CHAR=$(echo -n "$FIRST_LINE" | tail -c 1)
-FIRST_CHAR=$(echo -n "$FIRST_LINE" | head -c 1)
-
-if [[ "$FIRST_LINE" =~ ^(fixup|squash)\! ]]; then
- # If the commit starts with fixup! or squash! we ignore everything.
- # The message will be checked when the final commit is made.
- exit 0
-fi
-
-while read -r LINE
-do
- # In verbose mode, the diff of the commit is included in the commit message.
- # Since git looks for the following line and skips everything after it,
- # we also skip everything once this line is observed.
- if [ "$LINE" = "# ------------------------ >8 ------------------------" ]; then break; fi
- if [ "$(echo -n "$LINE" | head -c 1)" = "#" ]; then continue; fi
- LENGTH=$(echo -n "$LINE" | wc -c)
- if [ "$LENGTH" -gt "72" ]; then
- echo >&2 "Error: No line in the commit message summary may exceed 72 characters."
- exit 1
- fi
-done <<< "$COMMIT_MESSAGE"
-
-if [[ ! "$FIRST_CHAR" =~ [A-Z] ]]; then
- echo >&2 "Error: Commit message summary (the first line) must start with a capital letter."
- exit 1
-fi
-
-if [ "$LAST_CHAR" != "." ]; then
- echo >&2 "Error: Commit message summary (the first line) must end in a period."
- exit 1
-fi
diff --git a/support/hooks/post-rewrite b/support/hooks/post-rewrite
deleted file mode 100755
index 1ab14ab..0000000
--- a/support/hooks/post-rewrite
+++ /dev/null
@@ -1,40 +0,0 @@
-#!/bin/sh
-#
-# A hook script to verify what is about to be committed.
-# Called by "git commit --amend" or "git rebase". The hook exits with
-# non-zero status and warning messages if the files being rewritten do
-# not conform to the Mesos style.
-#
-# To enable this hook, do this from the root of the repo:
-#
-# $ ln -s ../../support/hooks/post_rewrite .git/hooks/post_rewrite
-
-# Redirect output to stderr.
-exec 1>&2
-
-# If there are whitespace errors, print the offending file names.
-## In git, '@~' represent previous commit. We check the whitespace between
-## current head and previous commit after a commit is rewritten.
-git diff-index --check @~ -- || exit 1
-
-# Check Mesos style.
-## In git, '@' represent current head, '@~' represent previous commit. We check
-## the style of changes between current head and previous commit after a commit
-## is rewritten.
-ADDED_OR_MODIFIED=$(git diff --name-only --diff-filter=AM @~..@)
-if [ "$ADDED_OR_MODIFIED" ]; then
- # NOTE: We choose to implement this as a conditional rather than a call to
- # `xargs` on purpose. Some implementations of `xargs` will call your script
- # even if the arguments you pass in are empty. In our case, this would
- # cause us to erroneously lint every file in the repository. Additionally,
- # many implementations do not support the `-r` flag, (which instructs
- # `xargs` to not run the script if the arguments are empty), so we also
- # cannot use that.
- ./support/mesos-style.py $ADDED_OR_MODIFIED || exit 1
-fi
-
-# Check that the commits are properly split between mesos, libprocess and stout.
-## In git, '@' represent current head, '@~' represent previous commit. We check
-## the style of changes between current head and previous commit after a commit
-## is rewritten.
-git diff --name-only --diff-filter=AMD @~..@ | xargs ./support/mesos-split.py || exit 1
diff --git a/support/hooks/pre-commit b/support/hooks/pre-commit
deleted file mode 100755
index 519567b..0000000
--- a/support/hooks/pre-commit
+++ /dev/null
@@ -1,41 +0,0 @@
-#!/bin/sh
-#
-# A hook script to verify what is about to be committed.
-# Called by "git commit" with no arguments. The hook exits with
-# non-zero status if the files being committed do not conform to
-# the Mesos style.
-#
-# To enable this hook, do this from the root of the repo:
-#
-# $ ln -s ../../support/hooks/pre-commit .git/hooks/pre-commit
-
-if git rev-parse --verify HEAD >/dev/null 2>&1
-then
- against=HEAD
-else
- # Initial commit: diff against an empty tree object
- against=4b825dc642cb6eb9a060e54bf8d69288fbee4904
-fi
-
-# Redirect output to stderr.
-exec 1>&2
-
-# If there are whitespace errors, print the offending file names and fail.
-git diff-index --check --cached $against -- || exit 1
-
-# Check Mesos style.
-ADDED_OR_MODIFIED=$(git diff --cached --name-only --diff-filter=AM)
-if [ -n "$ADDED_OR_MODIFIED" ]; then
- # NOTE: We choose to implement this as a conditional rather than a call to
- # `xargs` on purpose. Some implementations of `xargs` will call your script
- # even if the arguments you pass in are empty. In our case, this would
- # cause us to erroneously lint every file in the repository. Additionally,
- # many implementations do not support the `-r` flag, (which instructs
- # `xargs` to not run the script if the arguments are empty), so we also
- # cannot use that.
- ./support/mesos-style.py $ADDED_OR_MODIFIED || exit 1
-fi
-
-# Check that the commits are properly split between mesos, libprocess and stout.
-# TODO(ArmandGrillet): Remove the if to really switch to Python 3.
-git diff --cached --name-only --diff-filter=AMD | xargs ./support/mesos-split.py || exit 1
diff --git a/support/mesos-split.py b/support/mesos-split.py
index 0a77c25..aecad3f 100755
--- a/support/mesos-split.py
+++ b/support/mesos-split.py
@@ -58,8 +58,6 @@ def find_project(filename):
def main():
"""
Expects a list of filenames on the command line.
-
- See `support/hooks/pre-commit` for the canonical usage of this method.
"""
touched_projects = defaultdict(list)
for filename in sys.argv[1:]:
diff --git a/support/mesos-style.py b/support/mesos-style.py
index b3e20fd..4d9d3b4 100755
--- a/support/mesos-style.py
+++ b/support/mesos-style.py
@@ -1,5 +1,5 @@
#!/usr/bin/env python3
-#
+
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
@@ -15,551 +15,26 @@
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
+"""This is a dummy script to ease the transition from mesos-style to pre-commit
+hooks. This script can be removed after we have given contributors enough time
+to adjust their checkouts."""
-"""Runs checks for mesos style."""
-
-import os
-import platform
-import re
-import string
-import subprocess
import sys
-from pathlib import PurePath
-
-
-class LinterBase():
- """
- This is an abstract class that provides the base functionality for
- linting files in the mesos project. Its 'main()' function
- walks through the set of files passed to it and runs some
- standard linting over them. This includes checking for license headers
- and checking for non-supported characters. From there it calls a
- 'run_lint()' function that can be overridden to provide
- customizable style checks for a specific class of files (e.g. C++,
- Python, etc.).
-
- Any class that extends from 'LinterBase' should override the
- following class variables / functions:
-
- linter_type
- source_dirs
- exclude_files
- source_files
- comment_prefix
-
- run_lint()
-
- Please see the comments below for details on how to override each
- variable.
- """
- # The name of the linter to help with printing which linter files
- # are currently being processed by.
- linter_type = ''
-
- # Root source paths (will be traversed recursively).
- source_dirs = []
-
- # Add file paths and patterns which should not be checked
- # This should include 3rdparty libraries, includes and machine generated
- # source.
- exclude_files = ''
-
- # A regex of possible matches for your source files.
- source_files = ''
-
- # A prefix at the beginning of the line to demark comments (e.g. '//')
- comment_prefix = ''
-
- def check_encoding(self, source_paths):
- """
- Checks for encoding errors in the given files. Source
- code files must contain only printable ascii characters.
- This excludes the extended ascii characters 128-255.
- http://www.asciitable.com/
- """
- error_count = 0
- for path in source_paths:
- with open(path) as source_file:
- for line_number, line in enumerate(source_file):
- # If we find an error, add 1 to both the character and
- # the line offset to give them 1-based indexing
- # instead of 0 (as is common in most editors).
- char_errors = [offset for offset, char in enumerate(line)
- if char not in string.printable]
- if char_errors:
- sys.stderr.write(
- "{path}:{line_number}: Non-printable characters"
- " found at [{offsets}]: \"{line}\"\n".format(
- path=path,
- line_number=line_number + 1,
- offsets=', '.join([str(offset + 1) for offset
- in char_errors]),
- line=line.rstrip()))
- error_count += 1
-
- return error_count
-
- def check_license_header(self, source_paths):
- """Checks the license headers of the given files."""
- error_count = 0
- for path in source_paths:
- with open(path) as source_file:
- # We read the three first lines of the file as the
- # first line could be a shebang and the second line empty.
- head = "".join([next(source_file) for _ in range(3)])
-
- # TODO(bbannier) We allow `Copyright` for
- # currently deviating files. This should be
- # removed one we have a uniform license format.
- regex = r'^{comment_prefix} [Licensed|Copyright]'.format(
- comment_prefix=self.comment_prefix)
- # pylint: disable=no-member
- regex = re.compile(regex, re.MULTILINE)
-
- if not regex.search(head):
- sys.stderr.write(
- "{path}:1: A license header should appear's on one of"
- " the first line of the file starting with"
- " '{comment_prefix} Licensed'.: {head}".format(
- path=path,
- head=head,
- comment_prefix=self.comment_prefix))
- error_count += 1
-
- return error_count
-
- def find_candidates(self, root_dir):
- """
- Search through the all files rooted at 'root_dir' and compare
- them against 'self.exclude_files' and 'self.source_files' to
- come up with a set of candidate files to lint.
- """
- exclude_file_regex = re.compile(self.exclude_files)
- source_criteria_regex = re.compile(self.source_files)
- for root, _, files in os.walk(root_dir):
- for name in files:
- path = os.path.join(root, name)
- if exclude_file_regex.search(path) is not None:
- continue
-
- if source_criteria_regex.search(name) is not None:
- yield path
-
- def run_command_in_virtualenv(self, command):
- """
- Activate the virtual environment, run the
- given command and return its output.
- """
- virtualenv = os.path.join('support', '.virtualenv')
-
- if platform.system() == 'Windows':
- command = r'{virtualenv_path}\Scripts\activate.bat & {cmd}'.format(
- virtualenv_path=virtualenv, cmd=command)
- else:
- command = '. {virtualenv_path}/bin/activate; {cmd}'.format(
- virtualenv_path=virtualenv, cmd=command)
-
- return subprocess.Popen(command, shell=True, stdout=subprocess.PIPE)
-
- # pylint: disable=unused-argument
- def run_lint(self, source_paths):
- """
- A custom function to provide linting for 'linter_type'.
- It takes a list of source files to lint and returns the number
- of errors found during the linting process.
-
- It should print any errors as it encounters them to provide
- feedback to the caller.
- """
- return 0
-
- def main(self, modified_files):
- """
- This function takes a list of files and lints them for the
- class of files defined by 'linter_type'.
- """
-
- # Verify that source roots are accessible from current
- # working directory. A common error could be to call
- # the style checker from other (possibly nested) paths.
- for source_dir in self.source_dirs:
- if not os.path.exists(source_dir):
- print("Could not find '{dir}'".format(dir=source_dir))
- print('Please run from the root of the mesos source directory')
- exit(1)
-
- # Add all source file candidates to candidates list.
- candidates = []
- for source_dir in self.source_dirs:
- for candidate in self.find_candidates(source_dir):
- candidates.append(candidate)
-
- # Normalize paths of any files given.
- modified_files = [os.fspath(PurePath(f)) for f in modified_files]
-
- # If file paths are specified, check all file paths that are
- # candidates; else check all candidates.
- file_paths = modified_files if modified_files else candidates
-
- # Compute the set intersect of the input file paths and candidates.
- # This represents the reduced set of candidates to run lint on.
- candidates_set = set(candidates)
- clean_file_paths_set = set(path.rstrip() for path in file_paths)
- filtered_candidates_set = clean_file_paths_set.intersection(
- candidates_set)
-
- if filtered_candidates_set:
- plural = '' if len(filtered_candidates_set) == 1 else 's'
- print('Checking {num_files} {linter} file{plural}'.format(
- num_files=len(filtered_candidates_set),
- linter=self.linter_type,
- plural=plural))
-
- license_errors = self.check_license_header(filtered_candidates_set)
- encoding_errors = self.check_encoding(list(filtered_candidates_set))
- lint_errors = self.run_lint(list(filtered_candidates_set))
- total_errors = license_errors + encoding_errors + lint_errors
-
- if total_errors > 0:
- sys.stderr.write('Total errors found: {num_errors}\n'.format(
- num_errors=total_errors))
-
- return total_errors
-
- return 0
-
-
-class CppLinter(LinterBase):
- """The linter for C++ files, uses cpplint."""
- linter_type = 'C++'
-
- source_dirs = ['src',
- 'include',
- os.path.join('3rdparty', 'libprocess'),
- os.path.join('3rdparty', 'stout')]
-
- exclude_files = '(' \
- r'elfio\-3\.2|' \
- r'protobuf\-2\.4\.1|' \
- r'googletest\-release\-1\.8\.0|' \
- r'glog\-0\.3\.3|' \
- r'boost\-1\.53\.0|' \
- r'libev\-4\.15|' \
- r'\.pb\.cc|\.pb\.h|\.md|\.virtualenv' \
- ')'
-
- source_files = r'\.(cpp|hpp|cc|h)$'
-
- comment_prefix = r'\/\/'
-
- def run_lint(self, source_paths):
- """
- Runs cpplint over given files.
-
- http://google-styleguide.googlecode.com/svn/trunk/cpplint/cpplint.py
- """
- # We do not use a version of cpplint available through pip as
- # we use a custom version (see cpplint.path) to lint C++ files.
- process = subprocess.Popen(
- [sys.executable, 'support/cpplint.py', '--extensions=hpp,cpp'] +
- source_paths,
- stderr=subprocess.PIPE,
- close_fds=True)
-
- # Lines are stored and filtered, only showing found errors instead
- # of e.g., 'Done processing XXX.' which tends to be dominant output.
- for line in process.stderr:
- line = line.decode(sys.stdout.encoding)
- if re.match('^(Done processing |Total errors found: )', line):
- continue
- sys.stderr.write(line)
-
- process.wait()
- return process.returncode
-
-
-class JsLinter(LinterBase):
- """The linter for JavaScript files, uses eslint."""
- linter_type = 'JavaScript'
-
- source_dirs = [os.path.join('src', 'webui')]
-
- exclude_files = '(' \
- r'angular\-1\.2\.32|' \
- r'angular\-route\-1\.2\.32|' \
- r'bootstrap\-table\-1\.11\.1|' \
- r'clipboard\-1\.5\.16|' \
- r'jquery\-3\.2\.1|' \
- r'relative\-date|' \
- r'ui\-bootstrap\-tpls\-0\.9\.0|' \
- r'angular\-route\-1\.2\.32|' \
- r'underscore\-1\.4\.3' \
- ')'
-
- source_files = r'\.(js)$'
-
- comment_prefix = '//'
-
- def run_lint(self, source_paths):
- """
- Runs eslint over given files.
-
- https://eslint.org/docs/user-guide/configuring
- """
-
- num_errors = 0
-
- source_files = ' '.join(source_paths)
- config_path = os.path.join('support', '.eslintrc.js')
-
- process = self.run_command_in_virtualenv(
- 'eslint {files} -c {config} -f compact'.format(
- files=source_files,
- config=config_path
- )
- )
-
- for line in process.stdout:
- line = line.decode(sys.stdout.encoding)
- if "Error -" in line or "Warning -" in line:
- sys.stderr.write(line)
- if "Error -" in line:
- num_errors += 1
-
- return num_errors
-
-
-class PyLinter(LinterBase):
- """The linter for Python files, uses pylint."""
- linter_type = 'Python'
-
- cli_dir = os.path.join('src', 'python', 'cli_new')
- lib_dir = os.path.join('src', 'python', 'lib')
- support_dir = os.path.join('support')
- source_dirs_to_lint_with_venv = [support_dir]
- source_dirs_to_lint_with_tox = [cli_dir, lib_dir]
- source_dirs = source_dirs_to_lint_with_tox + source_dirs_to_lint_with_venv
-
- exclude_files = '(' \
- r'protobuf\-2\.4\.1|' \
- r'googletest\-release\-1\.8\.0|' \
- r'glog\-0\.3\.3|' \
- r'boost\-1\.53\.0|' \
- r'libev\-4\.15|' \
- r'\.virtualenv|' \
- r'\.tox' \
- ')'
-
- source_files = r'\.(py)$'
-
- comment_prefix = '#'
-
- pylint_config = os.path.abspath(os.path.join('support', 'pylint.config'))
-
- def run_tox(self, configfile, args, tox_env=None, recreate=False):
- """
- Runs tox with given configfile and args. Optionally set tox env
- and/or recreate the tox-managed virtualenv.
- """
- support_dir = os.path.dirname(__file__)
- bin_dir = 'Script' if platform.system() == 'Windows' else 'bin'
-
- cmd = [os.path.join(support_dir, '.virtualenv', bin_dir, 'tox')]
- cmd += ['-qq']
- cmd += ['-c', configfile]
- if tox_env is not None:
- cmd += ['-e', tox_env]
- if recreate:
- cmd += ['--recreate']
- cmd += ['--']
- cmd += args
-
- # We do not use `run_command_in_virtualenv()` here, as we
- # directly call `tox` from inside the virtual environment bin
- # directory without activating the virtualenv.
- return subprocess.Popen(cmd, stdout=subprocess.PIPE)
-
- def filter_source_files(self, source_dir, source_files):
- """
- Filters out files starting with source_dir.
- """
- return [f for f in source_files if f.startswith(source_dir)]
-
- def lint_source_files_under_source_dir(self, source_dir, source_files):
- """
- Runs pylint directly or indirectly throgh tox on source_files which
- are under source_dir. If tox is to be used, it must be configured
- in source_dir, i.e. a tox.ini must be present.
- """
- filtered_source_files = self.filter_source_files(
- source_dir, source_files)
-
- if not filtered_source_files:
- return 0
-
- if source_dir in self.source_dirs_to_lint_with_tox:
- process = self.run_tox(
- configfile=os.path.join(source_dir, 'tox.ini'),
- args=['--rcfile='+self.pylint_config] + filtered_source_files,
- tox_env='py3-lint')
- else:
- process = self.run_command_in_virtualenv(
- 'pylint --score=n --rcfile={rcfile} {files}'.format(
- rcfile=self.pylint_config,
- files=' '.join(filtered_source_files)))
-
- num_errors = 0
- for line in process.stdout:
- line = line.decode(sys.stdout.encoding)
- if re.search(r'[RCWEF][0-9]{4}:', line):
- num_errors += 1
- sys.stderr.write(line)
-
- return num_errors
-
- def run_lint(self, source_paths):
- """
- Runs pylint over given files.
-
- https://google.github.io/styleguide/pyguide.html
- """
- num_errors = 0
-
- for source_dir in self.source_dirs:
- num_errors += self.lint_source_files_under_source_dir(
- source_dir, source_paths)
-
- return num_errors
-
-
-def should_build_virtualenv(modified_files):
- """
- Check if we should build the virtual environment required.
- This is the case if the requirements of the environment
- have changed or if the support script is run with no
- arguments (meaning that the entire codebase should be linted).
- """
- # NOTE: If the file list is empty, we are linting the entire test
- # codebase. We should always rebuild the virtualenv in this case.
- if not modified_files:
- return True
-
- support_dir = os.path.dirname(__file__)
- bin_dir = 'Script' if platform.system() == 'Windows' else 'bin'
-
- interpreter = os.path.basename(sys.executable)
- interpreter = os.path.join(support_dir, '.virtualenv', bin_dir, interpreter)
- if not os.path.isfile(interpreter):
- return True
-
- basenames = [os.path.basename(path) for path in modified_files]
-
- if 'pip-requirements.txt' in basenames:
- print('The "pip-requirements.txt" file has changed.')
- return True
-
- if 'build-virtualenv' in basenames:
- print('The "build-virtualenv" file has changed.')
- return True
-
- # The JS and Python linters require a virtual environment.
- # If all the files modified are not JS or Python files,
- # we do not need to build the virtual environment.
- # TODO(ArmandGrillet): There should be no duplicated logic to know
- # which linters to instantiate depending on the files to analyze.
- if not os.path.isdir(os.path.join('support', '.virtualenv')):
- js_and_python_files = [JsLinter().source_files, PyLinter().source_files]
- js_and_python_files_regex = re.compile('|'.join(js_and_python_files))
-
- for basename in basenames:
- if js_and_python_files_regex.search(basename) is not None:
- print('Virtualenv not detected and required... building')
- return True
-
- return False
-
-
-def build_virtualenv():
- """
- Rebuild the virtualenv by running a bootstrap script.
- This will exit the program if there is a failure.
- """
- print('Rebuilding virtualenv...')
-
- python3_env = os.environ.copy()
- python3_env["PYTHON"] = sys.executable
-
- build_virtualenv_file = [os.path.join('support', 'build-virtualenv')]
-
- if platform.system() == 'Windows':
- # TODO(andschwa): Port more of the `build-virtualenv` Bash script.
- python_dir = os.path.dirname(sys.executable)
- virtualenv = os.path.join(python_dir, 'Scripts', 'virtualenv.exe')
- build_virtualenv_file = [virtualenv,
- '--no-site-packages',
- 'support/.virtualenv']
-
- process = subprocess.Popen(
- build_virtualenv_file,
- env=python3_env,
- stdout=subprocess.PIPE)
-
- output = ''
- for line in process.stdout:
- output += line.decode(sys.stdout.encoding)
-
- process.wait()
-
- if process.returncode != 0:
- sys.stderr.write(output)
- sys.exit(1)
-
- # TODO(andschwa): Move this into a script like above.
- if platform.system() == 'Windows':
- def run_command_in_virtualenv(command):
- """
- Stolen from `PyLinter`, runs command in virtualenv.
- """
- virtualenv = os.path.join('support',
- '.virtualenv',
- 'Scripts',
- 'activate.bat')
- command = '{virtualenv_path} & {cmd}'.format(
- virtualenv_path=virtualenv, cmd=command)
-
- return subprocess.Popen(command, shell=True, stdout=subprocess.PIPE)
-
- pip_install_pip = 'python.exe -m pip install --upgrade pip'
- process = run_command_in_virtualenv(pip_install_pip)
- for line in process.stdout:
- output += line.decode(sys.stdout.encoding)
- process.wait()
+error = """\
+'mesos-style.py' was removed in favor of hooks managed by pre-commit.
- if process.returncode != 0:
- sys.stderr.write(output)
- sys.exit(1)
+Linting requires an installation of pre-commit, see
+https://pre-commit.com/#install, e.g.,
- pip_reqs = 'python.exe -m pip install -r support/pip-requirements.txt'
- process = run_command_in_virtualenv(pip_reqs)
- for line in process.stdout:
- output += line.decode(sys.stdout.encoding)
- process.wait()
+ $ pip3 install pre-commit
- if process.returncode != 0:
- sys.stderr.write(output)
- sys.exit(1)
+After installing pre-commit, remove existing hooks in '.git/hooks'
+and reinstall hooks with './support/setup-dev.sh'.
-if __name__ == '__main__':
- if should_build_virtualenv(sys.argv[1:]):
- build_virtualenv()
+ $ rm -rfv .git/hooks/*
+ $ ./support/setup-dev.sh
+"""
- # TODO(ArmandGrillet): We should only instantiate the linters
- # required to lint the files to analyze. See MESOS-8351.
- CPP_LINTER = CppLinter()
- CPP_ERRORS = CPP_LINTER.main(sys.argv[1:])
- JS_LINTER = JsLinter()
- JS_ERRORS = JS_LINTER.main(sys.argv[1:])
- PY_LINTER = PyLinter()
- PY_ERRORS = PY_LINTER.main(sys.argv[1:])
- sys.exit(CPP_ERRORS + JS_ERRORS + PY_ERRORS)
+print(error, file=sys.stderr)
+sys.exit(1)
[mesos] 05/09: Switched commit hooks to pre-commit.
Posted by bb...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
bbannier pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/mesos.git
commit 454661dd0dcbb7a7bc87ac58ad74fd6dd04c5c15
Author: Benjamin Bannier <bb...@apache.org>
AuthorDate: Wed Sep 18 11:37:14 2019 +0200
Switched commit hooks to pre-commit.
This patch switches commit hooks to be orchestrated by the pre-commit
tool mirroring the previous linters invoked through git commit
hooks (orchestrated by `support/mesos-style.py` or standalone hooks).
Using pre-commit removes the burden of maintaining
`support/mesos-style.py`, making sure that hooks have the expected
environment (e.g., Python version, Node installed). Additionally,
upstream provides a number of additional linters which are not hard to
add to Mesos' hooks.
Review: https://reviews.apache.org/r/71205/
---
docs/advanced-contribution.md | 3 +-
support/gitignore | 1 +
support/pre-commit-config.yaml | 48 ++++++++++++++++++++++++++++
support/setup-dev.bat | 18 +++++------
support/setup-dev.sh | 72 ++++++++++--------------------------------
5 files changed, 77 insertions(+), 65 deletions(-)
diff --git a/docs/advanced-contribution.md b/docs/advanced-contribution.md
index 6582849..5fbfcb9 100644
--- a/docs/advanced-contribution.md
+++ b/docs/advanced-contribution.md
@@ -15,6 +15,7 @@ If you'd like to make significant contributions to Mesos, then you'll want to be
<br/><br/>
* Required installations
+ RBtools (0.7.10 is known to work, 1.0.1 appears to have an issue)
+ + [pre-commit](https://pre-commit.com/#install)
## Issue Tracking, Code Review, and Mailing Lists
@@ -64,7 +65,7 @@ Here is the standard procedure for proposing and making changes to Mesos:
1. Create one or more test cases to exercise the bug or the feature (the Mesos team uses [test-driven development](http://en.wikipedia.org/wiki/Test-driven_development)). Before you start coding, make sure these test cases all fail.
1. The [testing patterns](testing-patterns.md) page has some suggestions for writing test cases.
2. Make your changes to the code (using whatever IDE/editor you choose) to actually fix the bug or implement the feature.
- 1. Before beginning, please read the [Mesos C++ Style Guide](c++-style-guide.md). It is recommended to use the git pre-commit hook (`support/hooks/pre-commit`) to automatically check for style errors. See the hook script for instructions to enable it.
+ 1. Before beginning, please read the [Mesos C++ Style Guide](c++-style-guide.md). It is recommended to use the git pre-commit hook to automatically check for style errors. The hooks are set up by invoking `./support/setup_dev.sh`.
2. Most of your changes will probably be to files inside of `BASE_MESOS_DIR`
3. From inside of the root Mesos directory: `./bootstrap` and `./support/setup-dev.sh`.
4. To build, we recommend that you don't build inside of the src directory. We recommend you do the following:
diff --git a/support/gitignore b/support/gitignore
index 703b14b..08280cd 100644
--- a/support/gitignore
+++ b/support/gitignore
@@ -23,6 +23,7 @@ missing
# Files created by ./support/setup-dev.sh
.clang-format
.clang-tidy
+.pre-commit-config.yaml
.reviewboardrc
.gitlint
CPPLINT.cfg
diff --git a/support/pre-commit-config.yaml b/support/pre-commit-config.yaml
new file mode 100644
index 0000000..547a4fb
--- /dev/null
+++ b/support/pre-commit-config.yaml
@@ -0,0 +1,48 @@
+default_stages: [commit]
+repos:
+- repo: local
+ hooks:
+ - id: split
+ name: Check that commits are split by project
+ entry: python ./support/mesos-split.py
+ language: python
+ language_version: 'python3'
+ stages: [commit]
+ require_serial: true
+ - id: license
+ name: Check required license headers
+ description: license
+ language: python
+ language_version: 'python3'
+ entry: python ./support/check-license.py
+ types: [text]
+ files: '.*\.(cpp|hpp|py)$' # TODO(bbannier): also check shell scripts.
+ - id: cpplint
+ name: cpplint
+ entry: python ./support/cpplint.py
+ language: python
+ language_version: 'python2'
+ types: [c++]
+ args: ['--extensions=hpp,cpp']
+- repo: https://github.com/jorisroovers/gitlint
+ rev: v0.12.0
+ hooks:
+ - id: gitlint
+ stages: [commit-msg]
+- repo: https://github.com/pre-commit/mirrors-pylint
+ rev: v2.3.1
+ hooks:
+ - id: pylint
+ args: ['--score=n', '--rcfile=support/pylint.config']
+ files: '(support).*'
+- repo: https://github.com/pre-commit/mirrors-eslint
+ rev: v6.3.0
+ hooks:
+ - id: eslint
+ args: ['-c', 'support/.eslintrc.js']
+ exclude: '^(site/source/assets/js/|src/webui/assets/libs/)'
+- repo: https://github.com/jumanjihouse/pre-commit-hooks
+ rev: 1.11.0
+ hooks:
+ - id: require-ascii
+ exclude: '^(docs/(committers.md|contributors.yaml))|(site|src/webui)/'
diff --git a/support/setup-dev.bat b/support/setup-dev.bat
index 3f4154a..df3382b 100644
--- a/support/setup-dev.bat
+++ b/support/setup-dev.bat
@@ -35,15 +35,6 @@ fsutil reparsepoint query "support" | find "Symbolic Link" >nul && (
goto not_in_root
)
-:: Install mesos default hooks and gitignore template.
-if not exist .git\hooks\pre-commit (
- mklink .git\hooks\pre-commit ..\..\support\hooks\pre-commit
-)
-
-if not exist .git\hooks\post-rewrite (
- mklink .git\hooks\post-rewrite ..\..\support\hooks\post-rewrite
-)
-
if not exist .gitignore (
mklink .gitignore support\gitignore
)
@@ -64,6 +55,15 @@ if not exist .gitlint (
mklink .gitlint support\gitlint
)
+if not exist .pre-commit-config.yaml (
+ mklink .pre-commit-config.yaml support\pre-commit-config.yaml
+)
+
+:: Install mesos default hooks and gitignore template.
+pre-commit install-hooks
+pre-commit install --hook-type pre-commit
+pre-commit install --hook-type commit-msg
+
goto:eof
diff --git a/support/setup-dev.sh b/support/setup-dev.sh
index 183effa..d171234 100755
--- a/support/setup-dev.sh
+++ b/support/setup-dev.sh
@@ -26,61 +26,6 @@ __EOF__
exit 1
fi
-# Helper functions to get the absolute path of a directory as well as compute
-# the relative path between two directories.
-# Unfortunately these functions are not as readable as, say, python's
-# os.path.relpath method, but they allow us to avoid a hard dependence on
-# python or other tools during bootstrap.
-abspath() {
- cd "`dirname "${1}"`"
- echo "${PWD}"/"`basename "${1}"`"
- cd "${OLDPWD}"
-}
-relpath() {
- local from to up
- from="`abspath "${1%/}"`" to="`abspath "${2%/}"/`"
- while test "${to}" = "${to#"${from}"/}" \
- -a "${to}" != "${from}"; do
- from="${from%/*}" up="../${up}"
- done
- to="${up%/}${to#${from}}"
- echo "${to:-.}"
-}
-
-# Grab a reference to the repo's git directory. Usually this is simply .git in
-# the repo's top level directory. However, when submodules are used, it may
-# appear elsewhere. The most up-to-date way of finding this directory is to use
-# `git rev-parse --git-common-dir`. This is necessary to support things like
-# git worktree in addition to git submodules. However, as of January 2016,
-# support for the '--git-common-dir' flag is fairly new, forcing us to fall
-# back to the older '--git-dir' flag if '--git-common-dir' is not supported. We
-# do this by checking the output of `git rev-parse --git-common-dir` and seeing
-# if it gives us a valid directory back. If not, we set the git directory using
-# the '--git-dir' flag instead.
-_gitdir=`git rev-parse --git-common-dir`
-if test ! -d "${_gitdir}"; then
- _gitdir=`git rev-parse --git-dir`
-fi
-
-# Grab a reference to the git hooks directory as well as the relative path from
-# the git hooks directory to the current directory.
-_hooksdir=${_gitdir}/hooks
-_relpath=`relpath "${_hooksdir}" "${PWD}"`
-
-# Install mesos default hooks and gitignore template.
-if test ! -e "${_hooksdir}/pre-commit"; then
- ln -s "${_relpath}/support/hooks/pre-commit" "${_hooksdir}/pre-commit"
-fi
-
-if test ! -e "${_hooksdir}/post-rewrite"; then
- ln -s "${_relpath}/support/hooks/post-rewrite" "${_hooksdir}/post-rewrite"
-fi
-
-if test ! -e "${_hooksdir}/commit-msg"; then
- ln -s "${_relpath}/support/hooks/commit-msg" "${_hooksdir}/commit-msg"
-fi
-
-
if test ! -e .gitignore; then
ln -s support/gitignore .gitignore
fi
@@ -104,3 +49,20 @@ fi
if test ! -e .gitlint; then
ln -s support/gitlint .gitlint
fi
+
+if test ! -e .pre-commit-config.yaml; then
+ ln -s support/pre-commit-config.yaml .pre-commit-config.yaml
+fi
+
+# Install Mesos default hooks and gitignore template.
+if ! command pre-commit; then
+ cat >&2 <<__EOF__
+
+Please install pre-commit to excercise Mesos linters, https://pre-commit.com/#install.
+__EOF__
+ exit 1
+fi
+
+pre-commit install-hooks
+pre-commit install --hook-type pre-commit
+pre-commit install --hook-type commit-msg
[mesos] 01/09: Moved cpplint configuration into dedicated file.
Posted by bb...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
bbannier pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/mesos.git
commit 2232d48ce5b07c4e094c6850cb28212495824110
Author: Benjamin Bannier <bb...@apache.org>
AuthorDate: Wed Sep 18 11:37:09 2019 +0200
Moved cpplint configuration into dedicated file.
With this change we not only reduce the amount of code in
`support/mesos-style.py` in favor of a configuration supported by
upstream, but we also make it easier to interoperate with editor
integrations for cpplint.
Review: https://reviews.apache.org/r/70096/
---
bootstrap | 4 ++++
bootstrap.bat | 4 ++++
support/CPPLINT.cfg | 2 ++
support/gitignore | 1 +
support/mesos-style.py | 28 ++--------------------------
5 files changed, 13 insertions(+), 26 deletions(-)
diff --git a/bootstrap b/bootstrap
index 7be2cc9..a8b0da1 100755
--- a/bootstrap
+++ b/bootstrap
@@ -80,6 +80,10 @@ if test ! -e .clang-tidy; then
ln -s support/clang-tidy .clang-tidy
fi
+if test ! -e CPPLINT.cfg; then
+ ln -s support/CPPLINT.cfg CPPLINT.cfg
+fi
+
if [ -n "$AUTOMAKE" ] || [ -n "$ACLOCAL" ] ; then
if [ -z "$ACLOCAL" ] || [ -z "$AUTOMAKE" ] ; then
_present="AUTOMAKE"
diff --git a/bootstrap.bat b/bootstrap.bat
index 584b5c3..41447db 100644
--- a/bootstrap.bat
+++ b/bootstrap.bat
@@ -56,6 +56,10 @@ if not exist .clang-format (
mklink .clang-format support\clang-format
)
+if not exist CPPLINT.cfg (
+ mklink CPPLINT.cfg support\CPPLINT.cfg
+)
+
goto:eof
diff --git a/support/CPPLINT.cfg b/support/CPPLINT.cfg
new file mode 100644
index 0000000..b881c71
--- /dev/null
+++ b/support/CPPLINT.cfg
@@ -0,0 +1,2 @@
+set noparent
+filter=-,+build/class,+build/deprecated,+build/endif_comment,+readability/todo,+readability/namespace,+runtime/vlog,+whitespace/blank_line,+whitespace/comma,+whitespace/end_of_line,+whitespace/ending_newline,+whitespace/forcolon,+whitespace/indent,+whitespace/line_length,+whitespace/operators,+whitespace/semicolon,+whitespace/tab,+whitespace/comments,+whitespace/todo
diff --git a/support/gitignore b/support/gitignore
index 7218eda..484b912 100644
--- a/support/gitignore
+++ b/support/gitignore
@@ -5,6 +5,7 @@
.clang-format
.clang-tidy
.reviewboardrc
+CPPLINT.cfg
Makefile.in
aclocal.m4
ar-lib
diff --git a/support/mesos-style.py b/support/mesos-style.py
index cd490bd..b3e20fd 100755
--- a/support/mesos-style.py
+++ b/support/mesos-style.py
@@ -259,35 +259,11 @@ class CppLinter(LinterBase):
http://google-styleguide.googlecode.com/svn/trunk/cpplint/cpplint.py
"""
-
- # See cpplint.py for full list of rules.
- active_rules = [
- 'build/class',
- 'build/deprecated',
- 'build/endif_comment',
- 'readability/todo',
- 'readability/namespace',
- 'runtime/vlog',
- 'whitespace/blank_line',
- 'whitespace/comma',
- 'whitespace/end_of_line',
- 'whitespace/ending_newline',
- 'whitespace/forcolon',
- 'whitespace/indent',
- 'whitespace/line_length',
- 'whitespace/operators',
- 'whitespace/semicolon',
- 'whitespace/tab',
- 'whitespace/comments',
- 'whitespace/todo']
-
- rules_filter = '--filter=-,+' + ',+'.join(active_rules)
-
# We do not use a version of cpplint available through pip as
# we use a custom version (see cpplint.path) to lint C++ files.
process = subprocess.Popen(
- [sys.executable, 'support/cpplint.py', '--extensions=hpp,cpp',
- rules_filter] + source_paths,
+ [sys.executable, 'support/cpplint.py', '--extensions=hpp,cpp'] +
+ source_paths,
stderr=subprocess.PIPE,
close_fds=True)
[mesos] 09/09: Enabled a number of additional pre-commit checks.
Posted by bb...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
bbannier pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/mesos.git
commit fb467a03cb8a5bde5147dc06ca6b73c9df04ff48
Author: Benjamin Bannier <bb...@apache.org>
AuthorDate: Wed Sep 18 11:37:19 2019 +0200
Enabled a number of additional pre-commit checks.
This patch enables checkers for well-formed YAML and JSON, and a linter
which checks that all executable scripts have a valid shebang line.
Review: https://reviews.apache.org/r/71209/
---
support/pre-commit-config.yaml | 6 ++++++
1 file changed, 6 insertions(+)
diff --git a/support/pre-commit-config.yaml b/support/pre-commit-config.yaml
index 547a4fb..62ce1a4 100644
--- a/support/pre-commit-config.yaml
+++ b/support/pre-commit-config.yaml
@@ -46,3 +46,9 @@ repos:
hooks:
- id: require-ascii
exclude: '^(docs/(committers.md|contributors.yaml))|(site|src/webui)/'
+- repo: https://github.com/pre-commit/pre-commit-hooks
+ rev: v2.3.0
+ hooks:
+ - id: check-yaml
+ - id: check-json
+ - id: check-executables-have-shebangs
[mesos] 08/09: Revert "Updated cpplint to be compatible with Python
3."
Posted by bb...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
bbannier pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/mesos.git
commit 2af339668fd90212999bae06a050a05824f2971e
Author: Benjamin Bannier <bb...@apache.org>
AuthorDate: Wed Sep 18 11:37:18 2019 +0200
Revert "Updated cpplint to be compatible with Python 3."
This reverts commit 89db66e3df831eaa50fffb4149a3894097505c14.
This patch was necessary when we were running cpplint in the python3
environment used e.g., also for bindings and other scripts. With
pre-commit we have freedom to choose the Python environment needed so we
can undo our adjustments here to stay closer to upstream.
Review: https://reviews.apache.org/r/71208/
---
support/cpplint.py | 31 ++++++++++---------------------
1 file changed, 10 insertions(+), 21 deletions(-)
diff --git a/support/cpplint.py b/support/cpplint.py
index 5671873..4f551dd 100755
--- a/support/cpplint.py
+++ b/support/cpplint.py
@@ -62,24 +62,6 @@ try:
xrange # Python 2
except NameError:
xrange = range # Python 3
- unicode = str
- def iteritems(d):
- return d.items()
- def itervalues(d):
- return d.values()
-else:
- # Python 2
- def iteritems(d):
- return d.iteritems()
- def itervalues(d):
- return d.itervalues()
- # Change stderr to write with replacement characters so we don't die
- # if we try to print something containing non-ASCII characters.
- sys.stderr = codecs.StreamReaderWriter(sys.stderr,
- codecs.getreader('utf8'),
- codecs.getwriter('utf8'),
- 'replace')
-
_USAGE = """
@@ -975,7 +957,7 @@ class _CppLintState(object):
def PrintErrorCounts(self):
"""Print a summary of errors by category, and the total."""
- for category, count in iteritems(self.errors_by_category):
+ for category, count in self.errors_by_category.iteritems():
sys.stderr.write('Category \'%s\' errors found: %d\n' %
(category, count))
sys.stdout.write('Total errors found: %d\n' % self.error_count)
@@ -4640,7 +4622,7 @@ def _GetTextInside(text, start_pattern):
# Give opening punctuations to get the matching close-punctuations.
matching_punctuation = {'(': ')', '{': '}', '[': ']'}
- closing_punctuation = set(itervalues(matching_punctuation))
+ closing_punctuation = set(matching_punctuation.itervalues())
# Find the position to start extracting text.
match = re.search(start_pattern, text, re.M)
@@ -5588,7 +5570,7 @@ def CheckForIncludeWhatYouUse(filename, clean_lines, include_state, error,
# include_dict is modified during iteration, so we iterate over a copy of
# the keys.
- header_keys = list(include_dict)
+ header_keys = include_dict.keys()
for header in header_keys:
(same_module, common_path) = FilesBelongToSameModule(abs_filename, header)
fullpath = common_path + header
@@ -6241,6 +6223,13 @@ def ParseArguments(args):
def main():
filenames = ParseArguments(sys.argv[1:])
+ # Change stderr to write with replacement characters so we don't die
+ # if we try to print something containing non-ASCII characters.
+ sys.stderr = codecs.StreamReaderWriter(sys.stderr,
+ codecs.getreader('utf8'),
+ codecs.getwriter('utf8'),
+ 'replace')
+
_cpplint_state.ResetErrorCounts()
for filename in filenames:
ProcessFile(filename, _cpplint_state.verbose_level)
[mesos] 03/09: Added gitlint config.
Posted by bb...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
bbannier pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/mesos.git
commit cbaca81a54720771662c119c80aec6101f120afc
Author: Benjamin Bannier <bb...@apache.org>
AuthorDate: Wed Sep 18 11:37:11 2019 +0200
Added gitlint config.
This patch adds a config for the gitlint tool which is slated to replace
a custom commit-msg hook once we switch our hook infrastructure to the
pre-commit tool.
Review: https://reviews.apache.org/r/71204/
---
bootstrap | 4 ++++
bootstrap.bat | 4 ++++
support/gitignore | 1 +
support/gitlint | 48 ++++++++++++++++++++++++++++++++++++++++++++++++
4 files changed, 57 insertions(+)
diff --git a/bootstrap b/bootstrap
index a8b0da1..e88e8cb 100755
--- a/bootstrap
+++ b/bootstrap
@@ -84,6 +84,10 @@ if test ! -e CPPLINT.cfg; then
ln -s support/CPPLINT.cfg CPPLINT.cfg
fi
+if test ! -e .gitlint; then
+ ln -s support/gitlint .gitlint
+fi
+
if [ -n "$AUTOMAKE" ] || [ -n "$ACLOCAL" ] ; then
if [ -z "$ACLOCAL" ] || [ -z "$AUTOMAKE" ] ; then
_present="AUTOMAKE"
diff --git a/bootstrap.bat b/bootstrap.bat
index 41447db..7ca6449 100644
--- a/bootstrap.bat
+++ b/bootstrap.bat
@@ -60,6 +60,10 @@ if not exist CPPLINT.cfg (
mklink CPPLINT.cfg support\CPPLINT.cfg
)
+if not exist .gitlint (
+ mklink .gitlint support\gitlint
+)
+
goto:eof
diff --git a/support/gitignore b/support/gitignore
index 484b912..3d53411 100644
--- a/support/gitignore
+++ b/support/gitignore
@@ -5,6 +5,7 @@
.clang-format
.clang-tidy
.reviewboardrc
+.gitlint
CPPLINT.cfg
Makefile.in
aclocal.m4
diff --git a/support/gitlint b/support/gitlint
new file mode 100644
index 0000000..cac9253
--- /dev/null
+++ b/support/gitlint
@@ -0,0 +1,48 @@
+# All these sections are optional, edit this file as you like.
+[general]
+ignore=title-trailing-punctuation, T3, body-is-missing, B6, body-min-length
+# verbosity should be a value between 1 and 3, the commandline -v flags take precedence over this
+# verbosity = 2
+# By default gitlint will ignore merge commits. Set to 'false' to disable.
+# ignore-merge-commits=true
+# Enable debug mode (prints more output). Disabled by default.
+# debug=true
+
+# Set the extra-path where gitlint will search for user defined rules
+# See http://jorisroovers.github.io/gitlint/user_defined_rules for details
+# extra-path=examples/
+
+[title-max-length]
+line-length=72
+
+# [title-must-not-contain-word]
+# Comma-separated list of words that should not occur in the title. Matching is case
+# insensitive. It's fine if the keyword occurs as part of a larger word (so "WIPING"
+# will not cause a violation, but "WIP: my title" will.
+# words=wip
+
+[title-match-regex]
+# python like regex (https://docs.python.org/2/library/re.html) that the
+# commit-msg title must be matched to.
+# Note that the regex can contradict with other rules if not used correctly
+# (e.g. title-must-not-contain-word).
+# regex=^US[0-9]*
+regex=^[A-Z].*\.$
+
+[body-max-line-length]
+line-length=72
+
+# [body-min-length]
+# min-length=5
+
+# [body-is-missing]
+# Whether to ignore this rule on merge commits (which typically only have a title)
+# default = True
+# ignore-merge-commits=false
+
+# [body-changed-file-mention]
+# List of files that need to be explicitly mentioned in the body when they are changed
+# This is useful for when developers often erroneously edit certain files or git submodules.
+# By specifying this rule, developers can only change the file when they explicitly reference
+# it in the commit message.
+# files=gitlint/rules.py,README.md
[mesos] 02/09: Added check script to check for license headers.
Posted by bb...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
bbannier pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/mesos.git
commit 526043b586da0201fd7e374197139e75b249e299
Author: Benjamin Bannier <bb...@apache.org>
AuthorDate: Wed Sep 18 11:37:10 2019 +0200
Added check script to check for license headers.
This check adds a script which validates that source files have valid
license headers. This will allow us to reuse this functionality with
e.g., the pre-commit tool.
At the moment the code added here is not invoked from
`support/mesos-style.py` since it will be removed in a follow-up commit.
Review: https://reviews.apache.org/r/71203/
---
support/check-license.py | 28 ++++++++++++++++++++++++++++
1 file changed, 28 insertions(+)
diff --git a/support/check-license.py b/support/check-license.py
new file mode 100755
index 0000000..d416cb4
--- /dev/null
+++ b/support/check-license.py
@@ -0,0 +1,28 @@
+#!/usr/bin/env python3
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""Check if all files given as arguments contain a license header"""
+
+import re
+import sys
+
+for f in sys.argv[1:]:
+ if not any(
+ map(lambda x: re.match('.*(Licensed|Copyright)', x),
+ open(f).readlines())):
+ print("{} does not seem to contain a license header".format(f))
+ sys.exit(1)