You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@trafficserver.apache.org by dr...@apache.org on 2019/05/23 15:21:14 UTC
[trafficserver] branch master updated: Added pipenv config script
for Autest.
This is an automated email from the ASF dual-hosted git repository.
dragon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/trafficserver.git
The following commit(s) were added to refs/heads/master by this push:
new 2e680b7 Added pipenv config script for Autest.
2e680b7 is described below
commit 2e680b76be7211eb99f74c9d7323e0e0fd43f914
Author: Carl Zhang <ca...@verizonmedia.com>
AuthorDate: Wed May 22 11:16:57 2019 -0500
Added pipenv config script for Autest.
---
tests/{autest.sh => Pipfile} | 38 +++---
tests/README.md | 22 ++-
tests/autest.sh | 14 +-
tests/bootstrap.py | 240 ---------------------------------
tests/{autest.sh => test-env-check.sh} | 53 +++++---
5 files changed, 76 insertions(+), 291 deletions(-)
diff --git a/tests/autest.sh b/tests/Pipfile
old mode 100755
new mode 100644
similarity index 60%
copy from tests/autest.sh
copy to tests/Pipfile
index 33c4d21..37f2d3d
--- a/tests/autest.sh
+++ b/tests/Pipfile
@@ -1,5 +1,3 @@
-#!/bin/bash
-
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
@@ -16,20 +14,22 @@
# See the License for the specific language governing permissions and
# limitations under the License.
-pushd $(dirname $0) > /dev/null
-export PYTHONPATH=$(pwd):$PYTHONPATH
-RED='\033[0;31m'
-GREEN='\033[1;32m'
-NC='\033[0m' # No Color
-if [ ! -f ./env-test/bin/autest ]; then\
- echo -e "${RED}AuTest is not installed! Bootstrapping system...${NC}";\
- ./bootstrap.py;\
- echo -e "${GREEN}Done!${NC}";\
- fi
-# this is for rhel or centos systems
-test -r /opt/rh/rh-python35/enable && . /opt/rh/rh-python35/enable
-. env-test/bin/activate
-./env-test/bin/autest -D gold_tests "$@"
-ret=$?
-popd > /dev/null
-exit $ret
+[[source]]
+name = "pypi"
+url = "https://pypi.org/simple"
+verify_ssl = true
+
+[dev-packages]
+
+[packages]
+autest = "==1.7.3"
+traffic-replay = "*" # this should install TRLib, MicroServer, MicroDNS, Traffic-Replay
+# These are likely to be available via yum/dnf or apt-get
+hyper = "*"
+requests = "*"
+dnslib = "*"
+gunicorn = "*"
+httpbin = "*" # pulls in quite a lot
+
+[requires]
+python_version = "3"
diff --git a/tests/README.md b/tests/README.md
index 0e09266..b8632f2 100644
--- a/tests/README.md
+++ b/tests/README.md
@@ -17,14 +17,28 @@ The current layout is:
To help with easy running of the tests, there is autest.sh and bootstrap.py.
### autest.sh
-This file is a simple wrapper that will call the Reusable Gold Testing System (Autest) program in a python virtualenv. If the virtualenv is not setup, the script will try to install it on the system. That will set up the Autest on most systems in a Python virtual environment. The wrapper adds some basic options to the command to point to the location of the tests. Use --help for more details on options for running Autest.
+This file is a simple wrapper that will call the Reusable Gold Testing System (Autest) program in a pipenv. If the pipenv is not setup, the script will prompt user the missing components. That will set up the Autest on most systems in a Python virtual environment. The wrapper adds some basic options to the command to point to the location of the tests. Use --help for more details on options for running Autest.
-### bootstrap.py
-This script will try to install python35 or better on the system, and the needed python packages for running the tests.
+### test-env-check.sh
+This script will check for the necessary packages needed to create a pipenv that can run Autest. If any package is missing, the script will alert the user. If all packages are available, it install a virtual environment using the provided Pipfile.
+
+### Pipfile
+This file is used to setup a virtual environment using pipenv. It contains information including the packages needed for Autest.
+A set of commands for pipenv:
+ * **pipenv install**: create virtual environment from the Pipfile.
+ * **pipenv shell**: launch a shell with the environment running(type "exit" to leave the shell).
+ * **pipenv run cmd**: run command in the virtual environment without entering a shell, where cmd is the shell command to run.
+ * **pipenv --rm**: remove the environment.
# Basic setup
-AuTest can be run using the script file autest.sh listed above. Run the file from the tests/ directory followed by --ats-bin and the bin name. (ie ~/ats/bin) This will run the wrapper for the tests. See documentation for more details.
+AuTest can be run using the script file autest.sh listed above. Run the file from the tests/ directory followed by --ats-bin and the bin name. (ie ~/ats/bin) This will run the wrapper for the tests.
+
+To run autest manually, the recommended way is to follow these steps:
+1. **pipenv install**: create the virtual environment(only needed once).
+2. **pipenv shell**: enter a shell in the virtual environment(type "exit" to leave the shell).
+3. **cd gold_tests**: enter the directory containing the test files.
+4. **autest --ats-bin user_ats_bin**: run autest where user_ats_bin is the bin directory in the user's ats directory.
# Advanced setup
diff --git a/tests/autest.sh b/tests/autest.sh
index 33c4d21..36d4afb 100755
--- a/tests/autest.sh
+++ b/tests/autest.sh
@@ -18,18 +18,10 @@
pushd $(dirname $0) > /dev/null
export PYTHONPATH=$(pwd):$PYTHONPATH
-RED='\033[0;31m'
-GREEN='\033[1;32m'
-NC='\033[0m' # No Color
-if [ ! -f ./env-test/bin/autest ]; then\
- echo -e "${RED}AuTest is not installed! Bootstrapping system...${NC}";\
- ./bootstrap.py;\
- echo -e "${GREEN}Done!${NC}";\
- fi
+./test-env-check.sh;
# this is for rhel or centos systems
-test -r /opt/rh/rh-python35/enable && . /opt/rh/rh-python35/enable
-. env-test/bin/activate
-./env-test/bin/autest -D gold_tests "$@"
+echo "Environment config finished. Running AuTest..."
+pipenv run autest -D gold_tests "$@"
ret=$?
popd > /dev/null
exit $ret
diff --git a/tests/bootstrap.py b/tests/bootstrap.py
deleted file mode 100755
index 806fa44..0000000
--- a/tests/bootstrap.py
+++ /dev/null
@@ -1,240 +0,0 @@
-#!/usr/bin/env python
-# this script sets up the testing packages to allow the tests
-
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-from __future__ import absolute_import, division, print_function
-
-import argparse
-import os
-import subprocess
-import platform
-import sys
-
-pip_packages = [
- "autest==1.7.3",
- "hyper",
- "dnslib",
- "traffic-replay", # this should install TRLib, MicroServer, MicroDNS, Traffic-Replay
- # These are likely to be available via yum/dnf or apt-get
- "requests",
- "httpbin", # pulls in quite a lot
- "gunicorn",
- "pipenv"
-]
-
-
-distro_packages = {
- "RHEL": [
- "install epel-release",
- "install python35",
- "install rh-python35-python-virtualenv"
- ],
- "Fedora": [
- "install python3",
- "install python3-virtualenv",
- "install python-virtualenv",
- ],
- "Ubuntu": [
- "install python3",
- "install python3-virtualenv",
- "install virtualenv",
- "install python3-dev"
- ],
- "CentOS": [
- "install epel-release",
- "install rh-python35-python-virtualenv"
- ]
-}
-
-
-def command_output(cmd_str):
- print(cmd_str)
- proc = subprocess.Popen(
- cmd_str,
- shell=True,
- stdout=subprocess.PIPE,
- stderr=subprocess.STDOUT,
- universal_newlines=True)
-
- # while command runs get output
- while proc.poll() == None:
- tmp = proc.stdout.readline()
- sys.stdout.write(tmp)
-
- for last_output in proc.stdout.readlines():
- sys.stdout.write(last_output)
-
- return proc.returncode
-
-
-def get_distro():
- return platform.linux_distribution()
-
-
-def distro_version():
- return int(get_distro()[1].split(".")[0])
-
-
-def isFedora():
- return get_distro()[0].startswith("Fedora")
-
-
-def isCentOS():
- return get_distro()[0].startswith("CentOS")
-
-
-def distro():
- if isFedora():
- return "Fedora"
- if isCentOS():
- return "CentOS"
- if get_distro()[0].startswith("Red Hat"):
- return "RHEL"
- if get_distro()[0].startswith("Ubuntu"):
- return "Ubuntu"
-
-
-def isRedHatBased():
- return get_distro()[0].startswith("Red Hat") or get_distro()[0].startswith(
- "Fedora") or get_distro()[0].startswith("CentOS")
-
-
-def isInstalled(prog):
- out = subprocess.Popen(
- ["which", prog], stdout=subprocess.PIPE).communicate()
- if out[0] != '':
- return True
- return False
-
-
-def installManagerName():
- if isRedHatBased() and distro_version() >= 22:
- ret = "sudo dnf -y" # Fedora 22 or newer
- elif isRedHatBased():
- ret = "sudo yum -y" # Red Hat distro
- else:
- ret = "sudo apt-get -y" # Ubuntu/Debian
-
- return ret
-
-
-def installToolName():
- if isRedHatBased():
- ret = "rpm -ihv" # Red Hat Based
- else:
- ret = "dpkg -iv" # Ubuntu/Debian
-
- return ret
-
-
-def run_cmds(cmds):
- for cmd in cmds:
- # print (cmd.split[" "])
- # subprocess.call(cmd.split[" "])
- if command_output(cmd):
- print("'{0}'' - Failed".format(cmd))
-
-
-def gen_package_cmds(packages):
-
- # main install tool/manager (yum, dnf, apt-get, etc)
- mtool = installManagerName()
- # core install tool (rpm, dpkg, etc)
- itool = installToolName()
- ret = []
-
- for p in packages:
- if p.startswith("wget"):
- pth = p[5:]
- pack = os.path.split(pth)[1]
- cmd = ["wget {0}".format(pth), "{0} ./{1}".format(itool, pack)]
- else:
- cmd = ["{0} {1}".format(mtool, p)]
- ret.extend(cmd)
- return ret
-
-
-extra = ''
-if distro() == 'RHEL' or distro() == 'CentOS':
- extra = ". /opt/rh/rh-python35/enable ;"
-
-
-def venv_cmds(path):
- '''
- Create virtual environment and add it
- to the path being used for the script
- '''
-
- return [
- # first command only needed for rhel and centos systems at this time
- extra + " virtualenv --python=python3 {0}".format(path),
- extra + " {0}/bin/pip install pip --upgrade".format(path)
- ]
-
-
-def main():
- " main script logic"
- parser = argparse.ArgumentParser()
-
- parser.add_argument(
- "--use-pip", nargs='?', default="pip", help="Which pip to use")
-
- parser.add_argument(
- "venv_path",
- nargs='?',
- default="env-test",
- help="The directory to us to for the virtualenv")
-
- parser.add_argument(
- "--disable-virtualenv",
- default=False,
- action='store_true',
- help="Do not create virtual environment to install packages under")
-
- parser.add_argument(
- '-V', '--version', action='version', version='%(prog)s 1.0.0')
-
- args = parser.parse_args()
- # print(args)
- # print(get_distro())
-
- # do we know of packages to install for the given platform
- dist = distro()
- cmds = []
- if dist:
- cmds = gen_package_cmds(distro_packages[dist])
-
- # test to see if we should use a certain version of pip
- path_to_pip = None
- if args.use_pip != "pip":
- path_to_pip = args.use_pip
-
- # install on the system, or use virtualenv for pip based stuff
- if not args.disable_virtualenv:
- # Create virtual env
- cmds += venv_cmds(args.venv_path)
- if path_to_pip is None:
- path_to_pip = os.path.join(args.venv_path, "bin", args.use_pip)
-
- cmds += [extra + "{0} install {1}".format(path_to_pip, " ".join(pip_packages))]
-
- run_cmds(cmds)
-
-
-if __name__ == '__main__':
- main()
diff --git a/tests/autest.sh b/tests/test-env-check.sh
similarity index 55%
copy from tests/autest.sh
copy to tests/test-env-check.sh
index 33c4d21..2fabb56 100755
--- a/tests/autest.sh
+++ b/tests/test-env-check.sh
@@ -16,20 +16,39 @@
# See the License for the specific language governing permissions and
# limitations under the License.
-pushd $(dirname $0) > /dev/null
-export PYTHONPATH=$(pwd):$PYTHONPATH
-RED='\033[0;31m'
-GREEN='\033[1;32m'
-NC='\033[0m' # No Color
-if [ ! -f ./env-test/bin/autest ]; then\
- echo -e "${RED}AuTest is not installed! Bootstrapping system...${NC}";\
- ./bootstrap.py;\
- echo -e "${GREEN}Done!${NC}";\
- fi
-# this is for rhel or centos systems
-test -r /opt/rh/rh-python35/enable && . /opt/rh/rh-python35/enable
-. env-test/bin/activate
-./env-test/bin/autest -D gold_tests "$@"
-ret=$?
-popd > /dev/null
-exit $ret
+# check for python3
+python3 - << _END_
+import sys
+
+if sys.version_info.major < 3 or sys.version_info.minor < 5:
+ exit(1)
+_END_
+
+if [ $? = 1 ]
+then
+ echo "Python 3.5 or newer is not installed/enabled."
+ return
+else
+ echo "Python 3.5 or newer detected!"
+fi
+
+# check for python development header -- for autest
+python3-config &> /dev/null
+if [ $? = 1 ]
+then
+ echo "python3-dev/devel detected!"
+else
+ echo "python3-dev/devel is not installed. "
+ return
+fi
+
+# check for pipenv
+pipenv --version &> /dev/null
+if [ $? = 0 ]
+then
+ echo "pipenv detected!"
+ pipenv install
+ # pipenv shell
+else
+ echo "pipenv is not installed/enabled. "
+fi