[2/2] kudu git commit: python: upgrade pip when building client and restore old workarounds

[jd...@apache.org]

python: upgrade pip when building client and restore old workarounds

It appears that pypi recently deactivated their http:// endpoint, which is
the default in the older versions of pip found on el6. As a result, pip
installation requests fail:

  Downloading/unpacking pytest (from -r requirements.txt (line 1))
    Cannot fetch index base URL http://pypi.python.org/simple/
    Could not find any downloads that satisfy the requirement pytest (from -r requirements.txt (line 1))
  No distributions at all found for pytest (from -r requirements.txt (line 1))

Passing -i with the https:// endpoint works for the dependencies being
installed, but it doesn't appear to be passed down correctly to transitive
dependencies, and they fail to install with an inscrutable error [1].

So instead, let's go back to the old approach where we first upgrade pip.
This means effectively reverting [2] and restoring [3], but such is life.

I tested this by running build-and-test.sh on an el6 machine that created
virtualenvs containing pip 1.1 and setuptools 0.6c11.

1. https://stackoverflow.com/questions/5178535/easy-install-libmysqld-dev-error-nonetype-object-has-no-attribute-clone
2. https://github.com/apache/kudu/commit/b198ed8ff6a603816892c8bc7fd80679a014aaf1
3. https://github.com/apache/kudu/commit/d0acb55510c0552605953e07740f46d6be66c9c1

Change-Id: Ia75b65d66a1559ddaefb53dc1e67837faf9e7e6a
Reviewed-on: http://gerrit.cloudera.org:8080/8406
Tested-by: Kudu Jenkins
Reviewed-by: Jean-Daniel Cryans <jd...@apache.org>


Project: http://git-wip-us.apache.org/repos/asf/kudu/repo
Commit: http://git-wip-us.apache.org/repos/asf/kudu/commit/c08f27fc
Tree: http://git-wip-us.apache.org/repos/asf/kudu/tree/c08f27fc
Diff: http://git-wip-us.apache.org/repos/asf/kudu/diff/c08f27fc

Branch: refs/heads/master
Commit: c08f27fc583cbce2c99e4b28c9ddb91e9c9e1e69
Parents: 9a4ea1c
Author: Adar Dembo <ad...@cloudera.com>
Authored: Fri Oct 27 14:23:19 2017 -0700
Committer: Jean-Daniel Cryans <jd...@apache.org>
Committed: Mon Oct 30 16:02:20 2017 +0000

----------------------------------------------------------------------
 build-support/jenkins/build-and-test.sh | 44 ++++++++++++++++++++++++++--
 python/requirements.txt                 |  1 -
 2 files changed, 42 insertions(+), 3 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/kudu/blob/c08f27fc/build-support/jenkins/build-and-test.sh
----------------------------------------------------------------------
diff --git a/build-support/jenkins/build-and-test.sh b/build-support/jenkins/build-and-test.sh
index b303051..c2b0717 100755
--- a/build-support/jenkins/build-and-test.sh
+++ b/build-support/jenkins/build-and-test.sh
@@ -416,7 +416,27 @@ if [ "$BUILD_PYTHON" == "1" ]; then
   virtualenv $KUDU_BUILD/py_env
   source $KUDU_BUILD/py_env/bin/activate
 
-  # Download all necessary requirements.
+  # Old versions of pip (such as the one found in el6) default to pypi's http://
+  # endpoint which no longer exists. The -i option lets us switch to the
+  # https:// endpoint in such cases.
+  #
+  # Unfortunately, in these old versions of pip, -i doesn't appear to apply
+  # recursively to transitive dependencies installed via a direct dependency's
+  # "python setup.py" command. Therefore we have no choice but to upgrade to a
+  # new version of pip to proceed.
+  pip install -i https://pypi.python.org/simple --upgrade pip
+
+  # New versions of pip raise an exception when upgrading old versions of
+  # setuptools (such as the one found in el6). The workaround is to upgrade
+  # setuptools on its own, outside of requirements.txt, and with the pip version
+  # check disabled.
+  pip install --disable-pip-version-check --upgrade 'setuptools >= 0.8'
+
+  # We've got a new pip and new setuptools. We can now install the rest of the
+  # Python client's requirements.
+  #
+  # Installing the Cython dependency may involve some compiler work, so we must
+  # pass in the current values of CC and CXX.
   CC=$CLANG CXX=$CLANG++ pip install -r requirements.txt
 
   # Delete old Cython extensions to force them to be rebuilt.
@@ -454,7 +474,27 @@ if [ "$BUILD_PYTHON3" == "1" ]; then
   virtualenv -p python3 $KUDU_BUILD/py_env
   source $KUDU_BUILD/py_env/bin/activate
 
-  # Download all necessary requirements.
+  # Old versions of pip (such as the one found in el6) default to pypi's http://
+  # endpoint which no longer exists. The -i option lets us switch to the
+  # https:// endpoint in such cases.
+  #
+  # Unfortunately, in these old versions of pip, -i doesn't appear to apply
+  # recursively to transitive dependencies installed via a direct dependency's
+  # "python setup.py" command. Therefore we have no choice but to upgrade to a
+  # new version of pip to proceed.
+  pip install -i https://pypi.python.org/simple --upgrade pip
+
+  # New versions of pip raise an exception when upgrading old versions of
+  # setuptools (such as the one found in el6). The workaround is to upgrade
+  # setuptools on its own, outside of requirements.txt, and with the pip version
+  # check disabled.
+  pip install --disable-pip-version-check --upgrade 'setuptools >= 0.8'
+
+  # We've got a new pip and new setuptools. We can now install the rest of the
+  # Python client's requirements.
+  #
+  # Installing the Cython dependency may involve some compiler work, so we must
+  # pass in the current values of CC and CXX.
   CC=$CLANG CXX=$CLANG++ pip install -r requirements.txt
 
   # Delete old Cython extensions to force them to be rebuilt.

http://git-wip-us.apache.org/repos/asf/kudu/blob/c08f27fc/python/requirements.txt
----------------------------------------------------------------------
diff --git a/python/requirements.txt b/python/requirements.txt
index ecd5284..1740925 100644
--- a/python/requirements.txt
+++ b/python/requirements.txt
@@ -1,6 +1,5 @@
 pytest
 cython == 0.26.1
-setuptools >= 0.8
 six
 unittest2
 pytz