[4/4] impala git commit: IMPALA-6863: Make pip_download.py honor redirects.

[jo...@apache.org]

IMPALA-6863: Make pip_download.py honor redirects.

As part of our continuing woes with PyPi infrastructure, we've now seen
redirects. Following redirects seems like the right thing to do, so I've
changed the downloader code to follow them.

I checked that this is available in Python 2.6.

The build failure signature looks like:

   Downloading AllPairs-2.0.1.tar.gz from https://pypi.python.org/packages/04/44/cb85d029b33ddfa7b5a27ae523434c2230f11a89d1f0c73203e9007a6eb0/AllPairs-2.0.1.tar.gz
   ('http error', 302, 'Found', <httplib.HTTPMessage instance at 0x7fbf7819b050>)
   Download failed after several attempts.
   Warning: Unable to download Python requirements.
   Warning: bootstrap_virtualenv or other Python-based tooling may fail.

Change-Id: Ic7551cec43a2d378df7e3cc7d521ace338b56ba2
Reviewed-on: http://gerrit.cloudera.org:8080/10083
Reviewed-by: Philip Zeyliger <ph...@cloudera.com>
Tested-by: Philip Zeyliger <ph...@cloudera.com>
Reviewed-by: Lars Volker <lv...@cloudera.com>


Project: http://git-wip-us.apache.org/repos/asf/impala/repo
Commit: http://git-wip-us.apache.org/repos/asf/impala/commit/eaf66172
Tree: http://git-wip-us.apache.org/repos/asf/impala/tree/eaf66172
Diff: http://git-wip-us.apache.org/repos/asf/impala/diff/eaf66172

Branch: refs/heads/master
Commit: eaf66172df113dbf10cdb0a08a2bc51e4077ca38
Parents: ea698cd
Author: Philip Zeyliger <ph...@cloudera.com>
Authored: Mon Apr 16 16:19:57 2018 -0700
Committer: Philip Zeyliger <ph...@cloudera.com>
Committed: Tue Apr 17 19:49:55 2018 +0000

----------------------------------------------------------------------
 infra/python/deps/pip_download.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/impala/blob/eaf66172/infra/python/deps/pip_download.py
----------------------------------------------------------------------
diff --git a/infra/python/deps/pip_download.py b/infra/python/deps/pip_download.py
index 6fbb683..0cce9e9 100755
--- a/infra/python/deps/pip_download.py
+++ b/infra/python/deps/pip_download.py
@@ -28,7 +28,7 @@ import re
 import sys
 from random import randint
 from time import sleep
-from urllib import urlopen, URLopener
+from urllib import urlopen, FancyURLopener
 
 NUM_DOWNLOAD_ATTEMPTS = 8
 
@@ -114,7 +114,7 @@ def download_package(pkg_name, pkg_version):
       expected_digest):
     print 'File with matching digest already exists, skipping {0}'.format(file_name)
     return True
-  downloader = URLopener()
+  downloader = FancyURLopener()
   pkg_url = '{0}/packages/{1}'.format(PYPI_MIRROR, path)
   print 'Downloading {0} from {1}'.format(file_name, pkg_url)
   downloader.retrieve(pkg_url, file_name)