[jira] [Created] (IMPALA-6027) bootstrap_toolchain can fail to download toolchain components

["Michael Brown (JIRA)" <ji...@apache.org>]

Michael Brown created IMPALA-6027:
-------------------------------------

             Summary: bootstrap_toolchain can fail to download toolchain components
                 Key: IMPALA-6027
                 URL: https://issues.apache.org/jira/browse/IMPALA-6027
             Project: IMPALA
          Issue Type: Bug
          Components: Infrastructure
    Affects Versions: Impala 2.9.0, Impala 2.8.0, Impala 2.10.0, Impala 2.11.0
            Reporter: Michael Brown


Automated systems that build Impala can fail when downloading toolchain components.

{noformat}
Downloading avro-1.7.4-p4-gcc-4.9.2-ec2-package-sles-12.tar.gz to toolchain
Traceback (most recent call last):
  File "bin/bootstrap_toolchain.py", line 357, in <module>
    bootstrap(toolchain_root, packages)
  File "bin/bootstrap_toolchain.py", line 118, in bootstrap
    download_package(toolchain_root, pkg_name, pkg_version, compiler)
  File "bin/bootstrap_toolchain.py", line 100, in download_package
    wget_and_unpack_package(download_path, file_name, destination, True)
  File "bin/bootstrap_toolchain.py", line 82, in wget_and_unpack_package
    sh.wget(download_path, directory_prefix=destination, no_clobber=wget_no_clobber)
  File "infra/python/env/lib/python2.7/site-packages/sh.py", line 1021, in __call__
    return RunningCommand(cmd, call_args, stdin, stdout, stderr)
  File "infra/python/env/lib/python2.7/site-packages/sh.py", line 486, in __init__
    self.wait()
  File "infra/python/env/lib/python2.7/site-packages/sh.py", line 500, in wait
    self.handle_command_exit_code(exit_code)
  File "infra/python/env/lib/python2.7/site-packages/sh.py", line 516, in handle_command_exit_code
    raise exc(self.ran, self.process.stdout, self.process.stderr)
sh.ErrorReturnCode_8:

  RAN: '/usr/bin/wget https://native-toolchain.s3.amazonaws.com/build/474-6c406b4a88/avro/1.7.4-p4-gcc-4.9.2/avro-1.7.4-p4-gcc-4.9.2-ec2-package-sles-12.tar.gz --directory-prefix=toolchain --no-clobber'

  STDOUT:


  STDERR:
--2017-10-09 05:33:07--  https://native-toolchain.s3.amazonaws.com/build/474-6c406b4a88/avro/1.7.4-p4-gcc-4.9.2/avro-1.7.4-p4-gcc-4.9.2-ec2-package-sles-12.tar.gz
Resolving native-toolchain.s3.amazonaws.com (native-toolchain.s3.amazonaws.com)... 54.231.237.47
Connecting to native-toolchain.s3.amazonaws.com (native-toolchain.s3.amazonaws.com)|54.231.237.47|:443... connected.
HTTP request sent, awaiting response... 500 Internal Server Error
2017-10-09 05:33:07 ERROR 500: Internal Server Error.
{noformat}

Seems like this 500-series of error could be retried after some delay. A comprehensive solution has to consider all supported platforms. That means any {{wget}} solution presented must work in CentOS 5 / SLES 11 in addition to Ubuntu 16. An alternative is to check error code of the {{wget}} process and retry in Python.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)