kudu git commit: KUDU-2528: thirdparty downloads should be more robust

[ad...@apache.org]

Repository: kudu
Updated Branches:
  refs/heads/master 37d0c35bc -> 3bdc32054


KUDU-2528: thirdparty downloads should be more robust

Commit 37f3a95d8 added --retry to our curl invocations, but it turns out
that this only retries failures that curl deems to be "transient". And if
curl exits with a non-zero status, the entire script grinds to a halt
because of set -e.

This patch expands the class of retriable errors by parking the curl
statement in a pipeline, which means it is excused from set -e. After that,
the existing fetch_and_expand retry loop takes over.

Change-Id: Ic970afdb7ec23d3e04b2d7a9bf77d32774ca4d57
Reviewed-on: http://gerrit.cloudera.org:8080/11152
Reviewed-by: Alexey Serbin <as...@cloudera.com>
Tested-by: Kudu Jenkins
Reviewed-by: Grant Henke <gr...@apache.org>


Project: http://git-wip-us.apache.org/repos/asf/kudu/repo
Commit: http://git-wip-us.apache.org/repos/asf/kudu/commit/3bdc3205
Tree: http://git-wip-us.apache.org/repos/asf/kudu/tree/3bdc3205
Diff: http://git-wip-us.apache.org/repos/asf/kudu/diff/3bdc3205

Branch: refs/heads/master
Commit: 3bdc320542579be018212320ce556ed1220b3110
Parents: 37d0c35
Author: Adar Dembo <ad...@cloudera.com>
Authored: Tue Aug 7 16:22:10 2018 -0700
Committer: Adar Dembo <ad...@cloudera.com>
Committed: Wed Aug 8 21:28:21 2018 +0000

----------------------------------------------------------------------
 thirdparty/download-thirdparty.sh | 13 ++++++++++---
 1 file changed, 10 insertions(+), 3 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/kudu/blob/3bdc3205/thirdparty/download-thirdparty.sh
----------------------------------------------------------------------
diff --git a/thirdparty/download-thirdparty.sh b/thirdparty/download-thirdparty.sh
index 0ad768e..e1389c1 100755
--- a/thirdparty/download-thirdparty.sh
+++ b/thirdparty/download-thirdparty.sh
@@ -67,13 +67,20 @@ fetch_and_expand() {
 
   FULL_URL="${CLOUDFRONT_URL_PREFIX}/${FILENAME}"
   SUCCESS=0
-  # Loop in case we encounter a corrupted archive and we need to re-download it.
-  for attempt in 1 2; do
+  # Loop in case we encounter an error.
+  for attempt in 1 2 3; do
     if [ -r "$FILENAME" ]; then
       echo "Archive $FILENAME already exists. Not re-downloading archive."
     else
       echo "Fetching $FILENAME from $FULL_URL"
-      curl --retry 3 -L -O "$FULL_URL"
+      if ! curl --retry 3 -L -O "$FULL_URL"; then
+        echo "Error downloading $FILENAME"
+        rm -f "$FILENAME"
+
+        # Pause for a bit before looping in case the server throttled us.
+        sleep 5
+        continue
+      fi
     fi
 
     echo "Unpacking $FILENAME to $SOURCE"