[arrow] branch master updated: GH-15087: [Release] Slow down downloading RC binaries from GitHub (#15090)

[ko...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

kou pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/arrow.git


The following commit(s) were added to refs/heads/master by this push:
     new 9b22860030 GH-15087: [Release] Slow down downloading RC binaries from GitHub (#15090)
9b22860030 is described below

commit 9b22860030932ca752532b849c1f0ec9d06a62cd
Author: David Li <li...@gmail.com>
AuthorDate: Sun Dec 25 18:59:28 2022 -0500

    GH-15087: [Release] Slow down downloading RC binaries from GitHub (#15090)
    
    GitHub appears to have a rate limit on downloading artifacts from a release, which we easily run into. Adding a bit of a delay appears to avoid this. Also, force cURL itself retry failed downloads.
    * Closes: #15087
    
    Authored-by: David Li <li...@gmail.com>
    Signed-off-by: Sutou Kouhei <ko...@clear-code.com>
---
 dev/release/download_rc_binaries.py | 10 +++++++++-
 1 file changed, 9 insertions(+), 1 deletion(-)

diff --git a/dev/release/download_rc_binaries.py b/dev/release/download_rc_binaries.py
index a2c715e227..40829563e1 100755
--- a/dev/release/download_rc_binaries.py
+++ b/dev/release/download_rc_binaries.py
@@ -22,8 +22,10 @@ import concurrent.futures as cf
 import functools
 import json
 import os
+import random
 import re
 import subprocess
+import time
 import urllib.request
 
 
@@ -190,12 +192,18 @@ class GitHub(Downloader):
             print("Already downloaded", dest_path)
             return
 
-        return self._download_url(
+        delay = random.randint(0, 3)
+        print(f"Waiting {delay} seconds to avoid rate limit")
+        time.sleep(delay)
+
+        self._download_url(
             url,
             dest_path,
             extra_args=[
                 "--header",
                 "Accept: application/octet-stream",
+                # Also retry 403s
+                "--retry-all-errors",
             ],
         )