[jira] [Updated] (THRIFT-6234) Configure apt retries and timeouts in GitHub Actions workflows

["Sylwester Lachiewicz (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/THRIFT-6234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Sylwester Lachiewicz updated THRIFT-6234:
-----------------------------------------
    Description: 
Every Linux job in the GitHub Actions workflows ({{build.yml}}, {{cmake.yml}}, {{sca.yml}}, {{make-dist.yml}}) installs packages with {{apt-get}} at apt's defaults.

The GitHub runner images point the sources at a mirror list ({{mirror+file:/etc/apt/apt-mirrors.txt}}: azure.archive.ubuntu.com, then archive.ubuntu.com, then security.ubuntu.com). apt only moves an item to the next mirror once it has failed on the current one, and on a transient failure (timeout, connection refused) it first retries the same mirror {{Acquire::Retries}} times, which defaults to 3. The http method additionally reconnects and resends once before it reports a failure. With the default 30 second inactivity timeout a stalled mirror therefore costs 4 attempts x 2 x 30 s plus back-off, about four minutes per item, before apt falls over.

Example: [Build with CMake run 34574616061|https://github.com/apache/thrift/actions/runs/34574616061/job/103184144542] on master fetched 8 of 132 packages in 10 minutes, with gaps of 2 to 3 minutes between single downloads, and hit the 10 minute step timeout before compiling anything. The same workflow passed on the commits before and after.

Proposal: ship one apt configuration fragment in the repository and copy it into {{/etc/apt/apt.conf.d/}} before each {{apt-get update}}:

{code}
Acquire::Retries "0";
Acquire::http::Timeout "20";
Acquire::https::Timeout "20";
DPkg::Lock::Timeout "120";
{code}

Without same-mirror retries and with the shorter timeout a stalled item falls over to the next mirror after 40 seconds (measured with a local mirror that accepts connections and never answers: first fetch from the fallback mirror at 40 s on Ubuntu 22.04 and 24.04, versus 60 s with the timeout alone and more than 300 s with the defaults). The three mirrors in the list provide the redundancy that the retries were providing before. The lock timeout makes apt-get wait for a concurrent apt process instead of failing immediately.

  was:
Every Linux job in the GitHub Actions workflows ({{build.yml}}, {{cmake.yml}}, {{sca.yml}}, {{make-dist.yml}}) installs packages with {{apt-get}} at apt's defaults: no retries and a 120 second inactivity timeout per connection.

The GitHub runner images point {{sources.list}} at a mirror list ({{mirror+file:/etc/apt/apt-mirrors.txt}}: azure.archive.ubuntu.com, then archive.ubuntu.com, then security.ubuntu.com). apt only moves to the next mirror when a fetch fails, so a stalled primary mirror runs the step into its {{timeout-minutes}} instead of failing over.

Example: [Build with CMake run 34574616061|https://github.com/apache/thrift/actions/runs/34574616061/job/103184144542] on master fetched 8 of 132 packages in 10 minutes, with gaps of 2 to 3 minutes between single downloads, and hit the 10 minute step timeout before compiling anything. The same workflow passed on the commits before and after.

Proposal: ship one apt configuration fragment in the repository and copy it into {{/etc/apt/apt.conf.d/}} before each {{apt-get update}}:

{code}
Acquire::Retries "3";
Acquire::http::Timeout "30";
Acquire::https::Timeout "30";
DPkg::Lock::Timeout "120";
{code}

The shorter timeout lets the mirror fallback engage, the retries cover transient failures, and the lock timeout waits for unattended-upgrades instead of failing immediately when it holds the dpkg lock.


> Configure apt retries and timeouts in GitHub Actions workflows
> --------------------------------------------------------------
>
>                 Key: THRIFT-6234
>                 URL: https://issues.apache.org/jira/browse/THRIFT-6234
>             Project: Thrift
>          Issue Type: Improvement
>          Components: Build Process
>            Reporter: Sylwester Lachiewicz
>            Priority: Major
>
> Every Linux job in the GitHub Actions workflows ({{build.yml}}, {{cmake.yml}}, {{sca.yml}}, {{make-dist.yml}}) installs packages with {{apt-get}} at apt's defaults.
> The GitHub runner images point the sources at a mirror list ({{mirror+file:/etc/apt/apt-mirrors.txt}}: azure.archive.ubuntu.com, then archive.ubuntu.com, then security.ubuntu.com). apt only moves an item to the next mirror once it has failed on the current one, and on a transient failure (timeout, connection refused) it first retries the same mirror {{Acquire::Retries}} times, which defaults to 3. The http method additionally reconnects and resends once before it reports a failure. With the default 30 second inactivity timeout a stalled mirror therefore costs 4 attempts x 2 x 30 s plus back-off, about four minutes per item, before apt falls over.
> Example: [Build with CMake run 34574616061|https://github.com/apache/thrift/actions/runs/34574616061/job/103184144542] on master fetched 8 of 132 packages in 10 minutes, with gaps of 2 to 3 minutes between single downloads, and hit the 10 minute step timeout before compiling anything. The same workflow passed on the commits before and after.
> Proposal: ship one apt configuration fragment in the repository and copy it into {{/etc/apt/apt.conf.d/}} before each {{apt-get update}}:
> {code}
> Acquire::Retries "0";
> Acquire::http::Timeout "20";
> Acquire::https::Timeout "20";
> DPkg::Lock::Timeout "120";
> {code}
> Without same-mirror retries and with the shorter timeout a stalled item falls over to the next mirror after 40 seconds (measured with a local mirror that accepts connections and never answers: first fetch from the fallback mirror at 40 s on Ubuntu 22.04 and 24.04, versus 60 s with the timeout alone and more than 300 s with the defaults). The three mirrors in the list provide the redundancy that the retries were providing before. The lock timeout makes apt-get wait for a concurrent apt process instead of failing immediately.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)