You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "will-m-buchanan (via GitHub)" <gi...@apache.org> on 2026/04/01 14:00:27 UTC
[D] Can't build docker images based on slim-2.8.0 or higher [airflow]
GitHub user will-m-buchanan created a discussion: Can't build docker images based on slim-2.8.0 or higher
Our Airflow deployment is woefully behind the times and we're finally getting around to upgrading. First step before v3 is to upgrade from 2.7.3 to 2.11.2. We currently use the base image `apache/airflow:slim-2.7.3-python3.10` and run some `apt-get` commands to install other necessary utils. Unfortunately, when trying to build the image in Gitlab with a docker-in-docker runner, the 2.11.2 version of the base image (or any version 2.8.0 and above) breaks with the following error during `apt-get update`:
```
#11 127.9 Get:1 http://deb.debian.org/debian bookworm InRelease [151 kB]
#11 127.9 Get:2 http://deb.debian.org/debian bookworm-updates InRelease [55.4 kB]
#11 127.9 Get:3 http://deb.debian.org/debian-security bookworm-security InRelease [48.0 kB]
#11 127.9 Get:4 https://packages.microsoft.com/debian/12/prod bookworm InRelease [3618 B]
#11 127.9 Err:1 http://deb.debian.org/debian bookworm InRelease
#11 127.9 The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 6ED0E7B82643E131 NO_PUBKEY 78DBA3BC47EF2265 NO_PUBKEY F8D2585B8783D481
#11 128.0 Err:2 http://deb.debian.org/debian bookworm-updates InRelease
#11 128.0 The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 6ED0E7B82643E131 NO_PUBKEY 78DBA3BC47EF2265
#11 128.0 Err:3 http://deb.debian.org/debian-security bookworm-security InRelease
#11 128.0 The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 54404762BBB6E853 NO_PUBKEY BDE6D2B9216EC7A8
#11 128.0 Err:4 https://packages.microsoft.com/debian/12/prod bookworm InRelease
#11 128.0 At least one invalid signature was encountered.
#11 128.1 Get:5 https://download.docker.com/linux/debian bookworm InRelease [46.6 kB]
#11 128.1 Err:5 https://download.docker.com/linux/debian bookworm InRelease
#11 128.1 The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 7EA0A9C3F273FCD8
#11 128.1 Get:6 https://apt.postgresql.org/pub/repos/apt bookworm-pgdg InRelease [180 kB]
#11 128.2 Err:6 https://apt.postgresql.org/pub/repos/apt bookworm-pgdg InRelease
#11 128.2 At least one invalid signature was encountered.
#11 157.9 Ign:7 https://archive.mariadb.org/mariadb-10.11/repo/debian bookworm InRelease
#11 188.9 Ign:7 https://archive.mariadb.org/mariadb-10.11/repo/debian bookworm InRelease
#11 221.0 Ign:7 https://archive.mariadb.org/mariadb-10.11/repo/debian bookworm InRelease
#11 255.0 Err:7 https://archive.mariadb.org/mariadb-10.11/repo/debian bookworm InRelease
#11 255.0 Could not wait for server fd - select (11: Resource temporarily unavailable) [IP: 138.201.152.105 443]
#11 255.0 Reading package lists...
#11 255.0 W: GPG error: http://deb.debian.org/debian bookworm InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 6ED0E7B82643E131 NO_PUBKEY 78DBA3BC47EF2265 NO_PUBKEY F8D2585B8783D481
#11 255.0 E: The repository 'http://deb.debian.org/debian bookworm InRelease' is not signed.
#11 255.0 W: GPG error: http://deb.debian.org/debian bookworm-updates InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 6ED0E7B82643E131 NO_PUBKEY 78DBA3BC47EF2265
#11 255.0 E: The repository 'http://deb.debian.org/debian bookworm-updates InRelease' is not signed.
#11 255.0 W: GPG error: http://deb.debian.org/debian-security bookworm-security InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 54404762BBB6E853 NO_PUBKEY BDE6D2B9216EC7A8
#11 255.0 E: The repository 'http://deb.debian.org/debian-security bookworm-security InRelease' is not signed.
#11 255.0 W: GPG error: https://packages.microsoft.com/debian/12/prod bookworm InRelease: At least one invalid signature was encountered.
#11 255.0 E: The repository 'https://packages.microsoft.com/debian/12/prod bookworm InRelease' is not signed.
#11 255.0 W: GPG error: https://download.docker.com/linux/debian bookworm InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 7EA0A9C3F273FCD8
#11 255.0 E: The repository 'https://download.docker.com/linux/debian bookworm InRelease' is not signed.
#11 255.0 W: GPG error: https://apt.postgresql.org/pub/repos/apt bookworm-pgdg InRelease: At least one invalid signature was encountered.
#11 255.0 E: The repository 'https://apt.postgresql.org/pub/repos/apt bookworm-pgdg InRelease' is not signed.
#11 255.0 E: Problem executing scripts APT::Update::Post-Invoke 'rm -f /var/cache/apt/archives/*.deb /var/cache/apt/archives/partial/*.deb /var/cache/apt/*.bin || true'
#11 255.0 E: Sub-process returned an error code
```
The build runs successfully on my local machine (running docker v28.1.1), but only fails in Gitlab. The Gitlab CI/CD job is defined with
```yaml
image: docker:28.5.2-cli
services:
- name: docker:28.5.2-dind
```
I've seen many answers to similar issues that suggest something along the lines of including
```
sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys <list of missing keys>
```
in the Dockerfile, but this fails with "Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8))." (supposedly just a warning but it fails without any other message)
I've also tried
```
apt-get install -y --no-install-recommends debian-archive-keyring
```
But that just tells me "debian-archive-keyring is already the newest version (2023.3+deb12u2). debian-archive-keyring set to manually installed." then continues to fail as normal.
Finally, I attempted to just run `curl -v http://deb.debian.org/debian/dists/bookworm/InRelease` to see if I could even download one of the troublesome files directly and got this error
```
#5 [2/3] RUN curl -v https://deb.debian.org/debian/dists/bookworm/InRelease
#5 0.365 * getaddrinfo() thread failed to start
#5 0.365 * Could not resolve host: deb.debian.org
#5 0.365 * Closing connection 0
#5 0.365 curl: (6) getaddrinfo() thread failed to start
```
So it seems maybe there's some kind of issue with DNS inside the docker-in-docker container. What I'm curious about is what changed between 2.7.3 and 2.8.0 so that earlier images were not affected by this DNS issue?
Also, if anybody has any suggestions I would be very grateful.
GitHub link: https://github.com/apache/airflow/discussions/64583
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] Can't build docker images based on slim-2.8.0 or higher [airflow]
Posted by "will-m-buchanan (via GitHub)" <gi...@apache.org>.
GitHub user will-m-buchanan added a comment to the discussion: Can't build docker images based on slim-2.8.0 or higher
@Nickalus12, I just ran a build on an upgraded Docker runner and it worked. Thank you for the help!
GitHub link: https://github.com/apache/airflow/discussions/64583#discussioncomment-16427234
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] Can't build docker images based on slim-2.8.0 or higher [airflow]
Posted by "will-m-buchanan (via GitHub)" <gi...@apache.org>.
GitHub user will-m-buchanan added a comment to the discussion: Can't build docker images based on slim-2.8.0 or higher
Thanks @Nickalus12. Not having access to the runner itself, I tried your `--security-opt` suggestion and it made no difference. I also tried using a higher version of the DinD image. As we were already at a version over 27, I tried using the latest version, 29.3.1, and got an error:
```
ERROR: failed to build: Error response from daemon: client version 1.53 is too new. Maximum supported API version is 1.40: driver not connecting
```
It seems that the runner's Docker version is too low, so I've requested it be upgraded to the latest. Once that goes through I will try again and update here.
GitHub link: https://github.com/apache/airflow/discussions/64583#discussioncomment-16413270
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] Can't build docker images based on slim-2.8.0 or higher [airflow]
Posted by "Nickalus12 (via GitHub)" <gi...@apache.org>.
GitHub user Nickalus12 added a comment to the discussion: Can't build docker images based on slim-2.8.0 or higher
Yeah, the `Maximum supported API version is 1.40` error confirms it — your runner's Docker daemon is too old to talk to the newer DinD client. Once the upgrade goes through, the Bookworm `apt` signature issues should clear up since the newer seccomp profile supports the syscalls that Debian 12's tooling needs.
If the upgrade stalls or takes a while, one other thing you could try in the meantime: pin your base image back to a Bullseye variant if one is still available for your Airflow version. Something like `apache/airflow:slim-2.11.2-python3.10-bullseye` — though I'm not sure they still publish those for 2.11.x.
Drop me a tag (@Nickalus12) once the runner upgrade lands and I'll help troubleshoot if anything else comes up.
GitHub link: https://github.com/apache/airflow/discussions/64583#discussioncomment-16414001
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] Can't build docker images based on slim-2.8.0 or higher [airflow]
Posted by "Nickalus12 (via GitHub)" <gi...@apache.org>.
GitHub user Nickalus12 added a comment to the discussion: Can't build docker images based on slim-2.8.0 or higher
This is a classic symptom of the transition from Debian 11 (Bullseye) to Debian 12 (Bookworm). Airflow 2.7.3 images were Bullseye-based, but 2.8.0 shifted the default to Bookworm.
The GPG "NO_PUBKEY" and "invalid signature" errors are usually a red herring in CI environments. The real issue is almost always that your GitLab runner's host is running an older version of Docker or `libseccomp` that doesn't support the syscalls (like `faccessat2` or `clone3`) that Bookworm’s `apt` and `gpg` now rely on. When the syscall is blocked by an outdated seccomp profile, `apt` fails to verify signatures and throws these misleading errors.
You can usually fix this by updating your GitLab runner's Docker version (to 20.10.10+ at a minimum) or ensuring the `docker:dind` service version you're using in your `.gitlab-ci.yml` is current. I believe switching to `docker:24-dind` or `docker:27-dind` often resolves this immediately.
If you can't touch the runner infrastructure, you might be able to work around it by temporarily using a more permissive security policy in your build command, though this depends on your runner's configuration:
```bash
docker build --security-opt seccomp=unconfined .
```
Also, a quick check on your Dockerfile: make sure you're switching to `USER root` before running those `apt` commands. Airflow's slim images default to `USER airflow` (UID 50000), and while that usually causes a "Permission denied" error, it can occasionally cause weirdness with local keyrings during `apt-get update`.
```dockerfile
FROM apache/airflow:slim-2.11.2-python3.10
USER root
RUN apt-get update && apt-get install -y ...
USER airflow
```
I'd bet on the Docker-in-Docker version/host compatibility being the primary culprit here, though. Bookworm is much pickier about the container runtime than Bullseye was.
GitHub link: https://github.com/apache/airflow/discussions/64583#discussioncomment-16412645
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] Can't build docker images based on slim-2.8.0 or higher [airflow]
Posted by "Nickalus12 (via GitHub)" <gi...@apache.org>.
GitHub user Nickalus12 added a comment to the discussion: Can't build docker images based on slim-2.8.0 or higher
No worries!! Glad to help. Ping me anytime if you need something!
GitHub link: https://github.com/apache/airflow/discussions/64583#discussioncomment-16427397
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org