You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@impala.apache.org by Aleksandr Efimov <dz...@gmail.com> on 2026/08/26 07:40:14 UTC
jenkins.impala.io: built-in node fails to spawn processes (auto-critic and dryrun are down)
Hi all,
Something happened to the Jenkins controller between 03:39 and 06:44 UTC
today: every job that runs on the built-in node now dies on its first shell
step. The good news is that they fail fast - duration 0s)
Building on the built-in node in workspace /var/lib/jenkins/workspace/...
java.io.IOException: Failed to exec spawn helper
Caused: java.io.IOException: Cannot run program "/bin/bash": Failed to exec
spawn helper
Example: https://jenkins.impala.io/job/gerrit-auto-critic/24918/console
So far this hits gerrit-auto-critic #24918, gerrit-auto-critic-test #6011
and gerrit-verify-dryrun-external #3786, #3787,#3788 - all of them on the
built-in node. That node was fine a few hours earlier: gerrit-auto-critic
#24917 passed at 03:28 UTC in 87s, gerrit-auto-critic-test #6010 at 03:39
UTC. Jobs on EC2 agents are unaffected - ubuntu-20.04-build-only,
rat-check-ub2004 and python-check-ub2004 all passed around 07:10 UTC.
Two practical effects. Pre-commit dryruns can't be started at all, and I
don't see a way around that, since gerrit-verify-dryrun sits on the
built-in node as well. The other one is quieter and easier to miss:
auto-critic has stopped commenting, so a patch set with no auto-critic
remarks currently looks exactly like a patch set it had nothing to say
about.
The trace ends with Jenkins own list of possible causes and fixes,
restarting the controller JVM being the first of them. If someone with
admin access has a moment, could you take a look?
Thanks!
Re: jenkins.impala.io: built-in node fails to spawn processes (auto-critic and dryrun are down)
Posted by Laszlo Gaal <la...@cloudera.com>.
Thanks for letting us know, Alesandr,
I'll take a look a bit later today (I'll be commuting, away from actionable
keyboards),
and update the community.
Best,
- LaszloG
On Wed, Aug 26, 2026 at 9:40 AM Aleksandr Efimov <dz...@gmail.com> wrote:
> Hi all,
>
> Something happened to the Jenkins controller between 03:39 and 06:44 UTC
> today: every job that runs on the built-in node now dies on its first shell
> step. The good news is that they fail fast - duration 0s)
>
> Building on the built-in node in workspace /var/lib/jenkins/workspace/...
> java.io.IOException: Failed to exec spawn helper
> Caused: java.io.IOException: Cannot run program "/bin/bash": Failed to exec
> spawn helper
>
> Example: https://jenkins.impala.io/job/gerrit-auto-critic/24918/console
>
> So far this hits gerrit-auto-critic #24918, gerrit-auto-critic-test #6011
> and gerrit-verify-dryrun-external #3786, #3787,#3788 - all of them on the
> built-in node. That node was fine a few hours earlier: gerrit-auto-critic
> #24917 passed at 03:28 UTC in 87s, gerrit-auto-critic-test #6010 at 03:39
> UTC. Jobs on EC2 agents are unaffected - ubuntu-20.04-build-only,
> rat-check-ub2004 and python-check-ub2004 all passed around 07:10 UTC.
>
> Two practical effects. Pre-commit dryruns can't be started at all, and I
> don't see a way around that, since gerrit-verify-dryrun sits on the
> built-in node as well. The other one is quieter and easier to miss:
> auto-critic has stopped commenting, so a patch set with no auto-critic
> remarks currently looks exactly like a patch set it had nothing to say
> about.
>
> The trace ends with Jenkins own list of possible causes and fixes,
> restarting the controller JVM being the first of them. If someone with
> admin access has a moment, could you take a look?
>
> Thanks!
>
Re: jenkins.impala.io: built-in node fails to spawn processes (auto-critic and dryrun are down)
Posted by Aleksandr Efimov <dz...@gmail.com>.
А guess at the cause.
There is a jenkins-weekly-restart job on the instance, described as
"Restart Jenkins when idle to pick up system library updates. Runs on
Saturdays." Its last run was #51 on 22 August at 12:53, so the controller
JVM has been up since then.
That fits what the trace complains about. The spawn helper is a small
binary that ships with the JDK, and Jenkins own hint singles out in-place
JDK updates. If the JDK on that machine was updated some time after 22
August, the running JVM would keep looking for the helper it started with,
and every fork would fail the way it does now. I can't check that from
outside, so this is a guess rather than a diagnosis.
If it holds, the fix is the restart already scheduled for Saturday, just
sooner. Either way the EC2 agents are unaffected - what is waiting on it is
auto-critic and the pre-commit dryruns.
ср, 26 авг. 2026 г. в 10:40, Aleksandr Efimov <dz...@gmail.com>:
> Hi all,
>
> Something happened to the Jenkins controller between 03:39 and 06:44 UTC
> today: every job that runs on the built-in node now dies on its first shell
> step. The good news is that they fail fast - duration 0s)
>
> Building on the built-in node in workspace /var/lib/jenkins/workspace/...
> java.io.IOException: Failed to exec spawn helper
> Caused: java.io.IOException: Cannot run program "/bin/bash": Failed to
> exec spawn helper
>
> Example: https://jenkins.impala.io/job/gerrit-auto-critic/24918/console
>
> So far this hits gerrit-auto-critic #24918, gerrit-auto-critic-test #6011
> and gerrit-verify-dryrun-external #3786, #3787,#3788 - all of them on the
> built-in node. That node was fine a few hours earlier: gerrit-auto-critic
> #24917 passed at 03:28 UTC in 87s, gerrit-auto-critic-test #6010 at 03:39
> UTC. Jobs on EC2 agents are unaffected - ubuntu-20.04-build-only,
> rat-check-ub2004 and python-check-ub2004 all passed around 07:10 UTC.
>
> Two practical effects. Pre-commit dryruns can't be started at all, and I
> don't see a way around that, since gerrit-verify-dryrun sits on the
> built-in node as well. The other one is quieter and easier to miss:
> auto-critic has stopped commenting, so a patch set with no auto-critic
> remarks currently looks exactly like a patch set it had nothing to say
> about.
>
> The trace ends with Jenkins own list of possible causes and fixes,
> restarting the controller JVM being the first of them. If someone with
> admin access has a moment, could you take a look?
>
> Thanks!
>