Node Errors ("Backing Channel... is Disconnected.")

[Jason Kuster <ja...@google.com>]

Hi all,

We've been seeing some instances of nodes failing with messages like
"Backing channel... is disconnected." In talking to Infra, it appears that
this is due to both out-of-memory errors on the VMs killing the Jenkins
process, as well as potentially some Puppet issues on the Infra side.

I've opened https://issues.apache.org/jira/browse/INFRA-16380 to track the
infra-side issues; the out of memory errors are more concerning. Does
anyone know if the recent move to Gradle would have caused memory usage to
increase? We may be able to bump machine RAM, or limit parallelism somehow
to mitigate the issue. In the meantime, if you see this issue, feel free to
add the failing build as a comment to the bug.

Best,

Jason

-- 
-------
Jason Kuster
Apache Beam / Google Cloud Dataflow

See something? Say something. go/jasonkuster-feedback

Re: Node Errors ("Backing Channel... is Disconnected.")

[Scott Wegner <sw...@google.com>]

The Gradle build is much more granular in its ability to execute tasks in
parallel, so it's possible that it's doing more parallel work and thus
using additional memory.

When I run a Gradle build locally with the same flags as Jenkins [1], I see
a main Java process using ~2GB of memory and lots of worker subprocesses
each taking ~0.5GB. By default Gradle uses the same number of workers as
CPU cores.

Do you know the specs on our Jenkins workers, and are they limited to
executing a single job at a time? We can configure the amount of
parallelism or JVM heap size. Note that scaling down will result in longer
build runtimes.

[1]
https://github.com/apache/beam/blob/cd2b709c820160a0549f6b5396eeefcd58b6ba51/.test-infra/jenkins/common_job_properties.groovy#L168

On Mon, Apr 16, 2018 at 4:42 PM Jason Kuster <ja...@google.com> wrote:

> Hi all,
>
> We've been seeing some instances of nodes failing with messages like
> "Backing channel... is disconnected." In talking to Infra, it appears that
> this is due to both out-of-memory errors on the VMs killing the Jenkins
> process, as well as potentially some Puppet issues on the Infra side.
>
> I've opened https://issues.apache.org/jira/browse/INFRA-16380 to track
> the infra-side issues; the out of memory errors are more concerning. Does
> anyone know if the recent move to Gradle would have caused memory usage to
> increase? We may be able to bump machine RAM, or limit parallelism somehow
> to mitigate the issue. In the meantime, if you see this issue, feel free to
> add the failing build as a comment to the bug.
>
> Best,
>
> Jason
>
>
> --
> -------
> Jason Kuster
> Apache Beam / Google Cloud Dataflow
>
> See something? Say something. go/jasonkuster-feedback
>
-- 


Got feedback? http://go/swegner-feedback