Jenkins worker Disk Space Utilization alerts

[Brian Hulette <bh...@google.com>]

I looked into these alerts a little bit. Note the alerts are coming from
/dev/loop2, not the actual disk (/dev/sda1). All of the loopX devices on
all nodes are at 100%.

I found this answer [1] with a brief explanation of what these devices are.
So it seems like having these at 100% is WAI. Does anyone know why our
monitoring has started watching these devices?

Brian

[1] https://askubuntu.com/questions/906581/what-is-dev-loopx

---------- Forwarded message ---------
From: Google Cloud Alerting <al...@google.com>
Date: Sun, Oct 17, 2021 at 4:27 PM
Subject: [ALERT] Disk Space Utilization (Apache Beam Jenkins) on
apache-beam-testing apache-ci-beam-jenkins-9
To: <de...@beam.apache.org>


[image: Google Cloud] VIEW INCIDENT
<https://console.cloud.google.com/monitoring/alerting/incidents/0.m8khtqbat79p?project=apache-beam-testing>
[image: exclamation mark] Alert firing
Disk Space Utilization (Apache Beam Jenkins)

Disk utilization for apache-beam-testing apache-ci-beam-jenkins-9 with
metric labels {device=loop2} and system labels
{instance_group=apache-ci-beam-jenkins-nodes} is above the threshold of
85.000 with a value of 100.000.
Summary

*Start time*
Oct 17, 2021 at 11:27PM UTC (less than 1 sec ago)

*Project*
apache-beam-testing
<https://console.cloud.google.com/?project=apache-beam-testing>

*Policy*
Apache Beam Jenkins Disk Utilization >85%
<https://console.cloud.google.com/monitoring/alerting/policies/11173713051479322061?project=apache-beam-testing>

*Condition*
Disk Space Utilization (Apache Beam Jenkins)

*Metric*
agent.googleapis.com/disk/percent_used

*Threshold*
above 85

*Observed*
100.000
Policy documentation

Check that the inventory jobs are running properly on Jenkins 1
<https://ci-beam.apache.org/view/Inventory/>. These jobs run periodically
to clean up the instances and print useful information.

If they’re not running properly, you may need to cancel any pending
inventory jobs (if any), and then run them manually from within the Jenkins
UI.
VIEW INCIDENT
<https://console.cloud.google.com/monitoring/alerting/incidents/0.m8khtqbat79p?project=apache-beam-testing>
[image: Google]

© 2020 Google LLC
1600 Amphitheatre Parkway, Mountain View, CA 94043

You have received this mandatory service announcement to update you about
important changes to Google Cloud Platform or your account.

Manage notifications
<https://console.cloud.google.com/monitoring/alerting/policies/11173713051479322061/edit?project=apache-beam-testing>