[jira] [Commented] (HIVE-28954) CI fails intermittently due to ephemeral-storage exhaustion

["Stamatis Zampetakis (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HIVE-28954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17951176#comment-17951176 ] 

Stamatis Zampetakis commented on HIVE-28954:
--------------------------------------------

The ephemeral-storage capacity of each node of the GKE cluster is  101.2 GB out of which 47.06 GB are allocatable. Currently, the precommit pods do not define request/limit for the ephemeral-storage resource so scheduling does not take this aspect into account. Spot checking the disk usage on some pods it seems that the hdb container are consuming at least ~10GB. Usually, there are 5 pods running on each node so it seems that we can easily reach and exceed the allocatable space.

In order to overcome the errors, we have various options:
 # Define resource request/limits for ephemeral storage to avoid scheduling pods on nodes with insufficient resources and stop pods/tests that are behaving badly from affecting good runs.
 # Increase the disk space for the nodes in the GKE cluster that will likely also increase the overall cost of each precommit run.
 # Decrease the disk space consumed by test execution by tuning logs, output, etc.

In the context, of this ticket we should rather focus on option 1 and 2. Option 3, can be considered an independent improvement that needs careful investigation and analysis that has little to do with the CI infrastructure.

> CI fails intermittently due to ephemeral-storage exhaustion
> -----------------------------------------------------------
>
>                 Key: HIVE-28954
>                 URL: https://issues.apache.org/jira/browse/HIVE-28954
>             Project: Hive
>          Issue Type: Bug
>          Components: Testing Infrastructure
>            Reporter: Stamatis Zampetakis
>            Priority: Major
>
> The CI fails intermittently due to ephemeral-storage exhaustion. The failures on master have been quite often recently:
> * https://ci.hive.apache.org/blue/organizations/jenkins/hive-precommit/detail/master/2512/pipeline
> * https://ci.hive.apache.org/blue/organizations/jenkins/hive-precommit/detail/master/2511/pipeline
> * https://ci.hive.apache.org/blue/organizations/jenkins/hive-precommit/detail/master/2510/pipeline
> * https://ci.hive.apache.org/blue/organizations/jenkins/hive-precommit/detail/master/2507/pipeline
> * https://ci.hive.apache.org/blue/organizations/jenkins/hive-precommit/detail/master/2505/pipeline
> * https://ci.hive.apache.org/blue/organizations/jenkins/hive-precommit/detail/master/2501/pipeline
> * https://ci.hive.apache.org/blue/organizations/jenkins/hive-precommit/detail/master/2474/pipeline
> The error message is shown below.
> {noformat}
> Unable to create live FilePath for hive-precommit-master-2512-g4v1j-l5dbf-5l773; hive-precommit-master-2512-g4v1j-l5dbf-5l773 was marked offline: Pod failed (Reason: Evicted, Message: The node was low on resource: ephemeral-storage. Threshold quantity: 10120387530, available: 9866372Ki. Container jnlp was using 88Ki, request is 0, has larger consumption of ephemeral-storage. Container dind was using 188Ki, request is 0, has larger consumption of ephemeral-storage. Container hdb was using 453252Ki, request is 0, has larger consumption of ephemeral-storage. )
> {noformat}
> As the message indicates the node where the pod is running is low on ephemeral-storage leading to the eviction of some pods from the Kubernetes scheduler.
> The ephemeral-storage among other things stores all the jars, logs, and output from the test execution.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)