[jira] [Updated] (SPARK-58057) Make Kubernetes VolumeSuite OnDemand PVC test robust to slow container startup

["Dongjoon Hyun (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-58057?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Dongjoon Hyun updated SPARK-58057:
----------------------------------
    Fix Version/s: 4.2.0
                       (was: 4.2.1)

> Make Kubernetes VolumeSuite OnDemand PVC test robust to slow container startup
> ------------------------------------------------------------------------------
>
>                 Key: SPARK-58057
>                 URL: https://issues.apache.org/jira/browse/SPARK-58057
>             Project: Spark
>          Issue Type: Bug
>          Components: Kubernetes, Tests
>    Affects Versions: 5.0.0
>            Reporter: Hyukjin Kwon
>            Assignee: Hyukjin Kwon
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: 4.2.0, 4.3.0
>
>
> The Kubernetes integration test VolumeSuite "A driver-only Spark job with an OnDemand PVC volume" is flaky (observed on the scheduled build_java21 workflow on branch-4.0).
> checkDisk() runs 'df <path>' inside the driver container via Utils.executeCommand, wrapped in eventually(...), but used a short fixed timeout of 10 seconds. For the OnDemand PVC cases, dynamic PVC provisioning and mounting can keep the driver container in ContainerCreating longer than that, so the in-pod command is not yet runnable and the assertion never passes within 10s. The same test passes on the in-run retry, which is characteristic of a too-tight readiness timeout rather than a product bug.
> In addition, logForFailedTest() fetches the driver pod log with .getLog without a guard. When a test fails while the driver pod is still in ContainerCreating, that .getLog throws KubernetesClientException (HTTP 400, "container ... is waiting to start: ContainerCreating"), which then surfaces as the reported failure and masks the actual cause. The executor-pod log fetch right below already wraps .getLog in a try/catch.
> Fix: use the shared TIMEOUT (3 minutes) for checkDisk, matching the other in-pod readiness waits, and apply the same try/catch guard to the driver-pod log fetch in logForFailedTest.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org