You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "nitrajen (via GitHub)" <gi...@apache.org> on 2026/03/23 21:22:20 UTC
[PR] [SPARK-55330][K8S] Make spark.kubernetes.legacy.useReadWriteOnceAccessMode a public config [spark]
nitrajen opened a new pull request, #54963:
URL: https://github.com/apache/spark/pull/54963
### What changes were proposed in this pull request?
`spark.kubernetes.legacy.useReadWriteOnceAccessMode` was introduced in SPARK-46945 as an `.internal()` config. This PR removes the `.internal()` marker and improves the doc string so the config appears in the public documentation and can be used by operators.
### Why are the changes needed?
SPARK-46786 (Spark 4.0) changed the default PVC access mode from `ReadWriteOnce` to `ReadWriteOncePod`. The Kubernetes documentation states that whether `fsGroup` is applied to a mounted volume depends on the CSI driver's `fsGroupPolicy` setting, and certain policies do not apply `fsGroup` for `ReadWriteOncePod` volumes. This can cause non-root executor processes to fail at job startup when trying to create the `blockmgr-xxx` directory under a PVC-backed local dir:
```
ERROR JavaUtils: Failed to create directory /apps/application/data/blockmgr-2469dd51-bfd2-478d-9b3c-d5593bf21c26
java.nio.file.AccessDeniedException: /apps/application/data/blockmgr-2469dd51-bfd2-478d-9b3c-d5593bf21c26
```
The escape hatch config (`spark.kubernetes.legacy.useReadWriteOnceAccessMode=true`) already exists in the codebase to restore `ReadWriteOnce` behavior, but because it is marked internal it does not appear in the docs and users cannot find it.
### Does this PR introduce _any_ user-facing change?
Yes. `spark.kubernetes.legacy.useReadWriteOnceAccessMode` becomes a public, documented configuration option (versioned as 4.2.0). Previously it was internal and invisible to users.
### How was this patch tested?
Added two unit tests to `MountVolumesFeatureStepSuite`:
- `SPARK-55330: OnDemand PVC uses ReadWriteOncePod access mode by default` — verifies the default PVC access mode is `ReadWriteOncePod`
- `SPARK-55330: OnDemand PVC uses ReadWriteOnce when legacy access mode is enabled` — verifies that setting `spark.kubernetes.legacy.useReadWriteOnceAccessMode=true` results in PVCs being created with `ReadWriteOnce`
Both tests use `getAdditionalKubernetesResources()` on `MountVolumesFeatureStep` and assert the `accessModes` field on the returned `PersistentVolumeClaim`.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Sonnet 4.6
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-55330][K8S] Make spark.kubernetes.legacy.useReadWriteOnceAccessMode a public config [spark]
Posted by "github-actions[bot] (via GitHub) via reviews" <re...@spark.apache.org>.
github-actions[bot] closed pull request #54963: [SPARK-55330][K8S] Make spark.kubernetes.legacy.useReadWriteOnceAccessMode a public config
URL: https://github.com/apache/spark/pull/54963
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-55330][K8S] Make spark.kubernetes.legacy.useReadWriteOnceAccessMode a public config [spark]
Posted by "Pluies (via GitHub)" <gi...@apache.org>.
Pluies commented on PR #54963:
URL: https://github.com/apache/spark/pull/54963#issuecomment-4788468063
Additional comment here: we use the [local-storage-provisioner](https://github.com/kubernetes-sigs/sig-storage-local-static-provisioner) to create PVs on AWS instances with instancestore, and for some reason this setup only supports `ReadWriteOnce`, and not `ReadWriteOncePod`.
I would welcome a documented way to set the `accessMode` as part of the `spark.kubernetes.*` options, either globally via the current suggestion, or directly as an option on `spark.kubernetes.executor.volumes.[VolumeType].[VolumeName].options.[OptionName]` 👍
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-55330][K8S] Make spark.kubernetes.legacy.useReadWriteOnceAccessMode a public config [spark]
Posted by "nitrajen (via GitHub)" <gi...@apache.org>.
nitrajen commented on PR #54963:
URL: https://github.com/apache/spark/pull/54963#issuecomment-4183953275
@dongjoon-hyun Thanks for the review and explanantion!
Understood on the ReadWriteOnce concern - multiple pods on the same node mounting the same PVC.
For users with `claimName: OnDemand` Spark 4.x and get AccessDeniedException creating the blockmgr directory on the PVC mount (same config worked fine on 3.x.). My understanding is with onDemand Spark assigns a unique PVC per executor and explicitly avoids reusing a PVC that's still in use - so sharing is unlikely, though not impossible during the brief window when an old executor is dying and a new one starts. The RWO config is opt-in and default remains ReadWriteOncePod. Apologies if I misunderstood something..
On hostPath - since it's tied to a specific node, if an executor restarts on a different node its data wouldn't be accessible. Would that be a concern for users relying on shuffle recovery across executor restarts?
One alternative is `fsGroupPolicy: File` on the CSI driver, but that's a cluster-wide setting requiring admin access and not always in the hands of the team running Spark jobs.
Happy to close this out for better path or any suggestions..
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-55330][K8S] Make spark.kubernetes.legacy.useReadWriteOnceAccessMode a public config [spark]
Posted by "github-actions[bot] (via GitHub) via reviews" <re...@spark.apache.org>.
github-actions[bot] commented on PR #54963:
URL: https://github.com/apache/spark/pull/54963#issuecomment-6551202560
We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
If you'd like to revive this PR, please reopen it and ask a committer to remove the Stale tag!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org