You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "turboFei (via GitHub)" <gi...@apache.org> on 2026/06/25 17:56:14 UTC
[PR] [SPARK-57698][KUBERNETES] Set MALLOC_ARENA_MAX in Kubernetes entrypoint [spark]
turboFei opened a new pull request, #56781:
URL: https://github.com/apache/spark/pull/56781
### What changes were proposed in this pull request?
Add `export MALLOC_ARENA_MAX="${MALLOC_ARENA_MAX:-4}"` to Spark's Kubernetes container entrypoint script (`entrypoint.sh`), placed just before the command dispatch (`case "$1" in`).
The value defaults to 4 but respects any pre-set value in the container environment, allowing operators to override it without rebuilding images.
### Why are the changes needed?
`MALLOC_ARENA_MAX` limits the number of glibc malloc arenas. Without this cap, glibc creates arenas proportional to the CPU count, which can cause excessive virtual memory usage. This is a well-known problem in containerized environments where cgroup memory limits are strictly enforced.
Hadoop already sets `MALLOC_ARENA_MAX=4` in its YARN startup scripts (`hadoop-functions.sh`). Spark on YARN therefore inherits this setting, but Spark on Kubernetes does not — creating inconsistent memory allocation behavior across deployment modes. This change closes that gap.
### Does this PR introduce _any_ user-facing change?
No. The default behavior is equivalent to what YARN users already experience. Users who need a different value can override it by setting `MALLOC_ARENA_MAX` in their pod spec environment.
### How was this patch tested?
No automated test added — the change is a single shell variable export with no logic to unit-test. Manual verification: confirmed the variable is exported correctly in a local shell session and that a pre-set value is preserved (not overwritten) by the `:-` default syntax.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Sonnet 4.6
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57698][KUBERNETES] Set MALLOC_ARENA_MAX in Kubernetes entrypoint [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #56781:
URL: https://github.com/apache/spark/pull/56781#issuecomment-4803728512
Sorry but we want to keep it simple, @turboFei .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57698][KUBERNETES] Set MALLOC_ARENA_MAX in Kubernetes entrypoint [spark]
Posted by "turboFei (via GitHub)" <gi...@apache.org>.
turboFei closed pull request #56781: [SPARK-57698][KUBERNETES] Set MALLOC_ARENA_MAX in Kubernetes entrypoint
URL: https://github.com/apache/spark/pull/56781
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57698][KUBERNETES] Set MALLOC_ARENA_MAX in Kubernetes entrypoint [spark]
Posted by "turboFei (via GitHub)" <gi...@apache.org>.
turboFei commented on PR #56781:
URL: https://github.com/apache/spark/pull/56781#issuecomment-4803667124
> Why don't we use the existing configurations, e.g., something like spark.kubernetes.driverEnv.MALLOC_ARENA_MAX=4 and spark.executorEnv.MALLOC_ARENA_MAX=4
Thanks for the review. I agree that we can set config for driver/executorEnv, this PR just set the default value to 4 in case some customer is not aware of this change after spark on yarn to kubernetes migration.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57698][KUBERNETES] Set MALLOC_ARENA_MAX in Kubernetes entrypoint [spark]
Posted by "turboFei (via GitHub)" <gi...@apache.org>.
turboFei commented on PR #56781:
URL: https://github.com/apache/spark/pull/56781#issuecomment-4803857375
OK，thanks for the review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org