You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "yerenkow (via GitHub)" <gi...@apache.org> on 2023/05/05 13:18:25 UTC
[GitHub] [spark] yerenkow opened a new pull request, #41067: [KUBERNETES] Add configuration for pod memory limits
yerenkow opened a new pull request, #41067:
URL: https://github.com/apache/spark/pull/41067
### What changes were proposed in this pull request?
Adding dedicated configuration for specifying driver/executor memory limits.
### Why are the changes needed?
Separate mem limit for the pod might be needed if sprak JVM launches extrernal processes.
Also, this could be useful in heterogenous loads when some pods may use more memory for bursty operations.
### Does this PR introduce _any_ user-facing change?
Without using new configurations - no
### How was this patch tested?
Unit tested
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-43496][KUBERNETES] Add configuration for pod memory limits [spark]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #41067: [SPARK-43496][KUBERNETES] Add configuration for pod memory limits
URL: https://github.com/apache/spark/pull/41067
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] holdenk commented on pull request #41067: [SPARK-43496][KUBERNETES] Add configuration for pod memory limits
Posted by "holdenk (via GitHub)" <gi...@apache.org>.
holdenk commented on PR #41067:
URL: https://github.com/apache/spark/pull/41067#issuecomment-1579604504
I think we can already handle the situation of non JVM usage through the memory overhead parameters, would that meet your needs or how is this configuration different from that?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] yerenkow commented on pull request #41067: [KUBERNETES] Add configuration for pod memory limits
Posted by "yerenkow (via GitHub)" <gi...@apache.org>.
yerenkow commented on PR #41067:
URL: https://github.com/apache/spark/pull/41067#issuecomment-1547064602
Filed:
https://issues.apache.org/jira/browse/SPARK-43496
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] julienlau commented on pull request #41067: [SPARK-43496][KUBERNETES] Add configuration for pod memory limits
Posted by "julienlau (via GitHub)" <gi...@apache.org>.
julienlau commented on PR #41067:
URL: https://github.com/apache/spark/pull/41067#issuecomment-1724949719
I don't think these new limits should be mandatory
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-43496][KUBERNETES] Add configuration for pod memory limits [spark]
Posted by "yerenkow (via GitHub)" <gi...@apache.org>.
yerenkow commented on PR #41067:
URL: https://github.com/apache/spark/pull/41067#issuecomment-1872414623
Can anyone PTAL at this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #41067: [KUBERNETES] Add configuration for pod memory limits
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on PR #41067:
URL: https://github.com/apache/spark/pull/41067#issuecomment-1537619340
Mind filing a JIRA? See also https://spark.apache.org/contributing.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] vasa47 commented on pull request #41067: [SPARK-43496][KUBERNETES] Add configuration for pod memory limits
Posted by "vasa47 (via GitHub)" <gi...@apache.org>.
vasa47 commented on PR #41067:
URL: https://github.com/apache/spark/pull/41067#issuecomment-1721506888
I need this feature. when can we expect this in main release branch?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] yerenkow commented on pull request #41067: [SPARK-43496][KUBERNETES] Add configuration for pod memory limits
Posted by "yerenkow (via GitHub)" <gi...@apache.org>.
yerenkow commented on PR #41067:
URL: https://github.com/apache/spark/pull/41067#issuecomment-1601959338
Not really - if we have some external process that prepares/transforms data, then it's not part of the JVM memory, and its size can't be predictable transformed into fraction of that memory.
Ideally, I'd propose separate controls for a) JVM xmx, b) pod request mem, c) pod limit mem
But to make it simpler, configuration for limits will do for case with burst-like load.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-43496][KUBERNETES] Add configuration for pod memory limits [spark]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #41067:
URL: https://github.com/apache/spark/pull/41067#issuecomment-1871639179
We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
If you'd like to revive this PR, please reopen it and ask a committer to remove the Stale tag!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org