You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "yma11 (via GitHub)" <gi...@apache.org> on 2024/06/24 07:13:50 UTC
[PR] [SPARK-48694][CORE]Manage memory used by external cache [spark]
yma11 opened a new pull request, #47067:
URL: https://github.com/apache/spark/pull/47067
### What changes were proposed in this pull request?
This PR proposes changes to count memory used by external cache in storage memory and include it in overall spill logic.
### Why are the changes needed?
We have a scenario that use Spark together with a 3rd party file source cache, which is an independent lib and has its internal logic for cache entry creation, eviction and remove. Currently we allocate dedicated memory for this cache but the problem is that the memory can't be shared along with Spark execution/storage memory. It will be more effective for memory usage if we can count it in the UnifiedMemoryManager and include it in memory spill logic.
We also have a requirement of memory management for a native RDD cache implementation. The existing interfaces in MemoryStore is generally bound with Spark SerializerManager and BlockEvictionHandler. It's not easy to extend for such customized RDD cache.
### Does this PR introduce _any_ user-facing change?
Yes. It introduces some configurations for memory management: `spark.memory.external.cache.enabled`, `spark.memory.external.storageFraction` and `spark.memory.storage.preferEvictExtCache`.
### How was this patch tested?
It's tested by tpcds workload with external file source cache.
### Was this patch authored or co-authored using generative AI tooling?
No
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-48694][CORE]Manage memory used by external cache [spark]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #47067:
URL: https://github.com/apache/spark/pull/47067#issuecomment-2403661782
We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
If you'd like to revive this PR, please reopen it and ask a committer to remove the Stale tag!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-48694][CORE]Manage memory used by external cache [spark]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #47067: [SPARK-48694][CORE]Manage memory used by external cache
URL: https://github.com/apache/spark/pull/47067
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org