You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@gluten.apache.org by "yma11 (via GitHub)" <gi...@apache.org> on 2024/05/27 13:28:14 UTC
[I] [VL] Support file cache spill in Gluten [incubator-gluten]
yma11 opened a new issue, #5884:
URL: https://github.com/apache/incubator-gluten/issues/5884
### Description
Velox backend provides 2-level file cache (`AsyncDataCache` and `SsdCache` and we have enabled it in [PR](https://github.com/apache/incubator-gluten/pull/1076/files), using a dedicated `MMapAllocator` initialized with configured capacity. This part of memory is not counted by execution memory or storage memory, and not managed by Spark `UnifiedMemoryManager`. In this ticket, we would like to fill this gap by following designs:
- Add `NativeStorageMemory` segment in vanilla `StorageMemory`. We will have a configuration `spark.memory.native.storageFraction` to define its size. Then we use this size `offheap.memory*spark.memory.storageFraction*spark.memory.native.storageFraction` to initialize `AsyncDataCache`.
- -Add configuration `spark.memory.storage.preferSpillNative` to determine preference of spilling RDD cache or FileCache(Native) when storage memory should be shrinked. For example, when queries are mostly executed on same data sources, we prefer to keep native file cache.
- Update vanilla storage memory pool size everywhere it's used by collecting stats of `NativeStorageMemory`.
- Implement/update `AsyncDataCache::usedBytes()`/`AsyncDataCache::shrink()`/`AsyncDataCache::findOrCreate`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Support file cache spill in Gluten [incubator-gluten]
Posted by "yma11 (via GitHub)" <gi...@apache.org>.
yma11 commented on issue #5884:
URL: https://github.com/apache/incubator-gluten/issues/5884#issuecomment-2153715084
@zhli1142015 @FelixYBW @zhouyuan @zhztheplayer The code changes are available in following PRs: [Spark](https://github.com/yma11/spark/pull/4/files), [Gluten](https://github.com/yma11/gluten/pull/2), [Velox](https://github.com/yma11/velox/pull/1), please take a review. Next step I will test it in E2E and add some docs for it. Here are some explanations about code change:
1) New files in `shims/common`: Existing memory allocator listeners such as `ManagedAllocationListener` are under package `gluten-data` and native JNIs are under `backends-velox`, but because of I need to call these classes/APIs in the injects, so I put them in `shims/common`.
2) Late initialization of file cache: We use `GlutenMemStoreInjects` to get the conf of cache and then do initialization after Velox backend initialized which assures the native libs are loaded.
3) Cache size setting: we need to pass a cache size when `setAsyncDataCache`, using the default `int64_t max` will cause a `std::bad_alloc`. But the size is sensitive since in Velox, data cache will use this value to control the memory allocation. If it is too small, allocation failure will happen at native side even Spark doesn't report it at java side. As We leverage Spark memory manager to control the memory logic, we'd resolve this confliction by giving a large fake size for AsyncDataCache, maybe same as offheap size.
4) SSD cache can't work well in my test as the file cache entry is easily larger than `8M` and will cause check failure. [Issue](https://github.com/facebookincubator/velox/issues/10098) is reported for tracking.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Support file cache spill in Gluten [incubator-gluten]
Posted by "zhztheplayer (via GitHub)" <gi...@apache.org>.
zhztheplayer commented on issue #5884:
URL: https://github.com/apache/incubator-gluten/issues/5884#issuecomment-2134583768
> Add configuration spark.memory.storage.preferSpillNative to determine preference of spilling RDD cache or FileCache(Native) when storage memory should be shrinked. For example, when queries are mostly executed on same data sources, we prefer to keep native file cache.
Will there be any better name than `preferSpillNative` ? One can't easily understand it's not about Gluten's RDD cache. User may think of all Gluten stuffs native.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Support file cache spill in Gluten [incubator-gluten]
Posted by "zhli1142015 (via GitHub)" <gi...@apache.org>.
zhli1142015 commented on issue #5884:
URL: https://github.com/apache/incubator-gluten/issues/5884#issuecomment-2136292333
Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Support file cache spill in Gluten [incubator-gluten]
Posted by "yma11 (via GitHub)" <gi...@apache.org>.
yma11 commented on issue #5884:
URL: https://github.com/apache/incubator-gluten/issues/5884#issuecomment-2191679510
Some updates:
- Based on previous implementation that we added a listener against `MmapMemoryAllocator` to trigger the `acquireStoargeMemory`/`releaseStorageMemory` by binding with `freeNonContiguous`/`allocateNonContiguousWithoutRetry`, there is a problem can't be resolved because of these 2 methods are not always called when memory usage changes in AsyncDataCache. For example, when do a `shrink(bytes)`, the released size is not always equal to the bytes returned by `freeNonContiguous`, like this log `freed size by freeAllocations: 1753088 largeEvicted is: 4374528 tinyEvicted is:0`. Velox memory management introduced several regions, like small data, pages data, etc. So if we use returned shrinked size to decrease the storage memory pool, while using `freeNonContiguous`/`allocateNonContiguousWithoutRetry` to change `memoryUsed` value, there will eventually be a mismatch.
- Instead, we now switch to add a listener against `AsyncDataCache` itself. When `shrink` happens, we will release the amount of storage memory. when a new entry created, we will acquire corresponding size memory. But may return some back as it may not actually increase such amount of bytes if cache shrink happens during a new cache entry creation. It also has a advantage that no special change needed at velox side with only the "findOrCreate()" method changes to virtual.
We have created [PR](https://github.com/apache/spark/pull/47067) for upstream Spark.
For velox, there is only one line change:
<img width="490" alt="image" src="https://github.com/apache/incubator-gluten/assets/11849056/5671f423-49ff-4b24-8017-45ecdb34a326">
Latest code change for Gluten is tracked by [PR](https://github.com/apache/incubator-gluten/pull/6239).
@zhli1142015 would like to try this implementation in your workload?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Support file cache spill in Gluten [incubator-gluten]
Posted by "yma11 (via GitHub)" <gi...@apache.org>.
yma11 commented on issue #5884:
URL: https://github.com/apache/incubator-gluten/issues/5884#issuecomment-2134559957
@zhouyuan @zhztheplayer Can you help take a review at this draft design? Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Support file cache spill in Gluten [incubator-gluten]
Posted by "FelixYBW (via GitHub)" <gi...@apache.org>.
FelixYBW commented on issue #5884:
URL: https://github.com/apache/incubator-gluten/issues/5884#issuecomment-2135956052
@zhli1142015
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org