[I] [VL] Velox memCache is invisible to Spark's memory manager [gluten]

["baibaichen (via GitHub)" <gi...@apache.org>]

baibaichen opened a new issue, #12671:
URL: https://github.com/apache/gluten/issues/12671

   ### Description
   
   When `spark.gluten.sql.columnar.backend.velox.cacheEnabled` is on, `VeloxBackend::initCache()`
   builds an `MmapAllocator` sized from `memCacheSize` and hands it to the `AsyncDataCache`:
   
   ```cpp
   options.capacity = memCacheSize;
   cacheAllocator_ = std::make_shared<velox::memory::MmapAllocator>(options);
   if (ssdCacheSize == 0) {
     // TODO: this is not tracked by Spark.
     asyncDataCache_ = velox::cache::AsyncDataCache::create(cacheAllocator_.get());
   } else {
     // TODO: this is not tracked by Spark.
     auto ssd = initSsdCache(ssdCacheSize);
     asyncDataCache_ = velox::cache::AsyncDataCache::create(cacheAllocator_.get(), std::move(ssd));
   }
   ```
   
   Spark never sees that memory. The two `// TODO: this is not tracked by Spark.` comments in
   `cpp/velox/compute/VeloxBackend.cc` record exactly this.
   
   #### Why it matters
   
   An executor ends up occupying its Spark budget **plus** the whole cache. Every decision Spark
   makes from its own books is then made against a number that is short by up to the cache size:
   
   - when to spill,
   - when to fail a task with an OOM,
   - how much a task may acquire.
   
   The cache is also fixed at its configured size for the lifetime of the executor. It cannot give
   anything back when queries need the memory, which is the wrong priority: a query that has to
   spill costs far more than the cache misses that shrinking would cause.
   
   #### Proposal
   
   Make the cache capacity something Spark can move at runtime, and register what the cache occupies
   with Spark's memory manager, so that:
   
   1. the cache's footprint is visible to Spark on **all** supported Spark versions;
   2. the cache yields memory when execution needs it, and reclaims it when the pressure lifts;
   3. usage stays bounded by what Spark has agreed to.
   
   An earlier attempt used Spark 4.1's `UnmanagedMemoryConsumer`, which only works on 4.1+ and reacts
   only at task boundaries.
   
   This description was written with the assistance of generative AI tooling (GitHub Copilot CLI).
   
   ### Gluten version
   
   main branch
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org