Re: [I] [Epic] Optionally Limit memory used by DataFusion plan [arrow-datafusion]

["SteveLauC (via GitHub)" <gi...@apache.org>]

SteveLauC commented on issue #587:
URL: https://github.com/apache/arrow-datafusion/issues/587#issuecomment-1895387394

   Hi, from this thread, it seems that DataFusion can ONLY limit the memory used by those resource-heavy operators, can it limit the memory used by the underlying FileScan operators, like `ParquetExec`?
   
   Let me give a demo with the following code:
   
   ```rs
   use datafusion::execution::memory_pool::{GreedyMemoryPool, MemoryPool};
   use datafusion::execution::runtime_env::{RuntimeConfig, RuntimeEnv};
   use datafusion::prelude::{ParquetReadOptions, SessionConfig, SessionContext};
   use std::sync::Arc;
   
   #[tokio::main(flavor = "current_thread")]
   async fn main() {
       let mem_pool: Arc<dyn MemoryPool> = Arc::new(GreedyMemoryPool::new(0)); // limit memory usage to 0
       let rt_cfg = RuntimeConfig::new().with_memory_pool(mem_pool);
       let rt = RuntimeEnv::new(rt_cfg).unwrap();
   
       let session_cfg = SessionConfig::new();
       let ctx = SessionContext::new_with_config_rt(session_cfg, Arc::new(rt));
   
       ctx.register_parquet("foo", "foo.parquet", ParquetReadOptions::default())
           .await
           .unwrap();
       let df = ctx.sql("select * from foo").await.unwrap();
       df.show().await.unwrap();
   }
   ```
   
   Even though we limit the available memory to 0, the query exeuctes without any issue:
   
   ```sh
   $ ls -l foo.parquet
   .rw-r--r-- 484 steve 17 Jan 17:08 foo.parquet
   
   $ cargo r -q
   +-----+
   | foo |
   +-----+
   | bar |
   | bar |
   | bar |
   | bar |
   | bar |
   | bar |
   | bar |
   | bar |
   +-----+
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org