You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "xunxunmimi5577 (via GitHub)" <gi...@apache.org> on 2026/04/21 03:35:09 UTC
[I] How to limit the maximum memory usage of the ballista executor [datafusion-ballista]
xunxunmimi5577 opened a new issue, #1563:
URL: https://github.com/apache/datafusion-ballista/issues/1563
How should I limit the maximum memory usage of the ballista executor.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] How to limit the maximum memory usage of the ballista executor? [datafusion-ballista]
Posted by "milenkovicm (via GitHub)" <gi...@apache.org>.
milenkovicm commented on issue #1563:
URL: https://github.com/apache/datafusion-ballista/issues/1563#issuecomment-4290483723
something like
```rust
use ballista_executor::executor_process::{
ExecutorProcessConfig, start_executor_process,
};
use datafusion::execution::{memory_pool::FairSpillPool, runtime_env::RuntimeEnvBuilder};
use std::sync::Arc;
#[tokio::main]
async fn main() -> ballista_core::error::Result<()> {
let memory_pool = Arc::new(FairSpillPool::new(2_000_000));
let runtime_env_producer: ballista_core::RuntimeProducer = Arc::new(move |_config| {
let runtime_env = RuntimeEnvBuilder::new()
.with_memory_pool(memory_pool.clone())
.build()?;
Ok(Arc::new(runtime_env))
});
let config: ExecutorProcessConfig = ExecutorProcessConfig {
override_runtime_producer: Some(runtime_env_producer),
..Default::default()
};
start_executor_process(Arc::new(config)).await
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] How to limit the maximum memory usage of the ballista executor? [datafusion-ballista]
Posted by "xunxunmimi5577 (via GitHub)" <gi...@apache.org>.
xunxunmimi5577 commented on issue #1563:
URL: https://github.com/apache/datafusion-ballista/issues/1563#issuecomment-4301331200
> something like
>
> use ballista_executor::executor_process::{
> ExecutorProcessConfig, start_executor_process,
> };
> use datafusion::execution::{memory_pool::FairSpillPool, runtime_env::RuntimeEnvBuilder};
> use std::sync::Arc;
>
> #[tokio::main]
> async fn main() -> ballista_core::error::Result<()> {
>
> let memory_pool = Arc::new(FairSpillPool::new(2_000_000));
>
> let runtime_env_producer: ballista_core::RuntimeProducer = Arc::new(move |_config| {
> let runtime_env = RuntimeEnvBuilder::new()
> .with_memory_pool(memory_pool.clone())
> .build()?;
>
> Ok(Arc::new(runtime_env))
> });
>
> let config: ExecutorProcessConfig = ExecutorProcessConfig {
> override_runtime_producer: Some(runtime_env_producer),
> ..Default::default()
> };
>
> start_executor_process(Arc::new(config)).await
> }
Thanks for your answer! Previously, I set the memory limit for the MemoryPool too high because I assumed each executor shared a single MemoryPool—until I later discovered that the MemoryPool is actually created each time a task is executed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] How to limit the maximum memory usage of the ballista executor? [datafusion-ballista]
Posted by "milenkovicm (via GitHub)" <gi...@apache.org>.
milenkovicm commented on issue #1563:
URL: https://github.com/apache/datafusion-ballista/issues/1563#issuecomment-4287077426
you would need to override runtime producer on the client adding memory manager, you can find example how to override it at https://github.com/apache/datafusion-ballista/blob/de0aca3da0c959149711d62cfd1266bc98903095/examples/examples/custom-executor.rs#L44
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] How to limit the maximum memory usage of the ballista executor? [datafusion-ballista]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove closed issue #1563: How to limit the maximum memory usage of the ballista executor?
URL: https://github.com/apache/datafusion-ballista/issues/1563
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] How to limit the maximum memory usage of the ballista executor? [datafusion-ballista]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on issue #1563:
URL: https://github.com/apache/datafusion-ballista/issues/1563#issuecomment-4344751398
I am working on adding a config option to the executor to use a memory pool
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] How to limit the maximum memory usage of the ballista executor? [datafusion-ballista]
Posted by "milenkovicm (via GitHub)" <gi...@apache.org>.
milenkovicm closed issue #1563: How to limit the maximum memory usage of the ballista executor?
URL: https://github.com/apache/datafusion-ballista/issues/1563
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org