You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "erenavsarogullari (via GitHub)" <gi...@apache.org> on 2026/02/15 21:18:02 UTC
[PR] Set expected runtime config in error message when the used disk space during the spilling process has exceeded the allocation limit [datafusion]
erenavsarogullari opened a new pull request, #20375:
URL: https://github.com/apache/datafusion/pull/20375
## Which issue does this PR close?
- Closes 20373.
## Rationale for this change
Minor refactoring on error message by exposing required config name for the end user. This is follow-up PR to both PR: #20226 and #20372 by runtime config: `datafusion.runtime.max_temp_directory_size`.
## What changes are included in this PR?
**Current:**
```
The used disk space during the spilling process has exceeded the allowable limit of {}. Try increasing the `max_temp_directory_size` in the disk manager configuration.
```
**New:**
```
The used disk space during the spilling process has exceeded the allowable limit of {}. \
Please try increasing the config: `datafusion.runtime.max_temp_directory_size`.",
```
## Are these changes tested?
Yes, legacy UT case has been updated by covering expected config name.
## Are there any user-facing changes?
Yes, error message has been updated which is exposed to end-users.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] refactor: Set expected runtime config in error message when the used disk space during the spilling process has exceeded the allocation limit [datafusion]
Posted by "erenavsarogullari (via GitHub)" <gi...@apache.org>.
erenavsarogullari commented on code in PR #20375:
URL: https://github.com/apache/datafusion/pull/20375#discussion_r2876107961
##########
datafusion/execution/src/disk_manager.rs:
##########
@@ -420,7 +420,8 @@ impl RefCountedTempFile {
let global_disk_usage = self.disk_manager.used_disk_space.load(Ordering::Relaxed);
if global_disk_usage > self.disk_manager.max_temp_directory_size {
return resources_err!(
- "The used disk space during the spilling process has exceeded the allowable limit of {}. Try increasing the `max_temp_directory_size` in the disk manager configuration.",
+ "The used disk space during the spilling process has exceeded the allowable limit of {}. \
+ Please try increasing the config: `datafusion.runtime.max_temp_directory_size`.",
Review Comment:
Does `centralizing` mean to be defined as `constant` (e.g: under `runtime_env.rs`)? If so, AFAIS, DF configs do not have constant reference for config names and i think it can be useful to avoid config name duplication and drift risk as you mentioned.
Also, can `version` property per `ConfigEntry` be useful by showing when the config is added (e.g: `52.0.0`)? Spark uses `version` property per config as [reference](https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala#L247).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] refactor: Set expected runtime config in error message when the used disk space during the spilling process has exceeded the allocation limit [datafusion]
Posted by "erenavsarogullari (via GitHub)" <gi...@apache.org>.
erenavsarogullari commented on code in PR #20375:
URL: https://github.com/apache/datafusion/pull/20375#discussion_r2876107961
##########
datafusion/execution/src/disk_manager.rs:
##########
@@ -420,7 +420,8 @@ impl RefCountedTempFile {
let global_disk_usage = self.disk_manager.used_disk_space.load(Ordering::Relaxed);
if global_disk_usage > self.disk_manager.max_temp_directory_size {
return resources_err!(
- "The used disk space during the spilling process has exceeded the allowable limit of {}. Try increasing the `max_temp_directory_size` in the disk manager configuration.",
+ "The used disk space during the spilling process has exceeded the allowable limit of {}. \
+ Please try increasing the config: `datafusion.runtime.max_temp_directory_size`.",
Review Comment:
Does `centralizing` mean to be defined as `constant` (e.g: under `runtime_env.rs`)? If so, AFAIS, DF configs do not have constant reference for config names and i think it can be useful to avoid config name duplication and drift risk as you mentioned.
Also, can `version` property per `ConfigEntry` be useful by showing when the config is added (e.g: `53.0.0`)? Spark uses `version` property per config as [reference](https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala#L247).
##########
datafusion/core/tests/memory_limit/mod.rs:
##########
@@ -602,10 +602,10 @@ async fn test_disk_spill_limit_reached() -> Result<()> {
.await
.unwrap();
- let err = df.collect().await.unwrap_err();
- assert_contains!(
- err.to_string(),
- "The used disk space during the spilling process has exceeded the allowable limit"
+ let error_message = df.collect().await.unwrap_err().to_string();
+ assert!(
+ error_message.contains("The used disk space during the spilling process has exceeded the allowable limit") &&
+ error_message.contains("datafusion.runtime.max_temp_directory_size"),
);
Review Comment:
Yes, expected error message substrings can be iterated and addressed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] refactor: Set expected runtime config in error message when the used disk space during the spilling process has exceeded the allocation limit [datafusion]
Posted by "kosiew (via GitHub)" <gi...@apache.org>.
kosiew commented on code in PR #20375:
URL: https://github.com/apache/datafusion/pull/20375#discussion_r2870612210
##########
datafusion/execution/src/disk_manager.rs:
##########
@@ -420,7 +420,8 @@ impl RefCountedTempFile {
let global_disk_usage = self.disk_manager.used_disk_space.load(Ordering::Relaxed);
if global_disk_usage > self.disk_manager.max_temp_directory_size {
return resources_err!(
- "The used disk space during the spilling process has exceeded the allowable limit of {}. Try increasing the `max_temp_directory_size` in the disk manager configuration.",
+ "The used disk space during the spilling process has exceeded the allowable limit of {}. \
+ Please try increasing the config: `datafusion.runtime.max_temp_directory_size`.",
Review Comment:
`datafusion.runtime.max_temp_directory_size` is now part of user-facing messaging; centralizing it would reduce drift risk if keys are renamed or normalized elsewhere.
##########
datafusion/core/tests/memory_limit/mod.rs:
##########
@@ -602,10 +602,10 @@ async fn test_disk_spill_limit_reached() -> Result<()> {
.await
.unwrap();
- let err = df.collect().await.unwrap_err();
- assert_contains!(
- err.to_string(),
- "The used disk space during the spilling process has exceeded the allowable limit"
+ let error_message = df.collect().await.unwrap_err().to_string();
+ assert!(
+ error_message.contains("The used disk space during the spilling process has exceeded the allowable limit") &&
+ error_message.contains("datafusion.runtime.max_temp_directory_size"),
);
Review Comment:
could simplify this into separate assertions (or iterate expected substrings) so failures clearly indicate which expected fragment is missing, eg
```
for expected in [
"The used disk space during the spilling process has exceeded the allowable limit",
"datafusion.runtime.max_temp_directory_size",
] {
assert!(
error_message.contains(expected),....
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] refactor: Set expected runtime config in error message when the used disk space during the spilling process has exceeded the allocation limit [datafusion]
Posted by "erenavsarogullari (via GitHub)" <gi...@apache.org>.
erenavsarogullari commented on code in PR #20375:
URL: https://github.com/apache/datafusion/pull/20375#discussion_r2876107961
##########
datafusion/execution/src/disk_manager.rs:
##########
@@ -420,7 +420,8 @@ impl RefCountedTempFile {
let global_disk_usage = self.disk_manager.used_disk_space.load(Ordering::Relaxed);
if global_disk_usage > self.disk_manager.max_temp_directory_size {
return resources_err!(
- "The used disk space during the spilling process has exceeded the allowable limit of {}. Try increasing the `max_temp_directory_size` in the disk manager configuration.",
+ "The used disk space during the spilling process has exceeded the allowable limit of {}. \
+ Please try increasing the config: `datafusion.runtime.max_temp_directory_size`.",
Review Comment:
@kosiew Firstly, thanks for the review. Does `centralizing` mean to be defined as `constant` (e.g: under `runtime_env.rs`)? If so, AFAIS, DF configs do not have constant reference for config names and i think it can be useful to avoid config name duplication and drift risk as you mentioned.
Also, can `version` property per `ConfigEntry` be useful by showing when the config is added (e.g: `53.0.0`)? Spark uses `version` property per config as [reference](https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala#L247).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] refactor: Set expected runtime config in error message when the used disk space during the spilling process has exceeded the allocation limit [datafusion]
Posted by "martin-g (via GitHub)" <gi...@apache.org>.
martin-g merged PR #20375:
URL: https://github.com/apache/datafusion/pull/20375
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] refactor: Set expected runtime config in error message when the used disk space during the spilling process has exceeded the allocation limit [datafusion]
Posted by "erenavsarogullari (via GitHub)" <gi...@apache.org>.
erenavsarogullari commented on code in PR #20375:
URL: https://github.com/apache/datafusion/pull/20375#discussion_r2876107961
##########
datafusion/execution/src/disk_manager.rs:
##########
@@ -420,7 +420,8 @@ impl RefCountedTempFile {
let global_disk_usage = self.disk_manager.used_disk_space.load(Ordering::Relaxed);
if global_disk_usage > self.disk_manager.max_temp_directory_size {
return resources_err!(
- "The used disk space during the spilling process has exceeded the allowable limit of {}. Try increasing the `max_temp_directory_size` in the disk manager configuration.",
+ "The used disk space during the spilling process has exceeded the allowable limit of {}. \
+ Please try increasing the config: `datafusion.runtime.max_temp_directory_size`.",
Review Comment:
@kosiew Firstly, thanks for the review. Does `centralizing` mean to be defined as `constant` (e.g: under `runtime_env.rs`)? If so, AFAIS, DF configs do not have constant reference for config names and i think it can be useful to avoid config name duplication and drift risk as you mentioned. It can be extended for all configs (specially used under Rust source files (`*.rs`))
Also, can `version` property per `ConfigEntry` be useful by showing when the config is added (e.g: `53.0.0`)? Spark uses `version` property per config as [reference](https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala#L247).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] refactor: Set expected runtime config in error message when the used disk space during the spilling process has exceeded the allocation limit [datafusion]
Posted by "erenavsarogullari (via GitHub)" <gi...@apache.org>.
erenavsarogullari commented on code in PR #20375:
URL: https://github.com/apache/datafusion/pull/20375#discussion_r2881800923
##########
datafusion/execution/src/disk_manager.rs:
##########
@@ -420,7 +420,8 @@ impl RefCountedTempFile {
let global_disk_usage = self.disk_manager.used_disk_space.load(Ordering::Relaxed);
if global_disk_usage > self.disk_manager.max_temp_directory_size {
return resources_err!(
- "The used disk space during the spilling process has exceeded the allowable limit of {}. Try increasing the `max_temp_directory_size` in the disk manager configuration.",
+ "The used disk space during the spilling process has exceeded the allowable limit of {}. \
+ Please try increasing the config: `datafusion.runtime.max_temp_directory_size`.",
Review Comment:
> What you pointed out is absolutely right – existing ConfigEntry::new_… calls just inline the string and there isn’t a “global constants” module that every caller uses.
Let's leave it as a separate wider issue to be addressed later.
I think `global constants` covering all DF configs (specially Rust source files (`*.rs`)) can be useful for the end-users as well because when they extend/maintain DF fork, they can also use this new infra by avoiding config name duplication and drift risk. In terms of these, does it make sense to create a separate issue to support global constants covering all DF configs? If so, i can create and work on this.
> It would be useful but it's not in the existing ConfigEntry struct and no convention.
I think it would have to be a separate PR to update ConfigEntry with additional version field and helpers.
Until then the only way to know when a key was added is to look at the changelog/commit history.
Yes, currently, `ConfigEntry` `struct` does not have `version` property and we introduce new releases frequently so to track when the config/feature is added, can also be useful. Does it also make sense to create a separate issue for this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] refactor: Set expected runtime config in error message when the used disk space during the spilling process has exceeded the allocation limit [datafusion]
Posted by "kosiew (via GitHub)" <gi...@apache.org>.
kosiew commented on code in PR #20375:
URL: https://github.com/apache/datafusion/pull/20375#discussion_r2876956470
##########
datafusion/execution/src/disk_manager.rs:
##########
@@ -420,7 +420,8 @@ impl RefCountedTempFile {
let global_disk_usage = self.disk_manager.used_disk_space.load(Ordering::Relaxed);
if global_disk_usage > self.disk_manager.max_temp_directory_size {
return resources_err!(
- "The used disk space during the spilling process has exceeded the allowable limit of {}. Try increasing the `max_temp_directory_size` in the disk manager configuration.",
+ "The used disk space during the spilling process has exceeded the allowable limit of {}. \
+ Please try increasing the config: `datafusion.runtime.max_temp_directory_size`.",
Review Comment:
> DF configs do not have constant reference for config name
What you pointed out is absolutely right – existing ConfigEntry::new_… calls just inline the string and there isn’t a “global constants” module that every caller uses.
Let's leave it as a separate wider issue to be addressed later.
> can version property per ConfigEntry be useful by showing when the config is added (e.g: 53.0.0)?
It would be useful but it's not in the existing ConfigEntry struct and no convention.
I think it would have to be a separate PR to update ConfigEntry with additional `version` field and helpers.
Until then the only way to know when a key was added is to look at the changelog/commit history.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org