You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "Satyr09 (via GitHub)" <gi...@apache.org> on 2026/10/03 08:54:28 UTC
[I] Parallel Parquet writes delay reporting column-worker failures [datafusion]
Satyr09 opened a new issue, #25992:
URL: https://github.com/apache/datafusion/issues/25992
### Describe the bug
When a column worker fails during a parallel Parquet write, the writer can continue consuming input before returning the error.
The worker closes its input channel, but the dispatcher treats the failed send as normal shutdown and returns success from that operation. The underlying error is reported later, when the row group is finalized or input ends.
This delays failure reporting and wastes work. With long-lived input and a large row-group limit, the write can appear stuck even though a column worker has already failed. This also happens when max_row_group_bytes is unset.
### To Reproduce
1. Enable parallel Parquet writing and configure a small memory pool, such as 128 KiB.
2. Use a repeating input stream containing batches of wide strings.
3. Set max_row_group_size to usize::MAX so a row-group boundary is not reached during the test.
4. Write through the public DataFrame write_parquet API.
A column worker exceeds its memory limit, but the write continues consuming input instead of promptly returning that error.
The [proposed regression test](https://github.com/Satyr09/datafusion/blob/450c5d779f3961f4823f616312d5f5e312c6110a/datafusion/core/tests/parquet/write_errors.rs) reproduces this: it times out on the original writer and passes with the fix.
### Expected behavior
When the dispatcher encounters the failed worker’s closed channel, it should return that worker’s original error without waiting for a row-group boundary or the end of input.
### Additional context
Discovered while working on [#25041](https://github.com/apache/datafusion/pull/25041), but independent of the byte-target support requested in [#22982](https://github.com/apache/datafusion/issues/22982).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Parallel Parquet writes delay reporting column-worker failures [datafusion]
Posted by "Satyr09 (via GitHub)" <gi...@apache.org>.
Satyr09 commented on issue #25992:
URL: https://github.com/apache/datafusion/issues/25992#issuecomment-5967385091
take
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org