You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "jeffreyssmith2nd (via GitHub)" <gi...@apache.org> on 2025/01/07 15:49:27 UTC
[I] Un-cancellable Query when hitting many large files. [datafusion]
jeffreyssmith2nd opened a new issue, #14036:
URL: https://github.com/apache/datafusion/issues/14036
### Describe the bug
**TLDR; Reading many large Parquet files can prevent a query from being cancelled.**
We have a customer that is running a query similar to the following (edited for privacy):
```
SELECT DISTINCT "A","B","C","D","E" FROM table where "time" > now() - INTERVAL '5 days';
```
This produces a fairly straightforward plan [explain.txt](https://github.com/user-attachments/files/18323208/explain.txt).
Simplified Plan:
```
AggregateExec mode=FinalPartitioned
CoalesceBatchesExec target_batch_size=8192
RepartitionExec input_partitions=4
AggregateExec mode=Partial
ParquetExec file_groups={4 groups}
```
This will read ~85 parquet files at ~100MB each. What we've seen is that even when a query is cancelled, the resources with that query (CPU/RAM) are still being utilized for almost the same amount of time as a typical execution of the query.
### To Reproduce
_No response_
### Expected behavior
Cancelling a query should (within some reasonable amount of time) truly cancel the query, freeing up system resources for other query executions.
### Additional context
This appears to be a problem with the interaction between the [`GroupedHashAggregateStream`](https://github.com/apache/arrow-datafusion/blob/9a808eb7f63385e3e3b02051ed30eee91daeb613/datafusion/physical-plan/src/aggregates/row_hash.rs#L620) and [`FileStream`](https://github.com/apache/arrow-datafusion/blob/9a808eb7f63385e3e3b02051ed30eee91daeb613/datafusion/core/src/datasource/physical_plan/file_stream.rs#L246) approaches to yielding.
The `GroupedHashAggregateStream` will infinitely loop until its child stream (in this case a `FileStream`) is exhausted, errors, or returns `Pending`.
The `FileStream` loops while attempting to read `RecordBatch`es from the underlying file, while also doing some book-keeping to stage the next `File` for reading. This Stream will return `Ready` when a `RecordBatch` is processed, or an `Error` encountered. However, it **does not** return `Pending` at any point. When a File is finished reading, the next File up is swapped in and reading continues from the new File.
The combination of these two behaviours means that if there are many large files being read by the `FileStream`, the `GroupedHashAggregateStream` doesn't effectively yield back to Tokio.
My [PR](https://github.com/apache/datafusion/pull/14028) to fix this is to have the `FileStream` return `Pending` when swapping over to a new file. This seems like a natural yield point, and resolves the issue with cancellation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Un-cancellable Query when hitting many large files. [datafusion]
Posted by "berkaysynnada (via GitHub)" <gi...@apache.org>.
berkaysynnada commented on issue #14036:
URL: https://github.com/apache/datafusion/issues/14036#issuecomment-2577045685
How do you cancel the query? You mean terminating the next()'s on the stream, or dropping the stream. If it is the former, the issue might be related with the RepartitionExec, not GroupedHashAggregateStream (AFAIK _it **won't** loop until its child stream (in this case a FileStream) is exhausted, errors, or returns Pending._, it returns once it can emit a result)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Un-cancellable Query when hitting many large files. [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #14036:
URL: https://github.com/apache/datafusion/issues/14036#issuecomment-2921953997
There is a new proposed fix in
- https://github.com/apache/datafusion/pull/16196
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Un-cancellable Query when hitting many large files. [datafusion]
Posted by "ozankabak (via GitHub)" <gi...@apache.org>.
ozankabak closed issue #14036: Un-cancellable Query when hitting many large files.
URL: https://github.com/apache/datafusion/issues/14036
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Un-cancellable Query when hitting many large files. [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #14036:
URL: https://github.com/apache/datafusion/issues/14036#issuecomment-2578322453
I tried a bit today to re-create this but was not able to
What I tried was to create a highly compressed parquet file (48MB that has 1B rows with all repeated strings) and then made 500 symlinks: [table.tar.gz](https://github.com/user-attachments/files/18351266/table.tar.gz)
This lets me run queries like the following in `datafusion-cli` and tested by hitting `CTRL-C`
```sql
SELECT DISTINCT "A","B","C","D","E" FROM 'table';
```
and
```sql
SELECT DISTINCT "A","B","C","D","E" FROM 'table' WHERE "A" <> 'FooBar';
```
I tried several variants but cancelling always seemed to work just fine 🤔
I also tried a few settings, but also cancelling always worked.
```sql
set datafusion.execution.parquet.pushdown_filters=true;
set datafusion.execution.target_partitions=4;
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Un-cancellable Query when hitting many large files. [datafusion]
Posted by "jeffreyssmith2nd (via GitHub)" <gi...@apache.org>.
jeffreyssmith2nd commented on issue #14036:
URL: https://github.com/apache/datafusion/issues/14036#issuecomment-2580216590
@berkaysynnada I'll take another stab at getting a reproducer for this that doesn't require customer data
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Un-cancellable Query when hitting many large files. [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #14036:
URL: https://github.com/apache/datafusion/issues/14036#issuecomment-2685750238
@carols10cents has made a benchmark here:
- https://github.com/apache/datafusion/pull/14818
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Un-cancellable Query when hitting many large files. [datafusion]
Posted by "berkaysynnada (via GitHub)" <gi...@apache.org>.
berkaysynnada commented on issue #14036:
URL: https://github.com/apache/datafusion/issues/14036#issuecomment-2579313671
We have also checkpoint tests which will drop the stream after some amount of time, and after the failure, FileStream offsets do not increment more.
I think the same problem should be observed without the RepartitionExec (in single partition)
Can you also provide a brief reproducer for the cancellation logic? Cancel requests are setting some tokens and those are checked during while next()?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Un-cancellable Query when hitting many large files. [datafusion]
Posted by "jeffreyssmith2nd (via GitHub)" <gi...@apache.org>.
jeffreyssmith2nd commented on issue #14036:
URL: https://github.com/apache/datafusion/issues/14036#issuecomment-2577862225
> How do you cancel the query? You mean terminating the next()'s on the stream, or dropping the stream.
The queries are running in the context of a gRPC request, which when cancelled drops the stream.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org