You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "alamb (via GitHub)" <gi...@apache.org> on 2024/06/17 19:21:53 UTC
[I] Change `StatisticsConverter::row_group_counts` to return `None` for non existent columns in parquet files [datafusion]
alamb opened a new issue, #10965:
URL: https://github.com/apache/datafusion/issues/10965
### Is your feature request related to a problem or challenge?
While working on https://github.com/apache/datafusion/issues/10926 @marvinlanhenke has [an excellent question](https://github.com/apache/datafusion/pull/10946#discussion_r1642676938):
> Why do we return row counts for a non existing column in row_group_row_counts?
(this was because there was some inconsistency between data pages and row counts)
The reason `StatisticsConverter::row_group_counts` returns row counts even for a non existent column is because it is the API needed for PruningStatistics here: https://github.com/apache/datafusion/blob/a923c659cf932f6369f2d5257e5b99128b67091a/datafusion/core/src/datasource/physical_plan/parquet/row_groups.rs#L386-L390
It is possible to do because the `ParquetMetadata` knows how many row groups there are even when there are no row group statistics, but it doesn't make logical sense. Furthermore, for data pages, it is different if the "page index" is not present as then it doesn't even make sense to ask how many rows are in each data page as we don't have any data pages
Thus I think `row_group_row_counts` should also default to returning `None` if the column is not present, as @marvinlanhenke has done for `null_counts_page_statistics` in
So I guess my new proposal would be to return `Option` like:
```rust
impl<'a> StatisticsConverter<'a> {
...
/// return OK(None) if the column does not exist
pub(crate) fn null_counts_page_statistics<'a, I>(iterator: I) -> Result<Option<UInt64Array>> {
...
}
/// return OK(None) if the column does not exist
pub fn data_page_row_counts<I>(
&self,
column_offset_index: &ParquetOffsetIndex,
row_group_metadatas: &[RowGroupMetaData],
row_group_indices: I,
) -> Result<Option<UInt64Array>>
where
I: IntoIterator<Item = &'a usize>,
{
...
}
}
```
The rationale to return an Option rather than an error is that creating and ignoring `DataFusionError` via `ok()` still requires a string allocation, which is wasteful
I realize this is done many places already in the statistics extraction code, but I think for those cases it is meant to make the code resilent to incorrectly encoded parquet files rather than something that is "expected" to happen
### Describe the solution you'd like
_No response_
### Describe alternatives you've considered
_No response_
### Additional context
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Change `StatisticsConverter::row_group_counts` to return `None` for non existent columns in parquet files [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb closed issue #10965: Change `StatisticsConverter::row_group_counts` to return `None` for non existent columns in parquet files
URL: https://github.com/apache/datafusion/issues/10965
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Change `StatisticsConverter::row_group_counts` to return `None` for non existent columns in parquet files [datafusion]
Posted by "marvinlanhenke (via GitHub)" <gi...@apache.org>.
marvinlanhenke commented on issue #10965:
URL: https://github.com/apache/datafusion/issues/10965#issuecomment-2174942964
take
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org