You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "korowa (via GitHub)" <gi...@apache.org> on 2024/12/17 20:29:25 UTC
[I] parquet RowGroup pruning for `Dictionary(Decimal)` type [datafusion]
korowa opened a new issue, #13821:
URL: https://github.com/apache/datafusion/issues/13821
### Describe the bug
Parquet RowGroup pruning by statistics works incorrectly for `Dictionary(Decimal)` type.
### To Reproduce
```rs
use arrow;
use arrow::array::{ArrayRef, Decimal128Array, DictionaryArray, Int32Array, RecordBatch};
use datafusion::error::Result;
use datafusion::prelude::*;
use parquet::arrow::ArrowWriter;
use parquet::file::properties::{EnabledStatistics, WriterProperties};
use std::fs::File;
use std::sync::Arc;
#[tokio::main]
async fn main() -> Result<()> {
// Prepare record batch
let array_values = Decimal128Array::from_iter_values(vec![10, 20, 30])
.with_precision_and_scale(4, 1)?;
let array_keys = Int32Array::from_iter_values(vec![0, 1, 2]);
let array = Arc::new(DictionaryArray::new(array_keys, Arc::new(array_values)));
let batch = RecordBatch::try_from_iter(vec![("col", array as ArrayRef)])?;
// Write batch to parquet
let file_path = "dictionary_decimal.parquet";
let file = File::create(file_path)?;
let properties = WriterProperties::builder()
.set_statistics_enabled(EnabledStatistics::Chunk)
.set_bloom_filter_enabled(true)
.build();
let mut writer = ArrowWriter::try_new(file, batch.schema(), Some(properties))?;
writer.write(&batch)?;
writer.flush()?;
writer.close()?;
// Prepare context
let config = SessionConfig::default()
.with_parquet_bloom_filter_pruning(true)
.with_collect_statistics(true);
let ctx = SessionContext::new_with_config(config);
ctx.register_parquet("t", file_path, ParquetReadOptions::default())
.await?;
// In case pruning predicate not created (due to cast), there is a record in resultset
ctx.sql("select * from t where col = 1")
.await?
.show()
.await?;
println!();
// In case of triggered RowGroup pruning -- the only RowGroup eliminated while pruning by statistics
ctx.sql("select * from t where col = cast(1 as decimal(4, 1))")
.await?
.show()
.await?;
Ok(())
}
```
### Expected behavior
Results from both queries from the script above should match
### Additional context
The problem also happens with bloom filters (if enable them in pattern matching expressions in `prune_by_bloom_filters`), so there is a chance that `ArrowWriter` produces incorrect metadata (statistics / bloom filters).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] parquet RowGroup pruning for `Dictionary(Decimal)` type incorrect [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb closed issue #13821: parquet RowGroup pruning for `Dictionary(Decimal)` type incorrect
URL: https://github.com/apache/datafusion/issues/13821
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] parquet RowGroup pruning for `Dictionary(Decimal)` type incorrect [datafusion]
Posted by "lichuang (via GitHub)" <gi...@apache.org>.
lichuang commented on issue #13821:
URL: https://github.com/apache/datafusion/issues/13821#issuecomment-2558485724
take
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] parquet RowGroup pruning for `Dictionary(Decimal)` type incorrect [datafusion]
Posted by "korowa (via GitHub)" <gi...@apache.org>.
korowa commented on issue #13821:
URL: https://github.com/apache/datafusion/issues/13821#issuecomment-2594590175
@kosiew the root cause of the issue is how arrow writer handles data for `Dictionary(Decimal)`, and I suppose it'll mostly be fixed by https://github.com/apache/arrow-rs/pull/6987
Regarding RG pruning not being enabled in case literal and column have different datatypes -- I think adding some safe casting to enable it would be a useful feature. For this, perhaps, [PruningPredicate](https://github.com/apache/datafusion/blob/main/datafusion/physical-optimizer/src/pruning.rs#L529) and its interactions with ParquetExec will be a good place to start dig in.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] parquet RowGroup pruning for `Dictionary(Decimal)` type incorrect [datafusion]
Posted by "lichuang (via GitHub)" <gi...@apache.org>.
lichuang commented on issue #13821:
URL: https://github.com/apache/datafusion/issues/13821#issuecomment-2577698922
after i dig into this bug, here is the report:
`cast(1 as decimal(4, 1))` will get a value `Decimal128(Some(10),4,1)`, and `eq` operator transfer to two operation:
* compare with the min value, which is `PrimitiveArray<Decimal128(4, 1)>[1]} lteq Decimal128(Some(10),4,1)`, and return true;
* compare with the max value, which is `Decimal128(Some(10),4,1) lteq PrimitiveArray<Decimal128(4, 1)>[4]}`, and return false;
so `select * from t where col = cast(1 as decimal(4, 1))` cannot filter any columns.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] parquet RowGroup pruning for `Dictionary(Decimal)` type incorrect [datafusion]
Posted by "kosiew (via GitHub)" <gi...@apache.org>.
kosiew commented on issue #13821:
URL: https://github.com/apache/datafusion/issues/13821#issuecomment-2586088480
Would it be a good approach to fix this by coercing both sides of BinaryExpr like
`col = CAST(1 AS DECIMAL(4, 1)))`
to a common type?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] parquet RowGroup pruning for `Dictionary(Decimal)` type incorrect [datafusion]
Posted by "kosiew (via GitHub)" <gi...@apache.org>.
kosiew commented on issue #13821:
URL: https://github.com/apache/datafusion/issues/13821#issuecomment-2556552343
Research findings:
with parquest_pruning turned off
```
let config = SessionConfig::default()
.with_parquet_bloom_filter_pruning(true)
.with_parquet_pruning(false)
.with_collect_statistics(true);
```
both queries return the same result:
+-----+
| col |
+-----+
| 1.0 |
+-----+
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org