You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "alamb (via GitHub)" <gi...@apache.org> on 2024/07/09 19:56:03 UTC
[I] Inconsistent value for `data_page_max_rows` setting in DataFusion `ParquetOptions` and in `ArrowWriterOptions` [datafusion]
alamb opened a new issue, #11367:
URL: https://github.com/apache/datafusion/issues/11367
### Describe the bug
@wiedld pointed out that the default value of `data_page_max_rows` is different in DataFusion than in arrow-rs
https://docs.rs/datafusion/latest/datafusion/common/config/struct.ParquetOptions.html sets it to `usize::max`
As of https://github.com/apache/arrow-rs/pull/5957 was released in 51.1.0 , arrow-rs sets it to `20,000`
### To Reproduce
N/A
### Expected behavior
The DataFusion defaults should be the same as the arrow-rs https://docs.rs/parquet/latest/parquet/arrow/arrow_writer/struct.ArrowWriterOptions.html, unless there is a good reason to deviate
In this case, I think we should go with the arrow-rs value
Ideally a fix for this ticket would both
1. Change the defaults of `ParquetOptions` to match the default in `ArrowWriterOptions`
2. Write a test that verified the default values were the same
### Additional context
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Inconsistent value for `data_page_max_rows` setting in DataFusion `ParquetOptions` and in `ArrowWriterOptions` [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #11367:
URL: https://github.com/apache/datafusion/issues/11367#issuecomment-2226319371
https://github.com/apache/datafusion/pull/11444 is similar
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Inconsistent value for `data_page_max_rows` setting in DataFusion `ParquetOptions` and in `ArrowWriterOptions` [datafusion]
Posted by "wiedld (via GitHub)" <gi...@apache.org>.
wiedld commented on issue #11367:
URL: https://github.com/apache/datafusion/issues/11367#issuecomment-2226326990
Ah, I should be clear (not trying to step on toes 😅 ). My goal was to delineate the differences btwn the APIs more, but to leave the actual fixing of the defaults (& tests to enforce that it remains in sync) as this issue assigned to @Rachelint .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Inconsistent value for `data_page_max_rows` setting in DataFusion `ParquetOptions` and in `ArrowWriterOptions` [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #11367:
URL: https://github.com/apache/datafusion/issues/11367#issuecomment-2234282902
@wiedld 's pr https://github.com/apache/datafusion/pull/11524 has a very nice table of the current state of affairs:
Here are the places where the current defaults differ:
| setting_name | applied | default, datafusion (ParquetOptions) | default, parquet (ArrowWriterOptions) |
| ---------------------------- | -------------- | ------------------------------------ | ------------------------------------- |
| data_page_row_count_limit | file | usize::MAX | 20_000 |
| column_index_truncate_length | file | None | Some(64) |
| compression | column default | zstd | uncompressed |
| dictionary_enabled | column default | None or true †| true |
| statistics_enabled | column default | None or page †| page |
| max_statistics_size | column default | None or 4096 †| 4096 |
†For these settings, datafusion has no default (None). However, once datafusion's ParquetOptions are used by the extern parquet (a.k.a. converted to parquet's ArrowWriterOptions) then it uses the extern parquet's defaults. Refer to the newly added tests.
.
Additionally, there are differences in the bloom filter configurations based upon partial definition (a.k.a. leaving some as default, and some as defined):
| bloom_filter_on | fpp | ndv | if build from datafusion's ParquetOptions | if build from (arrow-rs) parquet's WriterPropertiesBuilder |
| --------------- | ---- | ---- | -------------------------------------------- | ------------------------------------ |
| false | none | none | None | None |
| **TRUE** | none | none | Some(BloomFilterProperties::default) | None |
| true | **SOME** | none | Some(BloomFilterProperties: fpp,default_ndv) | None |
| true | none | **SOME** | Some(BloomFilterProperties: ndv,default_fpp) | None |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Inconsistent value for `data_page_max_rows` setting in DataFusion `ParquetOptions` and in `ArrowWriterOptions` [datafusion]
Posted by "Rachelint (via GitHub)" <gi...@apache.org>.
Rachelint commented on issue #11367:
URL: https://github.com/apache/datafusion/issues/11367#issuecomment-2219087163
take
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Inconsistent value for `data_page_max_rows` setting in DataFusion `ParquetOptions` and in `ArrowWriterOptions` [datafusion]
Posted by "Rachelint (via GitHub)" <gi...@apache.org>.
Rachelint commented on issue #11367:
URL: https://github.com/apache/datafusion/issues/11367#issuecomment-2227342171
> **Edited**
>
> My goal was to delineate the differences btwn the APIs more, but to leave the actual fixing of the defaults (& tests to enforce that it remains in sync) as this issue assigned to @Rachelint .
>
> I was hoping #11444 could actually help make this issue/PR (setting defaults) be easier to test. I could also add in the changes to the defaults as well, if @Rachelint is ok with that?
Ok, feel free to make it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Inconsistent value for `data_page_max_rows` setting in DataFusion `ParquetOptions` and in `ArrowWriterOptions` [datafusion]
Posted by "wiedld (via GitHub)" <gi...@apache.org>.
wiedld commented on issue #11367:
URL: https://github.com/apache/datafusion/issues/11367#issuecomment-2229015194
take
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Inconsistent value for `data_page_max_rows` setting in DataFusion `ParquetOptions` and in `ArrowWriterOptions` [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb closed issue #11367: Inconsistent value for `data_page_max_rows` setting in DataFusion `ParquetOptions` and in `ArrowWriterOptions`
URL: https://github.com/apache/datafusion/issues/11367
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org