You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "wgtmac (via GitHub)" <gi...@apache.org> on 2025/01/14 02:23:43 UTC

[PR] GH-45227: [C++][Parquet] Enable Size Stats and Page Index by default [arrow]

wgtmac opened a new pull request, #45249:
URL: https://github.com/apache/arrow/pull/45249

   ### Rationale for this change
   
   Benchmark data shows that enabling page index and size stats by default does not have significant penalty.
   
   ### What changes are included in this PR?
   
   Enable the parquet writer to generate page index and size stats by default.
   
   ### Are these changes tested?
   
   Pass CIs.
   
   ### Are there any user-facing changes?
   
   No.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-45227: [C++][Parquet] Enable Size Stats and Page Index by default [arrow]

Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on PR #45249:
URL: https://github.com/apache/arrow/pull/45249#issuecomment-2589111229

   @pitrou @mapleFU Would you please take a look?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-45227: [C++][Parquet] Enable Size Stats and Page Index by default [arrow]

Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac merged PR #45249:
URL: https://github.com/apache/arrow/pull/45249


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-45227: [C++][Parquet] Enable Size Stats and Page Index by default [arrow]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #45249:
URL: https://github.com/apache/arrow/pull/45249#issuecomment-2588716305

   :warning: GitHub issue #45227 **has been automatically assigned in GitHub** to PR creator.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-45227: [C++][Parquet] Enable Size Stats and Page Index by default [arrow]

Posted by "conbench-apache-arrow[bot] (via GitHub)" <gi...@apache.org>.
conbench-apache-arrow[bot] commented on PR #45249:
URL: https://github.com/apache/arrow/pull/45249#issuecomment-2605230519

   After merging your PR, Conbench analyzed the 4 benchmarking runs that have been run so far on merge-commit 1fcc89240db4fe0ad798498e7410668423846118.
   
   There were no benchmark performance regressions. 🎉
   
   The [full Conbench report](https://github.com/apache/arrow/runs/35943977751) has more details. It also includes information about 14 possible false positives for unstable benchmarks that are known to sometimes produce them.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-45227: [C++][Parquet] Enable Size Stats and Page Index by default [arrow]

Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on PR #45249:
URL: https://github.com/apache/arrow/pull/45249#issuecomment-2589421292

   For the record, the benchmark data on my dev box:
   ```
   Run on (32 X 2700 MHz CPU s)
   CPU Caches:
     L1 Data 48 KiB (x16)
     L1 Instruction 32 KiB (x16)
     L2 Unified 1280 KiB (x16)
     L3 Unified 49152 KiB (x1)
   Load Average: 1.01, 1.06, 0.60
   ------------------------------------------------------------------------------------------------------------------------------------------------------
   Benchmark                                                                                            Time             CPU   Iterations UserCounters...
   ------------------------------------------------------------------------------------------------------------------------------------------------------
   BM_WritePrimitiveColumn<SizeStatisticsLevel::None, ::arrow::Int64Type, false>                  9386745 ns      9386625 ns           74 bytes_per_second=865.593Mi/s items_per_second=111.71M/s output_size=546.08k page_index_size=0
   BM_WritePrimitiveColumn<SizeStatisticsLevel::None, ::arrow::Int64Type, true>                   9387974 ns      9387716 ns           75 bytes_per_second=865.493Mi/s items_per_second=111.697M/s output_size=546.091k page_index_size=33
   BM_WritePrimitiveColumn<SizeStatisticsLevel::ColumnChunk, ::arrow::Int64Type, true>            9392147 ns      9391889 ns           74 bytes_per_second=865.108Mi/s items_per_second=111.647M/s output_size=546.107k page_index_size=33
   BM_WritePrimitiveColumn<SizeStatisticsLevel::PageAndColumnChunk, ::arrow::Int64Type, true>     9363539 ns      9363364 ns           75 bytes_per_second=867.744Mi/s items_per_second=111.987M/s output_size=546.121k page_index_size=47
   BM_WritePrimitiveColumn<SizeStatisticsLevel::None, ::arrow::StringType, false>                12773243 ns     12772995 ns           55 bytes_per_second=362.473Mi/s items_per_second=82.0932M/s output_size=864.052k page_index_size=0
   BM_WritePrimitiveColumn<SizeStatisticsLevel::None, ::arrow::StringType, true>                 12777189 ns     12776940 ns           55 bytes_per_second=362.361Mi/s items_per_second=82.0678M/s output_size=864.083k page_index_size=30
   BM_WritePrimitiveColumn<SizeStatisticsLevel::ColumnChunk, ::arrow::StringType, true>          12826700 ns     12826328 ns           54 bytes_per_second=360.965Mi/s items_per_second=81.7518M/s output_size=864.103k page_index_size=30
   BM_WritePrimitiveColumn<SizeStatisticsLevel::PageAndColumnChunk, ::arrow::StringType, true>   12837516 ns     12837345 ns           54 bytes_per_second=360.656Mi/s items_per_second=81.6817M/s output_size=864.122k page_index_size=44
   BM_WriteListColumn<SizeStatisticsLevel::None, ::arrow::Int64Type, false>                      16068608 ns     16068385 ns           43 bytes_per_second=531.323Mi/s items_per_second=65.2571M/s output_size=625.904k page_index_size=0
   BM_WriteListColumn<SizeStatisticsLevel::None, ::arrow::Int64Type, true>                       16026976 ns     16026766 ns           44 bytes_per_second=532.703Mi/s items_per_second=65.4266M/s output_size=625.915k page_index_size=34
   BM_WriteListColumn<SizeStatisticsLevel::ColumnChunk, ::arrow::Int64Type, true>                16723670 ns     16723159 ns           42 bytes_per_second=510.52Mi/s items_per_second=62.702M/s output_size=625.937k page_index_size=34
   BM_WriteListColumn<SizeStatisticsLevel::PageAndColumnChunk, ::arrow::Int64Type, true>         16784041 ns     16783691 ns           41 bytes_per_second=508.678Mi/s items_per_second=62.4759M/s output_size=625.957k page_index_size=54
   BM_WriteListColumn<SizeStatisticsLevel::None, ::arrow::StringType, false>                     19524874 ns     19524500 ns           36 bytes_per_second=258.258Mi/s items_per_second=53.7057M/s output_size=944.092k page_index_size=0
   BM_WriteListColumn<SizeStatisticsLevel::None, ::arrow::StringType, true>                      19822498 ns     19822084 ns           35 bytes_per_second=254.381Mi/s items_per_second=52.8994M/s output_size=944.123k page_index_size=31
   BM_WriteListColumn<SizeStatisticsLevel::ColumnChunk, ::arrow::StringType, true>               20485041 ns     20484509 ns           34 bytes_per_second=246.155Mi/s items_per_second=51.1887M/s output_size=944.149k page_index_size=31
   BM_WriteListColumn<SizeStatisticsLevel::PageAndColumnChunk, ::arrow::StringType, true>        20354835 ns     20354291 ns           34 bytes_per_second=247.73Mi/s items_per_second=51.5162M/s output_size=944.174k page_index_size=51
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-45227: [C++][Parquet] Enable Size Stats and Page Index by default [arrow]

Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on PR #45249:
URL: https://github.com/apache/arrow/pull/45249#issuecomment-2589195446

   > Looks ok to me but I don't know what other think about this, do we report need this in maillist?
   
   We can probably mention it on the dev@parquet ML?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org