[jira] [Closed] (ARROW-9730) [C++][Dataset] Parsing statistics of Parquet FileMetadata is expensive

["Joris Van den Bossche (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/ARROW-9730?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Joris Van den Bossche closed ARROW-9730.
----------------------------------------
    Resolution: Not A Problem

> [C++][Dataset] Parsing statistics of Parquet FileMetadata is expensive
> ----------------------------------------------------------------------
>
>                 Key: ARROW-9730
>                 URL: https://issues.apache.org/jira/browse/ARROW-9730
>             Project: Apache Arrow
>          Issue Type: Improvement
>          Components: C++
>            Reporter: Joris Van den Bossche
>            Priority: Major
>              Labels: dataset
>
> From a discussion in dask (https://github.com/dask/dask/pull/6346/#issuecomment-656548675), we noticed that parsing all the statistics of a larger dataset is quite time consuming.
> Now, it might be that this is already optimized and one simply needs to live with the cost of parsing statistics if you want the benefit of those statistics for row group filtering. But, it might be worth profiling this to ensure there is not actually some performance bug / low hanging fruit lying around.
> *Example timing:*
> I was testing locally with a part of the NYC taxi data (for 2.5 years (2016-07 - end 2018), one file per month, total disk size of 4.3 GB):
> {code:python}
> >>> import pyarrow.dataset as ds 
> >>> dataset = ds.dataset("notebooks-arrow/nyc-taxi-data/original-partitioned/", format="parquet", partitioning=["year", "month"]) 
> >>> fragments = list(dataset.get_fragments()) 
> >>> len(fragments) 
> 30
> >>> %time [frag.ensure_complete_metadata() for frag in fragments] 
> {code}
> Timing results of the last line of master vs commenting out parsing statistics when collecting the metadata:
> {code:python}
> In [5]: %time [frag.ensure_complete_metadata() for frag in fragments] 
> # master
> CPU times: user 4.22 s, sys: 75.4 ms, total: 4.3 s
> Wall time: 4.41 s
> # master but with parsing statistics commented out (still reading the FileMetadata and row group information for num_rows, total_byte_size)
> CPU times: user 377 ms, sys: 4.47 ms, total: 381 ms
> Wall time: 404 ms
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)