[jira] [Created] (ARROW-9321) [C++][Dataset] Allow to "collect" statistics for ParquetFragment row groups if not constructed from _metadata

["Joris Van den Bossche (Jira)" <ji...@apache.org>]

Joris Van den Bossche created ARROW-9321:
--------------------------------------------

             Summary: [C++][Dataset] Allow to "collect" statistics for ParquetFragment row groups if not constructed from _metadata
                 Key: ARROW-9321
                 URL: https://issues.apache.org/jira/browse/ARROW-9321
             Project: Apache Arrow
          Issue Type: Improvement
          Components: C++
            Reporter: Joris Van den Bossche


Right now, the statistics of the {{RowGroupInfo}} of ParquetFileFragments are only available when the dataset was constructed from a {{_metadata}} file:

{code:python}
import pandas as pd
df = pd.DataFrame({"part": ['A', 'A', 'B', 'B'], "col": range(4)})                                                                                                                                        
# use dask to write partitioned dataset *with* _metadata file
import dask.dataframe as dd                                                                                                                                                                               
ddf = dd.from_pandas(df, npartitions=2) 
ddf.to_parquet("test_dataset", partition_on=["part"], engine="pyarrow")                                                                                                                     

import pyarrow.dataset as ds
dataset_no_metadata = ds.dataset("test_dataset/", format="parquet", partitioning="hive")
dataset_from_metadata = ds.parquet_dataset("test_dataset/_metadata", partitioning="hive")                                                                                                                 
{code}

{code}

In [28]: list(dataset_no_metadata.get_fragments())[0].row_groups                                                                                                                                                   

In [30]: list(dataset_from_metadata.get_fragments())[0].row_groups                                                                                                                                                 
Out[30]: [<pyarrow._dataset.RowGroupInfo at 0x7fd7882c0030>]

In [32]: list(dataset_from_metadata.get_fragments())[0].row_groups[0].statistics                                                                                                                                   
Out[32]: {'col': {'min': 2, 'max': 3}, 'index': {'min': 2, 'max': 3}}
{code}

For some applications (eg dask), one could want access to those statistics, even if the original dataset / fragments were not created from a {{_metadata}} file. This should not happen automatically since it's costly, but a method to trigger collecting all metadata would be useful.

cc [~rjzamora] 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)