You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "Joris Van den Bossche (Jira)" <ji...@apache.org> on 2020/03/10 18:01:00 UTC

[jira] [Created] (ARROW-8061) [C++][Dataset] Ability to specify granularity of ParquetFileFragment (support row groups)

Joris Van den Bossche created ARROW-8061:
--------------------------------------------

             Summary: [C++][Dataset] Ability to specify granularity of ParquetFileFragment (support row groups)
                 Key: ARROW-8061
                 URL: https://issues.apache.org/jira/browse/ARROW-8061
             Project: Apache Arrow
          Issue Type: Improvement
          Components: C++ - Dataset
            Reporter: Joris Van den Bossche


Specifically for parquet (not sure if it will be relevant for other file formats as well, for IPC/feather potentially ther record batch), it would be useful to target row groups instead of files as fragments.

Quoting the original design documents: _"In datasets consisting of many fragments, the dataset API must expose the granularity of fragments in a public way to enable parallel processing, if desired. "._   
And a comment from Wes on that: _"a single Parquet file can "export" one or more fragments based on settings. The default might be to split fragments based on row group"_

Currently, the level on which fragments are defined (at least in the typical partitioned parquet dataset) is "1 file == 1 fragment".

Would it be possible or desirable to make this more fine grained, where you could also opt to have a fragment per row group?   
We could have a ParquetFragment that has this option, and a ParquetFileFormat specific option to say what the granularity of a fragment is (file vs row group)?

cc [~fsaintjacques] [~bkietz]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)