[jira] [Commented] (ARROW-15310) [C++][Python][Dataset] Detect (and warn?) when DirectoryPartitioning is parsing an actually hive-style file path?

["Joris Van den Bossche (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/ARROW-15310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17474693#comment-17474693 ] 

Joris Van den Bossche commented on ARROW-15310:
-----------------------------------------------

It seems https://issues.apache.org/jira/browse/ARROW-10485 is discussing a very similar case on the R side

> [C++][Python][Dataset] Detect (and warn?) when DirectoryPartitioning is parsing an actually hive-style file path?
> -----------------------------------------------------------------------------------------------------------------
>
>                 Key: ARROW-15310
>                 URL: https://issues.apache.org/jira/browse/ARROW-15310
>             Project: Apache Arrow
>          Issue Type: Improvement
>          Components: C++, Python
>            Reporter: Joris Van den Bossche
>            Priority: Major
>              Labels: dataset
>
> When you have a hive-style partitioned dataset, with our current {{dataset(..)}} API, it's relatively easy to mess up the inferred partitioning and get confusing results. 
> For example, if you specify the partitioning field names with {{partitioning=[...]}} (which is not needed for hive style since those are inferred), we actually assume you want directory partitioning. This DirectoryPartitioning will then parse the hive-style file paths and take the full "key=value" as the data values for the field.  
> And then, doing a filter can result in a confusing empty result (because "value" doesn't match "key=value").
> I am wondering if we can't relatively cheaply detect this case, and eg give an informative warning about this to the user. 
> Basically what happens is this:
> {code:python}
> >>> part = ds.DirectoryPartitioning(pa.schema([("part", "string")]))
> >>> part.parse("part=a")
> <pyarrow.dataset.Expression (part == "part=a")>
> {code}
> If the parsed value is a string that contains a "=" (and in this case also contains the field name), that is I think a clear sign that (in the large majority of cases) the user is doing something wrong.
> I am not fully sure where and at what stage the check could be done though. Doing it for every path in the dataset might be too costly.
> ----
> Illustrative code example:
> {code:python}
> import pyarrow as pa
> import pyarrow.parquet as pq
> import pyarrow.dataset as ds
> import pathlib
> ## constructing a small dataset with 1 hive-style partitioning level
> basedir = pathlib.Path(".") / "dataset_wrong_partitioning"
> basedir.mkdir(exist_ok=True)
> (basedir / "part=a").mkdir(exist_ok=True)
> (basedir / "part=b").mkdir(exist_ok=True)
> table1 = pa.table({'a': [1, 2, 3], 'b': [1, 2, 3]})
> pq.write_table(table1, basedir / "part=a" / "data.parquet")
> table2 = pa.table({'a': [4, 5, 6], 'b': [1, 2, 3]})
> pq.write_table(table2, basedir / "part=b" / "data.parquet")
> {code}
> Reading as is (not specifying a partitioning, so default to no partitioning) will at least give an error about a missing field:
> {code: python}
> >>> dataset = ds.dataset(basedir)
> >>> dataset.to_table(filter=ds.field("part") == "a")
> ...
> ArrowInvalid: No match for FieldRef.Name(part) in a: int64
> {code}
> But specifying the partitioning field name (which currently gets (silently) interpreted as directory partitioning) gives a confusing empty result:
> {code:python}
> >>> dataset = ds.dataset(basedir, partitioning=["part"])
> >>> dataset.to_table(filter=ds.field("part") == "a")
> pyarrow.Table
> a: int64
> b: int64
> part: string
> ----
> a: []
> b: []
> part: []
> {code}
> This filter doesn't work because the values in the "part" column are not "a" but "part=a":
> {code:python}
> >>> dataset.to_table().to_pandas()
>    a  b    part
> 0  1  1  part=a
> 1  2  2  part=a
> 2  3  3  part=a
> 3  4  1  part=b
> 4  5  2  part=b
> 5  6  3  part=b
> {code}



--
This message was sent by Atlassian Jira
(v8.20.1#820001)