[jira] [Resolved] (ARROW-9146) [C++][Dataset] Scanning a Fragment with a filter + mismatching schema shouldn't abort

["Francois Saint-Jacques (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/ARROW-9146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Francois Saint-Jacques resolved ARROW-9146.
-------------------------------------------
    Resolution: Fixed

Issue resolved by pull request 7526
[https://github.com/apache/arrow/pull/7526]

> [C++][Dataset] Scanning a Fragment with a filter + mismatching schema shouldn't abort
> -------------------------------------------------------------------------------------
>
>                 Key: ARROW-9146
>                 URL: https://issues.apache.org/jira/browse/ARROW-9146
>             Project: Apache Arrow
>          Issue Type: Bug
>          Components: C++
>            Reporter: Joris Van den Bossche
>            Assignee: Ben Kietzman
>            Priority: Major
>              Labels: dataset, pull-request-available
>             Fix For: 1.0.0
>
>          Time Spent: 2h 50m
>  Remaining Estimate: 0h
>
> If you have filter on a column where the physical and dataset schema differs, scanning aborts (as right now the dataset schema, if specified, gets used for implicit casts, but then the expression might have a different type as the actual physical column):
> Small parquet file with one int32 column:
> {code:python}
> df = pd.DataFrame({"col": np.array([1, 2, 3, 4], dtype='int32')})
> df.to_parquet("test_filter_schema.parquet", engine="pyarrow")
> import pyarrow.dataset as ds
> dataset = ds.dataset("test_filter_schema.parquet", format="parquet")
> fragment = list(dataset.get_fragments())[0]
> {code}
> and then reading in a fragment with a filter on that column, without and with specifying a dataset/read schema:
> {code}
> In [48]: fragment.to_table(filter=ds.field("col") > 2).to_pandas() 
> Out[48]: 
>    col
> 0    3
> 1    4
> In [49]: fragment.to_table(filter=ds.field("col") > 2, schema=pa.schema([("col", pa.int64())])).to_pandas()                                                                                                        
> ../src/arrow/result.cc:28: ValueOrDie called on an error: Type error: Cannot compare scalars of differing type: int64 vs int32
> /home/joris/miniconda3/envs/arrow-dev/lib/libarrow.so.100(+0xee2f86)[0x7f6b56490f86]
> ...
> Aborted (core dumped)
> {code}
> Now this int32->int64 type change is something we don't support yet (in the schema evolution/normalization, when scanning a dataset), but it also shouldn't abort but raise a normal error about type mismatch.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)