[jira] [Created] (ARROW-16045) Version=7.0.0 introduces bug when filtering by empty set during load

["Damian Barabonkov (Jira)" <ji...@apache.org>]

Damian Barabonkov created ARROW-16045:
-----------------------------------------

             Summary: Version=7.0.0 introduces bug when filtering by empty set during load
                 Key: ARROW-16045
                 URL: https://issues.apache.org/jira/browse/ARROW-16045
             Project: Apache Arrow
          Issue Type: Bug
          Components: Python
    Affects Versions: 7.0.0
         Environment: pandas                    1.3.5
pyarrow                   7.0.0
python                    3.10.4

            Reporter: Damian Barabonkov
             Fix For: 6.0.1


Pyarrow errors when attempting to read from a parquet file with an empty filter on a string column. This issue is present in pyarrow v7.0.0, but not in v6.0.1. Also, interestingly the issue is not present when reading from an integer column (in v7.0.0 as well).

 

The following Python code presents a minimal example which reproduces the issue:
{code:python}
import pandas as pd
import numpy as np
path = './example_df.parquet'
df = pd.DataFrame(
    {
        "A": 1.0,
        "B": pd.Timestamp("20130102"),
        "C": pd.Series(1, index=list(range(4)), dtype="float32"),
        "D": np.array([3] * 4, dtype="int32"),
        "E": pd.Categorical(["test", "train", "test", "train"]),
        "F": "foo",
    }
)
df.to_parquet(path)

# Works!
df_read = pd.read_parquet(
    path,
    filters=[
        [
            ("A", "in", set())
        ]
    ]
)

# Pyarrow v6.0.1 and v7.0.0
#
# Empty DataFrame
# Columns: [A, B, C, D, E, F]
# Index: []
print(df_read)

# Fails!
df_read = pd.read_parquet(
    path,
    filters=[
        [
            ("F", "in", set())
        ]
    ]
)
# Pyarrow v6.0.1
#
# Empty DataFrame
# Columns: [A, B, C, D, E, F]
# Index: []

# Pyarrow v7.0.0
#
# pyarrow.lib.ArrowInvalid: Array type didn't match type of values set: string vs null
print(df_read) {code}



--
This message was sent by Atlassian Jira
(v8.20.1#820001)