[jira] [Created] (ARROW-9476) [C++][Dataset] HivePartitioning discovery with dictionary types fails for multiple fields

["Joris Van den Bossche (Jira)" <ji...@apache.org>]

Joris Van den Bossche created ARROW-9476:
--------------------------------------------

             Summary: [C++][Dataset] HivePartitioning discovery with dictionary types fails for multiple fields
                 Key: ARROW-9476
                 URL: https://issues.apache.org/jira/browse/ARROW-9476
             Project: Apache Arrow
          Issue Type: Bug
          Components: C++
            Reporter: Joris Van den Bossche


Apparently, ARROW-9288 was not fully / correctly fixing the issue. With a single string partition field, it now works fine. But once you have multiple string fields, you get parsing errors.

A reproducible example:

{code}
import numpy as np
import pyarrow as pa
import pyarrow.parquet as pq
import pyarrow.dataset as ds 

foo_keys = np.array(['a', 'b', 'c'], dtype=object)
bar_keys = np.array(['d', 'e', 'f'], dtype=object)
N = 30

table = pa.table({
    'foo': foo_keys.repeat(10),
    'bar': np.tile(np.tile(bar_keys, 5), 2),
    'values': np.random.randn(N)
})

base_path = "test_partition_directories3"
pq.write_to_dataset(table, base_path, partition_cols=["bar", "foo"])

# works
ds.dataset(base_path, partitioning="hive")
# fails
part = ds.HivePartitioning.discover(max_partition_dictionary_size=-1)
ds.dataset(base_path, partitioning=part)
{code}


cc [~bkietz]





--
This message was sent by Atlassian Jira
(v8.3.4#803005)