[jira] [Created] (ARROW-9345) [C++][Dataset] Expression with dictionary type should work with operand of value type

["Joris Van den Bossche (Jira)" <ji...@apache.org>]

Joris Van den Bossche created ARROW-9345:
--------------------------------------------

             Summary: [C++][Dataset] Expression with dictionary type should work with operand of value type 
                 Key: ARROW-9345
                 URL: https://issues.apache.org/jira/browse/ARROW-9345
             Project: Apache Arrow
          Issue Type: Improvement
          Components: C++
            Reporter: Joris Van den Bossche


Related to ARROW-8647, see comment at https://github.com/apache/arrow/pull/7536#issuecomment-653124260

When using dictionary type for the partition fields, this now creates partition expressions that also use a dictionary type. Which means that doing something like {{dataset.to_table(filter=ds.field("part") == "A")}} to filter on the partition field with a plain string expression doesn't work, limiting the usability of this option (and even with the new Python scalar stuff, it would not be easy to construct the correct expression):

{code}
In [9]: part = ds.HivePartitioning.discover(max_partition_dictionary_size=2)  

In [10]: dataset = ds.dataset("test_partitioned_filter/", format="parquet", partitioning=part)

In [11]: fragment = list(dataset.get_fragments())[0]   

In [12]: fragment.partition_expression  
Out[12]: 
<pyarrow.dataset.Expression (part == [
  "A",
  "B"
][0]:dictionary<values=string, indices=int32, ordered=0>)>

In [13]: dataset.to_table(filter=ds.field("part") == "A") 
...
ArrowNotImplementedError: cast from string
{code}

It might be an option to keep the `partition_expression` use the dictionary *value type* instead of dictionary type? Or alternatively, as [~fsaintjacques] proposed, ensure that any comparison involving the dict type should also work with the "effective" logical type (the value type of the dict).



--
This message was sent by Atlassian Jira
(v8.3.4#803005)