You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "Scott Taylor (JIRA)" <ji...@apache.org> on 2019/03/25 10:53:00 UTC
[jira] [Created] (ARROW-5004) Confusing behaviour with boolean
partition keys
Scott Taylor created ARROW-5004:
-----------------------------------
Summary: Confusing behaviour with boolean partition keys
Key: ARROW-5004
URL: https://issues.apache.org/jira/browse/ARROW-5004
Project: Apache Arrow
Issue Type: Bug
Components: Python
Affects Versions: 0.12.1
Reporter: Scott Taylor
[https://github.com/apache/arrow/blob/3129732a18210d0c8921b45f79be4f34eadf0cc3/python/pyarrow/parquet.py#L686]
Here the type of a partition key is converted to match the type of a filter variable.
using the *write_to_dataset* function allows *boolean* partition keys (*True* or *False)* but these silently break at the linked line as *bool('False')* evaluates as *True*.
I understand a docstring ([https://github.com/apache/arrow/blob/3129732a18210d0c8921b45f79be4f34eadf0cc3/python/pyarrow/parquet.py#L653)] refers to only string or int partition variables being supported although this is somewhat buried away from the user facing API.
It may be beneficial to detect the boolean case and raise a warning or to ensure the function returns a more intuitive output when partition key is *'False'* and the filter variable is *False.*
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)