You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "Joris Van den Bossche (Jira)" <ji...@apache.org> on 2020/04/28 09:26:00 UTC
[jira] [Commented] (ARROW-3388) [Python] boolean Partition keys in ParquetDataset are reconstructed as string
[ https://issues.apache.org/jira/browse/ARROW-3388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17094329#comment-17094329 ]
Joris Van den Bossche commented on ARROW-3388:
----------------------------------------------
An update on this issue: also with the new datasets API, they get read back as strings.
Now, I am not fully sure we necessarily should automatically infer True/False string values. With the new API you can also specify the schema of the partition columns, which seems the general solution for this. However, this also does not work yet, as it just gives nulls (will open a separate issue about that, since it is not specific to booleans)
> [Python] boolean Partition keys in ParquetDataset are reconstructed as string
> -----------------------------------------------------------------------------
>
> Key: ARROW-3388
> URL: https://issues.apache.org/jira/browse/ARROW-3388
> Project: Apache Arrow
> Issue Type: Bug
> Components: Python
> Reporter: Uwe Korn
> Priority: Major
> Labels: dataset, dataset-parquet-read, parquet
>
> Saving a {{ParquetDataset}} using a boolean column as a partitioning column will store {{True/False}} as the values in the path. On reload these columns will then be string columns with the values {{'True'}} and {{'False'}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)