You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@arrow.apache.org by "Carl Boettiger (Jira)" <ji...@apache.org> on 2022/10/12 16:02:00 UTC
[jira] [Created] (ARROW-18006) [R] integer coercion on parition variables
Carl Boettiger created ARROW-18006:
--------------------------------------
Summary: [R] integer coercion on parition variables
Key: ARROW-18006
URL: https://issues.apache.org/jira/browse/ARROW-18006
Project: Apache Arrow
Issue Type: Bug
Components: R
Affects Versions: 9.0.0
Reporter: Carl Boettiger
If we partition on a character-string column that is "integer-like", arrow assumes the resulting partition variable is an integer. This is particularly surprising because in most other cases, making a column a partition variable coerces it to a character (makes sense, path names are strings), but in this case we have the opposite coercion where a string becomes an integer. Consider this reprex:
{code:java}
library(arrow)f <- tempfile()
df <- data.frame(x = c("00", "06", "18"))
df |> write_dataset(f, partitioning = "x")
open_dataset(f) # x is now an integer{code}
{code:java}
## fails
open_dataset(f) |> filter(x == "00")
{code}
changing this behavior could break existing code I think, so maybe this shouldn't be considered a bug, but I think it merits discussion.
(For instance, we often see large datasets, e.g. from NOAA csv etc, that use zero-padded integers in path names to indicate file-specific metadata. One side effect of the type-coercion is that it also strips the zero-padding)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)