You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2021/02/24 18:06:06 UTC
[GitHub] [arrow] jonkeane opened a new pull request #9561: ARROW-11649: [R] Add support for null_fallback to R
jonkeane opened a new pull request #9561:
URL: https://github.com/apache/arrow/pull/9561
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] westonpace commented on a change in pull request #9561: ARROW-11649: [R] Add support for null_fallback to R
Posted by GitBox <gi...@apache.org>.
westonpace commented on a change in pull request #9561:
URL: https://github.com/apache/arrow/pull/9561#discussion_r582351003
##########
File path: r/R/dataset-partition.R
##########
@@ -25,12 +25,17 @@
#' `DirectoryPartitioning` describes how to interpret raw path segments, in
#' order. For example, `schema(year = int16(), month = int8())` would define
#' partitions for file paths like "2019/01/file.parquet",
-#' "2019/02/file.parquet", etc.
+#' "2019/02/file.parquet", etc. In this scheme `NULL` values will be skipped.
+#' In the previous example, if the month was `NULL`, the files would be placed
Review comment:
It's a bit indirect but this applies for reading too. In `open_dataset` there is this comment...
```
#' * a `HivePartitioning` or `HivePartitioningFactory`, as returned
#' by [hive_partition()] which parses explicit or autodetected fields from
#' Hive-style path segments
```
...and then in `hive_partition()` there is this comment https://github.com/apache/arrow/pull/9561/files/f042c66b1f53f6185e5a33949e9f5e148e39501a#diff-7f4a981c0f18320c1258082cf98249d3e6bc6d6f62ac561b731e724a6997c118R80
I think this is ok because, unlike python, there is no option to specify a "flavor string" like "hive" or "directory" and the default is to use directory. So if a user is working with hive partitioned data in the first place they are already having to use `hive_partition`. Also, I think it will be a pretty rare case where the user has to specify `null_fallback` and they probably already have some idea of what they are looking for (since they will have custom configured whatever generated their data).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] westonpace commented on a change in pull request #9561: ARROW-11649: [R] Add support for null_fallback to R
Posted by GitBox <gi...@apache.org>.
westonpace commented on a change in pull request #9561:
URL: https://github.com/apache/arrow/pull/9561#discussion_r582357349
##########
File path: r/R/dataset-partition.R
##########
@@ -72,19 +77,22 @@ HivePartitioning$create <- dataset___HivePartitioning
#' Because fields are named in the path segments, order of fields passed to
#' `hive_partition()` does not matter.
#' @param ... named list of [data types][data-type], passed to [schema()]
+#' @param null_fallback character to be used in place of `NA` and `NULL` values
+#' in columns that are being partitioned by. (default:
+#' `"__HIVE_DEFAULT_PARTITION__"`)
#' @return A [HivePartitioning][Partitioning], or a `HivePartitioningFactory` if
#' calling `hive_partition()` with no arguments.
#' @examples
#' \donttest{
#' hive_partition(year = int16(), month = int8())
#' }
#' @export
-hive_partition <- function(...) {
+hive_partition <- function(..., null_fallback = "__HIVE_DEFAULT_PARTITION__") {
Review comment:
In Hive & Spark it's a configuration property which you can set at runtime or via a configuration file.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jonkeane commented on a change in pull request #9561: ARROW-11649: [R] Add support for null_fallback to R
Posted by GitBox <gi...@apache.org>.
jonkeane commented on a change in pull request #9561:
URL: https://github.com/apache/arrow/pull/9561#discussion_r582994030
##########
File path: r/R/dataset-partition.R
##########
@@ -72,19 +77,22 @@ HivePartitioning$create <- dataset___HivePartitioning
#' Because fields are named in the path segments, order of fields passed to
#' `hive_partition()` does not matter.
#' @param ... named list of [data types][data-type], passed to [schema()]
+#' @param null_fallback character to be used in place of `NA` and `NULL` values
+#' in columns that are being partitioned by. (default:
+#' `"__HIVE_DEFAULT_PARTITION__"`)
Review comment:
Yeah, is it hive that originated the convention? I think there's value in saying who we are following explicitly
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] westonpace commented on a change in pull request #9561: ARROW-11649: [R] Add support for null_fallback to R
Posted by GitBox <gi...@apache.org>.
westonpace commented on a change in pull request #9561:
URL: https://github.com/apache/arrow/pull/9561#discussion_r583113776
##########
File path: r/R/dataset-partition.R
##########
@@ -72,19 +77,22 @@ HivePartitioning$create <- dataset___HivePartitioning
#' Because fields are named in the path segments, order of fields passed to
#' `hive_partition()` does not matter.
#' @param ... named list of [data types][data-type], passed to [schema()]
+#' @param null_fallback character to be used in place of `NA` and `NULL` values
+#' in columns that are being partitioned by. (default:
+#' `"__HIVE_DEFAULT_PARTITION__"`)
Review comment:
Lots of tools follow this convention, Spark, Athena, Redshift, Impala. They all refer to this style of partitioning as "hive partitioning". I don't think Hive itself is always involved. I'm not sure if Hive came first or if this was a Spark convention that spun off into Hive or how things were ordered.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] nealrichardson closed pull request #9561: ARROW-11649: [R] Add support for null_fallback to R
Posted by GitBox <gi...@apache.org>.
nealrichardson closed pull request #9561:
URL: https://github.com/apache/arrow/pull/9561
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] westonpace commented on a change in pull request #9561: ARROW-11649: [R] Add support for null_fallback to R
Posted by GitBox <gi...@apache.org>.
westonpace commented on a change in pull request #9561:
URL: https://github.com/apache/arrow/pull/9561#discussion_r582252920
##########
File path: r/R/dataset-partition.R
##########
@@ -25,12 +25,17 @@
#' `DirectoryPartitioning` describes how to interpret raw path segments, in
#' order. For example, `schema(year = int16(), month = int8())` would define
#' partitions for file paths like "2019/01/file.parquet",
-#' "2019/02/file.parquet", etc.
+#' "2019/02/file.parquet", etc. In this scheme null values will be skipped.
+#' In the previous example, if the month was null, the files would be placed
+#' in 2019/file.parquet. An error will be raised if an outer directory is
Review comment:
`An error will be raised` is what I would say in Python. Does R translate invalid Status into some kind of "raised error" or would it be more accurate to say "returned"? Is the R terminology "thrown"?
##########
File path: r/R/dataset-partition.R
##########
@@ -72,19 +77,22 @@ HivePartitioning$create <- dataset___HivePartitioning
#' Because fields are named in the path segments, order of fields passed to
#' `hive_partition()` does not matter.
#' @param ... named list of [data types][data-type], passed to [schema()]
+#' @param null_fallback character to be used in place of `NA` and `NULL` values
Review comment:
Self nit: In this comment I say "`NA` and `NULL`" and in the next comment I say `null`. I guess I wasn't being too consistent. I'm not sure what the proper R term for "a null array value" is.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] westonpace commented on a change in pull request #9561: ARROW-11649: [R] Add support for null_fallback to R
Posted by GitBox <gi...@apache.org>.
westonpace commented on a change in pull request #9561:
URL: https://github.com/apache/arrow/pull/9561#discussion_r582352870
##########
File path: r/R/dataset-partition.R
##########
@@ -72,19 +77,22 @@ HivePartitioning$create <- dataset___HivePartitioning
#' Because fields are named in the path segments, order of fields passed to
#' `hive_partition()` does not matter.
#' @param ... named list of [data types][data-type], passed to [schema()]
+#' @param null_fallback character to be used in place of `NA` and `NULL` values
+#' in columns that are being partitioned by. (default:
+#' `"__HIVE_DEFAULT_PARTITION__"`)
Review comment:
Something like `(default: "__HIVE_DEFAULT_PARTITION__" to maintain compatibility with other tools that read & write hive partitioned datasets)`?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jonkeane commented on pull request #9561: ARROW-11649: [R] Add support for null_fallback to R
Posted by GitBox <gi...@apache.org>.
jonkeane commented on pull request #9561:
URL: https://github.com/apache/arrow/pull/9561#issuecomment-790126766
Ok, I've accepted, re-documented, and pushed. This is good to go once CI passes
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] westonpace commented on a change in pull request #9561: ARROW-11649: [R] Add support for null_fallback to R
Posted by GitBox <gi...@apache.org>.
westonpace commented on a change in pull request #9561:
URL: https://github.com/apache/arrow/pull/9561#discussion_r582344610
##########
File path: r/R/dataset-write.R
##########
@@ -41,6 +41,9 @@
#' will yield `"part-0.feather", ...`.
#' @param hive_style logical: write partition segments as Hive-style
#' (`key1=value1/key2=value2/file.ext`) or as just bare values. Default is `TRUE`.
+#' @param hive_null_fallback If writing partition segments as Hive-style
Review comment:
I think omitting is fine. The line `partitioning <- HivePartitioning$create(partition_schema, hive_null_fallback)` would become `partitioning <- hive_partition(partition_schema)` as `HivePartitioning$create` cannot have a default value for `hive_null_fallback`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] nealrichardson commented on a change in pull request #9561: ARROW-11649: [R] Add support for null_fallback to R
Posted by GitBox <gi...@apache.org>.
nealrichardson commented on a change in pull request #9561:
URL: https://github.com/apache/arrow/pull/9561#discussion_r582308004
##########
File path: r/R/dataset-partition.R
##########
@@ -72,19 +77,22 @@ HivePartitioning$create <- dataset___HivePartitioning
#' Because fields are named in the path segments, order of fields passed to
#' `hive_partition()` does not matter.
#' @param ... named list of [data types][data-type], passed to [schema()]
+#' @param null_fallback character to be used in place of `NA` and `NULL` values
Review comment:
```suggestion
#' @param null_fallback character to be used in place of missing values (`NA` or `NULL`)
```
##########
File path: r/R/dataset-partition.R
##########
@@ -25,12 +25,17 @@
#' `DirectoryPartitioning` describes how to interpret raw path segments, in
#' order. For example, `schema(year = int16(), month = int8())` would define
#' partitions for file paths like "2019/01/file.parquet",
-#' "2019/02/file.parquet", etc.
+#' "2019/02/file.parquet", etc. In this scheme `NULL` values will be skipped.
+#' In the previous example, if the month was `NULL`, the files would be placed
+#' in 2019/file.parquet. An error will be raised if an outer directory is
+#' `NULL` and an inner directory is not.
#'
#' `HivePartitioning` is for Hive-style partitioning, which embeds field
#' names and values in path segments, such as
#' "/year=2019/month=2/data.parquet". Because fields are named in the path
-#' segments, order does not matter.
+#' segments, order does not matter. This partitioning scheme allows `NULL`
+#' values. They will be replaced by a configurable `null_fallback` which
+#' defaults to `__HIVE_DEFAULT_PARTITION__`.
Review comment:
```suggestion
#' defaults to the string `"__HIVE_DEFAULT_PARTITION__"`.
```
this?
##########
File path: r/R/dataset-partition.R
##########
@@ -25,12 +25,17 @@
#' `DirectoryPartitioning` describes how to interpret raw path segments, in
#' order. For example, `schema(year = int16(), month = int8())` would define
#' partitions for file paths like "2019/01/file.parquet",
-#' "2019/02/file.parquet", etc.
+#' "2019/02/file.parquet", etc. In this scheme `NULL` values will be skipped.
+#' In the previous example, if the month was `NULL`, the files would be placed
Review comment:
While partitioning specification is for both reading and writing, these doc additions seem implicitly focused on writing datasets. Should this be made explicit? Is there a converse statement to be made about how nulls are interpreted on read?
##########
File path: r/R/dataset-write.R
##########
@@ -41,6 +41,9 @@
#' will yield `"part-0.feather", ...`.
#' @param hive_style logical: write partition segments as Hive-style
#' (`key1=value1/key2=value2/file.ext`) or as just bare values. Default is `TRUE`.
+#' @param hive_null_fallback If writing partition segments as Hive-style
Review comment:
I would either (1) call this `null_fallback` so it matches the name of the arg elsewhere or (2) omit it from the signature and pull it out of `...` in the function below.
Is there null fallback behavior for non-hive?
##########
File path: r/R/dataset-partition.R
##########
@@ -72,19 +77,22 @@ HivePartitioning$create <- dataset___HivePartitioning
#' Because fields are named in the path segments, order of fields passed to
#' `hive_partition()` does not matter.
#' @param ... named list of [data types][data-type], passed to [schema()]
+#' @param null_fallback character to be used in place of `NA` and `NULL` values
+#' in columns that are being partitioned by. (default:
+#' `"__HIVE_DEFAULT_PARTITION__"`)
#' @return A [HivePartitioning][Partitioning], or a `HivePartitioningFactory` if
#' calling `hive_partition()` with no arguments.
#' @examples
#' \donttest{
#' hive_partition(year = int16(), month = int8())
#' }
#' @export
-hive_partition <- function(...) {
+hive_partition <- function(..., null_fallback = "__HIVE_DEFAULT_PARTITION__") {
Review comment:
Do other systems let you set an env var to govern this, or is it always in code?
##########
File path: r/R/dataset-partition.R
##########
@@ -72,19 +77,22 @@ HivePartitioning$create <- dataset___HivePartitioning
#' Because fields are named in the path segments, order of fields passed to
#' `hive_partition()` does not matter.
#' @param ... named list of [data types][data-type], passed to [schema()]
+#' @param null_fallback character to be used in place of `NA` and `NULL` values
+#' in columns that are being partitioned by. (default:
+#' `"__HIVE_DEFAULT_PARTITION__"`)
Review comment:
Should it be explained somewhere where this sentinel string comes from? I.e. we didn't make this up, we're following the convention that XYZ follow.
##########
File path: r/R/dataset-write.R
##########
@@ -41,6 +41,9 @@
#' will yield `"part-0.feather", ...`.
#' @param hive_style logical: write partition segments as Hive-style
#' (`key1=value1/key2=value2/file.ext`) or as just bare values. Default is `TRUE`.
+#' @param hive_null_fallback If writing partition segments as Hive-style
+#' directories then this value will be used in place of any null value in a
+#' patition column.
Review comment:
```suggestion
#' partition column.
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jonkeane commented on a change in pull request #9561: ARROW-11649: [R] Add support for null_fallback to R
Posted by GitBox <gi...@apache.org>.
jonkeane commented on a change in pull request #9561:
URL: https://github.com/apache/arrow/pull/9561#discussion_r583026368
##########
File path: r/R/dataset-write.R
##########
@@ -41,6 +41,9 @@
#' will yield `"part-0.feather", ...`.
#' @param hive_style logical: write partition segments as Hive-style
#' (`key1=value1/key2=value2/file.ext`) or as just bare values. Default is `TRUE`.
+#' @param hive_null_fallback If writing partition segments as Hive-style
Review comment:
Ok I've done some shuffling to make this a possibility
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jonkeane commented on a change in pull request #9561: ARROW-11649: [R] Add support for null_fallback to R
Posted by GitBox <gi...@apache.org>.
jonkeane commented on a change in pull request #9561:
URL: https://github.com/apache/arrow/pull/9561#discussion_r582993386
##########
File path: r/R/dataset-partition.R
##########
@@ -72,19 +77,22 @@ HivePartitioning$create <- dataset___HivePartitioning
#' Because fields are named in the path segments, order of fields passed to
#' `hive_partition()` does not matter.
#' @param ... named list of [data types][data-type], passed to [schema()]
+#' @param null_fallback character to be used in place of `NA` and `NULL` values
+#' in columns that are being partitioned by. (default:
+#' `"__HIVE_DEFAULT_PARTITION__"`)
#' @return A [HivePartitioning][Partitioning], or a `HivePartitioningFactory` if
#' calling `hive_partition()` with no arguments.
#' @examples
#' \donttest{
#' hive_partition(year = int16(), month = int8())
#' }
#' @export
-hive_partition <- function(...) {
+hive_partition <- function(..., null_fallback = "__HIVE_DEFAULT_PARTITION__") {
Review comment:
I'm happy to make this an envvar as well if we think that's helpful
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] nealrichardson commented on a change in pull request #9561: ARROW-11649: [R] Add support for null_fallback to R
Posted by GitBox <gi...@apache.org>.
nealrichardson commented on a change in pull request #9561:
URL: https://github.com/apache/arrow/pull/9561#discussion_r585981323
##########
File path: r/R/dataset-write.R
##########
@@ -52,6 +55,8 @@
#' which case it will be V4.
#' - `codec`: A [Codec] which will be used to compress body buffers of written
#' files. Default (NULL) will not compress body buffers.
+#' - `null_fallback`: character to be used in place of missing values (`NA` or
+#' `NULL`) see [`hive_partition`] for more information.
Review comment:
```suggestion
#' `NULL`) when using Hive-style partitioning. See [hive_partition()].
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] nealrichardson commented on a change in pull request #9561: ARROW-11649: [R] Add support for null_fallback to R
Posted by GitBox <gi...@apache.org>.
nealrichardson commented on a change in pull request #9561:
URL: https://github.com/apache/arrow/pull/9561#discussion_r585980464
##########
File path: r/R/dataset-write.R
##########
@@ -41,6 +41,9 @@
#' will yield `"part-0.feather", ...`.
#' @param hive_style logical: write partition segments as Hive-style
#' (`key1=value1/key2=value2/file.ext`) or as just bare values. Default is `TRUE`.
+#' @param hive_null_fallback If writing partition segments as Hive-style
+#' directories then this value will be used in place of any null value in a
+#' partition column.
Review comment:
```suggestion
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] nealrichardson commented on a change in pull request #9561: ARROW-11649: [R] Add support for null_fallback to R
Posted by GitBox <gi...@apache.org>.
nealrichardson commented on a change in pull request #9561:
URL: https://github.com/apache/arrow/pull/9561#discussion_r585983314
##########
File path: r/R/dataset-partition.R
##########
@@ -72,19 +83,22 @@ HivePartitioning$create <- dataset___HivePartitioning
#' Because fields are named in the path segments, order of fields passed to
#' `hive_partition()` does not matter.
#' @param ... named list of [data types][data-type], passed to [schema()]
+#' @param null_fallback character to be used in place of missing values (`NA` or `NULL`)
+#' in columns that are being partitioned by. (default:
+#' `"__HIVE_DEFAULT_PARTITION__"`)
Review comment:
```suggestion
#' in partition columns. Default is `"__HIVE_DEFAULT_PARTITION__"`,
#' which is what Hive uses.
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jonkeane commented on a change in pull request #9561: ARROW-11649: [R] Add support for null_fallback to R
Posted by GitBox <gi...@apache.org>.
jonkeane commented on a change in pull request #9561:
URL: https://github.com/apache/arrow/pull/9561#discussion_r582281779
##########
File path: r/R/dataset-partition.R
##########
@@ -72,19 +77,22 @@ HivePartitioning$create <- dataset___HivePartitioning
#' Because fields are named in the path segments, order of fields passed to
#' `hive_partition()` does not matter.
#' @param ... named list of [data types][data-type], passed to [schema()]
+#' @param null_fallback character to be used in place of `NA` and `NULL` values
Review comment:
It's a bit complicated, in R most columns would have `NA`s for missing values (though it's possible that `NULL` would show up in some types, it's not particularly common). I added `NA` in here for the documentation on writing since that's most likely to be what's in the columns. I think it's just fine to have `NULL` above when talking about the hive partitioning. In (almost?) all cases those `NULL`s in the hive partitioning will get translated into `NA`s if/when the data is pulled into R.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] westonpace commented on a change in pull request #9561: ARROW-11649: [R] Add support for null_fallback to R
Posted by GitBox <gi...@apache.org>.
westonpace commented on a change in pull request #9561:
URL: https://github.com/apache/arrow/pull/9561#discussion_r583987579
##########
File path: r/R/dataset-partition.R
##########
@@ -72,19 +77,22 @@ HivePartitioning$create <- dataset___HivePartitioning
#' Because fields are named in the path segments, order of fields passed to
#' `hive_partition()` does not matter.
#' @param ... named list of [data types][data-type], passed to [schema()]
+#' @param null_fallback character to be used in place of `NA` and `NULL` values
+#' in columns that are being partitioned by. (default:
+#' `"__HIVE_DEFAULT_PARTITION__"`)
Review comment:
I think that is probably opening a can of worms. If you're within the spark execution context then you're probably using spark to read/write parquet. If you're outside the spark execution context then those configuration variables are probably not set/present/available.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] nealrichardson commented on a change in pull request #9561: ARROW-11649: [R] Add support for null_fallback to R
Posted by GitBox <gi...@apache.org>.
nealrichardson commented on a change in pull request #9561:
URL: https://github.com/apache/arrow/pull/9561#discussion_r583972560
##########
File path: r/R/dataset-partition.R
##########
@@ -72,19 +77,22 @@ HivePartitioning$create <- dataset___HivePartitioning
#' Because fields are named in the path segments, order of fields passed to
#' `hive_partition()` does not matter.
#' @param ... named list of [data types][data-type], passed to [schema()]
+#' @param null_fallback character to be used in place of `NA` and `NULL` values
+#' in columns that are being partitioned by. (default:
+#' `"__HIVE_DEFAULT_PARTITION__"`)
Review comment:
I guess my question was also along the lines of, if Spark uses this, where does it come from and how is it configured there? If there's an env var you set to configure how those pick up this value, we should probably also look for that env var.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jonkeane commented on a change in pull request #9561: ARROW-11649: [R] Add support for null_fallback to R
Posted by GitBox <gi...@apache.org>.
jonkeane commented on a change in pull request #9561:
URL: https://github.com/apache/arrow/pull/9561#discussion_r582998490
##########
File path: r/R/dataset-partition.R
##########
@@ -25,12 +25,17 @@
#' `DirectoryPartitioning` describes how to interpret raw path segments, in
#' order. For example, `schema(year = int16(), month = int8())` would define
#' partitions for file paths like "2019/01/file.parquet",
-#' "2019/02/file.parquet", etc.
+#' "2019/02/file.parquet", etc. In this scheme `NULL` values will be skipped.
+#' In the previous example, if the month was `NULL`, the files would be placed
Review comment:
I'm adding a bit more words and an explicit reading discussion.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jonkeane commented on a change in pull request #9561: ARROW-11649: [R] Add support for null_fallback to R
Posted by GitBox <gi...@apache.org>.
jonkeane commented on a change in pull request #9561:
URL: https://github.com/apache/arrow/pull/9561#discussion_r582267627
##########
File path: r/R/dataset-partition.R
##########
@@ -25,12 +25,17 @@
#' `DirectoryPartitioning` describes how to interpret raw path segments, in
#' order. For example, `schema(year = int16(), month = int8())` would define
#' partitions for file paths like "2019/01/file.parquet",
-#' "2019/02/file.parquet", etc.
+#' "2019/02/file.parquet", etc. In this scheme null values will be skipped.
+#' In the previous example, if the month was null, the files would be placed
+#' in 2019/file.parquet. An error will be raised if an outer directory is
Review comment:
Error will be raised sounds fine to me.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] westonpace commented on a change in pull request #9561: ARROW-11649: [R] Add support for null_fallback to R
Posted by GitBox <gi...@apache.org>.
westonpace commented on a change in pull request #9561:
URL: https://github.com/apache/arrow/pull/9561#discussion_r582343374
##########
File path: r/R/dataset-write.R
##########
@@ -41,6 +41,9 @@
#' will yield `"part-0.feather", ...`.
#' @param hive_style logical: write partition segments as Hive-style
#' (`key1=value1/key2=value2/file.ext`) or as just bare values. Default is `TRUE`.
+#' @param hive_null_fallback If writing partition segments as Hive-style
Review comment:
Right now "non-hive" means directory as those are the only two in common usage. Directory partitioning handles nulls by omitting directories or raising an error (if an outer directory is null but an inner directory is not). So there is no concept of a "null_fallback"
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #9561: ARROW-11649: [R] Add support for null_fallback to R
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #9561:
URL: https://github.com/apache/arrow/pull/9561#issuecomment-785332225
https://issues.apache.org/jira/browse/ARROW-11649
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org