You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2021/06/25 12:33:22 UTC
[GitHub] [arrow] jonkeane opened a new pull request #10601: ARROW-13149 [R]: Convert named lists to structs instead of (unnamed) lists
jonkeane opened a new pull request #10601:
URL: https://github.com/apache/arrow/pull/10601
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] romainfrancois commented on pull request #10601: ARROW-13149 [R]: Convert named lists to structs instead of (unnamed) lists
Posted by GitBox <gi...@apache.org>.
romainfrancois commented on pull request #10601:
URL: https://github.com/apache/arrow/pull/10601#issuecomment-872027133
I need to look into this in detail, but I believe that conceptually `arrow` structs have the same restrictions as data frames in that all elements have same size.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jonkeane commented on pull request #10601: ARROW-13149 [R]: Convert named lists to structs instead of (unnamed) lists
Posted by GitBox <gi...@apache.org>.
jonkeane commented on pull request #10601:
URL: https://github.com/apache/arrow/pull/10601#issuecomment-975816999
I'm going to close this for now — the branch might be revived or it might be (easier to be) re-done from first principles. https://issues.apache.org/jira/browse/ARROW-14378 is the overarching issue for this which we are hoping to tackle soon (though probably in stages)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jonkeane commented on a change in pull request #10601: ARROW-13149 [R]: Convert named lists to structs instead of (unnamed) lists
Posted by GitBox <gi...@apache.org>.
jonkeane commented on a change in pull request #10601:
URL: https://github.com/apache/arrow/pull/10601#discussion_r658846244
##########
File path: r/R/metadata.R
##########
@@ -56,7 +56,21 @@ apply_arrow_r_metadata <- function(x, r_metadata) {
if (is.data.frame(x)) {
if (length(names(x)) && !is.null(columns_metadata)) {
for (name in intersect(names(columns_metadata), names(x))) {
- x[[name]] <- apply_arrow_r_metadata(x[[name]], columns_metadata[[name]])
+ x[[name]] <- tryCatch({
+ x[[name]] <- apply_arrow_r_metadata(x[[name]], columns_metadata[[name]])
+ },
+ error = function(e) {
+ # if we are erroring because of incompatible data, try and make this
+ # a tibble
+ # TODO: also check if this is a list?
+ # TODO: only if there are exactly as many sub-list elements as rows?
+ # TODO: decide if this obviates the need for the option
+ # arrow.strucs_as_dfs (or if that is actually a better way to handle that)
+ if (grepl("must be compatible with existing data", e$message))
+ x[[name]] <- as.data.frame(x[[name]])
+ class(x[[name]]) <- c("tbl_df", "tbl", "data.frame")
+ apply_arrow_r_metadata(x[[name]], columns_metadata[[name]])
+ })
Review comment:
This code chunk will make it so that we can read in data saved in parquet files that don't also store the data.frame metadata needed to reconstruct them (since until this PR we always stripped that).
If we go this route, we probably should implement the todos listed here and also warn that this setup is deprecated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #10601: ARROW-13149 [R]: Convert named lists to structs instead of (unnamed) lists
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #10601:
URL: https://github.com/apache/arrow/pull/10601#issuecomment-868467415
https://issues.apache.org/jira/browse/ARROW-13149
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jonkeane commented on a change in pull request #10601: ARROW-13149 [R]: Convert named lists to structs instead of (unnamed) lists
Posted by GitBox <gi...@apache.org>.
jonkeane commented on a change in pull request #10601:
URL: https://github.com/apache/arrow/pull/10601#discussion_r658849841
##########
File path: r/R/metadata.R
##########
@@ -56,7 +56,21 @@ apply_arrow_r_metadata <- function(x, r_metadata) {
if (is.data.frame(x)) {
if (length(names(x)) && !is.null(columns_metadata)) {
for (name in intersect(names(columns_metadata), names(x))) {
- x[[name]] <- apply_arrow_r_metadata(x[[name]], columns_metadata[[name]])
+ x[[name]] <- tryCatch({
+ x[[name]] <- apply_arrow_r_metadata(x[[name]], columns_metadata[[name]])
+ },
+ error = function(e) {
+ # if we are erroring because of incompatible data, try and make this
+ # a tibble
+ # TODO: also check if this is a list?
+ # TODO: only if there are exactly as many sub-list elements as rows?
+ # TODO: decide if this obviates the need for the option
+ # arrow.strucs_as_dfs (or if that is actually a better way to handle that)
+ if (grepl("must be compatible with existing data", e$message))
+ x[[name]] <- as.data.frame(x[[name]])
+ class(x[[name]]) <- c("tbl_df", "tbl", "data.frame")
+ apply_arrow_r_metadata(x[[name]], columns_metadata[[name]])
+ })
Review comment:
This works along side the option restore-old-behavior I point out below. I don't think this could totally replace it. Though if we don't go this route maybe in the invalid message here we should mention trying to re-read with that option set
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jonkeane closed pull request #10601: ARROW-13149 [R]: Convert named lists to structs instead of (unnamed) lists
Posted by GitBox <gi...@apache.org>.
jonkeane closed pull request #10601:
URL: https://github.com/apache/arrow/pull/10601
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jonkeane commented on a change in pull request #10601: ARROW-13149 [R]: Convert named lists to structs instead of (unnamed) lists
Posted by GitBox <gi...@apache.org>.
jonkeane commented on a change in pull request #10601:
URL: https://github.com/apache/arrow/pull/10601#discussion_r658849273
##########
File path: r/src/array_to_vector.cpp
##########
@@ -723,9 +723,13 @@ class Converter_Struct : public Converter {
auto colnames = arrow::r::to_r_strings(
type->fields(),
[](const std::shared_ptr<Field>& field) { return field->name(); });
- out.attr(symbols::row_names) = arrow::r::short_row_names(n);
out.attr(R_NamesSymbol) = colnames;
- out.attr(R_ClassSymbol) = arrow::r::data::classes_tbl_df;
+
+ // backwards compatibility
+ if (arrow::r::GetBoolOption("arrow.structs_as_dfs", false)) {
+ out.attr(symbols::row_names) = arrow::r::short_row_names(n);
+ out.attr(R_ClassSymbol) = arrow::r::data::classes_tbl_df;
+ }
Review comment:
This is the other option for backwards compatibility which (if set) will restore the re-classing all structs as data.frames. It is more robust than the metadata one above, so is probably better.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jonkeane commented on a change in pull request #10601: ARROW-13149 [R]: Convert named lists to structs instead of (unnamed) lists
Posted by GitBox <gi...@apache.org>.
jonkeane commented on a change in pull request #10601:
URL: https://github.com/apache/arrow/pull/10601#discussion_r658859592
##########
File path: r/src/r_to_arrow.cpp
##########
@@ -1089,6 +1096,7 @@ class RStructConverter : public StructConverter<RConverter, RConverterTrait> {
auto fields = this->struct_type_->fields();
R_xlen_t n_columns = XLENGTH(values);
+ // TODO: this needs to be updated for the new struct situation like Extend above
Review comment:
I haven't touched the `DelayedExtend` definition at all yet. I need to propagate the changes made in `Extend` here, and will once that approach is more solidified
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on pull request #10601: ARROW-13149 [R]: Convert named lists to structs instead of (unnamed) lists
Posted by GitBox <gi...@apache.org>.
pitrou commented on pull request #10601:
URL: https://github.com/apache/arrow/pull/10601#issuecomment-975664229
@jonkeane Should this be updated?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jonkeane commented on a change in pull request #10601: ARROW-13149 [R]: Convert named lists to structs instead of (unnamed) lists
Posted by GitBox <gi...@apache.org>.
jonkeane commented on a change in pull request #10601:
URL: https://github.com/apache/arrow/pull/10601#discussion_r658853111
##########
File path: r/tests/testthat/test-Array.R
##########
@@ -839,3 +840,23 @@ test_that("Array to C-interface", {
delete_arrow_schema(schema_ptr)
delete_arrow_array(array_ptr)
})
+
+test_that("posixlt", {
+ skip("Arrays don't keep metadata about class type")
Review comment:
This is true both before and after this PR. I'm happy to take these additional tests out since they don't work today and won't work without additional work.
When I was chatting with Joris about this, he suggested that we might want to lean on using an extension type of struct here and that should give us enough info to reconstruct a posixlt from an Array of them directly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jonkeane commented on a change in pull request #10601: ARROW-13149 [R]: Convert named lists to structs instead of (unnamed) lists
Posted by GitBox <gi...@apache.org>.
jonkeane commented on a change in pull request #10601:
URL: https://github.com/apache/arrow/pull/10601#discussion_r658852266
##########
File path: r/tests/testthat/latin1.R
##########
@@ -35,12 +35,12 @@ expect_identical(Encoding(levels(df[[2]])), "latin1")
# Array
expect_identical(as.vector(Array$create(x)), x)
# struct
-expect_identical(as.vector(Array$create(df)), df)
+expect_identical(as.data.frame(Array$create(df)), df)
Review comment:
This is a change in interface, so we might also want to also make the backwards compatibility option change this behavior too. The new behavior seems more right here (especially in a world where not all structs are dataframes), but we might want to be more careful than this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jonkeane commented on a change in pull request #10601: ARROW-13149 [R]: Convert named lists to structs instead of (unnamed) lists
Posted by GitBox <gi...@apache.org>.
jonkeane commented on a change in pull request #10601:
URL: https://github.com/apache/arrow/pull/10601#discussion_r658850721
##########
File path: r/src/type_infer.cpp
##########
@@ -133,8 +133,14 @@ static inline std::shared_ptr<arrow::DataType> InferArrowTypeFromDataFrame(
template <>
std::shared_ptr<arrow::DataType> InferArrowTypeFromVector<VECSXP>(SEXP x) {
+ SEXP names = Rf_getAttrib(x, R_NamesSymbol);
if (Rf_inherits(x, "data.frame") || Rf_inherits(x, "POSIXlt")) {
return InferArrowTypeFromDataFrame(x);
+ } else if (!Rf_isNull(names)) {
+ // is there some more restrictive way to catch named lists? I've also tried
+ // adding Rf_inherits(x, "list") && !Rf_isNull(names), but that does not catch
+ // them for some reason.
+ return InferArrowTypeFromDataFrame(x);
Review comment:
I tried a few things here but none of them ended up working out — I'm sure that I'm missing something obvious
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jonkeane commented on a change in pull request #10601: ARROW-13149 [R]: Convert named lists to structs instead of (unnamed) lists
Posted by GitBox <gi...@apache.org>.
jonkeane commented on a change in pull request #10601:
URL: https://github.com/apache/arrow/pull/10601#discussion_r658854104
##########
File path: r/tests/testthat/test-json.R
##########
@@ -208,7 +208,7 @@ test_that("Can read json file with nested columns (ARROW-5503)", {
expect_equal(struct_array$GetFieldByName("ps"), ps)
expect_equal(struct_array$Flatten(), list(ps, hello))
expect_equal(
- as.vector(struct_array),
+ as.data.frame(struct_array),
Review comment:
Another example of a slight change in our `as.*` API. I think this is better now, but we will need to make sure the change is clearly communicated
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jonkeane commented on pull request #10601: ARROW-13149 [R]: Convert named lists to structs instead of (unnamed) lists
Posted by GitBox <gi...@apache.org>.
jonkeane commented on pull request #10601:
URL: https://github.com/apache/arrow/pull/10601#issuecomment-868570390
@github-actions autotune
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jonkeane commented on a change in pull request #10601: ARROW-13149 [R]: Convert named lists to structs instead of (unnamed) lists
Posted by GitBox <gi...@apache.org>.
jonkeane commented on a change in pull request #10601:
URL: https://github.com/apache/arrow/pull/10601#discussion_r658853111
##########
File path: r/tests/testthat/test-Array.R
##########
@@ -839,3 +840,23 @@ test_that("Array to C-interface", {
delete_arrow_schema(schema_ptr)
delete_arrow_array(array_ptr)
})
+
+test_that("posixlt", {
+ skip("Arrays don't keep metadata about class type")
Review comment:
This is true both before and after this PR. I'm happy to take these additional tests out since they don't work today and won't work without additional work.
When I was chatting with Joris about this, he suggested that we might want to lean on using an extension type of struct here and that should give us enough info to reconstruct a posixlt from an Array of them directly. Thought that's a separate ticket if we want to do that, I think.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jonkeane commented on a change in pull request #10601: ARROW-13149 [R]: Convert named lists to structs instead of (unnamed) lists
Posted by GitBox <gi...@apache.org>.
jonkeane commented on a change in pull request #10601:
URL: https://github.com/apache/arrow/pull/10601#discussion_r658857664
##########
File path: r/tests/testthat/test-metadata.R
##########
@@ -205,3 +205,47 @@ test_that("metadata of list elements (ARROW-10386)", {
expect_identical(attr(as.data.frame(tab)$x[[1]], "foo"), "bar")
expect_identical(attr(as.data.frame(tab)$x[[2]], "baz"), "qux")
})
+
+test_that("list types roundtrip", {
+ # first an unnamed, nested list
+ df <- tibble::tibble(
+ list = list(
+ list(c("one", "one"), c("two", "two")),
+ list(c("three", "three"), c("four", "four")),
+ list(c("five", "five"), c("six", "six"))
+ )
+ )
+ rb <- record_batch(df)
+ expect_identical(dim(rb), dim(df))
+ expect_equal(rb$schema$list$type, list_of(list_of(string())))
+ # as.data.frame(rb) adds arrow_list + vctrs classes, so we only check equivalence
+ expect_equivalent(as.data.frame(rb), df)
+
+ # Now, a named list
+ df <- tibble::tibble(
+ named_list = list(
+ list(a = c("one", "one"), b = c("two", "two")),
+ list(a = c("three", "three"), b = c("four", "four")),
+ list(a = c("five", "five"), b = c("six", "six"))
+ )
+ )
+ rb <- record_batch(df)
+ expect_identical(dim(rb), dim(df))
+ expect_equal(rb$schema$named_list$type, list_of(struct(a = string(), b = string())))
+ expect_identical(as.data.frame(rb), df)
+
+ skip("These are not yet supported, ")
Review comment:
I need to do more with this to support this. It converts ok, but it doesn't roundtrip correctly. In the conversion to record batches here, the named list of named lists gets turned into a 2 row, 1 column data frame (where each cell has three structs named "one_row", "two_row", "three_row"). When in fact what we want here is a 3 row, 1 column data frame (where each cell has two structs named "a" and "b").
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org