You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2021/01/06 23:13:20 UTC
[GitHub] [arrow] jonkeane opened a new pull request #9118: [WIP] Arrow 10623: [R] Version 1.0.1 breaks data.frame attributes when reading file written by 2.0.0
jonkeane opened a new pull request #9118:
URL: https://github.com/apache/arrow/pull/9118
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jonkeane commented on a change in pull request #9118: [WIP] Arrow 10623: [R] Version 1.0.1 breaks data.frame attributes when reading file written by 2.0.0
Posted by GitBox <gi...@apache.org>.
jonkeane commented on a change in pull request #9118:
URL: https://github.com/apache/arrow/pull/9118#discussion_r553018847
##########
File path: r/tests/testthat/test-parquet-compatibility.R
##########
@@ -0,0 +1,45 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# TODO: skip mostly? Or use this for backwards compat?
+
+pq_file <- test_path("parquets/data-arrow2.parquet")
+
+test_that("reading a known Parquet file to dataframe", {
+ df <- read_parquet(pq_file)
+ expect_data_frame(df, data.frame(col1 = 1:10))
+ expect_identical(dim(df), c(10L, 1L))
+
+ tab <- read_parquet(pq_file, as_data_frame = FALSE)
+ expect_s3_class(tab, "Table")
+ expect_equal(
+ # unserialize like .unserialize_arrow_r_metadata does (though we can't call
+ # it directly because it's not exported)
+ unserialize(charToRaw(tab$metadata$r)),
+ list(
+ attributes = list(class = "data.frame"),
+ columns = list(col1 = NULL)
+ )
+ )
+
+ # test the workaround
+ tab$metadata$r <- NULL
Review comment:
Yeah, this file isn't what it needs to be for the golden tests we were thinking about — that's my next step. This is basically as it was when I was using it for forward compatibility
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #9118: [WIP] Arrow 10623: [CI][R] Version 1.0.1 breaks data.frame attributes when reading file written by 2.0.0
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #9118:
URL: https://github.com/apache/arrow/pull/9118#issuecomment-756316918
Revision: c92939bfc102618d1e683ea192962061b41fa56e
Submitted crossbow builds: [ursa-labs/crossbow @ actions-837](https://github.com/ursa-labs/crossbow/branches/all?query=actions-837)
|Task|Status|
|----|------|
|test-r-version-compatibility|[](https://github.com/ursa-labs/crossbow/actions?query=branch:actions-837-github-test-r-version-compatibility)|
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] nealrichardson commented on a change in pull request #9118: ARROW-10623: [CI][R] Version 1.0.1 breaks data.frame attributes when reading file written by 2.0.0
Posted by GitBox <gi...@apache.org>.
nealrichardson commented on a change in pull request #9118:
URL: https://github.com/apache/arrow/pull/9118#discussion_r555174313
##########
File path: r/tests/testthat/test-backwards-compatibility.R
##########
@@ -0,0 +1,166 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# To write a new version of a test file for a current version:
+# write_parquet(example_with_metadata, test_path("golden-files/data-arrow_2.0.0.parquet"))
+
+# To write a new version of a test file for an old version, use docker(-compose)
+# to setup a linux distribution and use RStudio's public package manager binary
+# repo to install the old version. The following commands should be run at the
+# root of the arrow repo directory and might need slight adjusments.
+# R_ORG=rstudio R_IMAGE=r-base R_TAG=4.0-focal docker-compose build --no-cache r
+# R_ORG=rstudio R_IMAGE=r-base R_TAG=4.0-focal docker-compose run r /bin/bash
+# R
+# options(repos = "https://packagemanager.rstudio.com/all/__linux__/focal/latest")
+# remotes::install_version("arrow", version = "1.0.1")
+# # get example data into the global env
+# write_parquet(example_with_metadata, "arrow/r/tests/testthat/golden-files/data-arrow_1.0.1.parquet")
+# quit()/exit
+
+test_that("reading a known Parquet file to dataframe with 2.0.0", {
+ skip_if_not_available("snappy")
+ pq_file <- test_path("golden-files/data-arrow_2.0.0.parquet")
+
+ df <- read_parquet(pq_file)
+ expect_equal(df, example_with_metadata)
+ expect_identical(dim(df), c(1L, 4L))
+
+ expect_equal(
+ attributes(df),
+ list(
+ names = letters[1:4],
+ row.names = 1L,
+ top_level = list(field_one = 12, field_two = "more stuff"),
+ class = c("tbl_df", "tbl", "data.frame"))
+ )
+ expect_equal(attributes(df$a), list(class = "special_string"))
+ expect_null(attributes(df$b))
+ expect_equal(
+ attributes(df$c),
+ list(
+ row.names = 1L,
+ names = c("c1", "c2", "c3"),
+ class = c("tbl_df", "tbl", "data.frame")
+ )
+ )
+ expect_null(attributes(df$d))
+})
+
+test_that("reading a known Parquet file to dataframe with 1.0.1", {
+ skip_if_not_available("snappy")
+ pq_file <- test_path("golden-files/data-arrow_1.0.1.parquet")
+
+ df <- read_parquet(pq_file)
+ # 1.0.1 didn't save top-level metadata, so we need to remove it.
+ example_with_metadata_sans_toplevel <- example_with_metadata
+ attributes(example_with_metadata_sans_toplevel)$top_level <- NULL
+ expect_equal(df, example_with_metadata_sans_toplevel)
Review comment:
why not just `expect_identical` and skip the rest of the assertions?
##########
File path: r/extra-tests/test-read-files.R
##########
@@ -0,0 +1,165 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+library(arrow)
+library(testthat)
+
+pq_file <- "files/ex_data.parquet"
+
+test_that("Can read the file (parquet)", {
+ # We can read with no error, we assert metadata below
+ expect_error(
+ df <- read_parquet(pq_file),
+ NA
+ )
+})
+
+### Parquet
+test_that("Can see the metadata (parquet)", {
+ skip_if_version_less_than("2.0.0", "Version 1.0.1 can't read new version metadata.")
+
+ df <- read_parquet(pq_file)
+ expect_s3_class(df, "tbl")
+
+ expect_equal(
+ attributes(df),
+ list(
+ names = letters[1:4],
+ row.names = 1L,
+ top_level = list(
+ field_one = 12,
+ field_two = "more stuff"
+ ),
+ class = c("tbl_df", "tbl", "data.frame")
+ )
+ )
+
+ # column-level attributes
+ expect_equal(attributes(df$a), list(class = "special_string"))
+ expect_equal(
+ attributes(df$c),
+ list(
+ row.names = 1L,
+ names = c("c1", "c2", "c3"),
+ class = c("tbl_df", "tbl", "data.frame")
+ )
+ )
+})
Review comment:
What do you think about factoring this out to an `expect_something()` helper in this file, such that your tests here are just `expect_something(df)`?
Of course, the simplest one would be `expect_identical(df, example_with_metadata)`, right? Though there are tradeoffs with that.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #9118: [WIP] Arrow 10623: [CI][R] Version 1.0.1 breaks data.frame attributes when reading file written by 2.0.0
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #9118:
URL: https://github.com/apache/arrow/pull/9118#issuecomment-756941897
Revision: da48c83b1587e49b92c074096bae7e74db2e712f
Submitted crossbow builds: [ursa-labs/crossbow @ actions-848](https://github.com/ursa-labs/crossbow/branches/all?query=actions-848)
|Task|Status|
|----|------|
|test-r-version-compatibility|[](https://github.com/ursa-labs/crossbow/actions?query=branch:actions-848-github-test-r-version-compatibility)|
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jonkeane commented on pull request #9118: [WIP] Arrow 10623: [CI][R] Version 1.0.1 breaks data.frame attributes when reading file written by 2.0.0
Posted by GitBox <gi...@apache.org>.
jonkeane commented on pull request #9118:
URL: https://github.com/apache/arrow/pull/9118#issuecomment-756435682
@github-actions crossbow submit test-r-version-compatibility
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jonkeane commented on pull request #9118: [WIP] Arrow 10623: [CI][R] Version 1.0.1 breaks data.frame attributes when reading file written by 2.0.0
Posted by GitBox <gi...@apache.org>.
jonkeane commented on pull request #9118:
URL: https://github.com/apache/arrow/pull/9118#issuecomment-756923877
@github-actions crossbow submit test-r-version-compatibility
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] nealrichardson commented on a change in pull request #9118: Arrow 10623: [CI][R] Version 1.0.1 breaks data.frame attributes when reading file written by 2.0.0
Posted by GitBox <gi...@apache.org>.
nealrichardson commented on a change in pull request #9118:
URL: https://github.com/apache/arrow/pull/9118#discussion_r554211387
##########
File path: r/extra-tests/test-read-files.R
##########
@@ -0,0 +1,165 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+library(arrow)
+library(testthat)
+
+pq_file <- "files/ex_data.parquet"
+
+test_that("Can read the file (parquet)", {
+ # We can read with no error, we assert metadata below
+ expect_error(
+ df <- read_parquet(pq_file),
+ NA
+ )
+})
+
+### Parquet
+test_that("Can see the metadata (feather)", {
+ skip_if_version_less_than("2.0.0", "Version 1.0.1 can't read new version metadata.")
+
+ df <- read_parquet(pq_file)
+ expect_s3_class(df, "tbl")
+
+ expect_equal(
+ attributes(df),
+ list(
+ names = letters[1:4],
+ row.names = 1L,
+ top_level = list(
+ field_one = 12,
+ field_two = "more stuff"
+ ),
+ class = c("tbl_df", "tbl", "data.frame")
+ )
+ )
+
+ # column-level attributes
+ expect_equal(attributes(df$a), list(class = "special_string"))
+ expect_equal(
+ attributes(df$c),
+ list(
+ row.names = 1L,
+ names = c("c1", "c2", "c3"),
+ class = c("tbl_df", "tbl", "data.frame")
+ )
+ )
+})
+
+### Feather
+for (comp in c("lz4", "uncompressed", "zstd")) {
+ feather_file <- paste0("files/ex_data_", comp, ".feather")
+
+ test_that(paste0("Can read the file (feather ", comp, ")"), {
+ # We can read with no error, we assert metadata below
+ expect_error(
+ df <- read_feather(feather_file),
+ NA
+ )
+ })
+
+ test_that(paste0("Can see the metadata (feather ", comp, ")"), {
+ skip_if_version_less_than("2.0.0", "Version 1.0.1 can't read new version metadata.")
+
+ df <- read_feather(feather_file)
+ expect_s3_class(df, "tbl")
+
+ expect_equal(
+ attributes(df),
+ list(
+ names = letters[1:4],
+ row.names = 1L,
+ top_level = list(
+ field_one = 12,
+ field_two = "more stuff"
+ ),
+ class = c("tbl_df", "tbl", "data.frame")
+ )
+ )
+
+ # column-level attributes
+ expect_equal(attributes(df$a), list(class = "special_string"))
+ expect_equal(
+ attributes(df$c),
+ list(
+ row.names = 1L,
+ names = c("c1", "c2", "c3"),
+ class = c("tbl_df", "tbl", "data.frame")
+ )
+ )
+ })
+}
+
+test_that(paste0("Can read feather version 1"), {
Review comment:
```suggestion
test_that("Can read feather version 1", {
```
##########
File path: r/extra-tests/test-read-files.R
##########
@@ -0,0 +1,165 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+library(arrow)
+library(testthat)
+
+pq_file <- "files/ex_data.parquet"
+
+test_that("Can read the file (parquet)", {
+ # We can read with no error, we assert metadata below
+ expect_error(
+ df <- read_parquet(pq_file),
+ NA
+ )
+})
+
+### Parquet
+test_that("Can see the metadata (feather)", {
Review comment:
```suggestion
test_that("Can see the metadata (parquet)", {
```
?
##########
File path: r/tests/testthat/test-backwards-compatibility.R
##########
@@ -0,0 +1,159 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# To write a new version of a test file for a current version:
+# write_parquet(example_with_metadata, test_path("golden-files/data-arrow_2.0.0.parquet"))
+
+# To write a new version of a test file for an old version, use docker(-compose)
+# to setup a linux distribution and use RStudio's public package manager binary
+# repo to install the old version. The following commands should be run at the
+# root of the arrow repo directory and might need slight adjusments.
+# R_ORG=rstudio R_IMAGE=r-base R_TAG=4.0-focal docker-compose build --no-cache r
+# R_ORG=rstudio R_IMAGE=r-base R_TAG=4.0-focal docker-compose run r /bin/bash
+# R
+# options(repos = "https://packagemanager.rstudio.com/all/__linux__/focal/latest")
+# remotes::install_version("arrow", version = "1.0.1")
+# # get example data into the global env
+# write_parquet(example_with_metadata, "arrow/r/tests/testthat/golden-files/data-arrow_1.0.1.parquet")
+# quit()/exit
+
+test_that("reading a known Parquet file to dataframe with 2.0.0", {
+ skip_if_not_available("snappy")
+ pq_file <- test_path("golden-files/data-arrow_2.0.0.parquet")
+
+ df <- read_parquet(pq_file)
+ expect_equal(df, example_with_metadata)
+ expect_identical(dim(df), c(1L, 4L))
+
+ expect_equal(
+ attributes(df),
+ list(
+ names = letters[1:4],
+ row.names = 1L,
+ top_level = list(field_one = 12, field_two = "more stuff"),
+ class = c("tbl_df", "tbl", "data.frame"))
+ )
+ expect_equal(attributes(df$a), list(class = "special_string"))
+ expect_null(attributes(df$b))
+ expect_equal(
+ attributes(df$c),
+ list(
+ row.names = 1L,
+ names = c("c1", "c2", "c3"),
+ class = c("tbl_df", "tbl", "data.frame")
+ )
+ )
+ expect_null(attributes(df$d))
+})
+
+test_that("reading a known Parquet file to dataframe with 1.0.1", {
+ skip_if_not_available("snappy")
+ pq_file <- test_path("golden-files/data-arrow_1.0.1.parquet")
+
+ df <- read_parquet(pq_file)
+ # 1.0.1 didn't save top-level metadata, so we need to remove it.
+ example_with_metadata_sans_toplevel <- example_with_metadata
+ attributes(example_with_metadata_sans_toplevel)$top_level <- NULL
+ expect_equal(df, example_with_metadata_sans_toplevel)
+ expect_identical(dim(df), c(1L, 4L))
+
+ expect_equal(
+ attributes(df),
+ list(
+ names = letters[1:4],
+ row.names = 1L,
+ class = c("tbl_df", "tbl", "data.frame"))
+ )
+ expect_equal(attributes(df$a), list(class = "special_string"))
+ expect_null(attributes(df$b))
+ expect_equal(
+ attributes(df$c),
+ list(
+ row.names = 1L,
+ names = c("c1", "c2", "c3"),
+ class = c("tbl_df", "tbl", "data.frame")
+ )
+ )
+ expect_null(attributes(df$d))
+})
+
+for (comp in c("lz4", "uncompressed", "zstd")) {
+ # write_feather(example_with_metadata, test_path("golden-files/data-arrow_2.0.0_lz4.feather"), compression = "lz4")
+ # write_feather(example_with_metadata, test_path("golden-files/data-arrow_2.0.0_uncompressed.feather"), compression = "uncompressed")
+ # write_feather(example_with_metadata, test_path("golden-files/data-arrow_2.0.0_zstd.feather"), compression = "zstd")
+ test_that("reading a known Feather file to dataframe with 2.0.0", {
+ skip_if_not_available(comp)
+ feather_file <- test_path(paste0("golden-files/data-arrow_2.0.0_", comp,".feather"))
+
+ df <- read_feather(feather_file)
+ expect_equal(df, example_with_metadata)
+ expect_identical(dim(df), c(1L, 4L))
+
+ expect_equal(
+ attributes(df),
+ list(
+ names = letters[1:4],
+ row.names = 1L,
+ top_level = list(field_one = 12, field_two = "more stuff"),
+ class = c("tbl_df", "tbl", "data.frame"))
+ )
+ expect_equal(attributes(df$a), list(class = "special_string"))
+ expect_null(attributes(df$b))
+ expect_equal(
+ attributes(df$c),
+ list(
+ row.names = 1L,
+ names = c("c1", "c2", "c3"),
+ class = c("tbl_df", "tbl", "data.frame")
+ )
+ )
+ expect_null(attributes(df$d))
+ })
+
+ test_that("reading a known Feather file to dataframe with 1.0.1", {
Review comment:
Shouldn't this be failing on master per ARROW-10850?
##########
File path: r/extra-tests/helpers.R
##########
@@ -0,0 +1,32 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+if_version <- function(version, op = `==`) {
+ op(numeric_version(Sys.getenv("OLD_ARROW_VERSION", "0.0.0")), version)
Review comment:
why not `packageVersion("arrow")`?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #9118: [WIP] Arrow 10623: [CI][R] Version 1.0.1 breaks data.frame attributes when reading file written by 2.0.0
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #9118:
URL: https://github.com/apache/arrow/pull/9118#issuecomment-756362465
Revision: 1d7788cb1466438b143f8bd2d24b9338fd0c7ac9
Submitted crossbow builds: [ursa-labs/crossbow @ actions-839](https://github.com/ursa-labs/crossbow/branches/all?query=actions-839)
|Task|Status|
|----|------|
|test-r-version-compatibility|[](https://github.com/ursa-labs/crossbow/actions?query=branch:actions-839-github-test-r-version-compatibility)|
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jorisvandenbossche commented on a change in pull request #9118: ARROW-10623: [CI][R] Version 1.0.1 breaks data.frame attributes when reading file written by 2.0.0
Posted by GitBox <gi...@apache.org>.
jorisvandenbossche commented on a change in pull request #9118:
URL: https://github.com/apache/arrow/pull/9118#discussion_r555074943
##########
File path: r/tests/testthat/test-backwards-compatibility.R
##########
@@ -0,0 +1,171 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# To write a new version of a test file for a current version:
+# write_parquet(example_with_metadata, test_path("golden-files/data-arrow_2.0.0.parquet"))
+
+# To write a new version of a test file for an old version, use docker(-compose)
+# to setup a linux distribution and use RStudio's public package manager binary
+# repo to install the old version. The following commands should be run at the
+# root of the arrow repo directory and might need slight adjusments.
+# R_ORG=rstudio R_IMAGE=r-base R_TAG=4.0-focal docker-compose build --no-cache r
+# R_ORG=rstudio R_IMAGE=r-base R_TAG=4.0-focal docker-compose run r /bin/bash
+# R
+# options(repos = "https://packagemanager.rstudio.com/all/__linux__/focal/latest")
+# remotes::install_version("arrow", version = "1.0.1")
+# # get example data into the global env
+# write_parquet(example_with_metadata, "arrow/r/tests/testthat/golden-files/data-arrow_1.0.1.parquet")
+# quit()/exit
+
+test_that("reading a known Parquet file to dataframe with 2.0.0", {
+ skip_if_not_available("snappy")
+ pq_file <- test_path("golden-files/data-arrow_2.0.0.parquet")
+
+ df <- read_parquet(pq_file)
+ expect_equal(df, example_with_metadata)
+ expect_identical(dim(df), c(1L, 4L))
+
+ expect_equal(
+ attributes(df),
+ list(
+ names = letters[1:4],
+ row.names = 1L,
+ top_level = list(field_one = 12, field_two = "more stuff"),
+ class = c("tbl_df", "tbl", "data.frame"))
+ )
+ expect_equal(attributes(df$a), list(class = "special_string"))
+ expect_null(attributes(df$b))
+ expect_equal(
+ attributes(df$c),
+ list(
+ row.names = 1L,
+ names = c("c1", "c2", "c3"),
+ class = c("tbl_df", "tbl", "data.frame")
+ )
+ )
+ expect_null(attributes(df$d))
+})
+
+test_that("reading a known Parquet file to dataframe with 1.0.1", {
+ skip_if_not_available("snappy")
+ pq_file <- test_path("golden-files/data-arrow_1.0.1.parquet")
+
+ df <- read_parquet(pq_file)
+ # 1.0.1 didn't save top-level metadata, so we need to remove it.
+ example_with_metadata_sans_toplevel <- example_with_metadata
+ attributes(example_with_metadata_sans_toplevel)$top_level <- NULL
+ expect_equal(df, example_with_metadata_sans_toplevel)
+ expect_identical(dim(df), c(1L, 4L))
+
+ expect_equal(
+ attributes(df),
+ list(
+ names = letters[1:4],
+ row.names = 1L,
+ class = c("tbl_df", "tbl", "data.frame"))
+ )
+ expect_equal(attributes(df$a), list(class = "special_string"))
+ expect_null(attributes(df$b))
+ expect_equal(
+ attributes(df$c),
+ list(
+ row.names = 1L,
+ names = c("c1", "c2", "c3"),
+ class = c("tbl_df", "tbl", "data.frame")
+ )
+ )
+ expect_null(attributes(df$d))
+})
+
+for (comp in c("lz4", "uncompressed", "zstd")) {
+ # write_feather(example_with_metadata, test_path("golden-files/data-arrow_2.0.0_lz4.feather"), compression = "lz4")
+ # write_feather(example_with_metadata, test_path("golden-files/data-arrow_2.0.0_uncompressed.feather"), compression = "uncompressed")
+ # write_feather(example_with_metadata, test_path("golden-files/data-arrow_2.0.0_zstd.feather"), compression = "zstd")
+ test_that("reading a known Feather file to dataframe with 2.0.0", {
+ skip_if_not_available(comp)
+ feather_file <- test_path(paste0("golden-files/data-arrow_2.0.0_", comp,".feather"))
+
+ df <- read_feather(feather_file)
+ expect_equal(df, example_with_metadata)
+ expect_identical(dim(df), c(1L, 4L))
+
+ expect_equal(
+ attributes(df),
+ list(
+ names = letters[1:4],
+ row.names = 1L,
+ top_level = list(field_one = 12, field_two = "more stuff"),
+ class = c("tbl_df", "tbl", "data.frame"))
+ )
+ expect_equal(attributes(df$a), list(class = "special_string"))
+ expect_null(attributes(df$b))
+ expect_equal(
+ attributes(df$c),
+ list(
+ row.names = 1L,
+ names = c("c1", "c2", "c3"),
+ class = c("tbl_df", "tbl", "data.frame")
+ )
+ )
+ expect_null(attributes(df$d))
+ })
+
+ test_that("reading a known Feather file to dataframe with 1.0.1", {
+ skip_if_not_available(comp)
+ feather_file <- test_path(paste0("golden-files/data-arrow_1.0.1_", comp,".feather"))
+
+ df <- read_feather(feather_file)
+ # 1.0.1 didn't save top-level metadata, so we need to remove it.
+ example_with_metadata_sans_toplevel <- example_with_metadata
+ attributes(example_with_metadata_sans_toplevel)$top_level <- NULL
+ expect_equal(df, example_with_metadata_sans_toplevel)
+ expect_identical(dim(df), c(1L, 4L))
+
+ expect_equal(
+ attributes(df),
+ list(
+ names = letters[1:4],
+ row.names = 1L,
+ class = c("tbl_df", "tbl", "data.frame"))
+ )
+ expect_equal(attributes(df$a), list(class = "special_string"))
+ expect_null(attributes(df$b))
+ expect_equal(
+ attributes(df$c),
+ list(
+ row.names = 1L,
+ names = c("c1", "c2", "c3"),
+ class = c("tbl_df", "tbl", "data.frame")
+ )
+ )
+ expect_null(attributes(df$d))
+ })
+
+ test_that("reading a known Feather file to dataframe with 0.17.0", {
+ skip_if_not_available(comp)
+ feather_file <- test_path(paste0("golden-files/data-arrow_0.17.0_", comp,".feather"))
+
+ if (comp %in% c("lz4", "zstd")) {
+ # there is a case mis-match with versions 0.17.0 and before for the codec names
+ expect_error(df <- read_feather(feather_file), "Unrecognized compression type:")
Review comment:
So once https://github.com/apache/arrow/pull/9128 is merged, this should be fixed
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jonkeane commented on a change in pull request #9118: ARROW-10623: [CI][R] Version 1.0.1 breaks data.frame attributes when reading file written by 2.0.0
Posted by GitBox <gi...@apache.org>.
jonkeane commented on a change in pull request #9118:
URL: https://github.com/apache/arrow/pull/9118#discussion_r555126207
##########
File path: r/tests/testthat/test-backwards-compatibility.R
##########
@@ -0,0 +1,159 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# To write a new version of a test file for a current version:
+# write_parquet(example_with_metadata, test_path("golden-files/data-arrow_2.0.0.parquet"))
+
+# To write a new version of a test file for an old version, use docker(-compose)
+# to setup a linux distribution and use RStudio's public package manager binary
+# repo to install the old version. The following commands should be run at the
+# root of the arrow repo directory and might need slight adjusments.
+# R_ORG=rstudio R_IMAGE=r-base R_TAG=4.0-focal docker-compose build --no-cache r
+# R_ORG=rstudio R_IMAGE=r-base R_TAG=4.0-focal docker-compose run r /bin/bash
+# R
+# options(repos = "https://packagemanager.rstudio.com/all/__linux__/focal/latest")
+# remotes::install_version("arrow", version = "1.0.1")
+# # get example data into the global env
+# write_parquet(example_with_metadata, "arrow/r/tests/testthat/golden-files/data-arrow_1.0.1.parquet")
+# quit()/exit
+
+test_that("reading a known Parquet file to dataframe with 2.0.0", {
+ skip_if_not_available("snappy")
+ pq_file <- test_path("golden-files/data-arrow_2.0.0.parquet")
+
+ df <- read_parquet(pq_file)
+ expect_equal(df, example_with_metadata)
+ expect_identical(dim(df), c(1L, 4L))
+
+ expect_equal(
+ attributes(df),
+ list(
+ names = letters[1:4],
+ row.names = 1L,
+ top_level = list(field_one = 12, field_two = "more stuff"),
+ class = c("tbl_df", "tbl", "data.frame"))
+ )
+ expect_equal(attributes(df$a), list(class = "special_string"))
+ expect_null(attributes(df$b))
+ expect_equal(
+ attributes(df$c),
+ list(
+ row.names = 1L,
+ names = c("c1", "c2", "c3"),
+ class = c("tbl_df", "tbl", "data.frame")
+ )
+ )
+ expect_null(attributes(df$d))
+})
+
+test_that("reading a known Parquet file to dataframe with 1.0.1", {
+ skip_if_not_available("snappy")
+ pq_file <- test_path("golden-files/data-arrow_1.0.1.parquet")
+
+ df <- read_parquet(pq_file)
+ # 1.0.1 didn't save top-level metadata, so we need to remove it.
+ example_with_metadata_sans_toplevel <- example_with_metadata
+ attributes(example_with_metadata_sans_toplevel)$top_level <- NULL
+ expect_equal(df, example_with_metadata_sans_toplevel)
+ expect_identical(dim(df), c(1L, 4L))
+
+ expect_equal(
+ attributes(df),
+ list(
+ names = letters[1:4],
+ row.names = 1L,
+ class = c("tbl_df", "tbl", "data.frame"))
+ )
+ expect_equal(attributes(df$a), list(class = "special_string"))
+ expect_null(attributes(df$b))
+ expect_equal(
+ attributes(df$c),
+ list(
+ row.names = 1L,
+ names = c("c1", "c2", "c3"),
+ class = c("tbl_df", "tbl", "data.frame")
+ )
+ )
+ expect_null(attributes(df$d))
+})
+
+for (comp in c("lz4", "uncompressed", "zstd")) {
+ # write_feather(example_with_metadata, test_path("golden-files/data-arrow_2.0.0_lz4.feather"), compression = "lz4")
+ # write_feather(example_with_metadata, test_path("golden-files/data-arrow_2.0.0_uncompressed.feather"), compression = "uncompressed")
+ # write_feather(example_with_metadata, test_path("golden-files/data-arrow_2.0.0_zstd.feather"), compression = "zstd")
+ test_that("reading a known Feather file to dataframe with 2.0.0", {
+ skip_if_not_available(comp)
+ feather_file <- test_path(paste0("golden-files/data-arrow_2.0.0_", comp,".feather"))
+
+ df <- read_feather(feather_file)
+ expect_equal(df, example_with_metadata)
+ expect_identical(dim(df), c(1L, 4L))
+
+ expect_equal(
+ attributes(df),
+ list(
+ names = letters[1:4],
+ row.names = 1L,
+ top_level = list(field_one = 12, field_two = "more stuff"),
+ class = c("tbl_df", "tbl", "data.frame"))
+ )
+ expect_equal(attributes(df$a), list(class = "special_string"))
+ expect_null(attributes(df$b))
+ expect_equal(
+ attributes(df$c),
+ list(
+ row.names = 1L,
+ names = c("c1", "c2", "c3"),
+ class = c("tbl_df", "tbl", "data.frame")
+ )
+ )
+ expect_null(attributes(df$d))
+ })
+
+ test_that("reading a known Feather file to dataframe with 1.0.1", {
Review comment:
Thanks for confirming this. I went searching on Friday and hadn't yet fully reconstructed that time log from git history :)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jonkeane commented on pull request #9118: ARROW-10623: [CI][R] Version 1.0.1 breaks data.frame attributes when reading file written by 2.0.0
Posted by GitBox <gi...@apache.org>.
jonkeane commented on pull request #9118:
URL: https://github.com/apache/arrow/pull/9118#issuecomment-758040710
@github-actions crossbow submit test-r-version-compatibility
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] nealrichardson commented on a change in pull request #9118: [WIP] Arrow 10623: [R] Version 1.0.1 breaks data.frame attributes when reading file written by 2.0.0
Posted by GitBox <gi...@apache.org>.
nealrichardson commented on a change in pull request #9118:
URL: https://github.com/apache/arrow/pull/9118#discussion_r553016317
##########
File path: dev/tasks/r/azure.linux.yml
##########
@@ -37,6 +37,7 @@ jobs:
export R_IMAGE={{ r_image }}
export R_TAG={{ r_tag }}
export DEVTOOLSET_VERSION={{ devtoolset_version|default("-1") }}
+ export OLD_ARROW_VERSION={{ old_arrow_version }}
Review comment:
not needed?
##########
File path: r/tests/testthat/test-parquet-compatibility.R
##########
@@ -0,0 +1,45 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# TODO: skip mostly? Or use this for backwards compat?
Review comment:
resolve this. is this for testing that we can read files written by older versions?
##########
File path: r/extra-tests/write-parquet.R
##########
@@ -0,0 +1,23 @@
+library(arrow)
+
+if (!dir.exists("extra-tests/files")) {
+ dir.create("extra-tests/files")
+}
+
Review comment:
could this source tests/testthat/helper-data.R instead of copying the object here? just so these don't get out of sync (if we add more attrs or columns)
##########
File path: r/tests/testthat/test-parquet-compatibility.R
##########
@@ -0,0 +1,45 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# TODO: skip mostly? Or use this for backwards compat?
+
+pq_file <- test_path("parquets/data-arrow2.parquet")
+
+test_that("reading a known Parquet file to dataframe", {
+ df <- read_parquet(pq_file)
+ expect_data_frame(df, data.frame(col1 = 1:10))
+ expect_identical(dim(df), c(10L, 1L))
+
+ tab <- read_parquet(pq_file, as_data_frame = FALSE)
+ expect_s3_class(tab, "Table")
+ expect_equal(
+ # unserialize like .unserialize_arrow_r_metadata does (though we can't call
+ # it directly because it's not exported)
Review comment:
why is that a problem? testthat evals in the package namespace so you can call unexported functions
##########
File path: r/tests/testthat/test-parquet-compatibility.R
##########
@@ -0,0 +1,45 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# TODO: skip mostly? Or use this for backwards compat?
+
+pq_file <- test_path("parquets/data-arrow2.parquet")
+
+test_that("reading a known Parquet file to dataframe", {
+ df <- read_parquet(pq_file)
+ expect_data_frame(df, data.frame(col1 = 1:10))
+ expect_identical(dim(df), c(10L, 1L))
+
+ tab <- read_parquet(pq_file, as_data_frame = FALSE)
+ expect_s3_class(tab, "Table")
+ expect_equal(
+ # unserialize like .unserialize_arrow_r_metadata does (though we can't call
+ # it directly because it's not exported)
+ unserialize(charToRaw(tab$metadata$r)),
+ list(
+ attributes = list(class = "data.frame"),
+ columns = list(col1 = NULL)
+ )
+ )
+
+ # test the workaround
+ tab$metadata$r <- NULL
Review comment:
This workaround is for reading files written by 2.0 in 1.0, right? IIUC that should be tested in the extra-tests
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jonkeane commented on pull request #9118: [WIP] Arrow 10623: [R] Version 1.0.1 breaks data.frame attributes when reading file written by 2.0.0
Posted by GitBox <gi...@apache.org>.
jonkeane commented on pull request #9118:
URL: https://github.com/apache/arrow/pull/9118#issuecomment-755837763
@github-actions crossbow submit test-r-version-compatibility
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #9118: ARROW-10623: [CI][R] Version 1.0.1 breaks data.frame attributes when reading file written by 2.0.0
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #9118:
URL: https://github.com/apache/arrow/pull/9118#issuecomment-758010060
https://issues.apache.org/jira/browse/ARROW-10623
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #9118: [WIP] Arrow 10623: [CI][R] Version 1.0.1 breaks data.frame attributes when reading file written by 2.0.0
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #9118:
URL: https://github.com/apache/arrow/pull/9118#issuecomment-756238454
Revision: 1c58ce2e42fe9b1bf3ace430ca965b52fa079d09
Submitted crossbow builds: [ursa-labs/crossbow @ actions-835](https://github.com/ursa-labs/crossbow/branches/all?query=actions-835)
|Task|Status|
|----|------|
|test-r-version-compatibility|[](https://github.com/ursa-labs/crossbow/actions?query=branch:actions-835-github-test-r-version-compatibility)|
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jorisvandenbossche commented on a change in pull request #9118: ARROW-10623: [CI][R] Version 1.0.1 breaks data.frame attributes when reading file written by 2.0.0
Posted by GitBox <gi...@apache.org>.
jorisvandenbossche commented on a change in pull request #9118:
URL: https://github.com/apache/arrow/pull/9118#discussion_r555074292
##########
File path: r/tests/testthat/test-backwards-compatibility.R
##########
@@ -0,0 +1,159 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# To write a new version of a test file for a current version:
+# write_parquet(example_with_metadata, test_path("golden-files/data-arrow_2.0.0.parquet"))
+
+# To write a new version of a test file for an old version, use docker(-compose)
+# to setup a linux distribution and use RStudio's public package manager binary
+# repo to install the old version. The following commands should be run at the
+# root of the arrow repo directory and might need slight adjusments.
+# R_ORG=rstudio R_IMAGE=r-base R_TAG=4.0-focal docker-compose build --no-cache r
+# R_ORG=rstudio R_IMAGE=r-base R_TAG=4.0-focal docker-compose run r /bin/bash
+# R
+# options(repos = "https://packagemanager.rstudio.com/all/__linux__/focal/latest")
+# remotes::install_version("arrow", version = "1.0.1")
+# # get example data into the global env
+# write_parquet(example_with_metadata, "arrow/r/tests/testthat/golden-files/data-arrow_1.0.1.parquet")
+# quit()/exit
+
+test_that("reading a known Parquet file to dataframe with 2.0.0", {
+ skip_if_not_available("snappy")
+ pq_file <- test_path("golden-files/data-arrow_2.0.0.parquet")
+
+ df <- read_parquet(pq_file)
+ expect_equal(df, example_with_metadata)
+ expect_identical(dim(df), c(1L, 4L))
+
+ expect_equal(
+ attributes(df),
+ list(
+ names = letters[1:4],
+ row.names = 1L,
+ top_level = list(field_one = 12, field_two = "more stuff"),
+ class = c("tbl_df", "tbl", "data.frame"))
+ )
+ expect_equal(attributes(df$a), list(class = "special_string"))
+ expect_null(attributes(df$b))
+ expect_equal(
+ attributes(df$c),
+ list(
+ row.names = 1L,
+ names = c("c1", "c2", "c3"),
+ class = c("tbl_df", "tbl", "data.frame")
+ )
+ )
+ expect_null(attributes(df$d))
+})
+
+test_that("reading a known Parquet file to dataframe with 1.0.1", {
+ skip_if_not_available("snappy")
+ pq_file <- test_path("golden-files/data-arrow_1.0.1.parquet")
+
+ df <- read_parquet(pq_file)
+ # 1.0.1 didn't save top-level metadata, so we need to remove it.
+ example_with_metadata_sans_toplevel <- example_with_metadata
+ attributes(example_with_metadata_sans_toplevel)$top_level <- NULL
+ expect_equal(df, example_with_metadata_sans_toplevel)
+ expect_identical(dim(df), c(1L, 4L))
+
+ expect_equal(
+ attributes(df),
+ list(
+ names = letters[1:4],
+ row.names = 1L,
+ class = c("tbl_df", "tbl", "data.frame"))
+ )
+ expect_equal(attributes(df$a), list(class = "special_string"))
+ expect_null(attributes(df$b))
+ expect_equal(
+ attributes(df$c),
+ list(
+ row.names = 1L,
+ names = c("c1", "c2", "c3"),
+ class = c("tbl_df", "tbl", "data.frame")
+ )
+ )
+ expect_null(attributes(df$d))
+})
+
+for (comp in c("lz4", "uncompressed", "zstd")) {
+ # write_feather(example_with_metadata, test_path("golden-files/data-arrow_2.0.0_lz4.feather"), compression = "lz4")
+ # write_feather(example_with_metadata, test_path("golden-files/data-arrow_2.0.0_uncompressed.feather"), compression = "uncompressed")
+ # write_feather(example_with_metadata, test_path("golden-files/data-arrow_2.0.0_zstd.feather"), compression = "zstd")
+ test_that("reading a known Feather file to dataframe with 2.0.0", {
+ skip_if_not_available(comp)
+ feather_file <- test_path(paste0("golden-files/data-arrow_2.0.0_", comp,".feather"))
+
+ df <- read_feather(feather_file)
+ expect_equal(df, example_with_metadata)
+ expect_identical(dim(df), c(1L, 4L))
+
+ expect_equal(
+ attributes(df),
+ list(
+ names = letters[1:4],
+ row.names = 1L,
+ top_level = list(field_one = 12, field_two = "more stuff"),
+ class = c("tbl_df", "tbl", "data.frame"))
+ )
+ expect_equal(attributes(df$a), list(class = "special_string"))
+ expect_null(attributes(df$b))
+ expect_equal(
+ attributes(df$c),
+ list(
+ row.names = 1L,
+ names = c("c1", "c2", "c3"),
+ class = c("tbl_df", "tbl", "data.frame")
+ )
+ )
+ expect_null(attributes(df$d))
+ })
+
+ test_that("reading a known Feather file to dataframe with 1.0.1", {
Review comment:
Indeed, AFAIK, the bug only occurs for files written with 0.17, and when reading with 2.0.0.
In Arrow 0.17, we had experimental compression support specifically for feather (using some metadata string), while starting from Arrow 1.0, compression is part of the official format spec. So it's only those older pre-1.0.0 files that are the problem. And it only started failing to read them in Arrow 2.0, because we simply changed how the compressions are coded internally from lower case (as it was stored in the feather file) to upper case, in an unrelated commit (https://github.com/apache/arrow/commit/3694794bdfd0677b95b8c95681e392512f1c9237#r43851633), forgetting the compatibility issue with older feather files.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] nealrichardson closed pull request #9118: ARROW-10623: [CI][R] Version 1.0.1 breaks data.frame attributes when reading file written by 2.0.0
Posted by GitBox <gi...@apache.org>.
nealrichardson closed pull request #9118:
URL: https://github.com/apache/arrow/pull/9118
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jonkeane commented on pull request #9118: [WIP] Arrow 10623: [CI][R] Version 1.0.1 breaks data.frame attributes when reading file written by 2.0.0
Posted by GitBox <gi...@apache.org>.
jonkeane commented on pull request #9118:
URL: https://github.com/apache/arrow/pull/9118#issuecomment-756923877
@github-actions crossbow submit test-r-version-compatibility
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jonkeane commented on pull request #9118: [WIP] Arrow 10623: [R] Version 1.0.1 breaks data.frame attributes when reading file written by 2.0.0
Posted by GitBox <gi...@apache.org>.
jonkeane commented on pull request #9118:
URL: https://github.com/apache/arrow/pull/9118#issuecomment-755823111
@github-actions crossbow submit test-r-version-compatibility
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jonkeane commented on pull request #9118: Arrow 10623: [CI][R] Version 1.0.1 breaks data.frame attributes when reading file written by 2.0.0
Posted by GitBox <gi...@apache.org>.
jonkeane commented on pull request #9118:
URL: https://github.com/apache/arrow/pull/9118#issuecomment-756985440
And fork CI passing:
https://github.com/jonkeane/arrow/actions/runs/472646827
https://github.com/jonkeane/arrow/actions/runs/472646829
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jonkeane commented on pull request #9118: [WIP] Arrow 10623: [R] Version 1.0.1 breaks data.frame attributes when reading file written by 2.0.0
Posted by GitBox <gi...@apache.org>.
jonkeane commented on pull request #9118:
URL: https://github.com/apache/arrow/pull/9118#issuecomment-755773625
@github-actions crossbow submit test-r-version-compatibility
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] nealrichardson commented on a change in pull request #9118: Arrow 10623: [CI][R] Version 1.0.1 breaks data.frame attributes when reading file written by 2.0.0
Posted by GitBox <gi...@apache.org>.
nealrichardson commented on a change in pull request #9118:
URL: https://github.com/apache/arrow/pull/9118#discussion_r554211387
##########
File path: r/extra-tests/test-read-files.R
##########
@@ -0,0 +1,165 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+library(arrow)
+library(testthat)
+
+pq_file <- "files/ex_data.parquet"
+
+test_that("Can read the file (parquet)", {
+ # We can read with no error, we assert metadata below
+ expect_error(
+ df <- read_parquet(pq_file),
+ NA
+ )
+})
+
+### Parquet
+test_that("Can see the metadata (feather)", {
+ skip_if_version_less_than("2.0.0", "Version 1.0.1 can't read new version metadata.")
+
+ df <- read_parquet(pq_file)
+ expect_s3_class(df, "tbl")
+
+ expect_equal(
+ attributes(df),
+ list(
+ names = letters[1:4],
+ row.names = 1L,
+ top_level = list(
+ field_one = 12,
+ field_two = "more stuff"
+ ),
+ class = c("tbl_df", "tbl", "data.frame")
+ )
+ )
+
+ # column-level attributes
+ expect_equal(attributes(df$a), list(class = "special_string"))
+ expect_equal(
+ attributes(df$c),
+ list(
+ row.names = 1L,
+ names = c("c1", "c2", "c3"),
+ class = c("tbl_df", "tbl", "data.frame")
+ )
+ )
+})
+
+### Feather
+for (comp in c("lz4", "uncompressed", "zstd")) {
+ feather_file <- paste0("files/ex_data_", comp, ".feather")
+
+ test_that(paste0("Can read the file (feather ", comp, ")"), {
+ # We can read with no error, we assert metadata below
+ expect_error(
+ df <- read_feather(feather_file),
+ NA
+ )
+ })
+
+ test_that(paste0("Can see the metadata (feather ", comp, ")"), {
+ skip_if_version_less_than("2.0.0", "Version 1.0.1 can't read new version metadata.")
+
+ df <- read_feather(feather_file)
+ expect_s3_class(df, "tbl")
+
+ expect_equal(
+ attributes(df),
+ list(
+ names = letters[1:4],
+ row.names = 1L,
+ top_level = list(
+ field_one = 12,
+ field_two = "more stuff"
+ ),
+ class = c("tbl_df", "tbl", "data.frame")
+ )
+ )
+
+ # column-level attributes
+ expect_equal(attributes(df$a), list(class = "special_string"))
+ expect_equal(
+ attributes(df$c),
+ list(
+ row.names = 1L,
+ names = c("c1", "c2", "c3"),
+ class = c("tbl_df", "tbl", "data.frame")
+ )
+ )
+ })
+}
+
+test_that(paste0("Can read feather version 1"), {
Review comment:
```suggestion
test_that("Can read feather version 1", {
```
##########
File path: r/extra-tests/test-read-files.R
##########
@@ -0,0 +1,165 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+library(arrow)
+library(testthat)
+
+pq_file <- "files/ex_data.parquet"
+
+test_that("Can read the file (parquet)", {
+ # We can read with no error, we assert metadata below
+ expect_error(
+ df <- read_parquet(pq_file),
+ NA
+ )
+})
+
+### Parquet
+test_that("Can see the metadata (feather)", {
Review comment:
```suggestion
test_that("Can see the metadata (parquet)", {
```
?
##########
File path: r/tests/testthat/test-backwards-compatibility.R
##########
@@ -0,0 +1,159 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# To write a new version of a test file for a current version:
+# write_parquet(example_with_metadata, test_path("golden-files/data-arrow_2.0.0.parquet"))
+
+# To write a new version of a test file for an old version, use docker(-compose)
+# to setup a linux distribution and use RStudio's public package manager binary
+# repo to install the old version. The following commands should be run at the
+# root of the arrow repo directory and might need slight adjusments.
+# R_ORG=rstudio R_IMAGE=r-base R_TAG=4.0-focal docker-compose build --no-cache r
+# R_ORG=rstudio R_IMAGE=r-base R_TAG=4.0-focal docker-compose run r /bin/bash
+# R
+# options(repos = "https://packagemanager.rstudio.com/all/__linux__/focal/latest")
+# remotes::install_version("arrow", version = "1.0.1")
+# # get example data into the global env
+# write_parquet(example_with_metadata, "arrow/r/tests/testthat/golden-files/data-arrow_1.0.1.parquet")
+# quit()/exit
+
+test_that("reading a known Parquet file to dataframe with 2.0.0", {
+ skip_if_not_available("snappy")
+ pq_file <- test_path("golden-files/data-arrow_2.0.0.parquet")
+
+ df <- read_parquet(pq_file)
+ expect_equal(df, example_with_metadata)
+ expect_identical(dim(df), c(1L, 4L))
+
+ expect_equal(
+ attributes(df),
+ list(
+ names = letters[1:4],
+ row.names = 1L,
+ top_level = list(field_one = 12, field_two = "more stuff"),
+ class = c("tbl_df", "tbl", "data.frame"))
+ )
+ expect_equal(attributes(df$a), list(class = "special_string"))
+ expect_null(attributes(df$b))
+ expect_equal(
+ attributes(df$c),
+ list(
+ row.names = 1L,
+ names = c("c1", "c2", "c3"),
+ class = c("tbl_df", "tbl", "data.frame")
+ )
+ )
+ expect_null(attributes(df$d))
+})
+
+test_that("reading a known Parquet file to dataframe with 1.0.1", {
+ skip_if_not_available("snappy")
+ pq_file <- test_path("golden-files/data-arrow_1.0.1.parquet")
+
+ df <- read_parquet(pq_file)
+ # 1.0.1 didn't save top-level metadata, so we need to remove it.
+ example_with_metadata_sans_toplevel <- example_with_metadata
+ attributes(example_with_metadata_sans_toplevel)$top_level <- NULL
+ expect_equal(df, example_with_metadata_sans_toplevel)
+ expect_identical(dim(df), c(1L, 4L))
+
+ expect_equal(
+ attributes(df),
+ list(
+ names = letters[1:4],
+ row.names = 1L,
+ class = c("tbl_df", "tbl", "data.frame"))
+ )
+ expect_equal(attributes(df$a), list(class = "special_string"))
+ expect_null(attributes(df$b))
+ expect_equal(
+ attributes(df$c),
+ list(
+ row.names = 1L,
+ names = c("c1", "c2", "c3"),
+ class = c("tbl_df", "tbl", "data.frame")
+ )
+ )
+ expect_null(attributes(df$d))
+})
+
+for (comp in c("lz4", "uncompressed", "zstd")) {
+ # write_feather(example_with_metadata, test_path("golden-files/data-arrow_2.0.0_lz4.feather"), compression = "lz4")
+ # write_feather(example_with_metadata, test_path("golden-files/data-arrow_2.0.0_uncompressed.feather"), compression = "uncompressed")
+ # write_feather(example_with_metadata, test_path("golden-files/data-arrow_2.0.0_zstd.feather"), compression = "zstd")
+ test_that("reading a known Feather file to dataframe with 2.0.0", {
+ skip_if_not_available(comp)
+ feather_file <- test_path(paste0("golden-files/data-arrow_2.0.0_", comp,".feather"))
+
+ df <- read_feather(feather_file)
+ expect_equal(df, example_with_metadata)
+ expect_identical(dim(df), c(1L, 4L))
+
+ expect_equal(
+ attributes(df),
+ list(
+ names = letters[1:4],
+ row.names = 1L,
+ top_level = list(field_one = 12, field_two = "more stuff"),
+ class = c("tbl_df", "tbl", "data.frame"))
+ )
+ expect_equal(attributes(df$a), list(class = "special_string"))
+ expect_null(attributes(df$b))
+ expect_equal(
+ attributes(df$c),
+ list(
+ row.names = 1L,
+ names = c("c1", "c2", "c3"),
+ class = c("tbl_df", "tbl", "data.frame")
+ )
+ )
+ expect_null(attributes(df$d))
+ })
+
+ test_that("reading a known Feather file to dataframe with 1.0.1", {
Review comment:
Shouldn't this be failing on master per ARROW-10850?
##########
File path: r/extra-tests/helpers.R
##########
@@ -0,0 +1,32 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+if_version <- function(version, op = `==`) {
+ op(numeric_version(Sys.getenv("OLD_ARROW_VERSION", "0.0.0")), version)
Review comment:
why not `packageVersion("arrow")`?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #9118: [WIP] Arrow 10623: [CI][R] Version 1.0.1 breaks data.frame attributes when reading file written by 2.0.0
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #9118:
URL: https://github.com/apache/arrow/pull/9118#issuecomment-756274383
Revision: 1c58ce2e42fe9b1bf3ace430ca965b52fa079d09
Submitted crossbow builds: [ursa-labs/crossbow @ actions-836](https://github.com/ursa-labs/crossbow/branches/all?query=actions-836)
|Task|Status|
|----|------|
|test-r-version-compatibility|[](https://github.com/ursa-labs/crossbow/actions?query=branch:actions-836-github-test-r-version-compatibility)|
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jonkeane commented on pull request #9118: [WIP] Arrow 10623: [CI][R] Version 1.0.1 breaks data.frame attributes when reading file written by 2.0.0
Posted by GitBox <gi...@apache.org>.
jonkeane commented on pull request #9118:
URL: https://github.com/apache/arrow/pull/9118#issuecomment-756273669
@github-actions crossbow submit test-r-version-compatibility
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jonkeane commented on a change in pull request #9118: Arrow 10623: [CI][R] Version 1.0.1 breaks data.frame attributes when reading file written by 2.0.0
Posted by GitBox <gi...@apache.org>.
jonkeane commented on a change in pull request #9118:
URL: https://github.com/apache/arrow/pull/9118#discussion_r554263558
##########
File path: r/tests/testthat/test-backwards-compatibility.R
##########
@@ -0,0 +1,159 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# To write a new version of a test file for a current version:
+# write_parquet(example_with_metadata, test_path("golden-files/data-arrow_2.0.0.parquet"))
+
+# To write a new version of a test file for an old version, use docker(-compose)
+# to setup a linux distribution and use RStudio's public package manager binary
+# repo to install the old version. The following commands should be run at the
+# root of the arrow repo directory and might need slight adjusments.
+# R_ORG=rstudio R_IMAGE=r-base R_TAG=4.0-focal docker-compose build --no-cache r
+# R_ORG=rstudio R_IMAGE=r-base R_TAG=4.0-focal docker-compose run r /bin/bash
+# R
+# options(repos = "https://packagemanager.rstudio.com/all/__linux__/focal/latest")
+# remotes::install_version("arrow", version = "1.0.1")
+# # get example data into the global env
+# write_parquet(example_with_metadata, "arrow/r/tests/testthat/golden-files/data-arrow_1.0.1.parquet")
+# quit()/exit
+
+test_that("reading a known Parquet file to dataframe with 2.0.0", {
+ skip_if_not_available("snappy")
+ pq_file <- test_path("golden-files/data-arrow_2.0.0.parquet")
+
+ df <- read_parquet(pq_file)
+ expect_equal(df, example_with_metadata)
+ expect_identical(dim(df), c(1L, 4L))
+
+ expect_equal(
+ attributes(df),
+ list(
+ names = letters[1:4],
+ row.names = 1L,
+ top_level = list(field_one = 12, field_two = "more stuff"),
+ class = c("tbl_df", "tbl", "data.frame"))
+ )
+ expect_equal(attributes(df$a), list(class = "special_string"))
+ expect_null(attributes(df$b))
+ expect_equal(
+ attributes(df$c),
+ list(
+ row.names = 1L,
+ names = c("c1", "c2", "c3"),
+ class = c("tbl_df", "tbl", "data.frame")
+ )
+ )
+ expect_null(attributes(df$d))
+})
+
+test_that("reading a known Parquet file to dataframe with 1.0.1", {
+ skip_if_not_available("snappy")
+ pq_file <- test_path("golden-files/data-arrow_1.0.1.parquet")
+
+ df <- read_parquet(pq_file)
+ # 1.0.1 didn't save top-level metadata, so we need to remove it.
+ example_with_metadata_sans_toplevel <- example_with_metadata
+ attributes(example_with_metadata_sans_toplevel)$top_level <- NULL
+ expect_equal(df, example_with_metadata_sans_toplevel)
+ expect_identical(dim(df), c(1L, 4L))
+
+ expect_equal(
+ attributes(df),
+ list(
+ names = letters[1:4],
+ row.names = 1L,
+ class = c("tbl_df", "tbl", "data.frame"))
+ )
+ expect_equal(attributes(df$a), list(class = "special_string"))
+ expect_null(attributes(df$b))
+ expect_equal(
+ attributes(df$c),
+ list(
+ row.names = 1L,
+ names = c("c1", "c2", "c3"),
+ class = c("tbl_df", "tbl", "data.frame")
+ )
+ )
+ expect_null(attributes(df$d))
+})
+
+for (comp in c("lz4", "uncompressed", "zstd")) {
+ # write_feather(example_with_metadata, test_path("golden-files/data-arrow_2.0.0_lz4.feather"), compression = "lz4")
+ # write_feather(example_with_metadata, test_path("golden-files/data-arrow_2.0.0_uncompressed.feather"), compression = "uncompressed")
+ # write_feather(example_with_metadata, test_path("golden-files/data-arrow_2.0.0_zstd.feather"), compression = "zstd")
+ test_that("reading a known Feather file to dataframe with 2.0.0", {
+ skip_if_not_available(comp)
+ feather_file <- test_path(paste0("golden-files/data-arrow_2.0.0_", comp,".feather"))
+
+ df <- read_feather(feather_file)
+ expect_equal(df, example_with_metadata)
+ expect_identical(dim(df), c(1L, 4L))
+
+ expect_equal(
+ attributes(df),
+ list(
+ names = letters[1:4],
+ row.names = 1L,
+ top_level = list(field_one = 12, field_two = "more stuff"),
+ class = c("tbl_df", "tbl", "data.frame"))
+ )
+ expect_equal(attributes(df$a), list(class = "special_string"))
+ expect_null(attributes(df$b))
+ expect_equal(
+ attributes(df$c),
+ list(
+ row.names = 1L,
+ names = c("c1", "c2", "c3"),
+ class = c("tbl_df", "tbl", "data.frame")
+ )
+ )
+ expect_null(attributes(df$d))
+ })
+
+ test_that("reading a known Feather file to dataframe with 1.0.1", {
Review comment:
It looks like 1.0.1 works fine, but earlier versions (e.g. 0.17.0) saves files with the wrong case and does fail, but 1.0.1 (and 1.0.0) doesn't seem to have this issue. I need to dig more to inspect the files saved by 1.0.1 / through the source to find if there was a `tolower()` or the like that was introduced after 0.17
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #9118: ARROW-10623: [CI][R] Version 1.0.1 breaks data.frame attributes when reading file written by 2.0.0
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #9118:
URL: https://github.com/apache/arrow/pull/9118#issuecomment-758120287
Revision: 5c9fe3fefe69cc5f5af1179897ad64027274762c
Submitted crossbow builds: [ursa-labs/crossbow @ actions-870](https://github.com/ursa-labs/crossbow/branches/all?query=actions-870)
|Task|Status|
|----|------|
|test-r-version-compatibility|[](https://github.com/ursa-labs/crossbow/actions?query=branch:actions-870-github-test-r-version-compatibility)|
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jonkeane commented on pull request #9118: ARROW-10623: [CI][R] Version 1.0.1 breaks data.frame attributes when reading file written by 2.0.0
Posted by GitBox <gi...@apache.org>.
jonkeane commented on pull request #9118:
URL: https://github.com/apache/arrow/pull/9118#issuecomment-758082292
I'll consolidate those tests some more like you mentioned. The only thing left to do (if we want to) would be to add backwards compatibility tests for stream-to-filesystem. I can add them if they add substantially more coverage than we have with feather files themselves.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] nealrichardson commented on a change in pull request #9118: [WIP] Arrow 10623: [CI][R] Version 1.0.1 breaks data.frame attributes when reading file written by 2.0.0
Posted by GitBox <gi...@apache.org>.
nealrichardson commented on a change in pull request #9118:
URL: https://github.com/apache/arrow/pull/9118#discussion_r553530478
##########
File path: r/extra-tests/test-read-parquet.R
##########
@@ -0,0 +1,50 @@
+library(arrow)
+library(testthat)
+
+skip_if_version <- function(version, msg, op = `<=`) {
Review comment:
This `op` is a little non-obvious. I'd rather have explicit skip function names (skip_if_version_equal, skip_if_version_less_than, etc.)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jonkeane commented on pull request #9118: [WIP] Arrow 10623: [CI][R] Version 1.0.1 breaks data.frame attributes when reading file written by 2.0.0
Posted by GitBox <gi...@apache.org>.
jonkeane commented on pull request #9118:
URL: https://github.com/apache/arrow/pull/9118#issuecomment-756232737
@github-actions crossbow submit test-r-version-compatibility
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #9118: [WIP] Arrow 10623: [CI][R] Version 1.0.1 breaks data.frame attributes when reading file written by 2.0.0
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #9118:
URL: https://github.com/apache/arrow/pull/9118#issuecomment-756422750
Revision: 0cc1d259342f30c1e353f2680b7778c7c89d1f4e
Submitted crossbow builds: [ursa-labs/crossbow @ actions-840](https://github.com/ursa-labs/crossbow/branches/all?query=actions-840)
|Task|Status|
|----|------|
|test-r-version-compatibility|[](https://github.com/ursa-labs/crossbow/actions?query=branch:actions-840-github-test-r-version-compatibility)|
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jonkeane commented on pull request #9118: [WIP] Arrow 10623: [CI][R] Version 1.0.1 breaks data.frame attributes when reading file written by 2.0.0
Posted by GitBox <gi...@apache.org>.
jonkeane commented on pull request #9118:
URL: https://github.com/apache/arrow/pull/9118#issuecomment-756315837
@github-actions crossbow submit test-r-version-compatibility
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jonkeane commented on pull request #9118: [WIP] Arrow 10623: [CI][R] Version 1.0.1 breaks data.frame attributes when reading file written by 2.0.0
Posted by GitBox <gi...@apache.org>.
jonkeane commented on pull request #9118:
URL: https://github.com/apache/arrow/pull/9118#issuecomment-756422218
@github-actions crossbow submit test-r-version-compatibility
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jonkeane commented on a change in pull request #9118: Arrow 10623: [CI][R] Version 1.0.1 breaks data.frame attributes when reading file written by 2.0.0
Posted by GitBox <gi...@apache.org>.
jonkeane commented on a change in pull request #9118:
URL: https://github.com/apache/arrow/pull/9118#discussion_r554263558
##########
File path: r/tests/testthat/test-backwards-compatibility.R
##########
@@ -0,0 +1,159 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# To write a new version of a test file for a current version:
+# write_parquet(example_with_metadata, test_path("golden-files/data-arrow_2.0.0.parquet"))
+
+# To write a new version of a test file for an old version, use docker(-compose)
+# to setup a linux distribution and use RStudio's public package manager binary
+# repo to install the old version. The following commands should be run at the
+# root of the arrow repo directory and might need slight adjusments.
+# R_ORG=rstudio R_IMAGE=r-base R_TAG=4.0-focal docker-compose build --no-cache r
+# R_ORG=rstudio R_IMAGE=r-base R_TAG=4.0-focal docker-compose run r /bin/bash
+# R
+# options(repos = "https://packagemanager.rstudio.com/all/__linux__/focal/latest")
+# remotes::install_version("arrow", version = "1.0.1")
+# # get example data into the global env
+# write_parquet(example_with_metadata, "arrow/r/tests/testthat/golden-files/data-arrow_1.0.1.parquet")
+# quit()/exit
+
+test_that("reading a known Parquet file to dataframe with 2.0.0", {
+ skip_if_not_available("snappy")
+ pq_file <- test_path("golden-files/data-arrow_2.0.0.parquet")
+
+ df <- read_parquet(pq_file)
+ expect_equal(df, example_with_metadata)
+ expect_identical(dim(df), c(1L, 4L))
+
+ expect_equal(
+ attributes(df),
+ list(
+ names = letters[1:4],
+ row.names = 1L,
+ top_level = list(field_one = 12, field_two = "more stuff"),
+ class = c("tbl_df", "tbl", "data.frame"))
+ )
+ expect_equal(attributes(df$a), list(class = "special_string"))
+ expect_null(attributes(df$b))
+ expect_equal(
+ attributes(df$c),
+ list(
+ row.names = 1L,
+ names = c("c1", "c2", "c3"),
+ class = c("tbl_df", "tbl", "data.frame")
+ )
+ )
+ expect_null(attributes(df$d))
+})
+
+test_that("reading a known Parquet file to dataframe with 1.0.1", {
+ skip_if_not_available("snappy")
+ pq_file <- test_path("golden-files/data-arrow_1.0.1.parquet")
+
+ df <- read_parquet(pq_file)
+ # 1.0.1 didn't save top-level metadata, so we need to remove it.
+ example_with_metadata_sans_toplevel <- example_with_metadata
+ attributes(example_with_metadata_sans_toplevel)$top_level <- NULL
+ expect_equal(df, example_with_metadata_sans_toplevel)
+ expect_identical(dim(df), c(1L, 4L))
+
+ expect_equal(
+ attributes(df),
+ list(
+ names = letters[1:4],
+ row.names = 1L,
+ class = c("tbl_df", "tbl", "data.frame"))
+ )
+ expect_equal(attributes(df$a), list(class = "special_string"))
+ expect_null(attributes(df$b))
+ expect_equal(
+ attributes(df$c),
+ list(
+ row.names = 1L,
+ names = c("c1", "c2", "c3"),
+ class = c("tbl_df", "tbl", "data.frame")
+ )
+ )
+ expect_null(attributes(df$d))
+})
+
+for (comp in c("lz4", "uncompressed", "zstd")) {
+ # write_feather(example_with_metadata, test_path("golden-files/data-arrow_2.0.0_lz4.feather"), compression = "lz4")
+ # write_feather(example_with_metadata, test_path("golden-files/data-arrow_2.0.0_uncompressed.feather"), compression = "uncompressed")
+ # write_feather(example_with_metadata, test_path("golden-files/data-arrow_2.0.0_zstd.feather"), compression = "zstd")
+ test_that("reading a known Feather file to dataframe with 2.0.0", {
+ skip_if_not_available(comp)
+ feather_file <- test_path(paste0("golden-files/data-arrow_2.0.0_", comp,".feather"))
+
+ df <- read_feather(feather_file)
+ expect_equal(df, example_with_metadata)
+ expect_identical(dim(df), c(1L, 4L))
+
+ expect_equal(
+ attributes(df),
+ list(
+ names = letters[1:4],
+ row.names = 1L,
+ top_level = list(field_one = 12, field_two = "more stuff"),
+ class = c("tbl_df", "tbl", "data.frame"))
+ )
+ expect_equal(attributes(df$a), list(class = "special_string"))
+ expect_null(attributes(df$b))
+ expect_equal(
+ attributes(df$c),
+ list(
+ row.names = 1L,
+ names = c("c1", "c2", "c3"),
+ class = c("tbl_df", "tbl", "data.frame")
+ )
+ )
+ expect_null(attributes(df$d))
+ })
+
+ test_that("reading a known Feather file to dataframe with 1.0.1", {
Review comment:
It looks like 1.0.1 works fine, but earlier versions (e.g. 0.17.0) saves files with the wrong case and does fail, but 1.0.1 (and 1.0.0) doesn't seem to have this issue. I need to dig more to inspect the files saved by 1.0.1 / through the source to find if there was a `tolower()` or the like that was introduced after 0.17
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #9118: [WIP] Arrow 10623: [CI][R] Version 1.0.1 breaks data.frame attributes when reading file written by 2.0.0
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #9118:
URL: https://github.com/apache/arrow/pull/9118#issuecomment-756436048
Revision: 2d9cdf3042b2a2763a5d0e0f0d67685dc3def499
Submitted crossbow builds: [ursa-labs/crossbow @ actions-841](https://github.com/ursa-labs/crossbow/branches/all?query=actions-841)
|Task|Status|
|----|------|
|test-r-version-compatibility|[](https://github.com/ursa-labs/crossbow/actions?query=branch:actions-841-github-test-r-version-compatibility)|
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jonkeane commented on a change in pull request #9118: ARROW-10623: [CI][R] Version 1.0.1 breaks data.frame attributes when reading file written by 2.0.0
Posted by GitBox <gi...@apache.org>.
jonkeane commented on a change in pull request #9118:
URL: https://github.com/apache/arrow/pull/9118#discussion_r555125913
##########
File path: r/tests/testthat/test-backwards-compatibility.R
##########
@@ -0,0 +1,171 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# To write a new version of a test file for a current version:
+# write_parquet(example_with_metadata, test_path("golden-files/data-arrow_2.0.0.parquet"))
+
+# To write a new version of a test file for an old version, use docker(-compose)
+# to setup a linux distribution and use RStudio's public package manager binary
+# repo to install the old version. The following commands should be run at the
+# root of the arrow repo directory and might need slight adjusments.
+# R_ORG=rstudio R_IMAGE=r-base R_TAG=4.0-focal docker-compose build --no-cache r
+# R_ORG=rstudio R_IMAGE=r-base R_TAG=4.0-focal docker-compose run r /bin/bash
+# R
+# options(repos = "https://packagemanager.rstudio.com/all/__linux__/focal/latest")
+# remotes::install_version("arrow", version = "1.0.1")
+# # get example data into the global env
+# write_parquet(example_with_metadata, "arrow/r/tests/testthat/golden-files/data-arrow_1.0.1.parquet")
+# quit()/exit
+
+test_that("reading a known Parquet file to dataframe with 2.0.0", {
+ skip_if_not_available("snappy")
+ pq_file <- test_path("golden-files/data-arrow_2.0.0.parquet")
+
+ df <- read_parquet(pq_file)
+ expect_equal(df, example_with_metadata)
+ expect_identical(dim(df), c(1L, 4L))
+
+ expect_equal(
+ attributes(df),
+ list(
+ names = letters[1:4],
+ row.names = 1L,
+ top_level = list(field_one = 12, field_two = "more stuff"),
+ class = c("tbl_df", "tbl", "data.frame"))
+ )
+ expect_equal(attributes(df$a), list(class = "special_string"))
+ expect_null(attributes(df$b))
+ expect_equal(
+ attributes(df$c),
+ list(
+ row.names = 1L,
+ names = c("c1", "c2", "c3"),
+ class = c("tbl_df", "tbl", "data.frame")
+ )
+ )
+ expect_null(attributes(df$d))
+})
+
+test_that("reading a known Parquet file to dataframe with 1.0.1", {
+ skip_if_not_available("snappy")
+ pq_file <- test_path("golden-files/data-arrow_1.0.1.parquet")
+
+ df <- read_parquet(pq_file)
+ # 1.0.1 didn't save top-level metadata, so we need to remove it.
+ example_with_metadata_sans_toplevel <- example_with_metadata
+ attributes(example_with_metadata_sans_toplevel)$top_level <- NULL
+ expect_equal(df, example_with_metadata_sans_toplevel)
+ expect_identical(dim(df), c(1L, 4L))
+
+ expect_equal(
+ attributes(df),
+ list(
+ names = letters[1:4],
+ row.names = 1L,
+ class = c("tbl_df", "tbl", "data.frame"))
+ )
+ expect_equal(attributes(df$a), list(class = "special_string"))
+ expect_null(attributes(df$b))
+ expect_equal(
+ attributes(df$c),
+ list(
+ row.names = 1L,
+ names = c("c1", "c2", "c3"),
+ class = c("tbl_df", "tbl", "data.frame")
+ )
+ )
+ expect_null(attributes(df$d))
+})
+
+for (comp in c("lz4", "uncompressed", "zstd")) {
+ # write_feather(example_with_metadata, test_path("golden-files/data-arrow_2.0.0_lz4.feather"), compression = "lz4")
+ # write_feather(example_with_metadata, test_path("golden-files/data-arrow_2.0.0_uncompressed.feather"), compression = "uncompressed")
+ # write_feather(example_with_metadata, test_path("golden-files/data-arrow_2.0.0_zstd.feather"), compression = "zstd")
+ test_that("reading a known Feather file to dataframe with 2.0.0", {
+ skip_if_not_available(comp)
+ feather_file <- test_path(paste0("golden-files/data-arrow_2.0.0_", comp,".feather"))
+
+ df <- read_feather(feather_file)
+ expect_equal(df, example_with_metadata)
+ expect_identical(dim(df), c(1L, 4L))
+
+ expect_equal(
+ attributes(df),
+ list(
+ names = letters[1:4],
+ row.names = 1L,
+ top_level = list(field_one = 12, field_two = "more stuff"),
+ class = c("tbl_df", "tbl", "data.frame"))
+ )
+ expect_equal(attributes(df$a), list(class = "special_string"))
+ expect_null(attributes(df$b))
+ expect_equal(
+ attributes(df$c),
+ list(
+ row.names = 1L,
+ names = c("c1", "c2", "c3"),
+ class = c("tbl_df", "tbl", "data.frame")
+ )
+ )
+ expect_null(attributes(df$d))
+ })
+
+ test_that("reading a known Feather file to dataframe with 1.0.1", {
+ skip_if_not_available(comp)
+ feather_file <- test_path(paste0("golden-files/data-arrow_1.0.1_", comp,".feather"))
+
+ df <- read_feather(feather_file)
+ # 1.0.1 didn't save top-level metadata, so we need to remove it.
+ example_with_metadata_sans_toplevel <- example_with_metadata
+ attributes(example_with_metadata_sans_toplevel)$top_level <- NULL
+ expect_equal(df, example_with_metadata_sans_toplevel)
+ expect_identical(dim(df), c(1L, 4L))
+
+ expect_equal(
+ attributes(df),
+ list(
+ names = letters[1:4],
+ row.names = 1L,
+ class = c("tbl_df", "tbl", "data.frame"))
+ )
+ expect_equal(attributes(df$a), list(class = "special_string"))
+ expect_null(attributes(df$b))
+ expect_equal(
+ attributes(df$c),
+ list(
+ row.names = 1L,
+ names = c("c1", "c2", "c3"),
+ class = c("tbl_df", "tbl", "data.frame")
+ )
+ )
+ expect_null(attributes(df$d))
+ })
+
+ test_that("reading a known Feather file to dataframe with 0.17.0", {
+ skip_if_not_available(comp)
+ feather_file <- test_path(paste0("golden-files/data-arrow_0.17.0_", comp,".feather"))
+
+ if (comp %in% c("lz4", "zstd")) {
+ # there is a case mis-match with versions 0.17.0 and before for the codec names
+ expect_error(df <- read_feather(feather_file), "Unrecognized compression type:")
Review comment:
Great, I'll change these to skips and than add the Jira to the message then.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jonkeane commented on pull request #9118: [WIP] Arrow 10623: [CI][R] Version 1.0.1 breaks data.frame attributes when reading file written by 2.0.0
Posted by GitBox <gi...@apache.org>.
jonkeane commented on pull request #9118:
URL: https://github.com/apache/arrow/pull/9118#issuecomment-756361710
@github-actions crossbow submit test-r-version-compatibility
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #9118: [WIP] Arrow 10623: [CI][R] Version 1.0.1 breaks data.frame attributes when reading file written by 2.0.0
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #9118:
URL: https://github.com/apache/arrow/pull/9118#issuecomment-756941897
Revision: da48c83b1587e49b92c074096bae7e74db2e712f
Submitted crossbow builds: [ursa-labs/crossbow @ actions-848](https://github.com/ursa-labs/crossbow/branches/all?query=actions-848)
|Task|Status|
|----|------|
|test-r-version-compatibility|[](https://github.com/ursa-labs/crossbow/actions?query=branch:actions-848-github-test-r-version-compatibility)|
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org