You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "jonkeane (via GitHub)" <gi...@apache.org> on 2024/08/10 04:53:28 UTC

[PR] GH-43633: [R] Add tests for packages that might be tricky to roundtrip data to Tables + Parquet files [arrow]

jonkeane opened a new pull request, #43634:
URL: https://github.com/apache/arrow/pull/43634

   ### Rationale for this change
   
   Add coverage for objects that might have issues roundtripping to Arrow Tables or Parquet files 
   
   ### What changes are included in this PR?
   
   A new test file + a crossbow job that ensures these other packages are installed so the tests run.
   
   ### Are these changes tested?
   
   The changes are tests
   
   ### Are there any user-facing changes?
   
   No


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-43633: [R] Add tests for packages that might be tricky to roundtrip data to Tables + Parquet files [arrow]

Posted by "amoeba (via GitHub)" <gi...@apache.org>.
amoeba commented on code in PR #43634:
URL: https://github.com/apache/arrow/pull/43634#discussion_r1714503982


##########
r/tests/testthat/test-extra-package-roundtrip.R:
##########
@@ -0,0 +1,99 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+skip_on_cran()
+
+# Any additional pacakge that we test here that is not already in DESCRIPTION should be

Review Comment:
   ```suggestion
   # Any additional package that we test here that is not already in DESCRIPTION should be
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-43633: [R] Add tests for packages that might be tricky to roundtrip data to Tables + Parquet files [arrow]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #43634:
URL: https://github.com/apache/arrow/pull/43634#issuecomment-2282228436

   Revision: 319184f20c07e40da08c8d3f110e004f2c65a3df
   
   Submitted crossbow builds: [ursacomputing/crossbow @ actions-eceae40aa6](https://github.com/ursacomputing/crossbow/branches/all?query=actions-eceae40aa6)
   
   |Task|Status|
   |----|------|
   |test-r-extra-packages|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-eceae40aa6-github-test-r-extra-packages)](https://github.com/ursacomputing/crossbow/actions/runs/10333456439/job/28605826658)|


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-43633: [R] Add tests for packages that might be tricky to roundtrip data to Tables + Parquet files [arrow]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #43634:
URL: https://github.com/apache/arrow/pull/43634#issuecomment-2282339251

   Revision: a8aae5528c22ce3e674526dea7ec240b2dc53da9
   
   Submitted crossbow builds: [ursacomputing/crossbow @ actions-3e81433a91](https://github.com/ursacomputing/crossbow/branches/all?query=actions-3e81433a91)
   
   |Task|Status|
   |----|------|
   |test-r-extra-packages|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-3e81433a91-github-test-r-extra-packages)](https://github.com/ursacomputing/crossbow/actions/runs/10335890900/job/28611003100)|


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-43633: [R] Add tests for packages that might be tricky to roundtrip data to Tables + Parquet files [arrow]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #43634:
URL: https://github.com/apache/arrow/pull/43634#issuecomment-2282341312

   Revision: 32adc4aa8064dc4b685b1dc3fe0dfc6ec2f4e3d9
   
   Submitted crossbow builds: [ursacomputing/crossbow @ actions-5adf6b8636](https://github.com/ursacomputing/crossbow/branches/all?query=actions-5adf6b8636)
   
   |Task|Status|
   |----|------|
   |test-r-extra-packages|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-5adf6b8636-github-test-r-extra-packages)](https://github.com/ursacomputing/crossbow/actions/runs/10335951676/job/28611126145)|


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-43633: [R] Add tests for packages that might be tricky to roundtrip data to Tables + Parquet files [arrow]

Posted by "MichaelChirico (via GitHub)" <gi...@apache.org>.
MichaelChirico commented on code in PR #43634:
URL: https://github.com/apache/arrow/pull/43634#discussion_r1713293590


##########
r/tests/testthat/test-extra-package-roundtrip.R:
##########
@@ -65,6 +65,9 @@ test_that("readr read csvs roundtrip", {
 test_that("data.table objects roundtrip", {
   load_or_skip("data.table")
 
+  # https://rdatatable.gitlab.io/data.table/articles/datatable-importing.html#testing-using-testthat
+  .datatable.aware=TRUE

Review Comment:
   This works, but note the linked advice is for this to show up in R/, not here in tests/
   
   The advice you're following is here:
   
   https://github.com/Rdatatable/data.table/blob/83fd2c05ce2d8555ceb8ba417833956b1b574f7e/R/cedta.R#L25-L27



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-43633: [R] Add tests for packages that might be tricky to roundtrip data to Tables + Parquet files [arrow]

Posted by "jonkeane (via GitHub)" <gi...@apache.org>.
jonkeane commented on PR #43634:
URL: https://github.com/apache/arrow/pull/43634#issuecomment-2295258201

   Thanks for this reprex! Would you mind opening a new issue here (I'm also happy to copy / paste, but want to give you github points for reporting this).
   
   I've poked at this a bit and I don't believe that this is new or a regression, but I still would love to see if there's anything we can do to make this smoother. The metadata in the example is both large and in the information-theoretic worst case scenario for compression, so we don't get much compression out of it at all. But we might be able to do something else.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-43633: [R] Add tests for packages that might be tricky to roundtrip data to Tables + Parquet files [arrow]

Posted by "jonkeane (via GitHub)" <gi...@apache.org>.
jonkeane commented on PR #43634:
URL: https://github.com/apache/arrow/pull/43634#issuecomment-2294323522

   @github-actions crossbow submit test-r-extra-packages


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-43633: [R] Add tests for packages that might be tricky to roundtrip data to Tables + Parquet files [arrow]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #43634:
URL: https://github.com/apache/arrow/pull/43634#issuecomment-2290641495

   Revision: 4403b59536b8d8c9ca506c42d9ebb7cd4f853d76
   
   Submitted crossbow builds: [ursacomputing/crossbow @ actions-efb8875bbb](https://github.com/ursacomputing/crossbow/branches/all?query=actions-efb8875bbb)
   
   |Task|Status|
   |----|------|
   |test-r-extra-packages|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-efb8875bbb-github-test-r-extra-packages)](https://github.com/ursacomputing/crossbow/actions/runs/10399153429/job/28797621280)|


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-43633: [R] Add tests for packages that might be tricky to roundtrip data to Tables + Parquet files [arrow]

Posted by "jonkeane (via GitHub)" <gi...@apache.org>.
jonkeane commented on PR #43634:
URL: https://github.com/apache/arrow/pull/43634#issuecomment-2282232340

   @github-actions crossbow submit test-r-extra-packages


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-43633: [R] Add tests for packages that might be tricky to roundtrip data to Tables + Parquet files [arrow]

Posted by "jonkeane (via GitHub)" <gi...@apache.org>.
jonkeane commented on PR #43634:
URL: https://github.com/apache/arrow/pull/43634#issuecomment-2282340841

   @github-actions crossbow submit test-r-extra-packages


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-43633: [R] Add tests for packages that might be tricky to roundtrip data to Tables + Parquet files [arrow]

Posted by "jonkeane (via GitHub)" <gi...@apache.org>.
jonkeane commented on code in PR #43634:
URL: https://github.com/apache/arrow/pull/43634#discussion_r1717202653


##########
r/tests/testthat/test-extra-package-roundtrip.R:
##########
@@ -0,0 +1,99 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+skip_on_cran()
+
+# Any additional pacakge that we test here that is not already in DESCRIPTION should be
+# added to dev/tasks/r/github.linux.extra.packages.yml in the r-lib/actions/setup-r-dependencies@v2
+# step so that they are installed + available in that CI job.
+
+# So that we can force these in CI
+load_or_skip <- function(pkg) {
+  if (identical(tolower(Sys.getenv("ARROW_R_FORCE_EXTRA_PACKAGE_TESTS")), "true")) {
+    # because of this indirection on the package name we also avoid a CHECK note and 
+    # we don't otherwise need to Suggest this
+    requireNamespace(pkg, quietly = TRUE)
+  } else {
+    skip_if(!requireNamespace(pkg, quietly = TRUE))
+  }
+  attachNamespace(pkg)
+}
+
+library(tibble)
+
+test_that("readr read csvs roundtrip", {
+  load_or_skip("readr")
+
+  tbl <- example_data[, c("dbl", "lgl", "false", "chr")]
+
+  tf <- tempfile()
+  on.exit(unlink(tf))
+  write.csv(tbl, tf, row.names = FALSE)
+
+  # we should still be able to turn this into a table
+  new_df <- read_csv(tf, show_col_types = FALSE)
+  expect_equal(new_df, as_tibble(arrow_table(new_df)))    
+
+  # we should still be able to turn this into a table
+  new_df <- read_csv(tf, show_col_types = FALSE, lazy = TRUE)
+  expect_equal(new_df, as_tibble(arrow_table(new_df)))    
+
+  # and can roundtrip to a parquet file
+  pq_tmp_file <- tempfile()
+  write_parquet(new_df, pq_tmp_file)
+  new_df_read <- read_parquet(pq_tmp_file)
+
+  # we should still be able to turn this into a table
+  expect_equal(new_df, new_df_read)
+
+})
+
+test_that("data.table objects roundtrip", {
+  load_or_skip("data.table")
+
+  # https://rdatatable.gitlab.io/data.table/articles/datatable-importing.html#testing-using-testthat
+  .datatable.aware=TRUE
+
+  DT <- as.data.table(example_data)
+
+  # write to parquet
+  pq_tmp_file <- tempfile()
+  write_parquet(DT, pq_tmp_file)
+  DT_read <- read_parquet(pq_tmp_file)
+
+  # we should still be able to turn this into a table
+  expect_equal(DT, DT_read)
+
+  # attributes are the same, aside from the internal selfref pointer
+  expect_mapequal(attributes(DT_read), attributes(DT)[names(attributes(DT)) != ".internal.selfref"])

Review Comment:
   Ah, you're right. It also turns out that waldo ignores pointers regardless, so we can use `expect_equal` here.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-43633: [R] Add tests for packages that might be tricky to roundtrip data to Tables + Parquet files [arrow]

Posted by "jonkeane (via GitHub)" <gi...@apache.org>.
jonkeane merged PR #43634:
URL: https://github.com/apache/arrow/pull/43634


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-43633: [R] Add tests for packages that might be tricky to roundtrip data to Tables + Parquet files [arrow]

Posted by "MichaelChirico (via GitHub)" <gi...@apache.org>.
MichaelChirico commented on code in PR #43634:
URL: https://github.com/apache/arrow/pull/43634#discussion_r1713302813


##########
r/tests/testthat/test-extra-package-roundtrip.R:
##########
@@ -0,0 +1,99 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+skip_on_cran()
+
+# Any additional pacakge that we test here that is not already in DESCRIPTION should be
+# added to dev/tasks/r/github.linux.extra.packages.yml in the r-lib/actions/setup-r-dependencies@v2
+# step so that they are installed + available in that CI job.
+
+# So that we can force these in CI
+load_or_skip <- function(pkg) {
+  if (identical(tolower(Sys.getenv("ARROW_R_FORCE_EXTRA_PACKAGE_TESTS")), "true")) {
+    # because of this indirection on the package name we also avoid a CHECK note and 
+    # we don't otherwise need to Suggest this
+    requireNamespace(pkg, quietly = TRUE)
+  } else {
+    skip_if(!requireNamespace(pkg, quietly = TRUE))
+  }
+  attachNamespace(pkg)
+}
+
+library(tibble)
+
+test_that("readr read csvs roundtrip", {
+  load_or_skip("readr")
+
+  tbl <- example_data[, c("dbl", "lgl", "false", "chr")]
+
+  tf <- tempfile()
+  on.exit(unlink(tf))
+  write.csv(tbl, tf, row.names = FALSE)
+
+  # we should still be able to turn this into a table
+  new_df <- read_csv(tf, show_col_types = FALSE)
+  expect_equal(new_df, as_tibble(arrow_table(new_df)))    
+
+  # we should still be able to turn this into a table
+  new_df <- read_csv(tf, show_col_types = FALSE, lazy = TRUE)
+  expect_equal(new_df, as_tibble(arrow_table(new_df)))    
+
+  # and can roundtrip to a parquet file
+  pq_tmp_file <- tempfile()
+  write_parquet(new_df, pq_tmp_file)
+  new_df_read <- read_parquet(pq_tmp_file)
+
+  # we should still be able to turn this into a table
+  expect_equal(new_df, new_df_read)
+
+})
+
+test_that("data.table objects roundtrip", {
+  load_or_skip("data.table")
+
+  # https://rdatatable.gitlab.io/data.table/articles/datatable-importing.html#testing-using-testthat
+  .datatable.aware=TRUE
+
+  DT <- as.data.table(example_data)
+
+  # write to parquet
+  pq_tmp_file <- tempfile()
+  write_parquet(DT, pq_tmp_file)
+  DT_read <- read_parquet(pq_tmp_file)
+
+  # we should still be able to turn this into a table
+  expect_equal(DT, DT_read)
+
+  # attributes are the same, aside from the internal selfref pointer
+  expect_mapequal(attributes(DT_read), attributes(DT)[names(attributes(DT)) != ".internal.selfref"])

Review Comment:
   NB: data.table's `all.equal()` method should handle this attributes check (when `check.attributes=TRUE`, the default). That will make this test depend a little less on implementation details like `.internal.selfref`: `expect_true(all.equal(DT, DT_read))`.
   
   `expect_equal()` calls `all.equal()`, but only in testthat 2e; I see you're already on testthat 3e, so we need `expect_true()`. OTOH, in 3e we can do `expect_equal(DT, DT_read, ignore_attr=".internal.selfref")` which is more concise & very similar to what you have here.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-43633: [R] Add tests for packages that might be tricky to roundtrip data to Tables + Parquet files [arrow]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #43634:
URL: https://github.com/apache/arrow/pull/43634#issuecomment-2279247387

   :warning: GitHub issue #43633 **has been automatically assigned in GitHub** to PR creator.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-43633: [R] Add tests for packages that might be tricky to roundtrip data to Tables + Parquet files [arrow]

Posted by "jonkeane (via GitHub)" <gi...@apache.org>.
jonkeane commented on PR #43634:
URL: https://github.com/apache/arrow/pull/43634#issuecomment-2290637577

   @github-actions crossbow submit test-r-extra-packages


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-43633: [R] Add tests for packages that might be tricky to roundtrip data to Tables + Parquet files [arrow]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #43634:
URL: https://github.com/apache/arrow/pull/43634#issuecomment-2294342215

   Revision: ba3c1bfe298631cf0a6a426842ac869b7c53a193
   
   Submitted crossbow builds: [ursacomputing/crossbow @ actions-cc60961308](https://github.com/ursacomputing/crossbow/branches/all?query=actions-cc60961308)
   
   |Task|Status|
   |----|------|
   |test-r-extra-packages|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-cc60961308-github-test-r-extra-packages)](https://github.com/ursacomputing/crossbow/actions/runs/10426536351/job/28879638553)|


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-43633: [R] Add tests for packages that might be tricky to roundtrip data to Tables + Parquet files [arrow]

Posted by "jonkeane (via GitHub)" <gi...@apache.org>.
jonkeane commented on PR #43634:
URL: https://github.com/apache/arrow/pull/43634#issuecomment-2289207930

   @github-actions crossbow submit test-r-extra-packages


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-43633: [R] Add tests for packages that might be tricky to roundtrip data to Tables + Parquet files [arrow]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #43634:
URL: https://github.com/apache/arrow/pull/43634#issuecomment-2294325829

   Revision: 014e4be9115befed6aa729b8ebdb15a726e09895
   
   Submitted crossbow builds: [ursacomputing/crossbow @ actions-1b1641a9fb](https://github.com/ursacomputing/crossbow/branches/all?query=actions-1b1641a9fb)
   
   |Task|Status|
   |----|------|
   |test-r-extra-packages|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-1b1641a9fb-github-test-r-extra-packages)](https://github.com/ursacomputing/crossbow/actions/runs/10426355235/job/28879131527)|


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-43633: [R] Add tests for packages that might be tricky to roundtrip data to Tables + Parquet files [arrow]

Posted by "jonkeane (via GitHub)" <gi...@apache.org>.
jonkeane commented on PR #43634:
URL: https://github.com/apache/arrow/pull/43634#issuecomment-2282222254

   @github-actions crossbow submit test-r-extra-packages


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-43633: [R] Add tests for packages that might be tricky to roundtrip data to Tables + Parquet files [arrow]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #43634:
URL: https://github.com/apache/arrow/pull/43634#issuecomment-2282231195

   Revision: a6e0023452fcd51e3f55e85a3b2663bd0487595f
   
   Submitted crossbow builds: [ursacomputing/crossbow @ actions-8d154cd3a0](https://github.com/ursacomputing/crossbow/branches/all?query=actions-8d154cd3a0)
   
   |Task|Status|
   |----|------|
   |test-r-extra-packages|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-8d154cd3a0-github-test-r-extra-packages)](https://github.com/ursacomputing/crossbow/actions/runs/10333516325/job/28605960890)|


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-43633: [R] Add tests for packages that might be tricky to roundtrip data to Tables + Parquet files [arrow]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #43634:
URL: https://github.com/apache/arrow/pull/43634#issuecomment-2282344212

   Revision: 5c183dd81ef6db2fe18e40ecc4d6ab3e5352e79a
   
   Submitted crossbow builds: [ursacomputing/crossbow @ actions-822bab695d](https://github.com/ursacomputing/crossbow/branches/all?query=actions-822bab695d)
   
   |Task|Status|
   |----|------|
   |test-r-extra-packages|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-822bab695d-github-test-r-extra-packages)](https://github.com/ursacomputing/crossbow/actions/runs/10336034424/job/28611312110)|


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-43633: [R] Add tests for packages that might be tricky to roundtrip data to Tables + Parquet files [arrow]

Posted by "TysonStanley (via GitHub)" <gi...@apache.org>.
TysonStanley commented on code in PR #43634:
URL: https://github.com/apache/arrow/pull/43634#discussion_r1713029975


##########
r/tests/testthat/test-extra-package-roundtrip.R:
##########
@@ -0,0 +1,91 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+skip_on_cran()
+
+# So that we can force these in CI
+load_or_skip <- function(pkg) {
+  if (identical(tolower(Sys.getenv("ARROW_R_FORCE_EXTRA_PACKAGE_TESTS")), "true")) {
+    # because of this indirection on the package name we also avoid a CHECK note and 
+    # we don't otherwise need to Suggest this
+    requireNamespace(pkg, quietly = TRUE)
+  } else {
+    skip_if(!requireNamespace(pkg, quietly = TRUE))
+  }
+  attachNamespace(pkg)
+}
+
+library(tibble)
+
+test_that("readr read csvs roundtrip", {
+  load_or_skip("readr")
+
+  tbl <- example_data[, c("dbl", "lgl", "false", "chr")]
+
+  tf <- tempfile()
+  on.exit(unlink(tf))
+  write.csv(tbl, tf, row.names = FALSE)
+
+  # we should still be able to turn this into a table
+  new_df <- read_csv(tf, show_col_types = FALSE)
+  expect_equal(new_df, as_tibble(arrow_table(new_df)))    
+
+  # we should still be able to turn this into a table
+  new_df <- read_csv(tf, show_col_types = FALSE, lazy = TRUE)
+  expect_equal(new_df, as_tibble(arrow_table(new_df)))    
+
+  # and can roundtrip to a parquet file
+  pq_tmp_file <- tempfile()
+  write_parquet(new_df, pq_tmp_file)
+  new_df_read <- read_parquet(pq_tmp_file)
+
+  # we should still be able to turn this into a table
+  expect_equal(new_df, new_df_read)
+
+})
+
+test_that("data.table objects roundtrip", {
+  load_or_skip("data.table")
+
+  DT <- as.data.table(example_data)
+
+  # write to parquet
+  pq_tmp_file <- tempfile()
+  write_parquet(DT, pq_tmp_file)
+  DT_read <- read_parquet(pq_tmp_file)
+
+  # we should still be able to turn this into a table
+  expect_equal(DT, DT_read)
+
+  # attributes are the same, aside from the internal selfref pointer
+  expect_mapequal(attributes(DT_read), attributes(DT)[names(attributes(DT)) != ".internal.selfref"])
+
+  # and we can set keys + indices
+  setkey(DT, chr)
+  setindex(DT, dbl)
+
+  # write to parquet
+  pq_tmp_file <- tempfile()
+  write_parquet(DT, pq_tmp_file)
+  DT_read <- read_parquet(pq_tmp_file)
+
+  # we should still be able to turn this into a table
+  expect_equal(DT, DT_read)
+
+  # and the attributes are the same, aside from the internal selfref pointer
+  expect_mapequal(attributes(DT_read), attributes(DT)[names(attributes(DT)) != ".internal.selfref"])
+})

Review Comment:
   Those are the most important. Would also be useful to see if recently modified columns round trip well (eg, `DT[, new_col := ...]` where ... are some modification of the example data. 
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-43633: [R] Add tests for packages that might be tricky to roundtrip data to Tables + Parquet files [arrow]

Posted by "MichaelChirico (via GitHub)" <gi...@apache.org>.
MichaelChirico commented on code in PR #43634:
URL: https://github.com/apache/arrow/pull/43634#discussion_r1713302813


##########
r/tests/testthat/test-extra-package-roundtrip.R:
##########
@@ -0,0 +1,99 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+skip_on_cran()
+
+# Any additional pacakge that we test here that is not already in DESCRIPTION should be
+# added to dev/tasks/r/github.linux.extra.packages.yml in the r-lib/actions/setup-r-dependencies@v2
+# step so that they are installed + available in that CI job.
+
+# So that we can force these in CI
+load_or_skip <- function(pkg) {
+  if (identical(tolower(Sys.getenv("ARROW_R_FORCE_EXTRA_PACKAGE_TESTS")), "true")) {
+    # because of this indirection on the package name we also avoid a CHECK note and 
+    # we don't otherwise need to Suggest this
+    requireNamespace(pkg, quietly = TRUE)
+  } else {
+    skip_if(!requireNamespace(pkg, quietly = TRUE))
+  }
+  attachNamespace(pkg)
+}
+
+library(tibble)
+
+test_that("readr read csvs roundtrip", {
+  load_or_skip("readr")
+
+  tbl <- example_data[, c("dbl", "lgl", "false", "chr")]
+
+  tf <- tempfile()
+  on.exit(unlink(tf))
+  write.csv(tbl, tf, row.names = FALSE)
+
+  # we should still be able to turn this into a table
+  new_df <- read_csv(tf, show_col_types = FALSE)
+  expect_equal(new_df, as_tibble(arrow_table(new_df)))    
+
+  # we should still be able to turn this into a table
+  new_df <- read_csv(tf, show_col_types = FALSE, lazy = TRUE)
+  expect_equal(new_df, as_tibble(arrow_table(new_df)))    
+
+  # and can roundtrip to a parquet file
+  pq_tmp_file <- tempfile()
+  write_parquet(new_df, pq_tmp_file)
+  new_df_read <- read_parquet(pq_tmp_file)
+
+  # we should still be able to turn this into a table
+  expect_equal(new_df, new_df_read)
+
+})
+
+test_that("data.table objects roundtrip", {
+  load_or_skip("data.table")
+
+  # https://rdatatable.gitlab.io/data.table/articles/datatable-importing.html#testing-using-testthat
+  .datatable.aware=TRUE
+
+  DT <- as.data.table(example_data)
+
+  # write to parquet
+  pq_tmp_file <- tempfile()
+  write_parquet(DT, pq_tmp_file)
+  DT_read <- read_parquet(pq_tmp_file)
+
+  # we should still be able to turn this into a table
+  expect_equal(DT, DT_read)
+
+  # attributes are the same, aside from the internal selfref pointer
+  expect_mapequal(attributes(DT_read), attributes(DT)[names(attributes(DT)) != ".internal.selfref"])

Review Comment:
   NB: data.table's `all.equal()` method should handle this attributes check (when `check.attributes=TRUE`, the default). That will make this test depend a little less on implementation details like `.internal.selfref`: `expect_true(all.equal(DT, DT_read))`.
   
   `expect_equal()` calls `all.equal()`, but only in testthat 2e, I see you're already on testthat 3e, so we need `expect_true()`. OTOH, in 3e we can do `expect_equal(DT, DT_read, ignore_attr=".internal.selfref")` which is more concise & very similar to what you have here.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-43633: [R] Add tests for packages that might be tricky to roundtrip data to Tables + Parquet files [arrow]

Posted by "jonkeane (via GitHub)" <gi...@apache.org>.
jonkeane commented on PR #43634:
URL: https://github.com/apache/arrow/pull/43634#issuecomment-2282227777

   @github-actions crossbow submit test-r-extra-packages


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-43633: [R] Add tests for packages that might be tricky to roundtrip data to Tables + Parquet files [arrow]

Posted by "jonkeane (via GitHub)" <gi...@apache.org>.
jonkeane commented on PR #43634:
URL: https://github.com/apache/arrow/pull/43634#issuecomment-2282369749

   @github-actions crossbow submit test-r-extra-packages


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-43633: [R] Add tests for packages that might be tricky to roundtrip data to Tables + Parquet files [arrow]

Posted by "MichaelChirico (via GitHub)" <gi...@apache.org>.
MichaelChirico commented on code in PR #43634:
URL: https://github.com/apache/arrow/pull/43634#discussion_r1713295696


##########
r/tests/testthat/test-extra-package-roundtrip.R:
##########
@@ -0,0 +1,99 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+skip_on_cran()
+
+# Any additional pacakge that we test here that is not already in DESCRIPTION should be
+# added to dev/tasks/r/github.linux.extra.packages.yml in the r-lib/actions/setup-r-dependencies@v2
+# step so that they are installed + available in that CI job.
+
+# So that we can force these in CI
+load_or_skip <- function(pkg) {
+  if (identical(tolower(Sys.getenv("ARROW_R_FORCE_EXTRA_PACKAGE_TESTS")), "true")) {
+    # because of this indirection on the package name we also avoid a CHECK note and 
+    # we don't otherwise need to Suggest this
+    requireNamespace(pkg, quietly = TRUE)
+  } else {
+    skip_if(!requireNamespace(pkg, quietly = TRUE))
+  }
+  attachNamespace(pkg)
+}
+
+library(tibble)
+
+test_that("readr read csvs roundtrip", {
+  load_or_skip("readr")
+
+  tbl <- example_data[, c("dbl", "lgl", "false", "chr")]
+
+  tf <- tempfile()
+  on.exit(unlink(tf))
+  write.csv(tbl, tf, row.names = FALSE)
+
+  # we should still be able to turn this into a table
+  new_df <- read_csv(tf, show_col_types = FALSE)
+  expect_equal(new_df, as_tibble(arrow_table(new_df)))    
+
+  # we should still be able to turn this into a table
+  new_df <- read_csv(tf, show_col_types = FALSE, lazy = TRUE)
+  expect_equal(new_df, as_tibble(arrow_table(new_df)))    
+
+  # and can roundtrip to a parquet file
+  pq_tmp_file <- tempfile()
+  write_parquet(new_df, pq_tmp_file)
+  new_df_read <- read_parquet(pq_tmp_file)
+
+  # we should still be able to turn this into a table
+  expect_equal(new_df, new_df_read)
+
+})
+
+test_that("data.table objects roundtrip", {
+  load_or_skip("data.table")
+
+  # https://rdatatable.gitlab.io/data.table/articles/datatable-importing.html#testing-using-testthat
+  .datatable.aware=TRUE
+
+  DT <- as.data.table(example_data)
+
+  # write to parquet
+  pq_tmp_file <- tempfile()
+  write_parquet(DT, pq_tmp_file)
+  DT_read <- read_parquet(pq_tmp_file)
+
+  # we should still be able to turn this into a table
+  expect_equal(DT, DT_read)
+
+  # attributes are the same, aside from the internal selfref pointer
+  expect_mapequal(attributes(DT_read), attributes(DT)[names(attributes(DT)) != ".internal.selfref"])
+
+  # and we can set keys + indices + create new columns
+  setkey(DT, chr)
+  setindex(DT, dbl)
+  DT[, dblshift := data.table::shift(dbl, 1)]
+
+  # write to parquet
+  pq_tmp_file <- tempfile()
+  write_parquet(DT, pq_tmp_file)
+  DT_read <- read_parquet(pq_tmp_file)
+
+  # we should still be able to turn this into a table
+  expect_equal(DT, DT_read)
+
+  # and the attributes are the same, aside from the internal selfref pointer
+  expect_mapequal(attributes(DT_read), attributes(DT)[names(attributes(DT)) != ".internal.selfref"])

Review Comment:
   This did give me some initial pause given recent experience with:
   
   https://github.com/Rdatatable/data.table/issues/6349
   
   However, I think it's fine here because you generate the data.table object on the fly instead of using a cached one.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-43633: [R] Add tests for packages that might be tricky to roundtrip data to Tables + Parquet files [arrow]

Posted by "jonkeane (via GitHub)" <gi...@apache.org>.
jonkeane commented on code in PR #43634:
URL: https://github.com/apache/arrow/pull/43634#discussion_r1720378861


##########
dev/tasks/r/github.linux.extra.packages.yml:
##########
@@ -0,0 +1,52 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+{% import 'macros.jinja' as macros with context %}
+
+{{ macros.github_header() }}
+
+jobs:
+  extra-packages:
+    name: "extra package roundtrip tests"
+    runs-on: ubuntu-24.04
+    strategy:
+      fail-fast: false
+    env:
+      ARROW_R_DEV: "FALSE"
+      RSPM: "https://packagemanager.posit.co/cran/__linux__/noble/latest"

Review Comment:
   Nope, there is a different key in the setup-r but I've added it



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-43633: [R] Add tests for packages that might be tricky to roundtrip data to Tables + Parquet files [arrow]

Posted by "conbench-apache-arrow[bot] (via GitHub)" <gi...@apache.org>.
conbench-apache-arrow[bot] commented on PR #43634:
URL: https://github.com/apache/arrow/pull/43634#issuecomment-2294633475

   After merging your PR, Conbench analyzed the 4 benchmarking runs that have been run so far on merge-commit 801301ee22ce802fd000f9f4b919abb47ae1d6c3.
   
   There were no benchmark performance regressions. 🎉
   
   The [full Conbench report](https://github.com/apache/arrow/runs/28887728319) has more details. It also includes information about 1 possible false positive for unstable benchmarks that are known to sometimes produce them.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-43633: [R] Add tests for packages that might be tricky to roundtrip data to Tables + Parquet files [arrow]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #43634:
URL: https://github.com/apache/arrow/pull/43634#issuecomment-2282370371

   Revision: efad40a702530b524a3c5fddab7b6426746c554e
   
   Submitted crossbow builds: [ursacomputing/crossbow @ actions-12e6afc186](https://github.com/ursacomputing/crossbow/branches/all?query=actions-12e6afc186)
   
   |Task|Status|
   |----|------|
   |test-r-extra-packages|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-12e6afc186-github-test-r-extra-packages)](https://github.com/ursacomputing/crossbow/actions/runs/10336616691/job/28612559276)|


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-43633: [R] Add tests for packages that might be tricky to roundtrip data to Tables + Parquet files [arrow]

Posted by "jonkeane (via GitHub)" <gi...@apache.org>.
jonkeane commented on code in PR #43634:
URL: https://github.com/apache/arrow/pull/43634#discussion_r1712903532


##########
r/tests/testthat/test-extra-package-roundtrip.R:
##########
@@ -0,0 +1,91 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+skip_on_cran()
+
+# So that we can force these in CI
+load_or_skip <- function(pkg) {
+  if (identical(tolower(Sys.getenv("ARROW_R_FORCE_EXTRA_PACKAGE_TESTS")), "true")) {
+    # because of this indirection on the package name we also avoid a CHECK note and 
+    # we don't otherwise need to Suggest this
+    requireNamespace(pkg, quietly = TRUE)
+  } else {
+    skip_if(!requireNamespace(pkg, quietly = TRUE))
+  }
+  attachNamespace(pkg)
+}
+
+library(tibble)
+
+test_that("readr read csvs roundtrip", {
+  load_or_skip("readr")
+
+  tbl <- example_data[, c("dbl", "lgl", "false", "chr")]
+
+  tf <- tempfile()
+  on.exit(unlink(tf))
+  write.csv(tbl, tf, row.names = FALSE)
+
+  # we should still be able to turn this into a table
+  new_df <- read_csv(tf, show_col_types = FALSE)
+  expect_equal(new_df, as_tibble(arrow_table(new_df)))    
+
+  # we should still be able to turn this into a table
+  new_df <- read_csv(tf, show_col_types = FALSE, lazy = TRUE)
+  expect_equal(new_df, as_tibble(arrow_table(new_df)))    
+
+  # and can roundtrip to a parquet file
+  pq_tmp_file <- tempfile()
+  write_parquet(new_df, pq_tmp_file)
+  new_df_read <- read_parquet(pq_tmp_file)
+
+  # we should still be able to turn this into a table
+  expect_equal(new_df, new_df_read)
+
+})
+
+test_that("data.table objects roundtrip", {
+  load_or_skip("data.table")
+
+  DT <- as.data.table(example_data)
+
+  # write to parquet
+  pq_tmp_file <- tempfile()
+  write_parquet(DT, pq_tmp_file)
+  DT_read <- read_parquet(pq_tmp_file)
+
+  # we should still be able to turn this into a table
+  expect_equal(DT, DT_read)
+
+  # attributes are the same, aside from the internal selfref pointer
+  expect_mapequal(attributes(DT_read), attributes(DT)[names(attributes(DT)) != ".internal.selfref"])
+
+  # and we can set keys + indices
+  setkey(DT, chr)
+  setindex(DT, dbl)
+
+  # write to parquet
+  pq_tmp_file <- tempfile()
+  write_parquet(DT, pq_tmp_file)
+  DT_read <- read_parquet(pq_tmp_file)
+
+  # we should still be able to turn this into a table
+  expect_equal(DT, DT_read)
+
+  # and the attributes are the same, aside from the internal selfref pointer
+  expect_mapequal(attributes(DT_read), attributes(DT)[names(attributes(DT)) != ".internal.selfref"])
+})

Review Comment:
   @TysonStanley @MichaelChirico Any other attributes or other things I could add here that would stretch the ability to roundtrip data.table objects to parquet and back?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-43633: [R] Add tests for packages that might be tricky to roundtrip data to Tables + Parquet files [arrow]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #43634:
URL: https://github.com/apache/arrow/pull/43634#issuecomment-2289212826

   Revision: 62ced17f2676b9c3abcf19c2054db48c557e41a1
   
   Submitted crossbow builds: [ursacomputing/crossbow @ actions-33f2ea3feb](https://github.com/ursacomputing/crossbow/branches/all?query=actions-33f2ea3feb)
   
   |Task|Status|
   |----|------|
   |test-r-extra-packages|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-33f2ea3feb-github-test-r-extra-packages)](https://github.com/ursacomputing/crossbow/actions/runs/10391104513/job/28773208857)|


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-43633: [R] Add tests for packages that might be tricky to roundtrip data to Tables + Parquet files [arrow]

Posted by "jonkeane (via GitHub)" <gi...@apache.org>.
jonkeane commented on code in PR #43634:
URL: https://github.com/apache/arrow/pull/43634#discussion_r1720359278


##########
r/tests/testthat/test-extra-package-roundtrip.R:
##########
@@ -0,0 +1,108 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+skip_on_cran()
+
+# Any additional package that we test here that is not already in DESCRIPTION should be
+# added to dev/tasks/r/github.linux.extra.packages.yml in the r-lib/actions/setup-r-dependencies@v2
+# step so that they are installed + available in that CI job.
+
+# So that we can force these in CI
+load_or_skip <- function(pkg) {
+  if (identical(tolower(Sys.getenv("ARROW_R_FORCE_EXTRA_PACKAGE_TESTS")), "true")) {
+    # because of this indirection on the package name we also avoid a CHECK note and 
+    # we don't otherwise need to Suggest this
+    requireNamespace(pkg, quietly = TRUE)
+  } else {
+    skip_if(!requireNamespace(pkg, quietly = TRUE))
+  }
+  attachNamespace(pkg)
+}
+
+library(tibble)
+
+test_that("readr read csvs roundtrip", {
+  load_or_skip("readr")
+
+  tbl <- example_data[, c("dbl", "lgl", "false", "chr")]
+
+  tf <- tempfile()
+  on.exit(unlink(tf))
+  write.csv(tbl, tf, row.names = FALSE)
+
+  # we should still be able to turn this into a table
+  new_df <- read_csv(tf, show_col_types = FALSE)
+  expect_equal(new_df, as_tibble(arrow_table(new_df)))    
+
+  # we should still be able to turn this into a table
+  new_df <- read_csv(tf, show_col_types = FALSE, lazy = TRUE)
+  expect_equal(new_df, as_tibble(arrow_table(new_df)))    
+
+  # and can roundtrip to a parquet file

Review Comment:
   Yeah, you're right. When I first started this I did the roundtrip to parquet because `as.data.frame` doesn't attach all of the metadata (which makes sense, since it's expected to return a data.frame not some other thing!). I've switched this over to use `collect` which pulls things into R and then _also_ applies the right class(es). We might consider exporting some other function (in case folks aren't using dplyr otherwise) that rehydrates these objects.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-43633: [R] Add tests for packages that might be tricky to roundtrip data to Tables + Parquet files [arrow]

Posted by "amoeba (via GitHub)" <gi...@apache.org>.
amoeba commented on PR #43634:
URL: https://github.com/apache/arrow/pull/43634#issuecomment-2285135014

   Hi @jonkeane, this is great. The only extension I'm personally familiar with is `units`. Would that make sense to test here? See:
   
   ```r
   > library(arrow)
   > library(units)
   > df <- data.frame(x=1:10)
   > units(df$x) <- "s"
   > df
           x
   1   1 [s]
   2   2 [s]
   3   3 [s]
   4   4 [s]
   5   5 [s]
   6   6 [s]
   7   7 [s]
   8   8 [s]
   9   9 [s]
   10 10 [s]
   > class(df$x)
   [1] "units"
   > # write.csv/read.csv drops units
   > tf <- tempfile()
   > write.csv(df, tf)
   > read.csv(tf)
       X  x
   1   1  1
   2   2  2
   3   3  3
   4   4  4
   5   5  5
   6   6  6
   7   7  7
   8   8  8
   9   9  9
   10 10 10
   > # write_csv_arrow errors
   > tf <- tempfile()
   > arrow::write_csv_arrow(df, tf)
   Error: Invalid: Unsupported Type: [s]
   > # write/read_parquet works great
   > write_parquet(df, tf)
   > read_parquet(tf)
           x
   1   1 [s]
   2   2 [s]
   3   3 [s]
   4   4 [s]
   5   5 [s]
   6   6 [s]
   7   7 [s]
   8   8 [s]
   9   9 [s]
   10 10 [s]
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-43633: [R] Add tests for packages that might be tricky to roundtrip data to Tables + Parquet files [arrow]

Posted by "jonkeane (via GitHub)" <gi...@apache.org>.
jonkeane commented on PR #43634:
URL: https://github.com/apache/arrow/pull/43634#issuecomment-2282338659

   @github-actions crossbow submit test-r-extra-packages


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-43633: [R] Add tests for packages that might be tricky to roundtrip data to Tables + Parquet files [arrow]

Posted by "jonkeane (via GitHub)" <gi...@apache.org>.
jonkeane commented on code in PR #43634:
URL: https://github.com/apache/arrow/pull/43634#discussion_r1712549001


##########
r/tests/testthat/test-other-package-roundtrip.R:
##########
@@ -0,0 +1,53 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+skip_on_cran()
+
+library(tibble)
+
+test_that("readr read csvs roundtrip", {
+  # indirection to avoid CHECK note and we don't otherwise need to Suggest this
+  pkg <- "readr"
+  skip_if(!requireNamespace(pkg, quietly = TRUE))

Review Comment:
   This avoids `R CMD check` complaining on my local machine, which it does if I do `requireNamespace("readr", quietly = TRUE)` but is there a better way to do this?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-43633: [R] Add tests for packages that might be tricky to roundtrip data to Tables + Parquet files [arrow]

Posted by "jonkeane (via GitHub)" <gi...@apache.org>.
jonkeane commented on PR #43634:
URL: https://github.com/apache/arrow/pull/43634#issuecomment-2282335290

   @github-actions crossbow submit test-r-extra-packages
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-43633: [R] Add tests for packages that might be tricky to roundtrip data to Tables + Parquet files [arrow]

Posted by "jonkeane (via GitHub)" <gi...@apache.org>.
jonkeane commented on PR #43634:
URL: https://github.com/apache/arrow/pull/43634#issuecomment-2282343628

   @github-actions crossbow submit test-r-extra-packages
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-43633: [R] Add tests for packages that might be tricky to roundtrip data to Tables + Parquet files [arrow]

Posted by "jonkeane (via GitHub)" <gi...@apache.org>.
jonkeane commented on code in PR #43634:
URL: https://github.com/apache/arrow/pull/43634#discussion_r1712903532


##########
r/tests/testthat/test-extra-package-roundtrip.R:
##########
@@ -0,0 +1,91 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+skip_on_cran()
+
+# So that we can force these in CI
+load_or_skip <- function(pkg) {
+  if (identical(tolower(Sys.getenv("ARROW_R_FORCE_EXTRA_PACKAGE_TESTS")), "true")) {
+    # because of this indirection on the package name we also avoid a CHECK note and 
+    # we don't otherwise need to Suggest this
+    requireNamespace(pkg, quietly = TRUE)
+  } else {
+    skip_if(!requireNamespace(pkg, quietly = TRUE))
+  }
+  attachNamespace(pkg)
+}
+
+library(tibble)
+
+test_that("readr read csvs roundtrip", {
+  load_or_skip("readr")
+
+  tbl <- example_data[, c("dbl", "lgl", "false", "chr")]
+
+  tf <- tempfile()
+  on.exit(unlink(tf))
+  write.csv(tbl, tf, row.names = FALSE)
+
+  # we should still be able to turn this into a table
+  new_df <- read_csv(tf, show_col_types = FALSE)
+  expect_equal(new_df, as_tibble(arrow_table(new_df)))    
+
+  # we should still be able to turn this into a table
+  new_df <- read_csv(tf, show_col_types = FALSE, lazy = TRUE)
+  expect_equal(new_df, as_tibble(arrow_table(new_df)))    
+
+  # and can roundtrip to a parquet file
+  pq_tmp_file <- tempfile()
+  write_parquet(new_df, pq_tmp_file)
+  new_df_read <- read_parquet(pq_tmp_file)
+
+  # we should still be able to turn this into a table
+  expect_equal(new_df, new_df_read)
+
+})
+
+test_that("data.table objects roundtrip", {
+  load_or_skip("data.table")
+
+  DT <- as.data.table(example_data)
+
+  # write to parquet
+  pq_tmp_file <- tempfile()
+  write_parquet(DT, pq_tmp_file)
+  DT_read <- read_parquet(pq_tmp_file)
+
+  # we should still be able to turn this into a table
+  expect_equal(DT, DT_read)
+
+  # attributes are the same, aside from the internal selfref pointer
+  expect_mapequal(attributes(DT_read), attributes(DT)[names(attributes(DT)) != ".internal.selfref"])
+
+  # and we can set keys + indices
+  setkey(DT, chr)
+  setindex(DT, dbl)
+
+  # write to parquet
+  pq_tmp_file <- tempfile()
+  write_parquet(DT, pq_tmp_file)
+  DT_read <- read_parquet(pq_tmp_file)
+
+  # we should still be able to turn this into a table
+  expect_equal(DT, DT_read)
+
+  # and the attributes are the same, aside from the internal selfref pointer
+  expect_mapequal(attributes(DT_read), attributes(DT)[names(attributes(DT)) != ".internal.selfref"])
+})

Review Comment:
   @TysonStanley @ MichaelChirico Any other attributes or other things I could add here that would stretch the ability to roundtrip data.table objects to parquet and back?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-43633: [R] Add tests for packages that might be tricky to roundtrip data to Tables + Parquet files [arrow]

Posted by "jonkeane (via GitHub)" <gi...@apache.org>.
jonkeane commented on PR #43634:
URL: https://github.com/apache/arrow/pull/43634#issuecomment-2294339949

   @github-actions crossbow submit test-r-extra-packages


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-43633: [R] Add tests for packages that might be tricky to roundtrip data to Tables + Parquet files [arrow]

Posted by "jonkeane (via GitHub)" <gi...@apache.org>.
jonkeane commented on code in PR #43634:
URL: https://github.com/apache/arrow/pull/43634#discussion_r1720361817


##########
r/tests/testthat/test-extra-package-roundtrip.R:
##########
@@ -0,0 +1,108 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+skip_on_cran()
+
+# Any additional package that we test here that is not already in DESCRIPTION should be
+# added to dev/tasks/r/github.linux.extra.packages.yml in the r-lib/actions/setup-r-dependencies@v2
+# step so that they are installed + available in that CI job.
+
+# So that we can force these in CI
+load_or_skip <- function(pkg) {
+  if (identical(tolower(Sys.getenv("ARROW_R_FORCE_EXTRA_PACKAGE_TESTS")), "true")) {

Review Comment:
   > I thought a reason you put this in the regular test suite is so that they would run in local dev if you had the packages installed. 
   
   Yeah exactly 
   
   > But I won't have this env var set locally. Should this check NOT_CRAN maybe?
   
   This function is a little indirect: if you have the package installed, regardless of setting the envvar, it will run. If you have the envvar set and you don't have the package installed the tests will fail. If you have not set the envvar (or if it's false), and you don't have the packages installed, the tests will be skipped. This way in most CI setups we will silently move on and not need to install these packages, but in the one job where we want to ensure we are running, we can confirm that positively. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-43633: [R] Add tests for packages that might be tricky to roundtrip data to Tables + Parquet files [arrow]

Posted by "jonkeane (via GitHub)" <gi...@apache.org>.
jonkeane commented on code in PR #43634:
URL: https://github.com/apache/arrow/pull/43634#discussion_r1717164889


##########
r/tests/testthat/test-extra-package-roundtrip.R:
##########
@@ -0,0 +1,99 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+skip_on_cran()
+
+# Any additional pacakge that we test here that is not already in DESCRIPTION should be
+# added to dev/tasks/r/github.linux.extra.packages.yml in the r-lib/actions/setup-r-dependencies@v2
+# step so that they are installed + available in that CI job.
+
+# So that we can force these in CI
+load_or_skip <- function(pkg) {
+  if (identical(tolower(Sys.getenv("ARROW_R_FORCE_EXTRA_PACKAGE_TESTS")), "true")) {
+    # because of this indirection on the package name we also avoid a CHECK note and 
+    # we don't otherwise need to Suggest this
+    requireNamespace(pkg, quietly = TRUE)
+  } else {
+    skip_if(!requireNamespace(pkg, quietly = TRUE))
+  }
+  attachNamespace(pkg)
+}
+
+library(tibble)
+
+test_that("readr read csvs roundtrip", {
+  load_or_skip("readr")
+
+  tbl <- example_data[, c("dbl", "lgl", "false", "chr")]
+
+  tf <- tempfile()
+  on.exit(unlink(tf))
+  write.csv(tbl, tf, row.names = FALSE)
+
+  # we should still be able to turn this into a table
+  new_df <- read_csv(tf, show_col_types = FALSE)
+  expect_equal(new_df, as_tibble(arrow_table(new_df)))    
+
+  # we should still be able to turn this into a table
+  new_df <- read_csv(tf, show_col_types = FALSE, lazy = TRUE)
+  expect_equal(new_df, as_tibble(arrow_table(new_df)))    
+
+  # and can roundtrip to a parquet file
+  pq_tmp_file <- tempfile()
+  write_parquet(new_df, pq_tmp_file)
+  new_df_read <- read_parquet(pq_tmp_file)
+
+  # we should still be able to turn this into a table
+  expect_equal(new_df, new_df_read)
+
+})
+
+test_that("data.table objects roundtrip", {
+  load_or_skip("data.table")
+
+  # https://rdatatable.gitlab.io/data.table/articles/datatable-importing.html#testing-using-testthat
+  .datatable.aware=TRUE
+
+  DT <- as.data.table(example_data)
+
+  # write to parquet
+  pq_tmp_file <- tempfile()
+  write_parquet(DT, pq_tmp_file)
+  DT_read <- read_parquet(pq_tmp_file)
+
+  # we should still be able to turn this into a table
+  expect_equal(DT, DT_read)
+
+  # attributes are the same, aside from the internal selfref pointer
+  expect_mapequal(attributes(DT_read), attributes(DT)[names(attributes(DT)) != ".internal.selfref"])
+
+  # and we can set keys + indices + create new columns
+  setkey(DT, chr)
+  setindex(DT, dbl)
+  DT[, dblshift := data.table::shift(dbl, 1)]
+
+  # write to parquet
+  pq_tmp_file <- tempfile()
+  write_parquet(DT, pq_tmp_file)
+  DT_read <- read_parquet(pq_tmp_file)
+
+  # we should still be able to turn this into a table
+  expect_equal(DT, DT_read)
+
+  # and the attributes are the same, aside from the internal selfref pointer
+  expect_mapequal(attributes(DT_read), attributes(DT)[names(attributes(DT)) != ".internal.selfref"])

Review Comment:
   Yeah, there is no caching or cross-build / cross-version going on here. So _should_ be ok, and if not, it is worth investigation or indicative of something wrong



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-43633: [R] Add tests for packages that might be tricky to roundtrip data to Tables + Parquet files [arrow]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #43634:
URL: https://github.com/apache/arrow/pull/43634#issuecomment-2282233031

   Revision: 76eaed0155b58fb91e673e1d1faa1a298ea56897
   
   Submitted crossbow builds: [ursacomputing/crossbow @ actions-023f4e8fd4](https://github.com/ursacomputing/crossbow/branches/all?query=actions-023f4e8fd4)
   
   |Task|Status|
   |----|------|
   |test-r-extra-packages|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-023f4e8fd4-github-test-r-extra-packages)](https://github.com/ursacomputing/crossbow/actions/runs/10333554930/job/28606039250)|


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-43633: [R] Add tests for packages that might be tricky to roundtrip data to Tables + Parquet files [arrow]

Posted by "jonkeane (via GitHub)" <gi...@apache.org>.
jonkeane commented on PR #43634:
URL: https://github.com/apache/arrow/pull/43634#issuecomment-2282230620

   @github-actions crossbow submit test-r-extra-packages
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-43633: [R] Add tests for packages that might be tricky to roundtrip data to Tables + Parquet files [arrow]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #43634:
URL: https://github.com/apache/arrow/pull/43634#issuecomment-2282335802

   Revision: 76297e1cc9c825cec4f8bf296e7e3219044ee6cb
   
   Submitted crossbow builds: [ursacomputing/crossbow @ actions-fb5b3fec3f](https://github.com/ursacomputing/crossbow/branches/all?query=actions-fb5b3fec3f)
   
   |Task|Status|
   |----|------|
   |test-r-extra-packages|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-fb5b3fec3f-github-test-r-extra-packages)](https://github.com/ursacomputing/crossbow/actions/runs/10335795505/job/28610800120)|


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-43633: [R] Add tests for packages that might be tricky to roundtrip data to Tables + Parquet files [arrow]

Posted by "jonkeane (via GitHub)" <gi...@apache.org>.
jonkeane commented on code in PR #43634:
URL: https://github.com/apache/arrow/pull/43634#discussion_r1717163971


##########
r/tests/testthat/test-extra-package-roundtrip.R:
##########
@@ -65,6 +65,9 @@ test_that("readr read csvs roundtrip", {
 test_that("data.table objects roundtrip", {
   load_or_skip("data.table")
 
+  # https://rdatatable.gitlab.io/data.table/articles/datatable-importing.html#testing-using-testthat
+  .datatable.aware=TRUE

Review Comment:
   Thanks! I've updated this comment



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-43633: [R] Add tests for packages that might be tricky to roundtrip data to Tables + Parquet files [arrow]

Posted by "nealrichardson (via GitHub)" <gi...@apache.org>.
nealrichardson commented on code in PR #43634:
URL: https://github.com/apache/arrow/pull/43634#discussion_r1718749388


##########
dev/tasks/r/github.linux.extra.packages.yml:
##########
@@ -0,0 +1,52 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+{% import 'macros.jinja' as macros with context %}
+
+{{ macros.github_header() }}
+
+jobs:
+  extra-packages:
+    name: "extra package roundtrip tests"
+    runs-on: ubuntu-24.04

Review Comment:
   Future proof? (Assuming setup-r-dependencies handles PPPM setup) 
   
   ```suggestion
       runs-on: ubuntu-latest
   ```



##########
r/tests/testthat/test-extra-package-roundtrip.R:
##########
@@ -0,0 +1,108 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+skip_on_cran()
+
+# Any additional package that we test here that is not already in DESCRIPTION should be
+# added to dev/tasks/r/github.linux.extra.packages.yml in the r-lib/actions/setup-r-dependencies@v2
+# step so that they are installed + available in that CI job.
+
+# So that we can force these in CI
+load_or_skip <- function(pkg) {
+  if (identical(tolower(Sys.getenv("ARROW_R_FORCE_EXTRA_PACKAGE_TESTS")), "true")) {

Review Comment:
   I thought a reason you put this in the regular test suite is so that they would run in local dev if you had the packages installed. But I won't have this env var set locally. Should this check `NOT_CRAN` maybe?



##########
r/tests/testthat/test-extra-package-roundtrip.R:
##########
@@ -0,0 +1,108 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+skip_on_cran()
+
+# Any additional package that we test here that is not already in DESCRIPTION should be
+# added to dev/tasks/r/github.linux.extra.packages.yml in the r-lib/actions/setup-r-dependencies@v2
+# step so that they are installed + available in that CI job.
+
+# So that we can force these in CI
+load_or_skip <- function(pkg) {
+  if (identical(tolower(Sys.getenv("ARROW_R_FORCE_EXTRA_PACKAGE_TESTS")), "true")) {
+    # because of this indirection on the package name we also avoid a CHECK note and 
+    # we don't otherwise need to Suggest this
+    requireNamespace(pkg, quietly = TRUE)
+  } else {
+    skip_if(!requireNamespace(pkg, quietly = TRUE))
+  }
+  attachNamespace(pkg)
+}
+
+library(tibble)
+
+test_that("readr read csvs roundtrip", {
+  load_or_skip("readr")
+
+  tbl <- example_data[, c("dbl", "lgl", "false", "chr")]
+
+  tf <- tempfile()
+  on.exit(unlink(tf))
+  write.csv(tbl, tf, row.names = FALSE)
+
+  # we should still be able to turn this into a table
+  new_df <- read_csv(tf, show_col_types = FALSE)
+  expect_equal(new_df, as_tibble(arrow_table(new_df)))    
+
+  # we should still be able to turn this into a table
+  new_df <- read_csv(tf, show_col_types = FALSE, lazy = TRUE)
+  expect_equal(new_df, as_tibble(arrow_table(new_df)))    
+
+  # and can roundtrip to a parquet file

Review Comment:
   Minor observation: we already test elsewhere (in the R package and in C++) that writing an Arrow Table to Parquet and back preserves Arrow metadata. So you could just write arrow files.
   
   Following that reasoning further, there should be more than enough test coverage in C++ that writing and then reading IPC files preserves all information in memory. So would it be sufficient to just `df |> arrow_table() |> as.data.frame()`?



##########
dev/tasks/r/github.linux.extra.packages.yml:
##########
@@ -0,0 +1,52 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+{% import 'macros.jinja' as macros with context %}
+
+{{ macros.github_header() }}
+
+jobs:
+  extra-packages:
+    name: "extra package roundtrip tests"
+    runs-on: ubuntu-24.04
+    strategy:
+      fail-fast: false
+    env:
+      ARROW_R_DEV: "FALSE"
+      RSPM: "https://packagemanager.posit.co/cran/__linux__/noble/latest"

Review Comment:
   Do you need to set this manually? I thought that these days, `r-lib/actions/setup-r-dependencies` sorts this out for you by default.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-43633: [R] Add tests for packages that might be tricky to roundtrip data to Tables + Parquet files [arrow]

Posted by "TysonStanley (via GitHub)" <gi...@apache.org>.
TysonStanley commented on PR #43634:
URL: https://github.com/apache/arrow/pull/43634#issuecomment-2295015587

   @jonkeane I'm late to adding this, but notably my a team member sent me evidence that the index in data.table can cause problems in reading the parquet back in (`Error: IOError: Couldn't deserialize thrift: TProtocolException: Exceeded size limit`) and it explodes the size of the file (in her example from 400MB to 2GB with the only change being the index). See reprex below:
   
   ```r
   library(data.table)
   library(arrow)
   
   dt<-data.table(x=c(1:1e8), y = round(runif(n=1:1e8, min=1, max=5)))
   
   #Looking at rows where y == 3
   dt[y == 3,]
   
   #Creating a new variable, which is done uniformly across all rows (suggesting the previous row index isn't applicable?)
   dt[, z := 1]
   
   #Save the dt
   write_parquet(dt, "example.parquet")
   gc()
   
   #Cannot open the dt
   dt_open<-read_parquet("example.parquet")
   
   #Removing indexing that was created when looking at the y==3 subset before saving allows
   #the file to be opened after re-saving.
   setindex(dt, NULL)
   
   write_parquet(dt, "example2.parquet")
   dt_open<-read_parquet("example2.parquet")
   ```
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-43633: [R] Add tests for packages that might be tricky to roundtrip data to Tables + Parquet files [arrow]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #43634:
URL: https://github.com/apache/arrow/pull/43634#issuecomment-2282222873

   Revision: 04924fe96e3e273a0de2e9fa58b39a85dabd90af
   
   Submitted crossbow builds: [ursacomputing/crossbow @ actions-8a1f60248d](https://github.com/ursacomputing/crossbow/branches/all?query=actions-8a1f60248d)
   
   |Task|Status|
   |----|------|
   |test-r-extra-packages|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-8a1f60248d-github-test-r-extra-packages)](https://github.com/ursacomputing/crossbow/actions/runs/10333379117/job/28605657033)|


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-43633: [R] Add tests for packages that might be tricky to roundtrip data to Tables + Parquet files [arrow]

Posted by "jonkeane (via GitHub)" <gi...@apache.org>.
jonkeane commented on PR #43634:
URL: https://github.com/apache/arrow/pull/43634#issuecomment-2282428102

   @thisisnic @nealrichardson @amoeba I think y'all are probably the most likely to know other extensions of data.frame objects that folks have roundtripped to parquet (or arrow tables) and back. Any other objects / packages that I'm missing? I've set these tests up such that we don't need to add any of these to suggests, but they will run on crossbow runs. On the positive side this also would have caught the altrep bug we caught with revdep checks, so that's nice!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org