You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2022/06/20 15:26:44 UTC

[GitHub] [arrow] nealrichardson opened a new pull request, #13404: ARROW-16510: [R] Add bindings for GCS filesystem

nealrichardson opened a new pull request, #13404:
URL: https://github.com/apache/arrow/pull/13404

   To do (some may be deferred to followups): 
   
   - [ ] Basic R only tests (things that don't require a real or mock GCS backend)
   - [ ] Turn on ARROW_GCS in (some?) CI configurations
   - [ ] Set up testbench for fuller tests, like how we do with minio
   - [ ] C++ TODOs (type forwarding, Result return?)
   - [ ] Explore auth integration/compatibility with `gargle`, `googleAuthR`, etc.: can we pick up the same credentials they use 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] github-actions[bot] commented on pull request #13404: ARROW-16510: [R] Add bindings for GCS filesystem

Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #13404:
URL: https://github.com/apache/arrow/pull/13404#issuecomment-1166327994

   Revision: ee4e8e66b5f972bc40cde548b9eab7e491eac9dc
   
   Submitted crossbow builds: [ursacomputing/crossbow @ actions-fdfd53e5a1](https://github.com/ursacomputing/crossbow/branches/all?query=actions-fdfd53e5a1)
   
   |Task|Status|
   |----|------|
   |conda-linux-gcc-py37-cpu-r40|[![Azure](https://dev.azure.com/ursacomputing/crossbow/_apis/build/status/ursacomputing.crossbow?branchName=actions-fdfd53e5a1-azure-conda-linux-gcc-py37-cpu-r40)](https://dev.azure.com/ursacomputing/crossbow/_build/latest?definitionId=1&branchName=actions-fdfd53e5a1-azure-conda-linux-gcc-py37-cpu-r40)|
   |conda-linux-gcc-py37-cpu-r41|[![Azure](https://dev.azure.com/ursacomputing/crossbow/_apis/build/status/ursacomputing.crossbow?branchName=actions-fdfd53e5a1-azure-conda-linux-gcc-py37-cpu-r41)](https://dev.azure.com/ursacomputing/crossbow/_build/latest?definitionId=1&branchName=actions-fdfd53e5a1-azure-conda-linux-gcc-py37-cpu-r41)|
   |conda-osx-clang-py37-r40|[![Azure](https://dev.azure.com/ursacomputing/crossbow/_apis/build/status/ursacomputing.crossbow?branchName=actions-fdfd53e5a1-azure-conda-osx-clang-py37-r40)](https://dev.azure.com/ursacomputing/crossbow/_build/latest?definitionId=1&branchName=actions-fdfd53e5a1-azure-conda-osx-clang-py37-r40)|
   |conda-osx-clang-py37-r41|[![Azure](https://dev.azure.com/ursacomputing/crossbow/_apis/build/status/ursacomputing.crossbow?branchName=actions-fdfd53e5a1-azure-conda-osx-clang-py37-r41)](https://dev.azure.com/ursacomputing/crossbow/_build/latest?definitionId=1&branchName=actions-fdfd53e5a1-azure-conda-osx-clang-py37-r41)|
   |conda-win-vs2017-py37-r40|[![Azure](https://dev.azure.com/ursacomputing/crossbow/_apis/build/status/ursacomputing.crossbow?branchName=actions-fdfd53e5a1-azure-conda-win-vs2017-py37-r40)](https://dev.azure.com/ursacomputing/crossbow/_build/latest?definitionId=1&branchName=actions-fdfd53e5a1-azure-conda-win-vs2017-py37-r40)|
   |conda-win-vs2017-py37-r41|[![Azure](https://dev.azure.com/ursacomputing/crossbow/_apis/build/status/ursacomputing.crossbow?branchName=actions-fdfd53e5a1-azure-conda-win-vs2017-py37-r41)](https://dev.azure.com/ursacomputing/crossbow/_build/latest?definitionId=1&branchName=actions-fdfd53e5a1-azure-conda-win-vs2017-py37-r41)|
   |homebrew-r-autobrew|[![Github Actions](https://github.com/ursacomputing/crossbow/workflows/Crossbow/badge.svg?branch=actions-fdfd53e5a1-github-homebrew-r-autobrew)](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-fdfd53e5a1-github-homebrew-r-autobrew)|
   |homebrew-r-brew|[![Github Actions](https://github.com/ursacomputing/crossbow/workflows/Crossbow/badge.svg?branch=actions-fdfd53e5a1-github-homebrew-r-brew)](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-fdfd53e5a1-github-homebrew-r-brew)|
   |test-fedora-r-clang-sanitizer|[![Azure](https://dev.azure.com/ursacomputing/crossbow/_apis/build/status/ursacomputing.crossbow?branchName=actions-fdfd53e5a1-azure-test-fedora-r-clang-sanitizer)](https://dev.azure.com/ursacomputing/crossbow/_build/latest?definitionId=1&branchName=actions-fdfd53e5a1-azure-test-fedora-r-clang-sanitizer)|
   |test-r-arrow-backwards-compatibility|[![Github Actions](https://github.com/ursacomputing/crossbow/workflows/Crossbow/badge.svg?branch=actions-fdfd53e5a1-github-test-r-arrow-backwards-compatibility)](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-fdfd53e5a1-github-test-r-arrow-backwards-compatibility)|
   |test-r-depsource-bundled|[![Azure](https://dev.azure.com/ursacomputing/crossbow/_apis/build/status/ursacomputing.crossbow?branchName=actions-fdfd53e5a1-azure-test-r-depsource-bundled)](https://dev.azure.com/ursacomputing/crossbow/_build/latest?definitionId=1&branchName=actions-fdfd53e5a1-azure-test-r-depsource-bundled)|
   |test-r-depsource-system|[![Github Actions](https://github.com/ursacomputing/crossbow/workflows/Crossbow/badge.svg?branch=actions-fdfd53e5a1-github-test-r-depsource-system)](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-fdfd53e5a1-github-test-r-depsource-system)|
   |test-r-dev-duckdb|[![Github Actions](https://github.com/ursacomputing/crossbow/workflows/Crossbow/badge.svg?branch=actions-fdfd53e5a1-github-test-r-dev-duckdb)](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-fdfd53e5a1-github-test-r-dev-duckdb)|
   |test-r-devdocs|[![Github Actions](https://github.com/ursacomputing/crossbow/workflows/Crossbow/badge.svg?branch=actions-fdfd53e5a1-github-test-r-devdocs)](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-fdfd53e5a1-github-test-r-devdocs)|
   |test-r-gcc-11|[![Github Actions](https://github.com/ursacomputing/crossbow/workflows/Crossbow/badge.svg?branch=actions-fdfd53e5a1-github-test-r-gcc-11)](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-fdfd53e5a1-github-test-r-gcc-11)|
   |test-r-gcc-12|[![Github Actions](https://github.com/ursacomputing/crossbow/workflows/Crossbow/badge.svg?branch=actions-fdfd53e5a1-github-test-r-gcc-12)](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-fdfd53e5a1-github-test-r-gcc-12)|
   |test-r-install-local|[![Github Actions](https://github.com/ursacomputing/crossbow/workflows/Crossbow/badge.svg?branch=actions-fdfd53e5a1-github-test-r-install-local)](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-fdfd53e5a1-github-test-r-install-local)|
   |test-r-linux-as-cran|[![Github Actions](https://github.com/ursacomputing/crossbow/workflows/Crossbow/badge.svg?branch=actions-fdfd53e5a1-github-test-r-linux-as-cran)](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-fdfd53e5a1-github-test-r-linux-as-cran)|
   |test-r-linux-rchk|[![Github Actions](https://github.com/ursacomputing/crossbow/workflows/Crossbow/badge.svg?branch=actions-fdfd53e5a1-github-test-r-linux-rchk)](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-fdfd53e5a1-github-test-r-linux-rchk)|
   |test-r-linux-valgrind|[![Azure](https://dev.azure.com/ursacomputing/crossbow/_apis/build/status/ursacomputing.crossbow?branchName=actions-fdfd53e5a1-azure-test-r-linux-valgrind)](https://dev.azure.com/ursacomputing/crossbow/_build/latest?definitionId=1&branchName=actions-fdfd53e5a1-azure-test-r-linux-valgrind)|
   |test-r-minimal-build|[![Azure](https://dev.azure.com/ursacomputing/crossbow/_apis/build/status/ursacomputing.crossbow?branchName=actions-fdfd53e5a1-azure-test-r-minimal-build)](https://dev.azure.com/ursacomputing/crossbow/_build/latest?definitionId=1&branchName=actions-fdfd53e5a1-azure-test-r-minimal-build)|
   |test-r-offline-maximal|[![Github Actions](https://github.com/ursacomputing/crossbow/workflows/Crossbow/badge.svg?branch=actions-fdfd53e5a1-github-test-r-offline-maximal)](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-fdfd53e5a1-github-test-r-offline-maximal)|
   |test-r-offline-minimal|[![Azure](https://dev.azure.com/ursacomputing/crossbow/_apis/build/status/ursacomputing.crossbow?branchName=actions-fdfd53e5a1-azure-test-r-offline-minimal)](https://dev.azure.com/ursacomputing/crossbow/_build/latest?definitionId=1&branchName=actions-fdfd53e5a1-azure-test-r-offline-minimal)|
   |test-r-rhub-debian-gcc-devel-lto-latest|[![Azure](https://dev.azure.com/ursacomputing/crossbow/_apis/build/status/ursacomputing.crossbow?branchName=actions-fdfd53e5a1-azure-test-r-rhub-debian-gcc-devel-lto-latest)](https://dev.azure.com/ursacomputing/crossbow/_build/latest?definitionId=1&branchName=actions-fdfd53e5a1-azure-test-r-rhub-debian-gcc-devel-lto-latest)|
   |test-r-rhub-debian-gcc-release-custom-ccache|[![Azure](https://dev.azure.com/ursacomputing/crossbow/_apis/build/status/ursacomputing.crossbow?branchName=actions-fdfd53e5a1-azure-test-r-rhub-debian-gcc-release-custom-ccache)](https://dev.azure.com/ursacomputing/crossbow/_build/latest?definitionId=1&branchName=actions-fdfd53e5a1-azure-test-r-rhub-debian-gcc-release-custom-ccache)|
   |test-r-rhub-ubuntu-gcc-release-latest|[![Azure](https://dev.azure.com/ursacomputing/crossbow/_apis/build/status/ursacomputing.crossbow?branchName=actions-fdfd53e5a1-azure-test-r-rhub-ubuntu-gcc-release-latest)](https://dev.azure.com/ursacomputing/crossbow/_build/latest?definitionId=1&branchName=actions-fdfd53e5a1-azure-test-r-rhub-ubuntu-gcc-release-latest)|
   |test-r-rocker-r-base-latest|[![Azure](https://dev.azure.com/ursacomputing/crossbow/_apis/build/status/ursacomputing.crossbow?branchName=actions-fdfd53e5a1-azure-test-r-rocker-r-base-latest)](https://dev.azure.com/ursacomputing/crossbow/_build/latest?definitionId=1&branchName=actions-fdfd53e5a1-azure-test-r-rocker-r-base-latest)|
   |test-r-rstudio-r-base-4.1-opensuse153|[![Azure](https://dev.azure.com/ursacomputing/crossbow/_apis/build/status/ursacomputing.crossbow?branchName=actions-fdfd53e5a1-azure-test-r-rstudio-r-base-4.1-opensuse153)](https://dev.azure.com/ursacomputing/crossbow/_build/latest?definitionId=1&branchName=actions-fdfd53e5a1-azure-test-r-rstudio-r-base-4.1-opensuse153)|
   |test-r-rstudio-r-base-4.2-centos7-devtoolset-8|[![Azure](https://dev.azure.com/ursacomputing/crossbow/_apis/build/status/ursacomputing.crossbow?branchName=actions-fdfd53e5a1-azure-test-r-rstudio-r-base-4.2-centos7-devtoolset-8)](https://dev.azure.com/ursacomputing/crossbow/_build/latest?definitionId=1&branchName=actions-fdfd53e5a1-azure-test-r-rstudio-r-base-4.2-centos7-devtoolset-8)|
   |test-r-rstudio-r-base-4.2-focal|[![Azure](https://dev.azure.com/ursacomputing/crossbow/_apis/build/status/ursacomputing.crossbow?branchName=actions-fdfd53e5a1-azure-test-r-rstudio-r-base-4.2-focal)](https://dev.azure.com/ursacomputing/crossbow/_build/latest?definitionId=1&branchName=actions-fdfd53e5a1-azure-test-r-rstudio-r-base-4.2-focal)|
   |test-r-ubuntu-22.04|[![Github Actions](https://github.com/ursacomputing/crossbow/workflows/Crossbow/badge.svg?branch=actions-fdfd53e5a1-github-test-r-ubuntu-22.04)](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-fdfd53e5a1-github-test-r-ubuntu-22.04)|
   |test-r-versions|[![Github Actions](https://github.com/ursacomputing/crossbow/workflows/Crossbow/badge.svg?branch=actions-fdfd53e5a1-github-test-r-versions)](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-fdfd53e5a1-github-test-r-versions)|
   |test-ubuntu-18.04-r-sanitizer|[![Azure](https://dev.azure.com/ursacomputing/crossbow/_apis/build/status/ursacomputing.crossbow?branchName=actions-fdfd53e5a1-azure-test-ubuntu-18.04-r-sanitizer)](https://dev.azure.com/ursacomputing/crossbow/_build/latest?definitionId=1&branchName=actions-fdfd53e5a1-azure-test-ubuntu-18.04-r-sanitizer)|


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] nealrichardson commented on pull request #13404: ARROW-16510: [R] Add bindings for GCS filesystem

Posted by GitBox <gi...@apache.org>.
nealrichardson commented on PR #13404:
URL: https://github.com/apache/arrow/pull/13404#issuecomment-1166164809

   Cool, we should be able to pull https://github.com/coryan/google-cloud-cpp/commit/4b60f39b166855dc63bf85c382a5a47e23dd7817.diff into the patch we're applying already and solve both issues (for now, until we upgrade). I'm not sure how to construct the PATCH_COMMAND correctly to do that (my attempts are failing, I think), but it surely is possible. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] nealrichardson commented on a diff in pull request #13404: ARROW-16510: [R] Add bindings for GCS filesystem

Posted by GitBox <gi...@apache.org>.
nealrichardson commented on code in PR #13404:
URL: https://github.com/apache/arrow/pull/13404#discussion_r901794827


##########
cpp/cmake_modules/ThirdpartyToolchain.cmake:
##########
@@ -2794,18 +2795,17 @@ macro(resolve_dependency_absl)
     endforeach()
 
     # Extracted the dependency information using the Abseil pkg-config files:
-    #   grep Requires $PREFIX/pkgconfig/absl_*.pc | \
+    #   grep Requires $PREFIX/lib/pkgconfig/absl_*.pc | \
     #   sed -e 's;.*/absl_;set_property(TARGET absl::;' \
     #       -e 's/.pc:Requires:/ PROPERTY INTERFACE_LINK_LIBRARIES /' \
-    #       -e 's/ = 20210324,//g' \
-    #       -e 's/ = 20210324//g' \
+    #       -E -e 's/ = 20[0-9]{6},?//g' \

Review Comment:
   I simplified this regex so that it is robust to future abseil version bumps (at least until 2030!)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] nealrichardson commented on a diff in pull request #13404: ARROW-16510: [R] Add bindings for GCS filesystem

Posted by GitBox <gi...@apache.org>.
nealrichardson commented on code in PR #13404:
URL: https://github.com/apache/arrow/pull/13404#discussion_r901798660


##########
r/src/filesystem.cpp:
##########
@@ -33,6 +33,12 @@ const char* r6_class_name<fs::FileSystem>::get(
     return "LocalFileSystem";
   } else if (type_name == "s3") {
     return "S3FileSystem";
+  } else if (type_name == "gcs") {
+    return "GcsFileSystem";
+  } else if (type_name == "abfs") {
+    return "AzureBlobFileSystem";
+  } else if (type_name == "hdfs") {
+    return "HadoopFileSystem";

Review Comment:
   Put them here for completeness but we should probably comment this block out for now since these R6 classes don't exist yet and the default fallback to FileSystem might allow things to Just Work if you're linking to a C++ library that has support for these built.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] nealrichardson commented on pull request #13404: ARROW-16510: [R] Add bindings for GCS filesystem

Posted by GitBox <gi...@apache.org>.
nealrichardson commented on PR #13404:
URL: https://github.com/apache/arrow/pull/13404#issuecomment-1162466449

   Well, that didn't work, there must be some other piece I'm missing: 
   
   ```
   -- Configuring done
   CMake Error at cmake_modules/BuildUtils.cmake:112 (file):
     Error evaluating generator expression:
       $<TARGET_FILE:absl::memory>
     Target "absl::memory" is not an executable or library.
   Call Stack (most recent call first):
     src/arrow/CMakeLists.txt:608 (create_merged_static_lib)
   CMake Error at cmake_modules/BuildUtils.cmake:112 (file):
     Error evaluating generator expression:
       $<TARGET_FILE:absl::memory>
     Target "absl::memory" is not an executable or library.
   Call Stack (most recent call first):
     src/arrow/CMakeLists.txt:608 (create_merged_static_lib)
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] nealrichardson commented on pull request #13404: ARROW-16510: [R] Add bindings for GCS filesystem

Posted by GitBox <gi...@apache.org>.
nealrichardson commented on PR #13404:
URL: https://github.com/apache/arrow/pull/13404#issuecomment-1165551127

   
   
   
   > I confirmed that we can solve the Curl's symbol problem by defining `CURL_STATICLIB` in google-cloud-cpp. I also opened a pull request to google-cloud-cpp: [googleapis/google-cloud-cpp#9345](https://github.com/googleapis/google-cloud-cpp/pull/9345)
   > 
   
   🎉 
   
   > R 4.2 CI job succeeded and R 4.1 CI job failed because R 4.2 uses ucrt and R 4.1 uses msvcrt. ucrt provides `_dupenv_s()` but msvcrt doesn't. google-cloud-cpp uses `_dupenv_s()`. Our existing CI jobs disable GCS because of this: https://github.com/apache/arrow/blob/master/.github/workflows/cpp.yml#L279-L281
   > 
   > Error log: https://github.com/apache/arrow/runs/7036333396?check_suite_focus=true#step:11:589
   > 
   > ```
   > C:/rtools40/mingw32/bin/../lib/gcc/i686-w64-mingw32/8.3.0/../../../../i686-w64-mingw32/bin/ld.exe: ../windows/arrow-8.0.0.9000/lib/i386/libarrow_bundled_dependencies.a(getenv.cc.obj):(.text+0x22): undefined reference to `_imp___dupenv_s'
   > ```
   
   Is this not something that can be patched/shimmed in?
   
   > 
   > Can we disable GCS for R 4.1?
   
   Surely. I'll look for the cleanest way to do that.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] coryan commented on pull request #13404: ARROW-16510: [R] Add bindings for GCS filesystem

Posted by GitBox <gi...@apache.org>.
coryan commented on PR #13404:
URL: https://github.com/apache/arrow/pull/13404#issuecomment-1165558387

   > > I confirmed that we can solve the Curl's symbol problem by defining `CURL_STATICLIB` in google-cloud-cpp. I also opened a pull request to google-cloud-cpp: [googleapis/google-cloud-cpp#9345](https://github.com/googleapis/google-cloud-cpp/pull/9345)
   > 
   > 🎉
   
   FWIW, that PR just got merged, and will be part of the next release, due early in 2022-07.
   
   > 
   > > R 4.2 CI job succeeded and R 4.1 CI job failed because R 4.2 uses ucrt and R 4.1 uses msvcrt. ucrt provides `_dupenv_s()` but msvcrt doesn't. google-cloud-cpp uses `_dupenv_s()`.
   
   FYI: not anymore, that got fixed in in googleapis/google-cloud-cpp#9140 and is available in the latest release (v1.42.0).
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] nealrichardson commented on pull request #13404: ARROW-16510: [R] Add bindings for GCS filesystem

Posted by GitBox <gi...@apache.org>.
nealrichardson commented on PR #13404:
URL: https://github.com/apache/arrow/pull/13404#issuecomment-1166314931

   With the upgrade to google-cloud-cpp 1.42.0, the R Windows builds are now passing. 
   
   Python failure is ARROW-16873
   
   MingGW 32 failure is OOM https://github.com/apache/arrow/runs/7054579512?check_suite_focus=true#step:7:1019; maybe we need to turn off unity build there?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] kou commented on pull request #13404: ARROW-16510: [R] Add bindings for GCS filesystem

Posted by GitBox <gi...@apache.org>.
kou commented on PR #13404:
URL: https://github.com/apache/arrow/pull/13404#issuecomment-1166353926

   > MinGW 64 failure is something about boost in compiling the GCS filesystem tests: https://github.com/apache/arrow/runs/7054579528?check_suite_focus=true#step:7:1310
   
   We need a preparation to use boost/process.hpp with Mingw-w64 like https://github.com/apache/arrow/blob/master/cpp/src/arrow/filesystem/s3_test_util.cc#L24-L38 . We can handle it in a follow-up task.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] github-actions[bot] commented on pull request #13404: ARROW-16510: [R] Add bindings for GCS filesystem

Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #13404:
URL: https://github.com/apache/arrow/pull/13404#issuecomment-1166360875

   Revision: 226c31b3ec0fe94a55e10604bb4621b2fde3bbee
   
   Submitted crossbow builds: [ursacomputing/crossbow @ actions-5668d3fe2c](https://github.com/ursacomputing/crossbow/branches/all?query=actions-5668d3fe2c)
   
   |Task|Status|
   |----|------|
   |homebrew-r-brew|[![Github Actions](https://github.com/ursacomputing/crossbow/workflows/Crossbow/badge.svg?branch=actions-5668d3fe2c-github-homebrew-r-brew)](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-5668d3fe2c-github-homebrew-r-brew)|
   |test-r-depsource-system|[![Github Actions](https://github.com/ursacomputing/crossbow/workflows/Crossbow/badge.svg?branch=actions-5668d3fe2c-github-test-r-depsource-system)](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-5668d3fe2c-github-test-r-depsource-system)|
   |test-r-offline-maximal|[![Github Actions](https://github.com/ursacomputing/crossbow/workflows/Crossbow/badge.svg?branch=actions-5668d3fe2c-github-test-r-offline-maximal)](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-5668d3fe2c-github-test-r-offline-maximal)|


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] nealrichardson commented on pull request #13404: ARROW-16510: [R] Add bindings for GCS filesystem

Posted by GitBox <gi...@apache.org>.
nealrichardson commented on PR #13404:
URL: https://github.com/apache/arrow/pull/13404#issuecomment-1163637187

   For the R Windows packages, I need to use a static libcurl, and stack overflow tells me that I need to define `CURL_STATICLIB` for the google-cloud-cpp build. (The thing I just pushed won't work, it's not going to put it in the right place.) I don't see a way I can pass that in to the build macro from the outside (as in, from the `cmake` command that builds arrow), and I don't think we'd want to hard-code that. Is it possible in cmake to tell if `find_package(curl)` found a static or shared library? Then I think I could conditionally add that only when appropriate. @kou thoughts?
   
   cc @jeroen in case you have different ideas


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] jeroen commented on pull request #13404: ARROW-16510: [R] Add bindings for GCS filesystem

Posted by GitBox <gi...@apache.org>.
jeroen commented on PR #13404:
URL: https://github.com/apache/arrow/pull/13404#issuecomment-1163655426

   I think `find_package()` is supposed to shell out to `pkg-config --cflags libcurl` and `pkg-config --libs libcurl` which should automatically include the macro and static libraries when nee. For example with rtools40 I see:
   
   ```
   $ pkg-config --cflags libcurl
   -DCURL_STATICLIB -IC:/rtools40/ucrt64/include
   
   $ pkg-config --libs libcurl
   -LC:/rtools40/ucrt64/lib -LD:/a/_temp/msys/msys64/ucrt64/lib -lcurl -lnormaliz -lssh2 -lcrypt32 -lgdi32 -lws2_32 -lssl -lcrypto -lws2_32 -lgdi32 -lcrypt32 -lz -lssl -lcrypto -lssl -lcrypto -lws2_32 -lgdi32 -lcrypt32 -lgdi32 -lcrypt32 -lwldap32 -lz -lws2_32
   ```
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] nealrichardson commented on pull request #13404: ARROW-16510: [R] Add bindings for GCS filesystem

Posted by GitBox <gi...@apache.org>.
nealrichardson commented on PR #13404:
URL: https://github.com/apache/arrow/pull/13404#issuecomment-1164495701

   I turned on thirdparty build verbosity, and it looks like `-DCURL_STATICLIB` is only being added to the libarrow compilation, not the google_cloud_cpp compilation, so it still fails to link: https://github.com/apache/arrow/runs/7025275214?check_suite_focus=true#step:8:5397
   
   @kou seems like this is close, any idea now?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] github-actions[bot] commented on pull request #13404: ARROW-16510: [R] Add bindings for GCS filesystem

Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #13404:
URL: https://github.com/apache/arrow/pull/13404#issuecomment-1162381352

   Revision: df5033b5a4d4a9164526f0fbe6df07539e6ddf81
   
   Submitted crossbow builds: [ursacomputing/crossbow @ actions-b4426052af](https://github.com/ursacomputing/crossbow/branches/all?query=actions-b4426052af)
   
   |Task|Status|
   |----|------|
   |homebrew-r-autobrew|[![Github Actions](https://github.com/ursacomputing/crossbow/workflows/Crossbow/badge.svg?branch=actions-b4426052af-github-homebrew-r-autobrew)](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-b4426052af-github-homebrew-r-autobrew)|


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] nealrichardson commented on a diff in pull request #13404: ARROW-16510: [R] Add bindings for GCS filesystem

Posted by GitBox <gi...@apache.org>.
nealrichardson commented on code in PR #13404:
URL: https://github.com/apache/arrow/pull/13404#discussion_r901796837


##########
r/data-raw/codegen.R:
##########
@@ -44,7 +44,9 @@ get_exported_functions <- function(decorations, export_tag) {
   out <- decorations %>%
     filter(decoration %in% paste0(export_tag, "::export")) %>%
     mutate(functions = map(context, decor:::parse_cpp_function)) %>%
-    { vec_cbind(., vec_rbind(!!!pull(., functions))) } %>%
+    {

Review Comment:
   This and the rest of the changes in the file seem to be just `styler` running on the file (vscode did it on save). Since the code generated after these style changes looks identical (there is no massive diff in the arrowExports.* files), maybe we can remove the styler_exclusion on this file?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] nealrichardson commented on pull request #13404: ARROW-16510: [R] Add bindings for GCS filesystem

Posted by GitBox <gi...@apache.org>.
nealrichardson commented on PR #13404:
URL: https://github.com/apache/arrow/pull/13404#issuecomment-1166363266

   > > MinGW 64 failure is something about boost in compiling the GCS filesystem tests: https://github.com/apache/arrow/runs/7054579528?check_suite_focus=true#step:7:1310
   > 
   > We need a preparation to use boost/process.hpp with Mingw-w64 like https://github.com/apache/arrow/blob/master/cpp/src/arrow/filesystem/s3_test_util.cc#L24-L38 . We can handle it in a follow-up task.
   
   I made ARROW-16906 for that and added a TODO comment in the workflow.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] nealrichardson commented on a diff in pull request #13404: ARROW-16510: [R] Add bindings for GCS filesystem

Posted by GitBox <gi...@apache.org>.
nealrichardson commented on code in PR #13404:
URL: https://github.com/apache/arrow/pull/13404#discussion_r901794205


##########
cpp/cmake_modules/ThirdpartyToolchain.cmake:
##########
@@ -2690,15 +2690,16 @@ macro(resolve_dependency_absl)
         stacktrace
         status
         statusor
-        strerror
         str_format_internal
+        strerror
         strings
         strings_internal
         symbolize
         synchronization
         throw_delegate
         time
-        time_zone)
+        time_zone
+        wyhash)

Review Comment:
   Most of the cmake changes here seem to just be differences in sorting (due to my locale?), but the omission of `wyhash` seems to be legitimately new.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] coryan commented on pull request #13404: ARROW-16510: [R] Add bindings for GCS filesystem

Posted by GitBox <gi...@apache.org>.
coryan commented on PR #13404:
URL: https://github.com/apache/arrow/pull/13404#issuecomment-1163374197

   If you have a VM with abseil installed, you could do something like:
   
   ```sh
   pkg-config --libs absl_memory absl_strings absl_str_format absl_time absl_variant absl_base absl_memory absl_optional absl_span absl_time absl_variant
   -L/usr/local/lib64 -labsl_str_format_internal -labsl_bad_optional_access -labsl_time -labsl_civil_time -labsl_strings -labsl_strings_internal -lrt -labsl_base -labsl_spinlock_wait -labsl_int128 -labsl_throw_delegate -labsl_time_zone -labsl_bad_variant_access -labsl_raw_logging_internal -labsl_log_severity
   ```
   
   Which (a) removes the header-only libraries, and (b) saves you some whack-a-moling (maybe).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] nealrichardson commented on a diff in pull request #13404: ARROW-16510: [R] Add bindings for GCS filesystem

Posted by GitBox <gi...@apache.org>.
nealrichardson commented on code in PR #13404:
URL: https://github.com/apache/arrow/pull/13404#discussion_r906803022


##########
r/configure:
##########
@@ -229,44 +229,48 @@ if [ $? -eq 0 ]; then
   # Check for features
   LIB_DIR=`echo $PKG_DIRS | sed -e 's/^-L//'`
   ARROW_OPTS_CMAKE="$LIB_DIR/cmake/arrow/ArrowOptions.cmake"
-  # Check for Parquet
-  grep -i 'set(ARROW_PARQUET "ON")' $ARROW_OPTS_CMAKE >/dev/null 2>&1
-  if [ $? -eq 0 ]; then
+
+  arrow_built_with() {
+    # Function to check cmake options for features
+    grep -i 'set('"$1"' "ON")' $ARROW_OPTS_CMAKE >/dev/null 2>&1

Review Comment:
   `grep -q` doesn't suppress stderr though, and I want this to be completely silent



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] nealrichardson commented on pull request #13404: ARROW-16510: [R] Add bindings for GCS filesystem

Posted by GitBox <gi...@apache.org>.
nealrichardson commented on PR #13404:
URL: https://github.com/apache/arrow/pull/13404#issuecomment-1166360724

   @github-actions crossbow submit test-r-offline-maximal test-r-depsource-system homebrew-r-brew


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] nealrichardson commented on pull request #13404: ARROW-16510: [R] Add bindings for GCS filesystem

Posted by GitBox <gi...@apache.org>.
nealrichardson commented on PR #13404:
URL: https://github.com/apache/arrow/pull/13404#issuecomment-1166363297

   @github-actions crossbow submit homebrew-r-brew


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] wjones127 commented on a diff in pull request #13404: ARROW-16510: [R] Add bindings for GCS filesystem

Posted by GitBox <gi...@apache.org>.
wjones127 commented on code in PR #13404:
URL: https://github.com/apache/arrow/pull/13404#discussion_r904458342


##########
r/configure.win:
##########
@@ -104,6 +105,10 @@ function configure_dev() {
     PKG_CFLAGS="$PKG_CFLAGS -DARROW_R_WITH_S3"
   fi
 
+  if [ $(cmake_option ARROW_GCS) -eq 1 ]; then
+    PKG_CFLAGS="$PKG_CFLAGS -DARROW_R_WITH_GCS -DCURL_STATICLIB"
+  fi

Review Comment:
   Nevermind this might not help if libcurl is already statically linked in the arrow binaries.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] nealrichardson commented on a diff in pull request #13404: ARROW-16510: [R] Add bindings for GCS filesystem

Posted by GitBox <gi...@apache.org>.
nealrichardson commented on code in PR #13404:
URL: https://github.com/apache/arrow/pull/13404#discussion_r901799543


##########
r/src/filesystem.cpp:
##########
@@ -331,3 +337,77 @@ std::string fs___S3FileSystem__region(const std::shared_ptr<fs::S3FileSystem>& f
 }
 
 #endif
+
+#if defined(ARROW_R_WITH_GCS)
+
+#include <arrow/filesystem/gcsfs.h>
+
+std::shared_ptr<arrow::KeyValueMetadata> strings_to_kvm(cpp11::strings metadata);
+
+// [[gcs::export]]
+std::shared_ptr<fs::GcsFileSystem> fs___GcsFileSystem__Make(bool anonymous,
+                                                            cpp11::list options) {
+  fs::GcsOptions gcs_opts;
+
+  // Handle auth (anonymous, credentials, default)
+  // (validation/internal coherence handled in R)
+  if (anonymous) {
+    gcs_opts = fs::GcsOptions::Anonymous();
+  } else if (!Rf_isNull(options["access_token"])) {
+    // Convert POSIXct timestamp seconds to nanoseconds
+    std::chrono::nanoseconds ns_count(
+        static_cast<int64_t>(cpp11::as_cpp<double>(options["expiration"])) * 1000000000);
+    auto expiration_timepoint =
+        fs::TimePoint(std::chrono::duration_cast<fs::TimePoint::duration>(ns_count));
+    gcs_opts = fs::GcsOptions::FromAccessToken(
+        cpp11::as_cpp<std::string>(options["access_token"]), expiration_timepoint);
+    // TODO: implement FromImpersonatedServiceAccount
+    // } else if (base_credentials != "") {
+    //   // static GcsOptions FromImpersonatedServiceAccount(
+    //   // const GcsCredentials& base_credentials, const std::string&
+    //   target_service_account);
+    //   // TODO: construct GcsCredentials
+    //   gcs_opts = fs::GcsOptions::FromImpersonatedServiceAccount(base_credentials,
+    //                                                             target_service_account);
+  } else if (!Rf_isNull(options["json_credentials"])) {
+    gcs_opts = fs::GcsOptions::FromServiceAccountCredentials(
+        cpp11::as_cpp<std::string>(options["json_credentials"]));
+  } else {
+    gcs_opts = fs::GcsOptions::Defaults();
+  }
+
+  // Handle other attributes
+  if (!Rf_isNull(options["endpoint_override"])) {
+    gcs_opts.endpoint_override = cpp11::as_cpp<std::string>(options["endpoint_override"]);
+  }
+
+  if (!Rf_isNull(options["scheme"])) {
+    gcs_opts.scheme = cpp11::as_cpp<std::string>(options["scheme"]);
+  }
+
+  // /// \brief Location to use for creating buckets.
+  if (!Rf_isNull(options["default_bucket_location"])) {
+    gcs_opts.default_bucket_location =
+        cpp11::as_cpp<std::string>(options["default_bucket_location"]);
+  }
+  // /// \brief If set used to control total time allowed for retrying underlying
+  // /// errors.
+  // ///
+  // /// The default policy is to retry for up to 15 minutes.
+  if (!Rf_isNull(options["retry_limit_seconds"])) {
+    gcs_opts.retry_limit_seconds = cpp11::as_cpp<double>(options["retry_limit_seconds"]);
+  }
+
+  // /// \brief Default metadata for OpenOutputStream.
+  // ///
+  // /// This will be ignored if non-empty metadata is passed to OpenOutputStream.
+  if (!Rf_isNull(options["default_metadata"])) {
+    gcs_opts.default_metadata = strings_to_kvm(options["default_metadata"]);
+  }
+
+  auto io_context = arrow::io::IOContext(gc_memory_pool());
+  // TODO: S3FileSystem::Make returns a Result and uses ValueOrStop but this doesn't?

Review Comment:
   @emkornfield what do you think about this? Compare to https://github.com/apache/arrow/blob/master/cpp/src/arrow/filesystem/s3fs.h#L296-L297



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] nealrichardson commented on a diff in pull request #13404: ARROW-16510: [R] Add bindings for GCS filesystem

Posted by GitBox <gi...@apache.org>.
nealrichardson commented on code in PR #13404:
URL: https://github.com/apache/arrow/pull/13404#discussion_r901795365


##########
cpp/cmake_modules/ThirdpartyToolchain.cmake:
##########
@@ -3542,18 +3544,6 @@ macro(resolve_dependency_absl)
                    APPEND
                    PROPERTY INTERFACE_LINK_LIBRARIES ${CoreFoundation})
     endif()
-    set_property(TARGET absl::type_traits PROPERTY INTERFACE_LINK_LIBRARIES absl::config)

Review Comment:
   These are duplicated above so I removed them here; I suspect these were an artifact of a bad merge



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] nealrichardson commented on pull request #13404: ARROW-16510: [R] Add bindings for GCS filesystem

Posted by GitBox <gi...@apache.org>.
nealrichardson commented on PR #13404:
URL: https://github.com/apache/arrow/pull/13404#issuecomment-1162419856

   @coryan any ideas about that undefined abseil symbol mentioned above? `undefined symbol: _ZN4absl12lts_2021110212RFC3339_fullE`
   
   Similar issue in the homebrew build `Symbol not found: __ZN4absl12lts_2021110210FormatTimeENS0_11string_viewENS0_4TimeENS0_8TimeZoneE` See https://github.com/apache/arrow/pull/13407#issuecomment-1161923135


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] coryan commented on pull request #13404: ARROW-16510: [R] Add bindings for GCS filesystem

Posted by GitBox <gi...@apache.org>.
coryan commented on PR #13404:
URL: https://github.com/apache/arrow/pull/13404#issuecomment-1162474007

   I suspect `absl::memory` is a header-only library, so it does not have a `$<TARGET_FILE>`.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] wjones127 commented on a diff in pull request #13404: ARROW-16510: [R] Add bindings for GCS filesystem

Posted by GitBox <gi...@apache.org>.
wjones127 commented on code in PR #13404:
URL: https://github.com/apache/arrow/pull/13404#discussion_r904397786


##########
r/configure.win:
##########
@@ -104,6 +105,10 @@ function configure_dev() {
     PKG_CFLAGS="$PKG_CFLAGS -DARROW_R_WITH_S3"
   fi
 
+  if [ $(cmake_option ARROW_GCS) -eq 1 ]; then
+    PKG_CFLAGS="$PKG_CFLAGS -DARROW_R_WITH_GCS -DCURL_STATICLIB"
+  fi

Review Comment:
   I think we should be able to let pkg-config handle adding the flags and libraries for libcurl:
   
   ```suggestion
     if [ $(cmake_option ARROW_GCS) -eq 1 ]; then
       PKG_CFLAGS="$PKG_CFLAGS -DARROW_R_WITH_GCS"
       PKG_CONFIG_PACKAGES="$PKG_CONFIG_PACKAGES libcurl"
     fi
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] kou commented on pull request #13404: ARROW-16510: [R] Add bindings for GCS filesystem

Posted by GitBox <gi...@apache.org>.
kou commented on PR #13404:
URL: https://github.com/apache/arrow/pull/13404#issuecomment-1165203707

   I confirmed that we can solve the Curl's symbol problem by defining `CURL_STATICLIB` in google-cloud-cpp. I also opened a pull request to google-cloud-cpp: https://github.com/googleapis/google-cloud-cpp/pull/9345
   
   R 4.2 CI job succeeded and R 4.1 CI job failed because R 4.2 uses ucrt and R 4.1 uses msvcrt. ucrt provides `_dupenv_s()` but msvcrt doesn't. google-cloud-cpp uses `_dupenv_s()`. Our existing CI jobs disable GCS because of this: https://github.com/apache/arrow/blob/master/.github/workflows/cpp.yml#L279-L281
   
   Can we disable GCS for R 4.1?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] github-actions[bot] commented on pull request #13404: ARROW-16510: [R] Add bindings for GCS filesystem

Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #13404:
URL: https://github.com/apache/arrow/pull/13404#issuecomment-1166363431

   Revision: a2a5e87630000bc0577f651e4087f248eca8d332
   
   Submitted crossbow builds: [ursacomputing/crossbow @ actions-06d32e61d7](https://github.com/ursacomputing/crossbow/branches/all?query=actions-06d32e61d7)
   
   |Task|Status|
   |----|------|
   |homebrew-r-brew|[![Github Actions](https://github.com/ursacomputing/crossbow/workflows/Crossbow/badge.svg?branch=actions-06d32e61d7-github-homebrew-r-brew)](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-06d32e61d7-github-homebrew-r-brew)|


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] nealrichardson commented on a diff in pull request #13404: ARROW-16510: [R] Add bindings for GCS filesystem

Posted by GitBox <gi...@apache.org>.
nealrichardson commented on code in PR #13404:
URL: https://github.com/apache/arrow/pull/13404#discussion_r901797516


##########
r/src/arrow_types.h:
##########
@@ -69,6 +69,14 @@ namespace ds = ::arrow::dataset;
 namespace compute = ::arrow::compute;
 namespace fs = ::arrow::fs;
 
+#if defined(ARROW_R_WITH_GCS)
+namespace arrow {
+namespace fs {
+class GcsFileSystem;

Review Comment:
   TODO(me): I think this should be put in arrow/filesystem/type_fwd.h



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] nealrichardson commented on a diff in pull request #13404: ARROW-16510: [R] Add bindings for GCS filesystem

Posted by GitBox <gi...@apache.org>.
nealrichardson commented on code in PR #13404:
URL: https://github.com/apache/arrow/pull/13404#discussion_r906801722


##########
cpp/cmake_modules/ThirdpartyToolchain.cmake:
##########
@@ -4046,25 +4054,47 @@ macro(build_google_cloud_cpp_storage)
                                    "${GOOGLE_CLOUD_CPP_STATIC_LIBRARY_COMMON}"
                                    INTERFACE_INCLUDE_DIRECTORIES
                                    "${GOOGLE_CLOUD_CPP_INCLUDE_DIR}")
+  # Refer to https://github.com/googleapis/google-cloud-cpp/blob/main/google/cloud/google_cloud_cpp_common.cmake
+  # (subsitute `main` for the SHA of the version we use)
+  # Version 1.39.0 is at a different place (they refactored after):
+  # https://github.com/googleapis/google-cloud-cpp/blob/29e5af8ca9b26cec62106d189b50549f4dc1c598/google/cloud/CMakeLists.txt#L146-L155
   set_property(TARGET google-cloud-cpp::common
                PROPERTY INTERFACE_LINK_LIBRARIES
-                        absl::any
-                        absl::flat_hash_map
+                        absl::base
                         absl::memory
                         absl::optional
+                        absl::span
                         absl::time
+                        absl::variant
                         Threads::Threads
                         OpenSSL::Crypto)
 
+  add_library(google-cloud-cpp::rest-internal STATIC IMPORTED)
+  set_target_properties(google-cloud-cpp::rest-internal
+                        PROPERTIES IMPORTED_LOCATION
+                                   "${GOOGLE_CLOUD_CPP_STATIC_LIBRARY_REST_INTERNAL}"
+                                   INTERFACE_INCLUDE_DIRECTORIES
+                                   "${GOOGLE_CLOUD_CPP_INCLUDE_DIR}")
+  set_property(TARGET google-cloud-cpp::rest-internal
+               PROPERTY INTERFACE_LINK_LIBRARIES
+                        absl::span
+                        google-cloud-cpp::common
+                        CURL::libcurl
+                        nlohmann_json::nlohmann_json
+                        OpenSSL::SSL
+                        OpenSSL::Crypto)
+
   add_library(google-cloud-cpp::storage STATIC IMPORTED)
   set_target_properties(google-cloud-cpp::storage
                         PROPERTIES IMPORTED_LOCATION
                                    "${GOOGLE_CLOUD_CPP_STATIC_LIBRARY_STORAGE}"
                                    INTERFACE_INCLUDE_DIRECTORIES
                                    "${GOOGLE_CLOUD_CPP_INCLUDE_DIR}")
+  # Update this from https://github.com/googleapis/google-cloud-cpp/blob/main/google/cloud/storage/google_cloud_cpp_storage.cmake
   set_property(TARGET google-cloud-cpp::storage
                PROPERTY INTERFACE_LINK_LIBRARIES
                         google-cloud-cpp::common

Review Comment:
   I guess so, but since it's copied from upstream, I'd rather leave it there for completeness unless it's harming anything.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] nealrichardson merged pull request #13404: ARROW-16510: [R] Add bindings for GCS filesystem

Posted by GitBox <gi...@apache.org>.
nealrichardson merged PR #13404:
URL: https://github.com/apache/arrow/pull/13404


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] nealrichardson commented on pull request #13404: ARROW-16510: [R] Add bindings for GCS filesystem

Posted by GitBox <gi...@apache.org>.
nealrichardson commented on PR #13404:
URL: https://github.com/apache/arrow/pull/13404#issuecomment-1162173479

   R debian build is failing with an undefined symbol from absl, a different one from what I saw locally (on macOS): 
   
   https://github.com/apache/arrow/runs/6989838112?check_suite_focus=true#step:5:2990
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] coryan commented on pull request #13404: ARROW-16510: [R] Add bindings for GCS filesystem

Posted by GitBox <gi...@apache.org>.
coryan commented on PR #13404:
URL: https://github.com/apache/arrow/pull/13404#issuecomment-1162435724

   TL;DR; not much of an idea, sorry.
   
   > any ideas about that undefined abseil symbol mentioned above?
   
   I assume "above" refers to https://github.com/apache/arrow/runs/6989838112?check_suite_focus=true#step:5:2990 just let me know if I read that wrong.
   
   > `undefined symbol: _ZN4absl12lts_2021110212RFC3339_fullE`
   
   Seems like Abseil is not being linked?  The log says:
   
   > -- All bundled static libraries: Snappy::snappy;thrift::thrift;jemalloc;mimalloc::mimalloc;lz4::lz4;zstd::libzstd;re2::re2;utf8proc::utf8proc;google-cloud-cpp::storage;google-cloud-cpp::common;aws-cpp-sdk-identity-management;aws-cpp-sdk-sts;aws-cpp-sdk-cognito-identity;aws-cpp-sdk-s3;aws-cpp-sdk-core;AWS::aws-c-event-stream;AWS::aws-checksums;AWS::aws-c-common
   
   That seems to be missing Abseil, but I do not quite understand how the list of bundled dependencies is created.
   
   > 
   > Similar issue in the homebrew build `Symbol not found: __ZN4absl12lts_2021110210FormatTimeENS0_11string_viewENS0_4TimeENS0_8TimeZoneE` See [#13407 (comment)](https://github.com/apache/arrow/pull/13407#issuecomment-1161923135)
   
   Probably a similar problem as above.  Less likely, it could be Abseil's sensitivity to C++11 vs. C++17 building (see abseil/abseil-cpp#696), I always think it could be that issue if `string_view`, `variant` or `any` are involved.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] kou commented on pull request #13404: ARROW-16510: [R] Add bindings for GCS filesystem

Posted by GitBox <gi...@apache.org>.
kou commented on PR #13404:
URL: https://github.com/apache/arrow/pull/13404#issuecomment-1165027323

   Oh, sorry. My patch was wrong. We need to do the similar thing in google-cloud-cpp no in our `ThirdpartyToolchain.cmake`. I'll push a fix to this branch later.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] kou commented on a diff in pull request #13404: ARROW-16510: [R] Add bindings for GCS filesystem

Posted by GitBox <gi...@apache.org>.
kou commented on code in PR #13404:
URL: https://github.com/apache/arrow/pull/13404#discussion_r906740843


##########
cpp/cmake_modules/ThirdpartyToolchain.cmake:
##########
@@ -4075,11 +4105,39 @@ macro(build_google_cloud_cpp_storage)
                         CURL::libcurl
                         Threads::Threads
                         OpenSSL::SSL
-                        OpenSSL::Crypto)
+                        OpenSSL::Crypto
+                        ZLIB::ZLIB)
   add_dependencies(google-cloud-cpp::storage google_cloud_cpp_ep)
 
-  list(APPEND ARROW_BUNDLED_STATIC_LIBS google-cloud-cpp::storage
+  list(APPEND
+       ARROW_BUNDLED_STATIC_LIBS
+       google-cloud-cpp::rest-internal
+       google-cloud-cpp::storage

Review Comment:
   This may not be a problem but could you put `google-cloud-cpp::rest-internal` after `google-cloud-cpp::storage` because `google-cloud-cpp::storage` depends on `google-cloud-cpp::rest-internal`?
   
   ```suggestion
          google-cloud-cpp::storage
          google-cloud-cpp::rest-internal
   ```



##########
r/configure:
##########
@@ -229,44 +229,48 @@ if [ $? -eq 0 ]; then
   # Check for features
   LIB_DIR=`echo $PKG_DIRS | sed -e 's/^-L//'`
   ARROW_OPTS_CMAKE="$LIB_DIR/cmake/arrow/ArrowOptions.cmake"
-  # Check for Parquet
-  grep -i 'set(ARROW_PARQUET "ON")' $ARROW_OPTS_CMAKE >/dev/null 2>&1
-  if [ $? -eq 0 ]; then
+
+  arrow_built_with() {
+    # Function to check cmake options for features
+    grep -i 'set('"$1"' "ON")' $ARROW_OPTS_CMAKE >/dev/null 2>&1

Review Comment:
   FYI: We can use `grep -q ...` instead of `grep ... >/dev/null 2>&1`.



##########
cpp/cmake_modules/ThirdpartyToolchain.cmake:
##########
@@ -4046,25 +4054,47 @@ macro(build_google_cloud_cpp_storage)
                                    "${GOOGLE_CLOUD_CPP_STATIC_LIBRARY_COMMON}"
                                    INTERFACE_INCLUDE_DIRECTORIES
                                    "${GOOGLE_CLOUD_CPP_INCLUDE_DIR}")
+  # Refer to https://github.com/googleapis/google-cloud-cpp/blob/main/google/cloud/google_cloud_cpp_common.cmake
+  # (subsitute `main` for the SHA of the version we use)
+  # Version 1.39.0 is at a different place (they refactored after):
+  # https://github.com/googleapis/google-cloud-cpp/blob/29e5af8ca9b26cec62106d189b50549f4dc1c598/google/cloud/CMakeLists.txt#L146-L155
   set_property(TARGET google-cloud-cpp::common
                PROPERTY INTERFACE_LINK_LIBRARIES
-                        absl::any
-                        absl::flat_hash_map
+                        absl::base
                         absl::memory
                         absl::optional
+                        absl::span
                         absl::time
+                        absl::variant
                         Threads::Threads
                         OpenSSL::Crypto)
 
+  add_library(google-cloud-cpp::rest-internal STATIC IMPORTED)
+  set_target_properties(google-cloud-cpp::rest-internal
+                        PROPERTIES IMPORTED_LOCATION
+                                   "${GOOGLE_CLOUD_CPP_STATIC_LIBRARY_REST_INTERNAL}"
+                                   INTERFACE_INCLUDE_DIRECTORIES
+                                   "${GOOGLE_CLOUD_CPP_INCLUDE_DIR}")
+  set_property(TARGET google-cloud-cpp::rest-internal
+               PROPERTY INTERFACE_LINK_LIBRARIES
+                        absl::span
+                        google-cloud-cpp::common
+                        CURL::libcurl
+                        nlohmann_json::nlohmann_json
+                        OpenSSL::SSL
+                        OpenSSL::Crypto)
+
   add_library(google-cloud-cpp::storage STATIC IMPORTED)
   set_target_properties(google-cloud-cpp::storage
                         PROPERTIES IMPORTED_LOCATION
                                    "${GOOGLE_CLOUD_CPP_STATIC_LIBRARY_STORAGE}"
                                    INTERFACE_INCLUDE_DIRECTORIES
                                    "${GOOGLE_CLOUD_CPP_INCLUDE_DIR}")
+  # Update this from https://github.com/googleapis/google-cloud-cpp/blob/main/google/cloud/storage/google_cloud_cpp_storage.cmake
   set_property(TARGET google-cloud-cpp::storage
                PROPERTY INTERFACE_LINK_LIBRARIES
                         google-cloud-cpp::common

Review Comment:
   We can remove this because `google-cloud-cpp-rest-internal` depends on `google-cloud-cpp::common`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] github-actions[bot] commented on pull request #13404: ARROW-16510: [R] Add bindings for GCS filesystem

Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #13404:
URL: https://github.com/apache/arrow/pull/13404#issuecomment-1160585559

   https://issues.apache.org/jira/browse/ARROW-16510


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] nealrichardson commented on pull request #13404: ARROW-16510: [R] Add bindings for GCS filesystem

Posted by GitBox <gi...@apache.org>.
nealrichardson commented on PR #13404:
URL: https://github.com/apache/arrow/pull/13404#issuecomment-1162380105

   @github-actions crossbow submit homebrew-r-autobrew


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] emkornfield commented on a diff in pull request #13404: ARROW-16510: [R] Add bindings for GCS filesystem

Posted by GitBox <gi...@apache.org>.
emkornfield commented on code in PR #13404:
URL: https://github.com/apache/arrow/pull/13404#discussion_r903175660


##########
r/src/filesystem.cpp:
##########
@@ -331,3 +337,77 @@ std::string fs___S3FileSystem__region(const std::shared_ptr<fs::S3FileSystem>& f
 }
 
 #endif
+
+#if defined(ARROW_R_WITH_GCS)
+
+#include <arrow/filesystem/gcsfs.h>
+
+std::shared_ptr<arrow::KeyValueMetadata> strings_to_kvm(cpp11::strings metadata);
+
+// [[gcs::export]]
+std::shared_ptr<fs::GcsFileSystem> fs___GcsFileSystem__Make(bool anonymous,
+                                                            cpp11::list options) {
+  fs::GcsOptions gcs_opts;
+
+  // Handle auth (anonymous, credentials, default)
+  // (validation/internal coherence handled in R)
+  if (anonymous) {
+    gcs_opts = fs::GcsOptions::Anonymous();
+  } else if (!Rf_isNull(options["access_token"])) {
+    // Convert POSIXct timestamp seconds to nanoseconds
+    std::chrono::nanoseconds ns_count(
+        static_cast<int64_t>(cpp11::as_cpp<double>(options["expiration"])) * 1000000000);
+    auto expiration_timepoint =
+        fs::TimePoint(std::chrono::duration_cast<fs::TimePoint::duration>(ns_count));
+    gcs_opts = fs::GcsOptions::FromAccessToken(
+        cpp11::as_cpp<std::string>(options["access_token"]), expiration_timepoint);
+    // TODO: implement FromImpersonatedServiceAccount
+    // } else if (base_credentials != "") {
+    //   // static GcsOptions FromImpersonatedServiceAccount(
+    //   // const GcsCredentials& base_credentials, const std::string&
+    //   target_service_account);
+    //   // TODO: construct GcsCredentials
+    //   gcs_opts = fs::GcsOptions::FromImpersonatedServiceAccount(base_credentials,
+    //                                                             target_service_account);
+  } else if (!Rf_isNull(options["json_credentials"])) {
+    gcs_opts = fs::GcsOptions::FromServiceAccountCredentials(
+        cpp11::as_cpp<std::string>(options["json_credentials"]));
+  } else {
+    gcs_opts = fs::GcsOptions::Defaults();
+  }
+
+  // Handle other attributes
+  if (!Rf_isNull(options["endpoint_override"])) {
+    gcs_opts.endpoint_override = cpp11::as_cpp<std::string>(options["endpoint_override"]);
+  }
+
+  if (!Rf_isNull(options["scheme"])) {
+    gcs_opts.scheme = cpp11::as_cpp<std::string>(options["scheme"]);
+  }
+
+  // /// \brief Location to use for creating buckets.
+  if (!Rf_isNull(options["default_bucket_location"])) {
+    gcs_opts.default_bucket_location =
+        cpp11::as_cpp<std::string>(options["default_bucket_location"]);
+  }
+  // /// \brief If set used to control total time allowed for retrying underlying
+  // /// errors.
+  // ///
+  // /// The default policy is to retry for up to 15 minutes.
+  if (!Rf_isNull(options["retry_limit_seconds"])) {
+    gcs_opts.retry_limit_seconds = cpp11::as_cpp<double>(options["retry_limit_seconds"]);
+  }
+
+  // /// \brief Default metadata for OpenOutputStream.
+  // ///
+  // /// This will be ignored if non-empty metadata is passed to OpenOutputStream.
+  if (!Rf_isNull(options["default_metadata"])) {
+    gcs_opts.default_metadata = strings_to_kvm(options["default_metadata"]);
+  }
+
+  auto io_context = arrow::io::IOContext(gc_memory_pool());
+  // TODO: S3FileSystem::Make returns a Result and uses ValueOrStop but this doesn't?

Review Comment:
   I think this is because the GcsFileSystem does lazy connections so this can't currently error.  No objects to make them uniform though



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] wjones127 commented on pull request #13404: ARROW-16510: [R] Add bindings for GCS filesystem

Posted by GitBox <gi...@apache.org>.
wjones127 commented on PR #13404:
URL: https://github.com/apache/arrow/pull/13404#issuecomment-1163658023

   (I created docs follow up: https://issues.apache.org/jira/browse/ARROW-16887)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] nealrichardson commented on pull request #13404: ARROW-16510: [R] Add bindings for GCS filesystem

Posted by GitBox <gi...@apache.org>.
nealrichardson commented on PR #13404:
URL: https://github.com/apache/arrow/pull/13404#issuecomment-1163507517

   @github-actions crossbow submit homebrew-r-autobrew


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] pitrou commented on a diff in pull request #13404: ARROW-16510: [R] Add bindings for GCS filesystem

Posted by GitBox <gi...@apache.org>.
pitrou commented on code in PR #13404:
URL: https://github.com/apache/arrow/pull/13404#discussion_r903935405


##########
r/src/filesystem.cpp:
##########
@@ -331,3 +337,77 @@ std::string fs___S3FileSystem__region(const std::shared_ptr<fs::S3FileSystem>& f
 }
 
 #endif
+
+#if defined(ARROW_R_WITH_GCS)
+
+#include <arrow/filesystem/gcsfs.h>
+
+std::shared_ptr<arrow::KeyValueMetadata> strings_to_kvm(cpp11::strings metadata);
+
+// [[gcs::export]]
+std::shared_ptr<fs::GcsFileSystem> fs___GcsFileSystem__Make(bool anonymous,
+                                                            cpp11::list options) {
+  fs::GcsOptions gcs_opts;
+
+  // Handle auth (anonymous, credentials, default)
+  // (validation/internal coherence handled in R)
+  if (anonymous) {
+    gcs_opts = fs::GcsOptions::Anonymous();
+  } else if (!Rf_isNull(options["access_token"])) {
+    // Convert POSIXct timestamp seconds to nanoseconds
+    std::chrono::nanoseconds ns_count(
+        static_cast<int64_t>(cpp11::as_cpp<double>(options["expiration"])) * 1000000000);
+    auto expiration_timepoint =
+        fs::TimePoint(std::chrono::duration_cast<fs::TimePoint::duration>(ns_count));
+    gcs_opts = fs::GcsOptions::FromAccessToken(
+        cpp11::as_cpp<std::string>(options["access_token"]), expiration_timepoint);
+    // TODO: implement FromImpersonatedServiceAccount
+    // } else if (base_credentials != "") {
+    //   // static GcsOptions FromImpersonatedServiceAccount(
+    //   // const GcsCredentials& base_credentials, const std::string&
+    //   target_service_account);
+    //   // TODO: construct GcsCredentials
+    //   gcs_opts = fs::GcsOptions::FromImpersonatedServiceAccount(base_credentials,
+    //                                                             target_service_account);
+  } else if (!Rf_isNull(options["json_credentials"])) {
+    gcs_opts = fs::GcsOptions::FromServiceAccountCredentials(
+        cpp11::as_cpp<std::string>(options["json_credentials"]));
+  } else {
+    gcs_opts = fs::GcsOptions::Defaults();
+  }
+
+  // Handle other attributes
+  if (!Rf_isNull(options["endpoint_override"])) {
+    gcs_opts.endpoint_override = cpp11::as_cpp<std::string>(options["endpoint_override"]);
+  }
+
+  if (!Rf_isNull(options["scheme"])) {
+    gcs_opts.scheme = cpp11::as_cpp<std::string>(options["scheme"]);
+  }
+
+  // /// \brief Location to use for creating buckets.
+  if (!Rf_isNull(options["default_bucket_location"])) {
+    gcs_opts.default_bucket_location =
+        cpp11::as_cpp<std::string>(options["default_bucket_location"]);
+  }
+  // /// \brief If set used to control total time allowed for retrying underlying
+  // /// errors.
+  // ///
+  // /// The default policy is to retry for up to 15 minutes.
+  if (!Rf_isNull(options["retry_limit_seconds"])) {
+    gcs_opts.retry_limit_seconds = cpp11::as_cpp<double>(options["retry_limit_seconds"]);
+  }
+
+  // /// \brief Default metadata for OpenOutputStream.
+  // ///
+  // /// This will be ignored if non-empty metadata is passed to OpenOutputStream.
+  if (!Rf_isNull(options["default_metadata"])) {
+    gcs_opts.default_metadata = strings_to_kvm(options["default_metadata"]);
+  }
+
+  auto io_context = arrow::io::IOContext(gc_memory_pool());
+  // TODO: S3FileSystem::Make returns a Result and uses ValueOrStop but this doesn't?

Review Comment:
   Let's make it a `Result`, so that we can potentially raise errors on invalid parameters etc.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] nealrichardson commented on a diff in pull request #13404: ARROW-16510: [R] Add bindings for GCS filesystem

Posted by GitBox <gi...@apache.org>.
nealrichardson commented on code in PR #13404:
URL: https://github.com/apache/arrow/pull/13404#discussion_r904189404


##########
r/src/filesystem.cpp:
##########
@@ -331,3 +337,77 @@ std::string fs___S3FileSystem__region(const std::shared_ptr<fs::S3FileSystem>& f
 }
 
 #endif
+
+#if defined(ARROW_R_WITH_GCS)
+
+#include <arrow/filesystem/gcsfs.h>
+
+std::shared_ptr<arrow::KeyValueMetadata> strings_to_kvm(cpp11::strings metadata);
+
+// [[gcs::export]]
+std::shared_ptr<fs::GcsFileSystem> fs___GcsFileSystem__Make(bool anonymous,
+                                                            cpp11::list options) {
+  fs::GcsOptions gcs_opts;
+
+  // Handle auth (anonymous, credentials, default)
+  // (validation/internal coherence handled in R)
+  if (anonymous) {
+    gcs_opts = fs::GcsOptions::Anonymous();
+  } else if (!Rf_isNull(options["access_token"])) {
+    // Convert POSIXct timestamp seconds to nanoseconds
+    std::chrono::nanoseconds ns_count(
+        static_cast<int64_t>(cpp11::as_cpp<double>(options["expiration"])) * 1000000000);
+    auto expiration_timepoint =
+        fs::TimePoint(std::chrono::duration_cast<fs::TimePoint::duration>(ns_count));
+    gcs_opts = fs::GcsOptions::FromAccessToken(
+        cpp11::as_cpp<std::string>(options["access_token"]), expiration_timepoint);
+    // TODO: implement FromImpersonatedServiceAccount
+    // } else if (base_credentials != "") {
+    //   // static GcsOptions FromImpersonatedServiceAccount(
+    //   // const GcsCredentials& base_credentials, const std::string&
+    //   target_service_account);
+    //   // TODO: construct GcsCredentials
+    //   gcs_opts = fs::GcsOptions::FromImpersonatedServiceAccount(base_credentials,
+    //                                                             target_service_account);
+  } else if (!Rf_isNull(options["json_credentials"])) {
+    gcs_opts = fs::GcsOptions::FromServiceAccountCredentials(
+        cpp11::as_cpp<std::string>(options["json_credentials"]));
+  } else {
+    gcs_opts = fs::GcsOptions::Defaults();
+  }
+
+  // Handle other attributes
+  if (!Rf_isNull(options["endpoint_override"])) {
+    gcs_opts.endpoint_override = cpp11::as_cpp<std::string>(options["endpoint_override"]);
+  }
+
+  if (!Rf_isNull(options["scheme"])) {
+    gcs_opts.scheme = cpp11::as_cpp<std::string>(options["scheme"]);
+  }
+
+  // /// \brief Location to use for creating buckets.
+  if (!Rf_isNull(options["default_bucket_location"])) {
+    gcs_opts.default_bucket_location =
+        cpp11::as_cpp<std::string>(options["default_bucket_location"]);
+  }
+  // /// \brief If set used to control total time allowed for retrying underlying
+  // /// errors.
+  // ///
+  // /// The default policy is to retry for up to 15 minutes.
+  if (!Rf_isNull(options["retry_limit_seconds"])) {
+    gcs_opts.retry_limit_seconds = cpp11::as_cpp<double>(options["retry_limit_seconds"]);
+  }
+
+  // /// \brief Default metadata for OpenOutputStream.
+  // ///
+  // /// This will be ignored if non-empty metadata is passed to OpenOutputStream.
+  if (!Rf_isNull(options["default_metadata"])) {
+    gcs_opts.default_metadata = strings_to_kvm(options["default_metadata"]);
+  }
+
+  auto io_context = arrow::io::IOContext(gc_memory_pool());
+  // TODO: S3FileSystem::Make returns a Result and uses ValueOrStop but this doesn't?

Review Comment:
   Opened ARROW-16884 for that.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] nealrichardson commented on pull request #13404: ARROW-16510: [R] Add bindings for GCS filesystem

Posted by GitBox <gi...@apache.org>.
nealrichardson commented on PR #13404:
URL: https://github.com/apache/arrow/pull/13404#issuecomment-1166371707

   One observation: the fully bundled Linux build (`R / rhub/debian-gcc-devel:latest`) seems to be 10-15 minutes slower with ARROW_GCS=ON. We may want to reconsider turning ARROW_GCS=ON without an explicit request to turn it on in the Linux source build (we can include it in the binaries), but that raises some other UX questions. cc @jonkeane 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] nealrichardson commented on pull request #13404: ARROW-16510: [R] Add bindings for GCS filesystem

Posted by GitBox <gi...@apache.org>.
nealrichardson commented on PR #13404:
URL: https://github.com/apache/arrow/pull/13404#issuecomment-1162061355

   I turned on GCS wherever S3 was turned on (so NOT_CRAN=true will build GCS on linux). We should check the build times and see if it's worth the cost or whether it should be a more explicit opt-in. I think we should have it on in the binaries we build though.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] nealrichardson commented on a diff in pull request #13404: ARROW-16510: [R] Add bindings for GCS filesystem

Posted by GitBox <gi...@apache.org>.
nealrichardson commented on code in PR #13404:
URL: https://github.com/apache/arrow/pull/13404#discussion_r903724015


##########
r/src/filesystem.cpp:
##########
@@ -331,3 +337,77 @@ std::string fs___S3FileSystem__region(const std::shared_ptr<fs::S3FileSystem>& f
 }
 
 #endif
+
+#if defined(ARROW_R_WITH_GCS)
+
+#include <arrow/filesystem/gcsfs.h>
+
+std::shared_ptr<arrow::KeyValueMetadata> strings_to_kvm(cpp11::strings metadata);
+
+// [[gcs::export]]
+std::shared_ptr<fs::GcsFileSystem> fs___GcsFileSystem__Make(bool anonymous,
+                                                            cpp11::list options) {
+  fs::GcsOptions gcs_opts;
+
+  // Handle auth (anonymous, credentials, default)
+  // (validation/internal coherence handled in R)
+  if (anonymous) {
+    gcs_opts = fs::GcsOptions::Anonymous();
+  } else if (!Rf_isNull(options["access_token"])) {
+    // Convert POSIXct timestamp seconds to nanoseconds
+    std::chrono::nanoseconds ns_count(
+        static_cast<int64_t>(cpp11::as_cpp<double>(options["expiration"])) * 1000000000);
+    auto expiration_timepoint =
+        fs::TimePoint(std::chrono::duration_cast<fs::TimePoint::duration>(ns_count));
+    gcs_opts = fs::GcsOptions::FromAccessToken(
+        cpp11::as_cpp<std::string>(options["access_token"]), expiration_timepoint);
+    // TODO: implement FromImpersonatedServiceAccount
+    // } else if (base_credentials != "") {
+    //   // static GcsOptions FromImpersonatedServiceAccount(
+    //   // const GcsCredentials& base_credentials, const std::string&
+    //   target_service_account);
+    //   // TODO: construct GcsCredentials
+    //   gcs_opts = fs::GcsOptions::FromImpersonatedServiceAccount(base_credentials,
+    //                                                             target_service_account);
+  } else if (!Rf_isNull(options["json_credentials"])) {
+    gcs_opts = fs::GcsOptions::FromServiceAccountCredentials(
+        cpp11::as_cpp<std::string>(options["json_credentials"]));
+  } else {
+    gcs_opts = fs::GcsOptions::Defaults();
+  }
+
+  // Handle other attributes
+  if (!Rf_isNull(options["endpoint_override"])) {
+    gcs_opts.endpoint_override = cpp11::as_cpp<std::string>(options["endpoint_override"]);
+  }
+
+  if (!Rf_isNull(options["scheme"])) {
+    gcs_opts.scheme = cpp11::as_cpp<std::string>(options["scheme"]);
+  }
+
+  // /// \brief Location to use for creating buckets.
+  if (!Rf_isNull(options["default_bucket_location"])) {
+    gcs_opts.default_bucket_location =
+        cpp11::as_cpp<std::string>(options["default_bucket_location"]);
+  }
+  // /// \brief If set used to control total time allowed for retrying underlying
+  // /// errors.
+  // ///
+  // /// The default policy is to retry for up to 15 minutes.
+  if (!Rf_isNull(options["retry_limit_seconds"])) {
+    gcs_opts.retry_limit_seconds = cpp11::as_cpp<double>(options["retry_limit_seconds"]);
+  }
+
+  // /// \brief Default metadata for OpenOutputStream.
+  // ///
+  // /// This will be ignored if non-empty metadata is passed to OpenOutputStream.
+  if (!Rf_isNull(options["default_metadata"])) {
+    gcs_opts.default_metadata = strings_to_kvm(options["default_metadata"]);
+  }
+
+  auto io_context = arrow::io::IOContext(gc_memory_pool());
+  // TODO: S3FileSystem::Make returns a Result and uses ValueOrStop but this doesn't?

Review Comment:
   @pitrou do you have an opinion on this?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] nealrichardson commented on pull request #13404: ARROW-16510: [R] Add bindings for GCS filesystem

Posted by GitBox <gi...@apache.org>.
nealrichardson commented on PR #13404:
URL: https://github.com/apache/arrow/pull/13404#issuecomment-1163387088

   > If you have a VM with abseil installed, you could do something like:
   > 
   > ```shell
   > pkg-config --libs absl_memory absl_strings absl_str_format absl_time absl_variant absl_base absl_memory absl_optional absl_span absl_time absl_variant
   > -L/usr/local/lib64 -labsl_str_format_internal -labsl_bad_optional_access -labsl_time -labsl_civil_time -labsl_strings -labsl_strings_internal -lrt -labsl_base -labsl_spinlock_wait -labsl_int128 -labsl_throw_delegate -labsl_time_zone -labsl_bad_variant_access -labsl_raw_logging_internal -labsl_log_severity
   > ```
   > 
   > Which (a) removes the header-only libraries, and (b) saves you some whack-a-moling (maybe).
   
   Yeah I keep thinking I've reached the end (just one more library!) but that would be smarter. I have a local build of abseil already (from the arrow build), didn't think to point pkg-config at it. Thanks for the suggestion!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] kou commented on pull request #13404: ARROW-16510: [R] Add bindings for GCS filesystem

Posted by GitBox <gi...@apache.org>.
kou commented on PR #13404:
URL: https://github.com/apache/arrow/pull/13404#issuecomment-1163815653

   We need to define `CURL_STATICLIB` on building google-cloud-cpp not Apache Arrow C++. But CMake's bundled `FindCURL.cmake` doesn't support `CURL_STATICLIB`: https://gitlab.kitware.com/cmake/cmake/-/issues/18378
   
   How about the following?
   
   ```diff
   diff --git a/cpp/cmake_modules/ThirdpartyToolchain.cmake b/cpp/cmake_modules/ThirdpartyToolchain.cmake
   index 024859288b..299c131958 100644
   --- a/cpp/cmake_modules/ThirdpartyToolchain.cmake
   +++ b/cpp/cmake_modules/ThirdpartyToolchain.cmake
   @@ -1067,6 +1067,15 @@ macro(find_curl)
                                                                  "${CURL_LIBRARIES}")
        endif()
      endif()
   +  if(WIN32 AND NOT CURL_STATIC_CHECKED)
   +    get_target_property(CURL_LIBRARY CURL::libcurl IMPORTED_LOCATION)
   +    get_filename_component(CURL_LIBRARY_EXT "${CURL_LIBRARY}" LAST_EXT)
   +    if(CURL_LIBRARY_EXT STREQUAL "${CMAKE_STATIC_LIBRARY_SUFFIX}")
   +      set_target_properties(CURL::libcurl
   +                            PROPERTIES INTERFACE_COMPILE_DEFINITIONS "CURL_STATICLIB")
   +    endif()
   +    set(CURL_STATIC_CHECKED TRUE)
   +  endif()
    endmacro()
    
    # ----------------------------------------------------------------------
   
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] github-actions[bot] commented on pull request #13404: ARROW-16510: [R] Add bindings for GCS filesystem

Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #13404:
URL: https://github.com/apache/arrow/pull/13404#issuecomment-1163508561

   Revision: 4d789f06e0a4eb6621f42bb85784c12b41a8375d
   
   Submitted crossbow builds: [ursacomputing/crossbow @ actions-f35855420a](https://github.com/ursacomputing/crossbow/branches/all?query=actions-f35855420a)
   
   |Task|Status|
   |----|------|
   |homebrew-r-autobrew|[![Github Actions](https://github.com/ursacomputing/crossbow/workflows/Crossbow/badge.svg?branch=actions-f35855420a-github-homebrew-r-autobrew)](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-f35855420a-github-homebrew-r-autobrew)|


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] nealrichardson commented on pull request #13404: ARROW-16510: [R] Add bindings for GCS filesystem

Posted by GitBox <gi...@apache.org>.
nealrichardson commented on PR #13404:
URL: https://github.com/apache/arrow/pull/13404#issuecomment-1162448555

   @coryan thanks, that's helpful actually. I'll have to dig more on the bundled libraries thing. On the Homebrew build, abseil indeed is being built with C++17 by brew (https://github.com/Homebrew/homebrew-core/blob/master/Formula/abseil.rb#L31), and the arrow build is using C++11. What's odd is that we've been depending on it (transitively) for a while because Flight uses grpc (also built C++17 by brew), but I guess google-cloud-cpp hits different parts.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] nealrichardson commented on pull request #13404: ARROW-16510: [R] Add bindings for GCS filesystem

Posted by GitBox <gi...@apache.org>.
nealrichardson commented on PR #13404:
URL: https://github.com/apache/arrow/pull/13404#issuecomment-1166365875

   Ok @kou we've resolved all of the builds now, this is ready if you want to give it a final review/+1.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] nealrichardson commented on pull request #13404: ARROW-16510: [R] Add bindings for GCS filesystem

Posted by GitBox <gi...@apache.org>.
nealrichardson commented on PR #13404:
URL: https://github.com/apache/arrow/pull/13404#issuecomment-1166327794

   @github-actions crossbow submit -g r


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org