You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2020/08/26 21:44:17 UTC

[GitHub] [arrow] nealrichardson opened a new pull request #8058: ARROW-9854: [R] Support reading/writing data to/from S3

nealrichardson opened a new pull request #8058:
URL: https://github.com/apache/arrow/pull/8058


   - [x] read_parquet/feather/etc. from S3 (use FileSystem->OpenInputFile(path))
   - [x] write_$FORMAT via FileSystem->OpenOutputStream(path)
   - [x] write_dataset (done? at least via URI)
   - [ ] for linux, an argument to install_arrow to help, assuming you've installed aws-sdk-cpp already (turn on ARROW_S3, AWSSDK_SOURCE=SYSTEM)
   - [ ] testing with minio on CI
   - [x] set up a real test bucket and user for e2e testing (credentials available on request)
   - [ ] update docs, vignettes, news
   
   Out of the current scope:
   
   - [ ] download dataset, i.e. copy files/directory recursively (needs ARROW-9867, ARROW-9868)
   - [ ] friendlier methods for interacting with/viewing a filesystem (ls, mkdir, etc.)


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] nealrichardson closed pull request #8058: ARROW-9854: [R] Support reading/writing data to/from S3

Posted by GitBox <gi...@apache.org>.
nealrichardson closed pull request #8058:
URL: https://github.com/apache/arrow/pull/8058


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] jorisvandenbossche commented on a change in pull request #8058: ARROW-9854: [R] Support reading/writing data to/from S3

Posted by GitBox <gi...@apache.org>.
jorisvandenbossche commented on a change in pull request #8058:
URL: https://github.com/apache/arrow/pull/8058#discussion_r486167891



##########
File path: r/R/filesystem.R
##########
@@ -242,11 +242,31 @@ FileSystem <- R6Class("FileSystem", inherit = ArrowObject,
   )
 )
 FileSystem$from_uri <- function(uri) {
+  assert_that(is.string(uri))
   out <- fs___FileSystemFromUri(uri)
   out$fs <- shared_ptr(FileSystem, out$fs)$..dispatch()
   out
 }
 
+get_path_and_filesystem <- function(x, filesystem = NULL) {
+  # Wrapper around FileSystem$from_uri that handles local paths
+  # and an optional explicit filesystem
+  assert_that(is.string(x))
+  if (is_url(x)) {
+    if (!is.null(filesystem)) {
+      # Stop? Can't have URL (which yields a fs) and another fs
+    }
+    FileSystem$from_uri(x)
+  } else {
+    list(
+      fs = filesystem %||% LocalFileSystem$create(),
+      path = clean_path_abs(x)
+    )
+  }
+}
+
+is_url <- function(x) grepl("://", x)

Review comment:
       You might want to double check that the `get_path_and_filesystem` doesn't break local relative file paths (as I did in the python version .. ;-)), but I suppose this `is_url` check before calling `from_uri` will indeed prevent that issue.

##########
File path: r/R/filesystem.R
##########
@@ -242,11 +242,31 @@ FileSystem <- R6Class("FileSystem", inherit = ArrowObject,
   )
 )
 FileSystem$from_uri <- function(uri) {
+  assert_that(is.string(uri))
   out <- fs___FileSystemFromUri(uri)
   out$fs <- shared_ptr(FileSystem, out$fs)$..dispatch()
   out
 }
 
+get_path_and_filesystem <- function(x, filesystem = NULL) {
+  # Wrapper around FileSystem$from_uri that handles local paths
+  # and an optional explicit filesystem
+  assert_that(is.string(x))
+  if (is_url(x)) {
+    if (!is.null(filesystem)) {
+      # Stop? Can't have URL (which yields a fs) and another fs

Review comment:
       Indeed, the user shouldn't pass both a filesystem object and a URI, as then things like `filesystem$OpenInputFile(uri)` will fail (that was the root cause of the python parquet s3 test crashes -> https://github.com/apache/arrow/pull/8139). So I think can either let it pass here and rely on a later error in eg OpenInputFile, or otherwise raise a (potentially more informative) error message here.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] github-actions[bot] commented on pull request #8058: ARROW-9854: [R] Support reading/writing data to/from S3

Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #8058:
URL: https://github.com/apache/arrow/pull/8058#issuecomment-681140852


   https://issues.apache.org/jira/browse/ARROW-9854


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] nealrichardson commented on a change in pull request #8058: ARROW-9854: [R] Support reading/writing data to/from S3

Posted by GitBox <gi...@apache.org>.
nealrichardson commented on a change in pull request #8058:
URL: https://github.com/apache/arrow/pull/8058#discussion_r486656550



##########
File path: r/R/filesystem.R
##########
@@ -242,11 +242,31 @@ FileSystem <- R6Class("FileSystem", inherit = ArrowObject,
   )
 )
 FileSystem$from_uri <- function(uri) {
+  assert_that(is.string(uri))
   out <- fs___FileSystemFromUri(uri)
   out$fs <- shared_ptr(FileSystem, out$fs)$..dispatch()
   out
 }
 
+get_path_and_filesystem <- function(x, filesystem = NULL) {
+  # Wrapper around FileSystem$from_uri that handles local paths
+  # and an optional explicit filesystem
+  assert_that(is.string(x))
+  if (is_url(x)) {
+    if (!is.null(filesystem)) {
+      # Stop? Can't have URL (which yields a fs) and another fs

Review comment:
       What happens here is that `filesystem` will be ignored if `x` is a URL, i.e. we'll use the filesystem that `from_uri(x)` returns. So that should either Just Work if the filesystems happen to be the same, or you're right, it should error with some kind of File Not Found later. 




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org