You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "sonicseamus (via GitHub)" <gi...@apache.org> on 2024/12/10 16:08:05 UTC

[I] open_dataset with root directory inaccessible? [arrow]

sonicseamus opened a new issue, #44992:
URL: https://github.com/apache/arrow/issues/44992

   ### Describe the usage question you have. Please include as many useful details as  possible.
   
   
   Hello, I am hoping to use R to open the [MBTA LAMP Subway Performance Data](performancedata.mbta.com) which is stored in daily Parquet files as an Arrow `Table` object that I can query. The files are stored, separated by day, in the following format:
   
   > URL Construction: Replace [YYYY-MM-DD] with the YEAR, MONTH and DAY of the requested service date.
   > 
   > https://performancedata.mbta.com/lamp/subway-on-time-performance-v1/YYYY-MM-DD-subway-on-time-performance-v1.parquet
   
   The problem is that while the individual files are publicly accessible, the root directory, [](https://performancedata.mbta.com/lamp/subway-on-time-performance-v1/), is not. As a result, I cannot use the typical `open_dataset("https://performancedata.mbta.com/lamp/subway-on-time-performance-v1/", format = "parquet")` command to open the table.
   
   Of course, I can access individual files (e.g. yesterday's) with `read_parquet("https://performancedata.mbta.com/lamp/subway-on-time-performance-v1/2024-12-09-subway-on-time-performance-v1.parquet")`, and I could iterate through a bunch of dates, but I would prefer to use Arrow's more efficient `open_dataset()`.
   
   My question is: is there a way by specifying the `schema` or `partitioning` that I can open the dataset even if the root directory is inaccessible, considering that all the subsidiary files are indeed accessible?
   
   ### Component(s)
   
   Parquet, R


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [R] open_dataset with root directory inaccessible? [arrow]

Posted by "amoeba (via GitHub)" <gi...@apache.org>.
amoeba commented on issue #44992:
URL: https://github.com/apache/arrow/issues/44992#issuecomment-2536581569

   Sounds great. Please feel free to file new issues if you run into bugs or documentation gaps.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [R] open_dataset with root directory inaccessible? [arrow]

Posted by "sonicseamus (via GitHub)" <gi...@apache.org>.
sonicseamus commented on issue #44992:
URL: https://github.com/apache/arrow/issues/44992#issuecomment-2533302975

   Thank you for a helpful and thorough answer! I will need to dig into the functions you used to understand how this works but I can work with this, and it's around 10x faster than my old implementation with `pmap`.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [R] open_dataset with root directory inaccessible? [arrow]

Posted by "thisisnic (via GitHub)" <gi...@apache.org>.
thisisnic closed issue #44992: [R] open_dataset with root directory inaccessible?
URL: https://github.com/apache/arrow/issues/44992


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] open_dataset with root directory inaccessible? [arrow]

Posted by "amoeba (via GitHub)" <gi...@apache.org>.
amoeba commented on issue #44992:
URL: https://github.com/apache/arrow/issues/44992#issuecomment-2532636356

   Hi @sonicseamus, thanks for the question. There is a way to construct the Dataset manually, skipping the discovery process `open_dataset` initiates and can't do here, but I think you'll immediately run into an issue: Arrow doesn't support constructing datasets directly over HTTP, see https://github.com/apache/arrow/issues/23849.
   
   That said, you can get partway to what you want (with some workarounds) because the underlying Parquet files are on S3 or S3-like storage and Arrow does support S3. The one limitation is that you won't be able to take advantage of partitioning. If possible, requesting that the MBTA group open up access to the bucket root would be a nice improvement here.
   
   ```r
   library(dplyr)
   library(arrow)
   library(stringr)
   
   # First we need to get the list of files
   file_url_prefix <- "https://performancedata.mbta.com/lamp/subway-on-time-performance-v1/"
   index_csv_url <- paste0(file_url_prefix, "index.csv")
   
   index_tbl <- read_csv_arrow(index_csv_url) 
   index_tbl$relpath <- str_remove_all(index_tbl$file_url, file_url_prefix)
   
   # Then, to create a Dataset manually, we need to create a Filesystem first
   # Since we don't support HTTP filesystems but this dataset is backed by S3-like
   # storage, we can create a custom S3 Filesystem and still use our S3 driver
   fs <- S3FileSystem$create(
     anonymous = TRUE, 
     endpoint_override = "https://performancedata.mbta.com"
   )
   # cd into the right path
   ds_fs <- fs$cd("lamp/subway-on-time-performance-v1")
   
   dsf <- FileSystemDatasetFactory$create(
     filesystem = ds_fs, 
     paths = index_tbl$relpath, 
     format = FileFormat$create("parquet"),
   )
   ds <- dsf$Finish()
   
   # Now we can use the Dataset as normal
   ds |> 
     head() |> 
     collect()
   ```
   
   Gives me:
   
   ```
   # A tibble: 6 × 27
     stop_sequence stop_id   parent_station move_timestamp stop_timestamp travel_time_seconds dwell_time_seconds headway_trunk_seconds headway_branch_seconds service_date
             <int> <chr>     <chr>                   <int>          <int>               <int>              <int>                 <int>                  <int>        <int>
   1             1 Oak Grov… place-ogmnl                NA     1568643555                  NA                 NA                    NA                     NA     20190916
   2             1 70210     place-lech         1568643599             NA                  NA                 NA                    NA                     NA     20190916
   3           120 70095     place-jfk                  NA     1568643680                  NA                 NA                    NA                     NA     20190916
   4           650 70209     place-lech                 NA     1568643698                  NA                 NA                    NA                     NA     20190916
   5             1 Alewife-… place-alfcl                NA     1568643730                  NA                 NA                    NA                     NA     20190916
   6           310 70107     place-lake                 NA     1568643731                  NA                 NA                    NA                     NA     20190916
   # ℹ 17 more variables: route_id <chr>, direction_id <lgl>, start_time <int>, vehicle_id <chr>, branch_route_id <chr>, trunk_route_id <chr>, stop_count <int>,
   #   trip_id <chr>, vehicle_label <chr>, vehicle_consist <chr>, direction <chr>, direction_destination <chr>, scheduled_arrival_time <int>,
   #   scheduled_departure_time <int>, scheduled_travel_time <int>, scheduled_headway_branch <int>, scheduled_headway_trunk <int>
   ```
   
   Does the above seem like a workable solution for you?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [R] open_dataset with root directory inaccessible? [arrow]

Posted by "sonicseamus (via GitHub)" <gi...@apache.org>.
sonicseamus closed issue #44992: [R] open_dataset with root directory inaccessible?
URL: https://github.com/apache/arrow/issues/44992


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org