You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2021/10/07 12:49:54 UTC

[GitHub] [arrow-cookbook] GShotwell opened a new issue #83: [R] Recipe for random sampling

GShotwell opened a new issue #83:
URL: https://github.com/apache/arrow-cookbook/issues/83


   It would be great if there were a way to sample from an arrow dataset. I put together this somewhat hacky example, but I bet there's some thing a bit more elegant..
   
   ```
   library(arrow)
   library(dplyr)
   library(nycflights13)
   
   flights <- nycflights13::flights
   
   flights$id <- seq_len(nrow(flights))
   
   for(i in unique(flights$month)) {
     out <- filter(flights, month == i)
     arrow::write_parquet(out, paste0("flight_ds/", i, ".parquet"))
   }
   
   ds <- arrow::open_dataset("flight_ds")
   
   sample <- sample(flights$id, 100)
   
   ds %>% 
     filter(id %in% sample) %>% 
     collect()
   ```
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow-cookbook] thisisnic commented on issue #83: [R] Recipe for random sampling

Posted by GitBox <gi...@apache.org>.
thisisnic commented on issue #83:
URL: https://github.com/apache/arrow-cookbook/issues/83#issuecomment-937927421


   Thanks for opening this issue @GShotwell!  I agree that this would be a really useful thing to a) be able to do in arrow and b) have a recipe for in this cookbook.
   
   There's currently no optimised way of doing this in arrow, but I've opened up a request for this to be implemented at the C++ level; once this is done, we can look at implementing this in R.
   
   https://issues.apache.org/jira/browse/ARROW-14254
   
   In the short-term, the snippet below shows how you can achieve this if you have existing files without having to write any new data to them, though I don't think we want to add a recipe for this as apparently it's a bit slow - I'd like to wait until we have a nice way of doing it properly utilising the underlying C++ functionality before adding it to the cookbook.
   
   ```
   tf <- tempfile()
   dir.create(tf)
   
   iris %>% group_by(Species) %>%
     write_dataset(tf)
   
   iris_dataset <- open_dataset(tf)
   
   # sample as many values as needed
   rows <- sample(seq_len(nrow(iris_dataset)), 10)
   
   # return the sample
   collect(iris_dataset[rows,])
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow-cookbook] thisisnic edited a comment on issue #83: [R] Recipe for random sampling

Posted by GitBox <gi...@apache.org>.
thisisnic edited a comment on issue #83:
URL: https://github.com/apache/arrow-cookbook/issues/83#issuecomment-937927421


   Thanks for opening this issue @GShotwell!  I agree that this would be a really useful thing to a) be able to do in arrow and b) have a recipe for in this cookbook.
   
   There's currently no optimised way of doing this in arrow, but I've opened up a request for this to be implemented at the C++ level; once this is done, we can look at implementing this in R.
   
   https://issues.apache.org/jira/browse/ARROW-14254
   
   In the short-term, the snippet below shows how you can achieve this if you have existing files without having to write any new data to them, though I don't think we want to add a recipe for this as apparently it's a bit slow - I'd like to wait until we have a nice way of doing it properly utilising the underlying C++ functionality before adding it to the cookbook.
   
   ```
   tf <- tempfile()
   dir.create(tf)
   
   # I've added the grouping just so it's written as a partitioned dataset
   iris %>% group_by(Species) %>%
     write_dataset(tf)
   
   iris_dataset <- open_dataset(tf)
   
   # sample as many values as needed
   rows <- sample(seq_len(nrow(iris_dataset)), 10)
   
   # return the sample
   collect(iris_dataset[rows,])
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow-cookbook] thisisnic commented on issue #83: [R] Recipe for random sampling

Posted by GitBox <gi...@apache.org>.
thisisnic commented on issue #83:
URL: https://github.com/apache/arrow-cookbook/issues/83#issuecomment-937789620






-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow-cookbook] thisisnic commented on issue #83: [R] Recipe for random sampling

Posted by GitBox <gi...@apache.org>.
thisisnic commented on issue #83:
URL: https://github.com/apache/arrow-cookbook/issues/83#issuecomment-937789620


   Thanks for the request


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow-cookbook] wjones127 commented on issue #83: [R] Recipe for random sampling

Posted by GitBox <gi...@apache.org>.
wjones127 commented on issue #83:
URL: https://github.com/apache/arrow-cookbook/issues/83#issuecomment-1007584263


   As a part of apache/arrow#11894, I added an example to the datasets vignette to show how to create a sample from a dataset that doesn't fit into memory. This will be available in Arrow 7.0.0. Added #115 to add that to the cookbook.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow-cookbook] thisisnic removed a comment on issue #83: [R] Recipe for random sampling

Posted by GitBox <gi...@apache.org>.
thisisnic removed a comment on issue #83:
URL: https://github.com/apache/arrow-cookbook/issues/83#issuecomment-937789620


   Thanks for the request


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow-cookbook] nealrichardson commented on issue #83: [R] Recipe for random sampling

Posted by GitBox <gi...@apache.org>.
nealrichardson commented on issue #83:
URL: https://github.com/apache/arrow-cookbook/issues/83#issuecomment-937999701


   We could wire up the row selection used in `[` (which I added last year just to be able to run the vroom benchmarks, not because it was a good idea or done particularly well) to `slice_sample()` but I'm not sure it's a good idea given the inefficiency of the implementation.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow-cookbook] thisisnic edited a comment on issue #83: [R] Recipe for random sampling

Posted by GitBox <gi...@apache.org>.
thisisnic edited a comment on issue #83:
URL: https://github.com/apache/arrow-cookbook/issues/83#issuecomment-937927421


   Thanks for opening this issue @GShotwell!  I agree that this would be a really useful thing to a) be able to do in arrow and b) have a recipe for in this cookbook.
   
   There's currently no optimised way of doing this in arrow, but I've opened up a request for this to be implemented at the C++ level; once this is done, we can look at implementing this in R.
   
   https://issues.apache.org/jira/browse/ARROW-14254
   
   In the short-term, the snippet below shows how you can achieve this if you have existing files without having to write any new data to them, though I don't think we want to add a recipe for this as apparently it's a bit slow - I'd like to wait until we have a nice way of doing it properly utilising the underlying C++ functionality before adding it to the cookbook.
   
   ```
   tf <- tempfile()
   dir.create(tf)
   
   # I've added the grouping just so it's written as a partitioned dataset
   iris %>% group_by(Species) %>%
     write_dataset(tf)
   
   iris_dataset <- open_dataset(tf)
   
   # sample as many values as needed
   rows <- sample(seq_len(nrow(iris_dataset)), 10)
   
   # return the sample
   collect(iris_dataset[rows,])
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow-cookbook] thisisnic closed issue #83: [R] Recipe for random sampling

Posted by GitBox <gi...@apache.org>.
thisisnic closed issue #83:
URL: https://github.com/apache/arrow-cookbook/issues/83


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow-cookbook] thisisnic commented on issue #83: [R] Recipe for random sampling

Posted by GitBox <gi...@apache.org>.
thisisnic commented on issue #83:
URL: https://github.com/apache/arrow-cookbook/issues/83#issuecomment-937790010


   Apologies, closed this by accident!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow-cookbook] nealrichardson commented on issue #83: [R] Recipe for random sampling

Posted by GitBox <gi...@apache.org>.
nealrichardson commented on issue #83:
URL: https://github.com/apache/arrow-cookbook/issues/83#issuecomment-937999701


   We could wire up the row selection used in `[` (which I added last year just to be able to run the vroom benchmarks, not because it was a good idea or done particularly well) to `slice_sample()` but I'm not sure it's a good idea given the inefficiency of the implementation.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow-cookbook] thisisnic removed a comment on issue #83: [R] Recipe for random sampling

Posted by GitBox <gi...@apache.org>.
thisisnic removed a comment on issue #83:
URL: https://github.com/apache/arrow-cookbook/issues/83#issuecomment-937789620


   Thanks for the request


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow-cookbook] thisisnic closed issue #83: [R] Recipe for random sampling

Posted by GitBox <gi...@apache.org>.
thisisnic closed issue #83:
URL: https://github.com/apache/arrow-cookbook/issues/83


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow-cookbook] wjones127 commented on issue #83: [R] Recipe for random sampling

Posted by GitBox <gi...@apache.org>.
wjones127 commented on issue #83:
URL: https://github.com/apache/arrow-cookbook/issues/83#issuecomment-1007584263


   As a part of apache/arrow#11894, I added an example to the datasets vignette to show how to create a sample from a dataset that doesn't fit into memory. This will be available in Arrow 7.0.0. Added #115 to add that to the cookbook.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org