You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "mytarmail (via GitHub)" <gi...@apache.org> on 2023/11/28 18:12:39 UTC

[I] [R] How to delete rows out-of-memory in arrow [arrow]

mytarmail opened a new issue, #38935:
URL: https://github.com/apache/arrow/issues/38935

   ### Describe the usage question you have. Please include as many useful details as  possible.
   
   
   Suppose I have a huge dataset that does not fit into memory. How can I delete rows in an arrow without loading the dataset into memory, and update the already changed dataset on disk.
   
   ```
   library(arrow)
   path <- "C:\\Users\\....\\Desktop\\my_data"
   write_dataset(dataset = iris, path = path,partitioning = "Species")
   arr <- open_dataset(sources = path)
   ```
   ..
   ```
   remove_rows <- sample(1:nrow(arr), 50)
   
   #  Not work )
   arr <- arr[!remove_rows,]
   
   # Rewrite/update arr on disk
   # ...
   ```
   
   ### Component(s)
   
   R


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [R] How to delete rows out-of-memory in arrow [arrow]

Posted by "thisisnic (via GitHub)" <gi...@apache.org>.
thisisnic closed issue #38935: [R] How to delete rows out-of-memory in arrow
URL: https://github.com/apache/arrow/issues/38935


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [R] How to delete rows out-of-memory in arrow [arrow]

Posted by "thisisnic (via GitHub)" <gi...@apache.org>.
thisisnic commented on issue #38935:
URL: https://github.com/apache/arrow/issues/38935#issuecomment-2966701579

   This is likely far too late but responding here for anyone else who has this question; you'd need to filter the dataset to only contain the rows to retain and then rewrite it.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org