You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "paleolimbot (via GitHub)" <gi...@apache.org> on 2023/11/27 18:02:32 UTC
[PR] GH-38828: [R] Ensure that streams can be written to socket connections [arrow]
paleolimbot opened a new pull request, #38897:
URL: https://github.com/apache/arrow/pull/38897
### Rationale for this change
Currently we can't write to socket connection from R. This is a very useful way to send Arrow data around and should work!
### What changes are included in this PR?
Implements `Tell()` for non-seekable output streams. Apparently some Arrow code calls this to figure out how many bytes have been written.
### Are these changes tested?
Working on it!
### Are there any user-facing changes?
Yes! (Working on documenting that socket connections or general non-seekable connections should work as generic sinks).
``` r
tmp <- tempfile()
proc <- callr::r_bg(function() {
server <- function() {
library(arrow)
while (TRUE) {
writeLines("Listening...")
con <- socketConnection(host = "localhost", port = 6011, blocking = TRUE,
server = TRUE, open = "r+b")
socketTimeout(con, 3600)
data <- arrow::read_ipc_stream(con, as_data_frame = FALSE)
print(head(as.data.frame(data)))
}
}
server()
}, stdout = tmp)
Sys.sleep(0.5)
library(arrow, warn.conflicts = FALSE)
#> Some features are not enabled in this build of Arrow. Run `arrow_info()` for more information.
rb <- arrow::record_batch(iris)
socketDriver <- socketConnection(host = "localhost",
port = "6011",
blocking = TRUE,
server = FALSE,
open = "w+b")
write_ipc_stream(rb, socketDriver)
Sys.sleep(0.5)
cat(brio::read_file(tmp))
#> Listening...
#> Sepal.Length Sepal.Width Petal.Length Petal.Width Species
#> 1 5.1 3.5 1.4 0.2 setosa
#> 2 4.9 3.0 1.4 0.2 setosa
#> 3 4.7 3.2 1.3 0.2 setosa
#> 4 4.6 3.1 1.5 0.2 setosa
#> 5 5.0 3.6 1.4 0.2 setosa
#> 6 5.4 3.9 1.7 0.4 setosa
#> Listening...
# Shutdown server
proc$interrupt()
#> [1] TRUE
Sys.sleep(0.5)
proc$is_alive()
#> [1] FALSE
```
<sup>Created on 2023-11-27 with [reprex v2.0.2](https://reprex.tidyverse.org)</sup>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-38828: [R] Ensure that streams can be written to socket connections [arrow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #38897:
URL: https://github.com/apache/arrow/pull/38897#issuecomment-1828352365
:warning: GitHub issue #38828 **has been automatically assigned in GitHub** to PR creator.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-38828: [R] Ensure that streams can be written to socket connections [arrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot merged PR #38897:
URL: https://github.com/apache/arrow/pull/38897
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-38828: [R] Ensure that streams can be written to socket connections [arrow]
Posted by "amoeba (via GitHub)" <gi...@apache.org>.
amoeba commented on PR #38897:
URL: https://github.com/apache/arrow/pull/38897#issuecomment-1997936891
Hi @paleolimbot this looks good to me and I don't think an automated test is critical.
I'd like to see the documentation updated to reflect the support this adds (happy to do that in a follow-up PR). Is it safe to say `write_ipc_stream`'s sink arg can now take any `connection` and `read_ipc_stream`'s file arg can also now take any `connection`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-38828: [R] Ensure that streams can be written to socket connections [arrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on PR #38897:
URL: https://github.com/apache/arrow/pull/38897#issuecomment-1998321747
In general, we can use a connection in any place where we read or write anything! Most R functions work like this and I'm not sure we documented it anywhere. I'll find a few places where I can add it in while I've got the tab open.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-38828: [R] Ensure that streams can be written to socket connections [arrow]
Posted by "conbench-apache-arrow[bot] (via GitHub)" <gi...@apache.org>.
conbench-apache-arrow[bot] commented on PR #38897:
URL: https://github.com/apache/arrow/pull/38897#issuecomment-2035378330
After merging your PR, Conbench analyzed the 6 benchmarking runs that have been run so far on merge-commit e0d73c5fd521e9cf79f8c0bbe74285ab8fd4b6d2.
There were no benchmark performance regressions. 🎉
The [full Conbench report](https://github.com/apache/arrow/runs/23409604601) has more details. It also includes information about 2 possible false positives for unstable benchmarks that are known to sometimes produce them.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-38828: [R] Ensure that streams can be written to socket connections [arrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on PR #38897:
URL: https://github.com/apache/arrow/pull/38897#issuecomment-2032835612
@amoeba I think those errors were the duckdb failures (unless I'm looking at the wrong ones). I rebased, but now we have an entirely new set of errors (`docker compose` vs `docker-compose` or something).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-38828: [R] Ensure that streams can be written to socket connections [arrow]
Posted by "amoeba (via GitHub)" <gi...@apache.org>.
amoeba commented on PR #38897:
URL: https://github.com/apache/arrow/pull/38897#issuecomment-2032958391
Sounds like https://github.com/apache/arrow/pull/40949 which should be resolved soon.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org