You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "ianmcook (via GitHub)" <gi...@apache.org> on 2024/03/16 16:34:12 UTC
[I] [Python] Create Python examples of indirect HTTP GET Arrow client and server [arrow]
ianmcook opened a new issue, #40596:
URL: https://github.com/apache/arrow/issues/40596
### Describe the enhancement requested
Contribute Python client and server examples to the [indirect HTTP GET examples in the `arrow-experiments` repo](https://github.com/apache/arrow-experiments/tree/main/http/get_indirect). This should demonstrate how to use a two-step sequence to retrieve Arrow data:
1. The client sends a GET request to a server and receives a JSON response from the server containing one or more server URIs.
2. The client sends GET requests to each of those URIs and receives a response from each server containing an Arrow IPC stream of record batches (exactly as in the [simple GET examples](https://github.com/apache/arrow-experiments/tree/main/http/get_simple)).
### Component(s)
Python
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python] Create Python examples of indirect HTTP GET Arrow client and server [arrow]
Posted by "ianmcook (via GitHub)" <gi...@apache.org>.
ianmcook closed issue #40596: [Python] Create Python examples of indirect HTTP GET Arrow client and server
URL: https://github.com/apache/arrow/issues/40596
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python] Create Python examples of indirect HTTP GET Arrow client and server [arrow]
Posted by "ianmcook (via GitHub)" <gi...@apache.org>.
ianmcook commented on issue #40596:
URL: https://github.com/apache/arrow/issues/40596#issuecomment-2035344802
> we do not have any facilities in IPC, Flight, Flight SQL, or ADBC that encapsulate multiple different-schema streams into one logical unit
but there are some issues requesting this in ADBC: https://github.com/apache/arrow-adbc/issues/1447, https://github.com/apache/arrow-adbc/issues/1358
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python] Create Python examples of indirect HTTP GET Arrow client and server [arrow]
Posted by "felipecrv (via GitHub)" <gi...@apache.org>.
felipecrv commented on issue #40596:
URL: https://github.com/apache/arrow/issues/40596#issuecomment-2318604803
@CurtHagenlocher you should take a look at https://github.com/apache/arrow-experiments/pull/33 It specifies how Arrow streams can be served in `multipart/mixed` responses.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python] Create Python examples of indirect HTTP GET Arrow client and server [arrow]
Posted by "ianmcook (via GitHub)" <gi...@apache.org>.
ianmcook commented on issue #40596:
URL: https://github.com/apache/arrow/issues/40596#issuecomment-2035295660
Ah, I see.
IIUC we do not have any facilities in IPC, Flight, Flight SQL, or ADBC that encapsulate multiple different-schema streams into one logical unit. And I don't think we're super eager to create anything like that. So using whatever facilities HTTP provides seems like the way to go.
So I think a multipart/mixed response (as described in #40598) seems like probably the best way to do this if you can't maintain state on the server side. I think if you choose a sufficiently obscure delimiter, the delimiter problem is exceedingly unlikely to be a real problem in practice, but we should research this more to better understand the risks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python] Create Python examples of indirect HTTP GET Arrow client and server [arrow]
Posted by "ianmcook (via GitHub)" <gi...@apache.org>.
ianmcook commented on issue #40596:
URL: https://github.com/apache/arrow/issues/40596#issuecomment-2035198898
@CurtHagenlocher asked:
> Any thoughts on the "best" canonical way to return multiple record batches? Would that also be multipart/mixed or would it be better to avoid the delimiter problem and e.g. use an alternate Content-Type indicating that the response contains multiple streams?
By "multiple record batches" do you mean record batches with different schemas? (Or maybe they have the same schema but it's important to keep them logically separated in separate IPC streams?) If that's what you mean, then I think the two-step / indirect approach described here is probably what we should generally recommend.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python] Create Python examples of indirect HTTP GET Arrow client and server [arrow]
Posted by "CurtHagenlocher (via GitHub)" <gi...@apache.org>.
CurtHagenlocher commented on issue #40596:
URL: https://github.com/apache/arrow/issues/40596#issuecomment-2035255506
Yes, sorry, separate result sets with potentially-different schemas. I think the scenario here is visualization in the browser where the JS-based UI sends a single request for multiple results, each of which is relatively small. Having to do this as described here would mean having to maintain state on the server across multiple requests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org