You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "corasaurus-hex (via GitHub)" <gi...@apache.org> on 2025/07/05 07:35:17 UTC
[I] Add support for registering files in the Arrow IPC stream format as tables using `register_arrow` or similar [datafusion]
corasaurus-hex opened a new issue, #16688:
URL: https://github.com/apache/datafusion/issues/16688
### Is your feature request related to a problem or challenge?
Datafusion currently supports [registering files in the Arrow IPC file format as tables](https://gist.github.com/corasaurus-hex/a6a4b47acd047e4def359931b5714a2b):
```rust
ctx.register_arrow("my_table", "file.arrow", ArrowReadOptions::default())
.await
.unwrap();
ctx.sql("SELECT * FROM my_table LIMIT 10")
.await
.unwrap()
.show()
.await
.unwrap();
```
You can also just reference the file path from SQL in `datafusion-cli`:
```sh
> SELECT * FROM 'file.arrow' LIMIT 10;
```
You cannot, however, [do the same with files in the Arrow IPC stream format](https://gist.github.com/corasaurus-hex/d76a166112ecae57540f0f4a70c93b12). You get the error:
```
called `Result::unwrap()` on an `Err` value: ArrowError(ParseError("Arrow file does not contain correct footer"), None)
```
### Describe the solution you'd like
I would love if `register_arrow` supported files in the Arrow IPC stream format, or if another equivalent function would be added to do the same. Additionally, it would be great if `datafusion-cli` could just reference the files by name in the same way it can for the alternative Arrow IPC format.
### Describe alternatives you've considered
1. [Convert from the stream format to the file format](https://gist.github.com/corasaurus-hex/0dd4d7ef489ed49b5e45ea188c202aa9) and then query as shown above.
2. [Read all the record batches into memory and then register it as MemTable](https://gist.github.com/corasaurus-hex/23c9be7a415c6aede5a89c1d92a6cf47).
3. [Add a new `StreamProvider` impl and use a `StreamTable`](https://gist.github.com/corasaurus-hex/96574afd82780e48c4d0c679c116b23a).
There are probably others, too, but none as simple as just being able to register the arrow file with `register_arrow` or referencing the file directly in `datafusion-cli`.
### Additional context
I'm interested in taking a crack at this feature but, assuming y'all are interested in it, I would love some implementation guidance.
Thanks for your time!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Add support for registering files in the Arrow IPC stream format as tables using `register_arrow` or similar [datafusion]
Posted by "corasaurus-hex (via GitHub)" <gi...@apache.org>.
corasaurus-hex commented on issue #16688:
URL: https://github.com/apache/datafusion/issues/16688#issuecomment-3469630332
Still just my [WIP](https://github.com/apache/datafusion/compare/main...corasaurus-hex:datafusion:register-arrow-ipc-stream-format-files) branch. I haven't had a chance to pick it up again, work has been crazy, but I planned on it this weekend barring something coming up. Would love your thoughts on the direction!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Add support for registering files in the Arrow IPC stream format as tables using `register_arrow` or similar [datafusion]
Posted by "timsaucer (via GitHub)" <gi...@apache.org>.
timsaucer closed issue #16688: Add support for registering files in the Arrow IPC stream format as tables using `register_arrow` or similar
URL: https://github.com/apache/datafusion/issues/16688
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Add support for registering files in the Arrow IPC stream format as tables using `register_arrow` or similar [datafusion]
Posted by "corasaurus-hex (via GitHub)" <gi...@apache.org>.
corasaurus-hex commented on issue #16688:
URL: https://github.com/apache/datafusion/issues/16688#issuecomment-3478729909
would love a review if you have time https://github.com/apache/datafusion/pull/18457
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Add support for registering files in the Arrow IPC stream format as tables using `register_arrow` or similar [datafusion]
Posted by "corasaurus-hex (via GitHub)" <gi...@apache.org>.
corasaurus-hex commented on issue #16688:
URL: https://github.com/apache/datafusion/issues/16688#issuecomment-3369897963
I took a crack at this today and got pretty far but I'm running up against a couple instances of low-level arrow decoding that feel out of place. I'd love your thoughts on maybe upstreaming something or coming up with a better abstraction to handle the need to do selective low-level decoding. I'm willing to put in the work, I just don't want to put a lot of labor into something that might get rejected for going a bad direction.
1. [`ArrowOpener`](https://github.com/apache/datafusion/blob/82cd7f3cdb8dbe0b63b8b62f54543641598655a0/datafusion/core/src/datasource/physical_plan/arrow_file.rs#L159-L255): I'd need to duplicate this decoding/fetching for IPC streams as well. It feels like there's a missing abstraction somewhere for this...
2. [`infer_schema_from_file_stream`](https://github.com/apache/datafusion/blob/82cd7f3cdb8dbe0b63b8b62f54543641598655a0/datafusion/core/src/datasource/file_format/arrow.rs#L350-L433): this was a pretty straight-forward one to solve ([here's how I did it](https://github.com/corasaurus-hex/datafusion/blob/b53b67ca165514801d9aabb2df842da3b93c1a47/datafusion/core/src/datasource/file_format/arrow.rs#L350-L465)), but still feels like something that might be best to have in arrow-rs itself?
You can see my WIP branch diff [here](https://github.com/apache/datafusion/compare/main...corasaurus-hex:datafusion:register-arrow-ipc-stream-format-files). I'm not sure if I should be trying to combine things like I am or if I should be creating entirely separate handling for `arrow_stream` vs `arrow_file`. They're so similar and yet so very different in practice...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Add support for registering files in the Arrow IPC stream format as tables using `register_arrow` or similar [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #16688:
URL: https://github.com/apache/datafusion/issues/16688#issuecomment-3371890527
Thanks @corasaurus-hex
I agree it looks like we can improve the code in this area
What I would suggest is:
1. Proceed with your PR, even if it isn't ideal (e.g. replicate ArrowOpener if needed), and ensure all the code is well tested
2. File any relevant upstream tickets to improve the APIs
> I should be trying to combine things like I am
That sound great in theory, but I am not really sure until we see what it looks like. By default, I would defer to the person implementing the feature (you)
Also, as long as we have good test coverage of the feature, we can always refactor / improve the code as follow on PRs
Thank you for all your help in this area
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Add support for registering files in the Arrow IPC stream format as tables using `register_arrow` or similar [datafusion]
Posted by "jdcasale (via GitHub)" <gi...@apache.org>.
jdcasale commented on issue #16688:
URL: https://github.com/apache/datafusion/issues/16688#issuecomment-3469133824
Any movement on this? Happy to help with dev or review cycles if it's blocked.
Kind of a shame that if I want to read an ipc file (a buffer file or something) I have to hydrate the whole file into memory in order to call read_batches().
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Add support for registering files in the Arrow IPC stream format as tables using `register_arrow` or similar [datafusion]
Posted by "coracuity (via GitHub)" <gi...@apache.org>.
coracuity commented on issue #16688:
URL: https://github.com/apache/datafusion/issues/16688#issuecomment-3186081137
I think, from a superficial reading of the code, I would need to extend [ArrowOpener](https://github.com/apache/datafusion/blob/main/datafusion/core/src/datasource/physical_plan/arrow_file.rs) and [ArrowFormat](https://github.com/apache/datafusion/blob/main/datafusion/core/src/datasource/file_format/arrow.rs) to determine the underlying IPC variant and use the appropriate reader. I know the file format starts with the bytes `ARROW` and so that might be a good enough check.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Add support for registering files in the Arrow IPC stream format as tables using `register_arrow` or similar [datafusion]
Posted by "coracuity (via GitHub)" <gi...@apache.org>.
coracuity commented on issue #16688:
URL: https://github.com/apache/datafusion/issues/16688#issuecomment-3186069109
a StreamReader can't read an IPC file format, unfortunately, so we'd need to try both and see which works?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Add support for registering files in the Arrow IPC stream format as tables using `register_arrow` or similar [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #16688:
URL: https://github.com/apache/datafusion/issues/16688#issuecomment-3046580169
I reread the format guide again, and I think the Arrow file format is an extension of the IPC (stream) format https://arrow.apache.org/docs/format/Columnar.html#ipc-file-format
It looks like the file format simply has a few extra header bytes and then a footer. Otherwise it is the same as the stream. format
So we should be able to support both formats fairly straightforwardly
It may be possible to simply use the [StreamReader](https://docs.rs/arrow-ipc/55.2.0/arrow_ipc/reader/struct.StreamReader.html) rather than the [`FileReader`](https://docs.rs/arrow-ipc/55.2.0/arrow_ipc/reader/struct.FileReader.html) in the underlying code for `ArrowFormatFactory`
https://github.com/apache/datafusion/blob/2741c60ad9d637e5014d201802abb25562449491/datafusion/core/src/datasource/file_format/arrow.rs#L110-L109
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Add support for registering files in the Arrow IPC stream format as tables using `register_arrow` or similar [datafusion]
Posted by "jdcasale (via GitHub)" <gi...@apache.org>.
jdcasale commented on issue #16688:
URL: https://github.com/apache/datafusion/issues/16688#issuecomment-3486435029
Will take a look tonight (don't have merge permissions, but I can give a thorough review to help those who do)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Add support for registering files in the Arrow IPC stream format as tables using `register_arrow` or similar [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #16688:
URL: https://github.com/apache/datafusion/issues/16688#issuecomment-3189115601
That sounds right to me
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org