You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "alamb (via GitHub)" <gi...@apache.org> on 2025/01/12 13:13:50 UTC
[I] Consider using upstream arrow-avro reader [datafusion]
alamb opened a new issue, #14097:
URL: https://github.com/apache/datafusion/issues/14097
### Is your feature request related to a problem or challenge?
Currently DataFusion has its own avro --> arrow implementation in: https://github.com/apache/datafusion/blob/54a5d3fd3f98997f048c2e3e7ba67d7bfe2f124d/datafusion/core/src/datasource/avro_to_arrow/mod.rs
This was added way back in the day by @Igosuki and @jorgecarleitao in
- https://github.com/apache/datafusion/pull/910
Recently there is work upstream in arrow-rs to add additional avro support:
- https://github.com/apache/arrow-rs/issues/4886
For example, the recent PR from @jecsand838
- https://github.com/apache/arrow-rs/pull/6965
### Describe the solution you'd like
One arrow-avro is more mature, it would be great to consider removing the DataFusion specific implementation and using the upstream arrow-avro one
This would allow DataFusion to take advantage of the upstream work and keep our codebase a bit simpler
### Describe alternatives you've considered
_No response_
### Additional context
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Use `arrow-avro` for performance and improved type support [datafusion]
Posted by "jecsand838 (via GitHub)" <gi...@apache.org>.
jecsand838 commented on issue #14097:
URL: https://github.com/apache/datafusion/issues/14097#issuecomment-3583042236
> > Sure, would love to help, what is actually missing right now?
>
> I think [@getChan](https://github.com/getChan) and [@jecsand838](https://github.com/jecsand838) are working out some schema issues:
>
> * [WIP: using arrow-avro. remove own implementation #17861 (comment)](https://github.com/apache/datafusion/pull/17861#issuecomment-3553594979)
> Maybe you can help push that along?
We're making progress. The biggest issue so far is getting projection working as far I can tell atleast. @getChan
> [@alamb](https://github.com/alamb) I've also encountered some schema issues in my attempt to create a reader using arrow-avro, documented here: [apache/arrow-rs#8928](https://github.com/apache/arrow-rs/issues/8928)
>
> Regardless, I will open a PR soon in arrow-rs for an AsyncReader that uses object_store and should be very easily integrated into DataFusion in lieu of the current reader.
@EmilyMatt Thank you for creating these issues. I do like the idea of an OCF AsyncReader, this makes sense to me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Use `arrow-avro` for performance and improved type support [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb closed issue #14097: Use `arrow-avro` for performance and improved type support
URL: https://github.com/apache/datafusion/issues/14097
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Consider using upstream arrow-avro reader [datafusion]
Posted by "jecsand838 (via GitHub)" <gi...@apache.org>.
jecsand838 commented on issue #14097:
URL: https://github.com/apache/datafusion/issues/14097#issuecomment-3293064544
@getChan @alamb
I had a quick update I wanted to give.
Once https://github.com/apache/arrow-rs/pull/8293, https://github.com/apache/arrow-rs/pull/8348, and https://github.com/apache/arrow-rs/pull/8349 are reviewed and merged in, The `arrow-avro` `Reader` will be functionally ready for use in DataFusion. We are getting extremely close.
Also on the `arrow-avro` `Writer` side, we have about 4-5 PRs left as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Consider using upstream arrow-avro reader [datafusion]
Posted by "getChan (via GitHub)" <gi...@apache.org>.
getChan commented on issue #14097:
URL: https://github.com/apache/datafusion/issues/14097#issuecomment-3293191255
> @getChan @alamb
>
> I had a quick update I wanted to give.
>
> Once https://github.com/apache/arrow-rs/pull/8293, https://github.com/apache/arrow-rs/pull/8348, and https://github.com/apache/arrow-rs/pull/8349 are reviewed and merged in, The `arrow-avro` `Reader` will be functionally ready for use in DataFusion. We are getting extremely close.
>
> Also on the `arrow-avro` `Writer` side, we have about 4-5 PRs left as well.
Thank you for the update. I'll get started on the work soon. If you'd like to work on this, you can take this issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Consider using upstream arrow-avro reader [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #14097:
URL: https://github.com/apache/datafusion/issues/14097#issuecomment-3303951278
FYI @getChan I don't think we are going to release the arrow-avro crate for a while
You can start testing immediately using crates.io patch in Cargo.toml
An example of how to do so is here
- https://github.com/apache/datafusion/pull/14628
Something like
```toml
[patch.crates-io]
arrow-avro = { git = "https://github.com/apache/arrow-rs.git", rev = "..." }
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Consider using upstream arrow-avro reader [datafusion]
Posted by "getChan (via GitHub)" <gi...@apache.org>.
getChan commented on issue #14097:
URL: https://github.com/apache/datafusion/issues/14097#issuecomment-3301963628
waiting arrow-avro 56.0.0 release.
for use brand-new reader APIs. like ReaderBuilder
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Use `arrow-avro` for performance and improved type support [datafusion]
Posted by "EmilyMatt (via GitHub)" <gi...@apache.org>.
EmilyMatt commented on issue #14097:
URL: https://github.com/apache/datafusion/issues/14097#issuecomment-3580560775
@alamb I've also encountered some schema issues in my attempt to create a reader using arrow-avro, documented here.
Regardless, I will open a PR soon in arrow-rs for an AsyncReader that uses object_store and should be very easily integrated into DataFusion in lieu of the current reader.
https://github.com/apache/arrow-rs/issues/8928
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Consider using upstream arrow-avro reader [datafusion]
Posted by "getChan (via GitHub)" <gi...@apache.org>.
getChan commented on issue #14097:
URL: https://github.com/apache/datafusion/issues/14097#issuecomment-2614046188
I'm waiting for the arrow-avro PR below as it might include changes to the public API.
https://github.com/apache/arrow-rs/pull/6965
I will resume related work once this PR is merged and arrow-avro is released.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Use `arrow-avro` for performance and improved type support [datafusion]
Posted by "EmilyMatt (via GitHub)" <gi...@apache.org>.
EmilyMatt commented on issue #14097:
URL: https://github.com/apache/datafusion/issues/14097#issuecomment-3567935147
> [@getChan](https://github.com/getChan) and [@jecsand838](https://github.com/jecsand838) , [@EmilyMatt](https://github.com/EmilyMatt) has been looking into the current avro reader, see
>
> * [Support Ref types in Avro Reader. #18811](https://github.com/apache/datafusion/issues/18811)
>
> * [Improve Avro Reader Types Support #18810](https://github.com/apache/datafusion/issues/18810)
>
>
> Maybe you can work together to clean up avro support in DataFusion by consolidating on the new crate
Sure, would love to help, what is actually missing right now?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Use `arrow-avro` for performance and improved type support [datafusion]
Posted by "EmilyMatt (via GitHub)" <gi...@apache.org>.
EmilyMatt commented on issue #14097:
URL: https://github.com/apache/datafusion/issues/14097#issuecomment-3580686385
Note the following arrow-rs PR:
https://github.com/apache/arrow-rs/pull/8930
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Use `arrow-avro` for performance and improved type support [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #14097:
URL: https://github.com/apache/datafusion/issues/14097#issuecomment-3575436101
> Sure, would love to help, what is actually missing right now?
I think @getChan and @jecsand838 are working out some schema issues:
- https://github.com/apache/datafusion/pull/17861#issuecomment-3553594979
Maybe you can help push that along?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Use `arrow-avro` for performance and improved type support [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #14097:
URL: https://github.com/apache/datafusion/issues/14097#issuecomment-3559567983
@getChan and @jecsand838 , @EmilyMatt has been looking into the current avro reader, see
- https://github.com/apache/datafusion/issues/18811
- https://github.com/apache/datafusion/issues/18810
Maybe you can work together to clean up avro support in DataFusion by consolidating on the new crate
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org