You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "alamb (via GitHub)" <gi...@apache.org> on 2025/01/12 13:13:50 UTC

[I] Consider using upstream arrow-avro reader [datafusion]

alamb opened a new issue, #14097:
URL: https://github.com/apache/datafusion/issues/14097

   ### Is your feature request related to a problem or challenge?
   
   Currently DataFusion has its own avro --> arrow implementation in: https://github.com/apache/datafusion/blob/54a5d3fd3f98997f048c2e3e7ba67d7bfe2f124d/datafusion/core/src/datasource/avro_to_arrow/mod.rs
   
   This was added way back in the day by @Igosuki and @jorgecarleitao in 
   - https://github.com/apache/datafusion/pull/910
   
   Recently there is work upstream in arrow-rs  to add additional avro support:
   - https://github.com/apache/arrow-rs/issues/4886
   
   For example, the recent PR from @jecsand838
   - https://github.com/apache/arrow-rs/pull/6965
   
   
   
   ### Describe the solution you'd like
   
   One arrow-avro is more mature, it would be great to consider removing the DataFusion specific implementation and using the upstream arrow-avro one 
   
   This would allow DataFusion to take advantage of the upstream work and keep our codebase a bit simpler
   
   ### Describe alternatives you've considered
   
   _No response_
   
   ### Additional context
   
   _No response_


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Use `arrow-avro` for performance and improved type support [datafusion]

Posted by "jecsand838 (via GitHub)" <gi...@apache.org>.
jecsand838 commented on issue #14097:
URL: https://github.com/apache/datafusion/issues/14097#issuecomment-3583042236

   > > Sure, would love to help, what is actually missing right now?
   > 
   > I think [@getChan](https://github.com/getChan) and [@jecsand838](https://github.com/jecsand838) are working out some schema issues:
   > 
   > * [WIP: using arrow-avro. remove own implementation #17861 (comment)](https://github.com/apache/datafusion/pull/17861#issuecomment-3553594979)
   >   Maybe you can help push that along?
   
   We're making progress. The biggest issue so far is getting projection working as far I can tell atleast. @getChan 
   
   > [@alamb](https://github.com/alamb) I've also encountered some schema issues in my attempt to create a reader using arrow-avro, documented here: [apache/arrow-rs#8928](https://github.com/apache/arrow-rs/issues/8928)
   > 
   > Regardless, I will open a PR soon in arrow-rs for an AsyncReader that uses object_store and should be very easily integrated into DataFusion in lieu of the current reader.
   
   @EmilyMatt Thank you for creating these issues. I do like the idea of an OCF AsyncReader, this makes sense to me.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Use `arrow-avro` for performance and improved type support [datafusion]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb closed issue #14097: Use `arrow-avro` for performance and improved type support
URL: https://github.com/apache/datafusion/issues/14097


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Consider using upstream arrow-avro reader [datafusion]

Posted by "jecsand838 (via GitHub)" <gi...@apache.org>.
jecsand838 commented on issue #14097:
URL: https://github.com/apache/datafusion/issues/14097#issuecomment-3293064544

   @getChan @alamb 
   
   I had a quick update I wanted to give.
   
   Once https://github.com/apache/arrow-rs/pull/8293, https://github.com/apache/arrow-rs/pull/8348, and https://github.com/apache/arrow-rs/pull/8349 are reviewed and merged in, The `arrow-avro` `Reader` will be functionally ready for use in DataFusion. We are getting extremely close. 
   
   Also on the `arrow-avro` `Writer` side, we have about 4-5 PRs left as well. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Consider using upstream arrow-avro reader [datafusion]

Posted by "getChan (via GitHub)" <gi...@apache.org>.
getChan commented on issue #14097:
URL: https://github.com/apache/datafusion/issues/14097#issuecomment-3293191255

   > @getChan @alamb 
   > 
   > I had a quick update I wanted to give.
   > 
   > Once https://github.com/apache/arrow-rs/pull/8293, https://github.com/apache/arrow-rs/pull/8348, and https://github.com/apache/arrow-rs/pull/8349 are reviewed and merged in, The `arrow-avro` `Reader` will be functionally ready for use in DataFusion. We are getting extremely close. 
   > 
   > Also on the `arrow-avro` `Writer` side, we have about 4-5 PRs left as well. 
   
   Thank you for the update. I'll get started on the work soon. If you'd like to work on this, you can take this issue.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Consider using upstream arrow-avro reader [datafusion]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #14097:
URL: https://github.com/apache/datafusion/issues/14097#issuecomment-3303951278

   FYI @getChan I don't think we are going to release the arrow-avro crate for a while
   
   You can start testing immediately using  crates.io patch in Cargo.toml
   
   An example of how to do so is here
   
   - https://github.com/apache/datafusion/pull/14628
   
   
   Something like
   ```toml
   [patch.crates-io]
   arrow-avro = { git = "https://github.com/apache/arrow-rs.git", rev = "..." }
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Consider using upstream arrow-avro reader [datafusion]

Posted by "getChan (via GitHub)" <gi...@apache.org>.
getChan commented on issue #14097:
URL: https://github.com/apache/datafusion/issues/14097#issuecomment-3301963628

   waiting arrow-avro 56.0.0 release. 
   for use brand-new reader APIs. like ReaderBuilder


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Use `arrow-avro` for performance and improved type support [datafusion]

Posted by "EmilyMatt (via GitHub)" <gi...@apache.org>.
EmilyMatt commented on issue #14097:
URL: https://github.com/apache/datafusion/issues/14097#issuecomment-3580560775

   @alamb I've also encountered some schema issues in my attempt to create a reader using arrow-avro, documented here.
   Regardless, I will open a PR soon in arrow-rs for an AsyncReader that uses object_store and should be very easily integrated into DataFusion in lieu of the current reader.
   https://github.com/apache/arrow-rs/issues/8928


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Consider using upstream arrow-avro reader [datafusion]

Posted by "getChan (via GitHub)" <gi...@apache.org>.
getChan commented on issue #14097:
URL: https://github.com/apache/datafusion/issues/14097#issuecomment-2614046188

   I'm waiting for the arrow-avro PR below as it might include changes to the public API. 
   https://github.com/apache/arrow-rs/pull/6965
   I will resume related work once this PR is merged and arrow-avro is released.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Use `arrow-avro` for performance and improved type support [datafusion]

Posted by "EmilyMatt (via GitHub)" <gi...@apache.org>.
EmilyMatt commented on issue #14097:
URL: https://github.com/apache/datafusion/issues/14097#issuecomment-3567935147

   > [@getChan](https://github.com/getChan) and [@jecsand838](https://github.com/jecsand838) , [@EmilyMatt](https://github.com/EmilyMatt) has been looking into the current avro reader, see
   > 
   >     * [Support Ref types in Avro Reader. #18811](https://github.com/apache/datafusion/issues/18811)
   > 
   >     * [Improve Avro Reader Types Support #18810](https://github.com/apache/datafusion/issues/18810)
   > 
   > 
   > Maybe you can work together to clean up avro support in DataFusion by consolidating on the new crate
   
   Sure, would love to help, what is actually missing right now?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Use `arrow-avro` for performance and improved type support [datafusion]

Posted by "EmilyMatt (via GitHub)" <gi...@apache.org>.
EmilyMatt commented on issue #14097:
URL: https://github.com/apache/datafusion/issues/14097#issuecomment-3580686385

   Note the following arrow-rs PR:
   https://github.com/apache/arrow-rs/pull/8930


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Use `arrow-avro` for performance and improved type support [datafusion]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #14097:
URL: https://github.com/apache/datafusion/issues/14097#issuecomment-3575436101

   > Sure, would love to help, what is actually missing right now?
   
   I think @getChan and @jecsand838 are working out some schema issues: 
   - https://github.com/apache/datafusion/pull/17861#issuecomment-3553594979
   Maybe you can help push that along?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Use `arrow-avro` for performance and improved type support [datafusion]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #14097:
URL: https://github.com/apache/datafusion/issues/14097#issuecomment-3559567983

   @getChan  and @jecsand838 , @EmilyMatt has been looking into the current avro reader, see
   - https://github.com/apache/datafusion/issues/18811
   - https://github.com/apache/datafusion/issues/18810
   
   Maybe you can work together to clean up avro support in DataFusion by consolidating on the new crate


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org