You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2021/06/08 16:56:18 UTC
[GitHub] [arrow-datafusion] Dandandan opened a new issue #525: Add delta tableprovider
Dandandan opened a new issue #525:
URL: https://github.com/apache/arrow-datafusion/issues/525
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
Delta is used more and more as a storage format, and has some nice features like ACID operations, table statistics and storage optimization.
**Describe the solution you'd like**
Use `delta-rs` to add support for reading delta datasets. The library already has a `TableProvider` and some other features.
**Describe alternatives you've considered**
**Additional context**
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] Dandandan commented on issue #525: Add Delta Lake TableProvider
Posted by GitBox <gi...@apache.org>.
Dandandan commented on issue #525:
URL: https://github.com/apache/arrow-datafusion/issues/525#issuecomment-856943253
> fwiw, imo this should be discussed over the mailing list.
I agree, if we have some positive reactions I will send something over the mailing list.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] jorgecarleitao commented on issue #525: Add Delta Lake TableProvider
Posted by GitBox <gi...@apache.org>.
jorgecarleitao commented on issue #525:
URL: https://github.com/apache/arrow-datafusion/issues/525#issuecomment-856941493
fwiw, imo this should be discussed over the mailing list.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] nevi-me commented on issue #525: Add Delta Lake TableProvider
Posted by GitBox <gi...@apache.org>.
nevi-me commented on issue #525:
URL: https://github.com/apache/arrow-datafusion/issues/525#issuecomment-857942853
> I am also planning to promote datafusion as the default query engine for executing native delta lake queries in `delta-rs`. This will make it easier for us to provide deltalake query access to other languages and runtimes.
I like this approach, and I think there might be other approaches to adding IO support to datafusion.
How about separate crates implementing functionality through traits, then having a contrib section in the README listing them?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] houqp commented on issue #525: Add Delta Lake TableProvider
Posted by GitBox <gi...@apache.org>.
houqp commented on issue #525:
URL: https://github.com/apache/arrow-datafusion/issues/525#issuecomment-857006465
I am all for this. I think this is a good move, especially for ballista. I am happy to help maintain the deltalake support in datafusion going forward as well. If we go with this route, I would like to drop the table provider implementation in `delta-rs` so we can all focus on one official datafusion provider implementation in arrow-datafusion.
I am also planning to promote datafusion as the default query engine for executing native delta lake queries in `delta-rs`. This will make it easier for us to provide deltalake query access to other languages and runtimes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] Dandandan commented on issue #525: Add Delta Lake TableProvider
Posted by GitBox <gi...@apache.org>.
Dandandan commented on issue #525:
URL: https://github.com/apache/arrow-datafusion/issues/525#issuecomment-856936027
FYI @houqp
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] Dandandan edited a comment on issue #525: Add Delta Lake TableProvider
Posted by GitBox <gi...@apache.org>.
Dandandan edited a comment on issue #525:
URL: https://github.com/apache/arrow-datafusion/issues/525#issuecomment-856936027
FYI @houqp what do you think of integrating this into DataFusion?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org