You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "Xuanwo (via GitHub)" <gi...@apache.org> on 2023/07/22 04:18:46 UTC
[GitHub] [iceberg-rust] Xuanwo opened a new issue, #9: proposal: Roadmap of iceberg-rust v0.2
Xuanwo opened a new issue, #9:
URL: https://github.com/apache/iceberg-rust/issues/9
As discussed in our [previous meeting](https://hackmd.io/@xuanwo/iceberg-rust), we plan to provide the following feature in iceberg-rust v0.1:
## Required
- specs: all needed data types to represents an iceberg table in memory
- arrow interop: allow interop between arrow
- io: integrate with different storage services via [opendal](https://github.com/apache/incubator-opendal)
## Optional
- compilation to wasm/wasi so users can use iceberg from web browser.
## Possibility
- Integrate with [datafusion](https://github.com/apache/arrow-datafusion)
- Providing catalog trait to allow vendors can implement a catalog.
- Implement catalogs (REST, Glue, Databend, RisingWave, ...)
## Notes
- iceberg `v0.1` already been take, so we will take v0.2 as our first release.
- opendal covers all storage services that already implemented by iceberg and pyiceberg.
- opendal provides a compatible layer that can be used in datafusion (the query engine we can integrate in the future).
---
Welcome for any comments!
cc @liurenjie1024, @JanKaul, @ZENOTME for review.
Also cc @Fokko for ideas.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg-rust] Xuanwo commented on issue #9: proposal: Roadmap of iceberg-rust v0.2
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on issue #9:
URL: https://github.com/apache/iceberg-rust/issues/9#issuecomment-1661474075
No objections to this proposal. I have created a tracking issue at https://github.com/apache/iceberg-rust/issues/18. Let's get started!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg-rust] scsmithr commented on issue #9: proposal: Roadmap of iceberg-rust v0.2
Posted by "scsmithr (via GitHub)" <gi...@apache.org>.
scsmithr commented on issue #9:
URL: https://github.com/apache/iceberg-rust/issues/9#issuecomment-1650348282
Hello from glaredb!
We have a wip pr with reading icerberg tables here: https://github.com/GlareDB/glaredb/pull/1382. We lean pretty heavily on datafusion, and so there's likely some bits we'd be able to contribute in that area.
There's still some exploratory work that we're doing with this, but we're definitely looking at the best way to integrate iceberg-rust.
> > About datafusion integration, I think it would be better to put it in another crate, and an example usage of reader/writer api.
>
> I used to think we can hide them under a feature like `delta-rs` does. We can discuss the detail after v0.2 released.
For what it's worth, I'd prefer the feature over having a separate crate for datafusion integration.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg-rust] JanKaul commented on issue #9: proposal: Roadmap of iceberg-rust v0.2
Posted by "JanKaul (via GitHub)" <gi...@apache.org>.
JanKaul commented on issue #9:
URL: https://github.com/apache/iceberg-rust/issues/9#issuecomment-1650360491
Hi there, thanks for joining the discussion. I would also put the datafusion integration in another crate.
Since many projects rely on arrow, I think arrow support deserves a spot in iceberg-rust. And this should make it easier to integrate datafusion.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg-rust] Xuanwo commented on issue #9: proposal: Roadmap of iceberg-rust v0.2
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on issue #9:
URL: https://github.com/apache/iceberg-rust/issues/9#issuecomment-1647116371
> I think one missing part is the standalone reader/writer api, table transaction api. But I'm ok to postpone it to later releases.
Thanks for the comment. I will add them in `possibility` section to avoid missing them in next release.
> About datafusion integration, I think it would be better to put it in another crate, and an example usage of reader/writer api.
I used to think we can hide them under a feature like `delta-rs` does. We can discuss the detail after v0.2 released.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg-rust] liurenjie1024 commented on issue #9: proposal: Roadmap of iceberg-rust v0.2
Posted by "liurenjie1024 (via GitHub)" <gi...@apache.org>.
liurenjie1024 commented on issue #9:
URL: https://github.com/apache/iceberg-rust/issues/9#issuecomment-1647096006
Agree with the proposal.
I think one missing part is the standalone reader/writer api, table transaction api. But I'm ok to postpone it to later releases.
About datafusion integration, I think it would be better to put it in another crate, and an example usage of reader/writer api.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org