You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@iceberg.apache.org by Xuanwo <xu...@apache.org> on 2025/03/14 07:52:00 UTC
[DISCUSS] Move iceberg-cpp step forward
Hi, everyone
Today, Renjie Liu and I had a discussion with some of the iceberg-cpp contributors—Gang Wu, Junwang Zhao, and David Li—about ways to advance the project. We came up with some great ideas, so I'm sharing them here to gather more feedback.
## Background
The community continues to show interest in a C++ implementation of Iceberg. Some users are interested in C++ bindings with a well-designed API for data types and manifests, but they have their own runtime and I/O, so they prefer a binding that exclude these components. However, there are also users who require an all-in-one solution.
The iceberg-cpp project aims to provide a well-designed interface for users to plug in file I/O, readers, writers, and other components while offering a default implementation in other modules.
Luckily, we now have iceberg-rust, which is not fully feature-complete but undergoing rapid development. We can develop a C++ binding based on iceberg-rust and establish a robust C++ API within the iceberg-cpp project. The C++ project will define public interfaces and offer various build targets for users to choose from. The overall process will be similar to how pyiceberg interacts with iceberg-rust.
## Plan
1. Use cxx[^1] to build cpp bindings for iceberg-rust which exposes core APIs.
The binding will support different build profiles that users can choose to use fileio from iceberg-rust or not.
2. iceberg-cpp will define public interfaces and types for cpp users.
After this separation, iceberg-rust can focus on implementing features and exposing them through simpler, more direct bindings. Meanwhile, iceberg-cpp will be responsible for designing, testing, and maintaining the user-level APIs, allowing users to select the parts they need or plug in their own file I/O components.
## Conclusion
We believe this design combines the best parts of two projects and helps us iterate iceberg-cpp more quickly.
Welcome to leave your comments here!
[^1]: https://cxx.rs/
Xuanwo
https://xuanwo.io/
Re: [DISCUSS] Move iceberg-cpp step forward
Posted by Renjie Liu <li...@gmail.com>.
Thanks Xuanwo for the summary.
I think it's a good idea to move c/cpp related code to iceberg-cpp repo,
and have a try on the cxx binding approach.
On Wed, Mar 26, 2025 at 6:54 PM Junwang Zhao <zh...@gmail.com> wrote:
> Hi xuanwo
>
> On Tue, Mar 25, 2025 at 2:36 PM Xuanwo <xu...@apache.org> wrote:
>
>> After a deeper discussion with friends in the Slack #cpp channel, we
>> reached the following consensus:
>>
>> - iceberg-rust won't maintain any c/cpp related code.
>> - iceberg-cpp will have an internal crate based on iceberg-rust and
>> invokes cxx/cbingen to build bindings
>> - And finally, we will have a libiceberg-api and libiceberg-cpp ,
>> libicberg-rust in the iceberg-cpp repo.
>>
>> In which, iceberg-cpp will provide:
>>
>> - libiceberg-api : An headers that defines APIs available to users.
>> - libiceberg-cpp : An impl of libiceberg-api based on pure cpp (based on
>> nanoarrow, json, libz and so on)
>> - libicberg-rust : An impl of libiceberg-api based on iceberg-rust.
>>
>> What do you think?
>>
>
> Thank you for the summary!
>
> I think this approach will work well. Iceberg-cpp might also consider
> providing a
> C API wrapper to support pure C users.
>
> Since we are still at the early stage, we can experiment with this approach
> and revisit it for further discussion if any issues arise.
>
>
>>
>> On Tue, Mar 18, 2025, at 16:56, Fokko Driesprong wrote:
>>
>> Hi Xuanwo,
>>
>> Thanks for bringing this up. I'm not a cpp expert, but following this
>> with great interest. I think this is a very promising approach since the
>> memory model is already the same across projects (Arrow).
>>
>> Kind regards,
>> Fokko
>>
>> Op vr 14 mrt 2025 om 08:53 schreef Xuanwo <xu...@apache.org>:
>>
>>
>> Hi, everyone
>>
>> Today, Renjie Liu and I had a discussion with some of the iceberg-cpp
>> contributors—Gang Wu, Junwang Zhao, and David Li—about ways to advance the
>> project. We came up with some great ideas, so I'm sharing them here to
>> gather more feedback.
>>
>> ## Background
>>
>> The community continues to show interest in a C++ implementation of
>> Iceberg. Some users are interested in C++ bindings with a well-designed API
>> for data types and manifests, but they have their own runtime and I/O, so
>> they prefer a binding that exclude these components. However, there are
>> also users who require an all-in-one solution.
>>
>> The iceberg-cpp project aims to provide a well-designed interface for
>> users to plug in file I/O, readers, writers, and other components while
>> offering a default implementation in other modules.
>>
>> Luckily, we now have iceberg-rust, which is not fully feature-complete
>> but undergoing rapid development. We can develop a C++ binding based on
>> iceberg-rust and establish a robust C++ API within the iceberg-cpp project.
>> The C++ project will define public interfaces and offer various build
>> targets for users to choose from. The overall process will be similar to
>> how pyiceberg interacts with iceberg-rust.
>>
>> ## Plan
>>
>>
>> 1. Use cxx[^1] to build cpp bindings for iceberg-rust which exposes core
>> APIs.
>>
>>
>> The binding will support different build profiles that users can choose
>> to use fileio from iceberg-rust or not.
>>
>> 2. iceberg-cpp will define public interfaces and types for cpp users.
>>
>> After this separation, iceberg-rust can focus on implementing features
>> and exposing them through simpler, more direct bindings. Meanwhile,
>> iceberg-cpp will be responsible for designing, testing, and maintaining the
>> user-level APIs, allowing users to select the parts they need or plug in
>> their own file I/O components.
>>
>> ## Conclusion
>>
>> We believe this design combines the best parts of two projects and helps
>> us iterate iceberg-cpp more quickly.
>>
>> Welcome to leave your comments here!
>>
>> [^1]: https://cxx.rs/
>>
>> Xuanwo
>>
>> https://xuanwo.io/
>>
>> Xuanwo
>>
>> https://xuanwo.io/
>>
>>
>
> --
> Regards
> Junwang Zhao
>
Re: [DISCUSS] Move iceberg-cpp step forward
Posted by Junwang Zhao <zh...@gmail.com>.
Hi xuanwo
On Tue, Mar 25, 2025 at 2:36 PM Xuanwo <xu...@apache.org> wrote:
> After a deeper discussion with friends in the Slack #cpp channel, we
> reached the following consensus:
>
> - iceberg-rust won't maintain any c/cpp related code.
> - iceberg-cpp will have an internal crate based on iceberg-rust and
> invokes cxx/cbingen to build bindings
> - And finally, we will have a libiceberg-api and libiceberg-cpp ,
> libicberg-rust in the iceberg-cpp repo.
>
> In which, iceberg-cpp will provide:
>
> - libiceberg-api : An headers that defines APIs available to users.
> - libiceberg-cpp : An impl of libiceberg-api based on pure cpp (based on
> nanoarrow, json, libz and so on)
> - libicberg-rust : An impl of libiceberg-api based on iceberg-rust.
>
> What do you think?
>
Thank you for the summary!
I think this approach will work well. Iceberg-cpp might also consider
providing a
C API wrapper to support pure C users.
Since we are still at the early stage, we can experiment with this approach
and revisit it for further discussion if any issues arise.
>
> On Tue, Mar 18, 2025, at 16:56, Fokko Driesprong wrote:
>
> Hi Xuanwo,
>
> Thanks for bringing this up. I'm not a cpp expert, but following this with
> great interest. I think this is a very promising approach since the memory
> model is already the same across projects (Arrow).
>
> Kind regards,
> Fokko
>
> Op vr 14 mrt 2025 om 08:53 schreef Xuanwo <xu...@apache.org>:
>
>
> Hi, everyone
>
> Today, Renjie Liu and I had a discussion with some of the iceberg-cpp
> contributors—Gang Wu, Junwang Zhao, and David Li—about ways to advance the
> project. We came up with some great ideas, so I'm sharing them here to
> gather more feedback.
>
> ## Background
>
> The community continues to show interest in a C++ implementation of
> Iceberg. Some users are interested in C++ bindings with a well-designed API
> for data types and manifests, but they have their own runtime and I/O, so
> they prefer a binding that exclude these components. However, there are
> also users who require an all-in-one solution.
>
> The iceberg-cpp project aims to provide a well-designed interface for
> users to plug in file I/O, readers, writers, and other components while
> offering a default implementation in other modules.
>
> Luckily, we now have iceberg-rust, which is not fully feature-complete but
> undergoing rapid development. We can develop a C++ binding based on
> iceberg-rust and establish a robust C++ API within the iceberg-cpp project.
> The C++ project will define public interfaces and offer various build
> targets for users to choose from. The overall process will be similar to
> how pyiceberg interacts with iceberg-rust.
>
> ## Plan
>
>
> 1. Use cxx[^1] to build cpp bindings for iceberg-rust which exposes core
> APIs.
>
>
> The binding will support different build profiles that users can choose to
> use fileio from iceberg-rust or not.
>
> 2. iceberg-cpp will define public interfaces and types for cpp users.
>
> After this separation, iceberg-rust can focus on implementing features and
> exposing them through simpler, more direct bindings. Meanwhile, iceberg-cpp
> will be responsible for designing, testing, and maintaining the user-level
> APIs, allowing users to select the parts they need or plug in their own
> file I/O components.
>
> ## Conclusion
>
> We believe this design combines the best parts of two projects and helps
> us iterate iceberg-cpp more quickly.
>
> Welcome to leave your comments here!
>
> [^1]: https://cxx.rs/
>
> Xuanwo
>
> https://xuanwo.io/
>
> Xuanwo
>
> https://xuanwo.io/
>
>
--
Regards
Junwang Zhao
Re: [DISCUSS] Move iceberg-cpp step forward
Posted by Xuanwo <xu...@apache.org>.
After a deeper discussion with friends in the Slack #cpp channel, we reached the following consensus:
• iceberg-rust won't maintain any c/cpp related code.
• iceberg-cpp will have an internal crate based on iceberg-rust and invokes cxx/cbingen to build bindings
• And finally, we will have a `libiceberg-api` and `libiceberg-cpp` ,`libicberg-rust` in the iceberg-cpp repo.
In which, iceberg-cpp will provide:
- `libiceberg-api` : An headers that defines APIs available to users.
- `libiceberg-cpp` : An impl of libiceberg-api based on pure cpp (based on nanoarrow, json, libz and so on)
- `libicberg-rust` : An impl of libiceberg-api based on iceberg-rust.
What do you think?
On Tue, Mar 18, 2025, at 16:56, Fokko Driesprong wrote:
> Hi Xuanwo,
>
> Thanks for bringing this up. I'm not a cpp expert, but following this with great interest. I think this is a very promising approach since the memory model is already the same across projects (Arrow).
>
> Kind regards,
> Fokko
>
> Op vr 14 mrt 2025 om 08:53 schreef Xuanwo <xu...@apache.org>:
>> __
>> Hi, everyone
>>
>> Today, Renjie Liu and I had a discussion with some of the iceberg-cpp contributors—Gang Wu, Junwang Zhao, and David Li—about ways to advance the project. We came up with some great ideas, so I'm sharing them here to gather more feedback.
>>
>> ## Background
>>
>> The community continues to show interest in a C++ implementation of Iceberg. Some users are interested in C++ bindings with a well-designed API for data types and manifests, but they have their own runtime and I/O, so they prefer a binding that exclude these components. However, there are also users who require an all-in-one solution.
>>
>> The iceberg-cpp project aims to provide a well-designed interface for users to plug in file I/O, readers, writers, and other components while offering a default implementation in other modules.
>>
>> Luckily, we now have iceberg-rust, which is not fully feature-complete but undergoing rapid development. We can develop a C++ binding based on iceberg-rust and establish a robust C++ API within the iceberg-cpp project. The C++ project will define public interfaces and offer various build targets for users to choose from. The overall process will be similar to how pyiceberg interacts with iceberg-rust.
>>
>> ## Plan
>>
>>
>> 1. Use cxx[^1] to build cpp bindings for iceberg-rust which exposes core APIs.
>>
>>
>> The binding will support different build profiles that users can choose to use fileio from iceberg-rust or not.
>>
>> 2. iceberg-cpp will define public interfaces and types for cpp users.
>>
>> After this separation, iceberg-rust can focus on implementing features and exposing them through simpler, more direct bindings. Meanwhile, iceberg-cpp will be responsible for designing, testing, and maintaining the user-level APIs, allowing users to select the parts they need or plug in their own file I/O components.
>>
>> ## Conclusion
>>
>> We believe this design combines the best parts of two projects and helps us iterate iceberg-cpp more quickly.
>>
>> Welcome to leave your comments here!
>>
>> [^1]: https://cxx.rs/
>>
>> Xuanwo
>>
>> https://xuanwo.io/
Xuanwo
https://xuanwo.io/
Re: [DISCUSS] Move iceberg-cpp step forward
Posted by Fokko Driesprong <fo...@apache.org>.
Hi Xuanwo,
Thanks for bringing this up. I'm not a cpp expert, but following this with
great interest. I think this is a very promising approach since the memory
model is already the same across projects (Arrow).
Kind regards,
Fokko
Op vr 14 mrt 2025 om 08:53 schreef Xuanwo <xu...@apache.org>:
> Hi, everyone
>
> Today, Renjie Liu and I had a discussion with some of the iceberg-cpp
> contributors—Gang Wu, Junwang Zhao, and David Li—about ways to advance the
> project. We came up with some great ideas, so I'm sharing them here to
> gather more feedback.
>
> ## Background
>
> The community continues to show interest in a C++ implementation of
> Iceberg. Some users are interested in C++ bindings with a well-designed API
> for data types and manifests, but they have their own runtime and I/O, so
> they prefer a binding that exclude these components. However, there are
> also users who require an all-in-one solution.
>
> The iceberg-cpp project aims to provide a well-designed interface for
> users to plug in file I/O, readers, writers, and other components while
> offering a default implementation in other modules.
>
> Luckily, we now have iceberg-rust, which is not fully feature-complete but
> undergoing rapid development. We can develop a C++ binding based on
> iceberg-rust and establish a robust C++ API within the iceberg-cpp project.
> The C++ project will define public interfaces and offer various build
> targets for users to choose from. The overall process will be similar to
> how pyiceberg interacts with iceberg-rust.
>
> ## Plan
>
>
> 1. Use cxx[^1] to build cpp bindings for iceberg-rust which exposes core
> APIs.
>
> The binding will support different build profiles that users can choose to
> use fileio from iceberg-rust or not.
>
> 2. iceberg-cpp will define public interfaces and types for cpp users.
>
> After this separation, iceberg-rust can focus on implementing features and
> exposing them through simpler, more direct bindings. Meanwhile, iceberg-cpp
> will be responsible for designing, testing, and maintaining the user-level
> APIs, allowing users to select the parts they need or plug in their own
> file I/O components.
>
> ## Conclusion
>
> We believe this design combines the best parts of two projects and helps
> us iterate iceberg-cpp more quickly.
>
> Welcome to leave your comments here!
>
> [^1]: https://cxx.rs/
>
> Xuanwo
>
> https://xuanwo.io/
>