You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@paimon.apache.org by 刘欣瑀 <15...@163.com> on 2025/09/01 03:14:56 UTC
[DISCUSS] Proposal to Contribute C++ Version of Paimon
Dear Paimon Development Team,
We are a team of C++ developers who have recently developed a C++ version of Paimon. I am reaching out to propose contributing this C++ version to the Paimon community. Considering its unique architecture and functionalities, we suggest that it may be advantageous to host it as a new sub-repo of Paimon, potentially under the name paimon-cpp. Paimon-cpp aims to provide a native, high-performance and extensible implementation that allows native engines to access the Paimon datalake format. Some key features of the C++ version include:
1. Basic operations—including write, commit, scan, and read—implemented for both append-only and primary key tables.
2. Interfaces for batch reading and writing utilizing the Arrow Columnar In-Memory Format to enhance throughput.
3. A plugin mechanism enabling users to flexibly define file formats, file systems, memory pools, thread pools, etc.
4. Default built-in plugins for ORC and Parquet file formats; local and OSS file systems; integrated memory pool and thread pool plugins.
Furthermore, we assure compatibility with Paimon communication protocols and meta formats, providing users with the flexibility to seamlessly switch between multiple languages. We hope that the C++ version will contribute to the enrichment of the Paimon community, enhancing its diversity and expanding its capabilities.
I am excited about the possibility of collaborating with the community to refine and expand this version further. Please let me know if the development team is interested in this contribution or any specific procedures I need to follow for submission.
Thank you for considering this proposal. I am looking forward to your rely.
Best regards,
Xinyu Liu
Re: Re: [DISCUSS] Proposal to Contribute C++ Version of Paimon
Posted by Jingsong Li <ji...@gmail.com>.
Thanks Xinyu,
Sounds good initially the C++ SDK within Alibaba repository. I think
it can also be marked on the ecological webpage [1].
[1] https://paimon.apache.org/docs/master/ecosystem/overview/
Best,
Jingsong
On Tue, Sep 16, 2025 at 9:19 AM 刘欣瑀 <15...@163.com> wrote:
>
> Thank you, Jinsong, for your insightful response.
>
> Currently, community engagement with C++ development still has room for growth. Following internal discussions, we have decided to initially open-source the C++ SDK within the Alibaba repository. If there is demand from the community in the future, we are willing to donate the library. Paimon-cpp is committed to delivering a high-performance lib for reading and writing Paimon data. We will continually enhance Paimon-cpp, promptly integrating the latest Paimon features and ensuring protocol compatibility. We warmly invite interested individuals to follow our progress at Alibaba and contribute to its development.
>
> Best regards
> Xinyu Liu
>
> At 2025-09-03 19:01:19, "Jingsong Li" <ji...@gmail.com> wrote:
> >Thanks Xinyu for raising this discussion!
> >
> >I believe that maintaining a complete C++ rewrite is not an easy task.
> >All code needs to be reviewed and discussed again before it can enter
> >the Apache codebase.
> >
> >I think before contributing, what is more important is what the
> >community can gain from developing C++, and do you have enough input?
> >
> >Best,
> >Jingsong
> >
> >On Wed, Sep 3, 2025 at 2:41 PM wj wang <ho...@gmail.com> wrote:
> >>
> >> Good job!
> >>
> >> On Mon, Sep 1, 2025 at 11:16 AM 刘欣瑀 <15...@163.com> wrote:
> >> >
> >> > Dear Paimon Development Team,
> >> >
> >> >
> >> > We are a team of C++ developers who have recently developed a C++ version of Paimon. I am reaching out to propose contributing this C++ version to the Paimon community. Considering its unique architecture and functionalities, we suggest that it may be advantageous to host it as a new sub-repo of Paimon, potentially under the name paimon-cpp. Paimon-cpp aims to provide a native, high-performance and extensible implementation that allows native engines to access the Paimon datalake format. Some key features of the C++ version include:
> >> > 1. Basic operations—including write, commit, scan, and read—implemented for both append-only and primary key tables.
> >> > 2. Interfaces for batch reading and writing utilizing the Arrow Columnar In-Memory Format to enhance throughput.
> >> > 3. A plugin mechanism enabling users to flexibly define file formats, file systems, memory pools, thread pools, etc.
> >> > 4. Default built-in plugins for ORC and Parquet file formats; local and OSS file systems; integrated memory pool and thread pool plugins.
> >> >
> >> >
> >> > Furthermore, we assure compatibility with Paimon communication protocols and meta formats, providing users with the flexibility to seamlessly switch between multiple languages. We hope that the C++ version will contribute to the enrichment of the Paimon community, enhancing its diversity and expanding its capabilities.
> >> >
> >> >
> >> > I am excited about the possibility of collaborating with the community to refine and expand this version further. Please let me know if the development team is interested in this contribution or any specific procedures I need to follow for submission.
> >> >
> >> >
> >> > Thank you for considering this proposal. I am looking forward to your rely.
> >> >
> >> >
> >> > Best regards,
> >> > Xinyu Liu
Re:Re: [DISCUSS] Proposal to Contribute C++ Version of Paimon
Posted by 刘欣瑀 <15...@163.com>.
Thank you, Jinsong, for your insightful response.
Currently, community engagement with C++ development still has room for growth. Following internal discussions, we have decided to initially open-source the C++ SDK within the Alibaba repository. If there is demand from the community in the future, we are willing to donate the library. Paimon-cpp is committed to delivering a high-performance lib for reading and writing Paimon data. We will continually enhance Paimon-cpp, promptly integrating the latest Paimon features and ensuring protocol compatibility. We warmly invite interested individuals to follow our progress at Alibaba and contribute to its development.
Best regards
Xinyu Liu
At 2025-09-03 19:01:19, "Jingsong Li" <ji...@gmail.com> wrote:
>Thanks Xinyu for raising this discussion!
>
>I believe that maintaining a complete C++ rewrite is not an easy task.
>All code needs to be reviewed and discussed again before it can enter
>the Apache codebase.
>
>I think before contributing, what is more important is what the
>community can gain from developing C++, and do you have enough input?
>
>Best,
>Jingsong
>
>On Wed, Sep 3, 2025 at 2:41 PM wj wang <ho...@gmail.com> wrote:
>>
>> Good job!
>>
>> On Mon, Sep 1, 2025 at 11:16 AM 刘欣瑀 <15...@163.com> wrote:
>> >
>> > Dear Paimon Development Team,
>> >
>> >
>> > We are a team of C++ developers who have recently developed a C++ version of Paimon. I am reaching out to propose contributing this C++ version to the Paimon community. Considering its unique architecture and functionalities, we suggest that it may be advantageous to host it as a new sub-repo of Paimon, potentially under the name paimon-cpp. Paimon-cpp aims to provide a native, high-performance and extensible implementation that allows native engines to access the Paimon datalake format. Some key features of the C++ version include:
>> > 1. Basic operations—including write, commit, scan, and read—implemented for both append-only and primary key tables.
>> > 2. Interfaces for batch reading and writing utilizing the Arrow Columnar In-Memory Format to enhance throughput.
>> > 3. A plugin mechanism enabling users to flexibly define file formats, file systems, memory pools, thread pools, etc.
>> > 4. Default built-in plugins for ORC and Parquet file formats; local and OSS file systems; integrated memory pool and thread pool plugins.
>> >
>> >
>> > Furthermore, we assure compatibility with Paimon communication protocols and meta formats, providing users with the flexibility to seamlessly switch between multiple languages. We hope that the C++ version will contribute to the enrichment of the Paimon community, enhancing its diversity and expanding its capabilities.
>> >
>> >
>> > I am excited about the possibility of collaborating with the community to refine and expand this version further. Please let me know if the development team is interested in this contribution or any specific procedures I need to follow for submission.
>> >
>> >
>> > Thank you for considering this proposal. I am looking forward to your rely.
>> >
>> >
>> > Best regards,
>> > Xinyu Liu
Re: [DISCUSS] Proposal to Contribute C++ Version of Paimon
Posted by Jingsong Li <ji...@gmail.com>.
Thanks Xinyu for raising this discussion!
I believe that maintaining a complete C++ rewrite is not an easy task.
All code needs to be reviewed and discussed again before it can enter
the Apache codebase.
I think before contributing, what is more important is what the
community can gain from developing C++, and do you have enough input?
Best,
Jingsong
On Wed, Sep 3, 2025 at 2:41 PM wj wang <ho...@gmail.com> wrote:
>
> Good job!
>
> On Mon, Sep 1, 2025 at 11:16 AM 刘欣瑀 <15...@163.com> wrote:
> >
> > Dear Paimon Development Team,
> >
> >
> > We are a team of C++ developers who have recently developed a C++ version of Paimon. I am reaching out to propose contributing this C++ version to the Paimon community. Considering its unique architecture and functionalities, we suggest that it may be advantageous to host it as a new sub-repo of Paimon, potentially under the name paimon-cpp. Paimon-cpp aims to provide a native, high-performance and extensible implementation that allows native engines to access the Paimon datalake format. Some key features of the C++ version include:
> > 1. Basic operations—including write, commit, scan, and read—implemented for both append-only and primary key tables.
> > 2. Interfaces for batch reading and writing utilizing the Arrow Columnar In-Memory Format to enhance throughput.
> > 3. A plugin mechanism enabling users to flexibly define file formats, file systems, memory pools, thread pools, etc.
> > 4. Default built-in plugins for ORC and Parquet file formats; local and OSS file systems; integrated memory pool and thread pool plugins.
> >
> >
> > Furthermore, we assure compatibility with Paimon communication protocols and meta formats, providing users with the flexibility to seamlessly switch between multiple languages. We hope that the C++ version will contribute to the enrichment of the Paimon community, enhancing its diversity and expanding its capabilities.
> >
> >
> > I am excited about the possibility of collaborating with the community to refine and expand this version further. Please let me know if the development team is interested in this contribution or any specific procedures I need to follow for submission.
> >
> >
> > Thank you for considering this proposal. I am looking forward to your rely.
> >
> >
> > Best regards,
> > Xinyu Liu
Re: [DISCUSS] Proposal to Contribute C++ Version of Paimon
Posted by wj wang <ho...@gmail.com>.
Good job!
On Mon, Sep 1, 2025 at 11:16 AM 刘欣瑀 <15...@163.com> wrote:
>
> Dear Paimon Development Team,
>
>
> We are a team of C++ developers who have recently developed a C++ version of Paimon. I am reaching out to propose contributing this C++ version to the Paimon community. Considering its unique architecture and functionalities, we suggest that it may be advantageous to host it as a new sub-repo of Paimon, potentially under the name paimon-cpp. Paimon-cpp aims to provide a native, high-performance and extensible implementation that allows native engines to access the Paimon datalake format. Some key features of the C++ version include:
> 1. Basic operations—including write, commit, scan, and read—implemented for both append-only and primary key tables.
> 2. Interfaces for batch reading and writing utilizing the Arrow Columnar In-Memory Format to enhance throughput.
> 3. A plugin mechanism enabling users to flexibly define file formats, file systems, memory pools, thread pools, etc.
> 4. Default built-in plugins for ORC and Parquet file formats; local and OSS file systems; integrated memory pool and thread pool plugins.
>
>
> Furthermore, we assure compatibility with Paimon communication protocols and meta formats, providing users with the flexibility to seamlessly switch between multiple languages. We hope that the C++ version will contribute to the enrichment of the Paimon community, enhancing its diversity and expanding its capabilities.
>
>
> I am excited about the possibility of collaborating with the community to refine and expand this version further. Please let me know if the development team is interested in this contribution or any specific procedures I need to follow for submission.
>
>
> Thank you for considering this proposal. I am looking forward to your rely.
>
>
> Best regards,
> Xinyu Liu