You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "zhjwpku (via GitHub)" <gi...@apache.org> on 2024/12/18 14:09:45 UTC
[I] [Discussion] Add Avro library [iceberg-cpp]
zhjwpku opened a new issue, #17:
URL: https://github.com/apache/iceberg-cpp/issues/17
Iceberg's [manifests](https://iceberg.apache.org/spec/#manifests) files are stored using Apache Avro, I just give a quick look at the avro's code base, there is a C++ implementation and a C implementation, as the document stated,
- C implementation has been tested on MacOSX and Linux, not mentioning Windows
- C++ implementation seems support all three OS platforms we are targeting.
C implementation has a dependence lib jansson while C++ implementation need Boost
I'd like to know your opinions about what's the best way to integrate Avro into iceberg-cpp.
[0] https://avro.apache.org/docs/1.12.0/api/c/
[1] https://avro.apache.org/docs/1.12.0/api/cpp/html/#Installing
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] [Discussion] Add Avro library [iceberg-cpp]
Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on issue #17:
URL: https://github.com/apache/iceberg-cpp/issues/17#issuecomment-2551697006
I have created https://issues.apache.org/jira/browse/AVRO-4095 to work on removing the boost dependency from non-testing code path (at least binary-encoding). I'm inclined to use the C++ library with boost dependency removed. I happened to see that DuckDB has leveraged the Avro C library for its Avro extension: https://duckdb.org/2024/12/09/duckdb-avro-extension.html. So I don't think we need to reinvent a Avro C++ library at the moment.
> I was thinking about picking some files we need from Avro and implement our own.
I don't think it is a good approach. We can define a good interface for Avro reader/writer in the iceberg-core library and provide a default implementation via iceberg-avro library with help of the open source Avro C++ library.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] [Discussion] Add Avro library [iceberg-cpp]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on issue #17:
URL: https://github.com/apache/iceberg-cpp/issues/17#issuecomment-2551451270
I'm surprised that Avro C++ depends so much on Boost.
> Apart from the header-only libraries of Boost, Avro C++ requires filesystem, iostreams, system and program_options libraries.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] [Discussion] Add Avro library [iceberg-cpp]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on issue #17:
URL: https://github.com/apache/iceberg-cpp/issues/17#issuecomment-2551568611
> For PyIceberg we made a bold decision to re-implement the Avro reader/writer which was pretty fun to implement.
In Python or in C++? :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] [Discussion] Add Avro library [iceberg-cpp]
Posted by "zhjwpku (via GitHub)" <gi...@apache.org>.
zhjwpku commented on issue #17:
URL: https://github.com/apache/iceberg-cpp/issues/17#issuecomment-2551611911
I was thinking about picking some files we need from Avro and implement our own. Since PyIceberg goes this way, I will try to dig more in this direction, thanks for the inputs :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] [Discussion] Add Avro library [iceberg-cpp]
Posted by "zhjwpku (via GitHub)" <gi...@apache.org>.
zhjwpku commented on issue #17:
URL: https://github.com/apache/iceberg-cpp/issues/17#issuecomment-2597412056
> After [apache/avro#3293](https://github.com/apache/avro/pull/3293) is merged, I think we are ready to use it. [@zhjwpku](https://github.com/zhjwpku)
Great, will keep an eye on it, do you want me to import avro to iceberg-cpp or you do it yourself?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] [Discussion] Add Avro library [iceberg-cpp]
Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on issue #17:
URL: https://github.com/apache/iceberg-cpp/issues/17#issuecomment-2564339103
Yes, I also share the same feeling. Boost dependencies listed below are tightly used for gzip/zlib codec. I think it needs non-trivial work to replace them with the standard zlib library.
```
boost/crc.hpp
boost/iostreams/device/file.hpp
boost/iostreams/filter/gzip.hpp
boost/iostreams/filter/zlib.hpp
boost/iostreams/filtering_stream.hpp
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] [Discussion] Add Avro library [iceberg-cpp]
Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko commented on issue #17:
URL: https://github.com/apache/iceberg-cpp/issues/17#issuecomment-2551567572
cc @wgtmac I know that he was also working on removing boost from the C++ library.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] [Discussion] Add Avro library [iceberg-cpp]
Posted by "zhjwpku (via GitHub)" <gi...@apache.org>.
zhjwpku commented on issue #17:
URL: https://github.com/apache/iceberg-cpp/issues/17#issuecomment-2564282758
> I have created https://issues.apache.org/jira/browse/AVRO-4095 to work on removing the boost dependency from non-testing code path (at least binary-encoding). I'm inclined to use the C++ library with boost dependency removed.
I studied the avro c++ code base a little bit, it seems to me that the boost/iostreams dependency in DataFile.cc is the most difficult part to get rid of, do you have any solution in mind?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] [Discussion] Add Avro library [iceberg-cpp]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on issue #17:
URL: https://github.com/apache/iceberg-cpp/issues/17#issuecomment-2551581682
We only care about the binary encoding of Avro, right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] [Discussion] Add Avro library [iceberg-cpp]
Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on issue #17:
URL: https://github.com/apache/iceberg-cpp/issues/17#issuecomment-2597400734
After https://github.com/apache/avro/pull/3293 is merged, I think we are ready to use it. @zhjwpku
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] [Discussion] Add Avro library [iceberg-cpp]
Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko commented on issue #17:
URL: https://github.com/apache/iceberg-cpp/issues/17#issuecomment-2551585408
Yes, that's correct 👍
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] [Discussion] Add Avro library [iceberg-cpp]
Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on issue #17:
URL: https://github.com/apache/iceberg-cpp/issues/17#issuecomment-2590059561
Update: I have removed most of the Boost dependency from Apache Avro C++ library: https://issues.apache.org/jira/browse/AVRO-4095. Now the one last thing is to replace the Boost zlib/gzip implementation with the one from https://zlib.net/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] [Discussion] Add Avro library [iceberg-cpp]
Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko closed issue #17: [Discussion] Add Avro library
URL: https://github.com/apache/iceberg-cpp/issues/17
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] [Discussion] Add Avro library [iceberg-cpp]
Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko closed issue #17: [Discussion] Add Avro library
URL: https://github.com/apache/iceberg-cpp/issues/17
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] [Discussion] Add Avro library [iceberg-cpp]
Posted by "zhjwpku (via GitHub)" <gi...@apache.org>.
zhjwpku commented on issue #17:
URL: https://github.com/apache/iceberg-cpp/issues/17#issuecomment-2552611597
> I have created https://issues.apache.org/jira/browse/AVRO-4095 to work on removing the boost dependency from non-testing code path (at least binary-encoding). I'm inclined to use the C++ library with boost dependency removed.
Yeah, removing the boost dependency is better, will keep an eye on it, thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] [Discussion] Add Avro library [iceberg-cpp]
Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on issue #17:
URL: https://github.com/apache/iceberg-cpp/issues/17#issuecomment-2597419682
I believe I have a lot of TODOs on my end, especially the internal project from my employer :)
It would be great if you have time to do it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] [Discussion] Add Avro library [iceberg-cpp]
Posted by "zhjwpku (via GitHub)" <gi...@apache.org>.
zhjwpku commented on issue #17:
URL: https://github.com/apache/iceberg-cpp/issues/17#issuecomment-2597469266
> I believe I have a lot of TODOs on my end, especially the internal project from my employer :)
>
> It would be great if you have time to do it.
Ok, I will prepare a PR later ;)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] [Discussion] Add Avro library [iceberg-cpp]
Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko commented on issue #17:
URL: https://github.com/apache/iceberg-cpp/issues/17#issuecomment-2551566258
For PyIceberg we made a bold decision to re-implement the Avro reader/writer which was pretty fun to implement. We also made the reader rely on an Iceberg schema, rather than an Avro schema. This gave some nice advantages, such as native field-ID-based projections and initial defaults that makes it easier to project Iceberg V1 metadata into V2.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] [Discussion] Add Avro library [iceberg-cpp]
Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko commented on issue #17:
URL: https://github.com/apache/iceberg-cpp/issues/17#issuecomment-2551573554
[In Cython](https://github.com/apache/iceberg-python/blob/main/pyiceberg/avro/decoder_fast.pyx) :) The performance of the zig-zag (de/en)coding of ints was pretty terrible in Python.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org