You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "PingLiuPing (via GitHub)" <gi...@apache.org> on 2026/10/16 13:55:53 UTC
[I] Scan planning needs iceberg_bundle (and Arrow C++) only for the Avro reader [iceberg-cpp]
PingLiuPing opened a new issue, #1027:
URL: https://github.com/apache/iceberg-cpp/issues/1027
We are building an Iceberg connector for [Axiom](https://github.com/facebookincubator/axiom), which runs on Velox. We plan to use iceberg-cpp for the REST catalog and for scan planning, and Velox reads the data files.
To plan a scan, `ManifestReader` reads the manifest list and the manifests, which are Avro files. It gets the reader from ReaderFactoryRegistry for FileFormatType::kAvro, and the only Avro reader is the one `iceberg::avro::RegisterAll()` registers from iceberg_bundle. The bundle also links Arrow C++ and Parquet. The other libraries we use, iceberg, iceberg_rest and iceberg_data, have no Arrow C++ in them, they contain no arrow:: symbols. Without the bundle, though, every scan fails with `Missing reader factory for file format: avro.` So we have to link Arrow C++ just to read manifests.
For us that means two Arrow versions in one process. Velox pins Arrow 18.0.0 and iceberg-cpp builds Arrow 25.0.0. They define the same symbols with different class layouts, and code compiled against one version ends up calling the other. We hit this with `arrow::DictionaryArray`, which is 72 bytes in Arrow 18 and 80 bytes in Arrow 25 (Arrow 25 added a std::once_flag member). Velox's Parquet writer is compiled against Arrow 18 and allocates 72 bytes for one, but the constructor that runs is Arrow 25's, and it writes the new member 8 bytes past the end of the object. The process then crashes later, in some unrelated malloc.
Right now we work around this by building iceberg-cpp and its Arrow into a separate shared library, with per-platform linker flags that hide every Arrow symbol and a build-time check that none leak out. It works, but it's fragile, and the Avro reader is the only reason we need it.
The Avro code doesn’t need Arrow C++ at its interface, and the public Avro headers don’t include it. The reader returns `ArrowArray` and `ArrowSchema` C structs, the writer takes an ArrowArray, and both open files through FileIO. The core already builds and parses Arrow data with nanoarrow, for example in ManifestEntryAdapter and ManifestReader.
Inside src/iceberg/avro/, Arrow C++ is only an implementation detail, and each use of it can be replaced with something Arrow-free. The reader builds its output with Arrow C++ builders and the writer reads its input as Arrow C++ arrays, decimals and default values included; nanoarrow, which the core already uses, can do both. Files are read and written through Arrow IO streams, which FileIO’s own streams can replace. A couple of small helpers come from the bundle’s iceberg/arrow/ directory and could move to the core. The field-id projection (Project and MakeAvroNodeWithFieldIds) doesn’t use Arrow C++ at all, and the only Arrow call in avro_schema_util.cc is arrow::DecimalType::DecimalSize, a byte-width calculation that is easy to do without Arrow.
Would you be open to rewriting the Avro reader and writer on nanoarrow and moving them into their own library, say `iceberg_avro`, that depends only on the core and avro-cpp? iceberg_bundle would link it, so nothing changes for people who use the bundle. Engines like us could then link iceberg, iceberg_rest, iceberg_data and iceberg_avro, with only one Arrow in the process.
Thank you.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Scan planning needs iceberg_bundle (and Arrow C++) only for the Avro reader [iceberg-cpp]
Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on issue #1027:
URL: https://github.com/apache/iceberg-cpp/issues/1027#issuecomment-6313300406
Is it possible to add a shim layer to iceberg-bundle by detecting Arrow version under the hood to hide the difference among supported versions? @PingLiuPing
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Scan planning needs iceberg_bundle (and Arrow C++) only for the Avro reader [iceberg-cpp]
Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on issue #1027:
URL: https://github.com/apache/iceberg-cpp/issues/1027#issuecomment-6258370163
In that case, I would prefer splitting `iceberg-bundle` into `iceberg-arrow` and `iceberg-avro` explicitly. My concern is that it will just add a lot of complexity to Avro adapter implementation. May I ask why Velox pins Arrow 18.0.0? Is it possible and a smaller effort to bump the Arrow version on the Velox side?
cc @manuzhang @zhjwpku @HuaHuaY
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Scan planning needs iceberg_bundle (and Arrow C++) only for the Avro reader [iceberg-cpp]
Posted by "PingLiuPing (via GitHub)" <gi...@apache.org>.
PingLiuPing commented on issue #1027:
URL: https://github.com/apache/iceberg-cpp/issues/1027#issuecomment-6336536309
@wgtmac Thank you, let me explore it first and get back to you.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Scan planning needs iceberg_bundle (and Arrow C++) only for the Avro reader [iceberg-cpp]
Posted by "PingLiuPing (via GitHub)" <gi...@apache.org>.
PingLiuPing commented on issue #1027:
URL: https://github.com/apache/iceberg-cpp/issues/1027#issuecomment-6312347727
@wgtmac Thank you for looking at this issue.
> May I ask why Velox pins Arrow 18.0.0? Is it possible and a smaller effort to bump the Arrow version on the Velox side?
Dig little deeper on this.
Velox has tried to move past Arrow 18 a few times since 2025. The upgrade from 15 to 18 landed in February 2026 ([#16366](https://github.com/facebookincubator/velox/pull/16366)). The move to Arrow 21 ([#14102](https://github.com/facebookincubator/velox/pull/14102)) was merged and then reverted, and a later attempt to revive it also ran into build problems. Moving to 21 or later is more than a version bump, there are four reasons:
1. **Velox's Parquet code uses Arrow's private helpers.** The Velox Parquet reader and writer call internal Arrow functions, such as the bit-unpacking code in the reader. Newer Arrow releases renamed or removed these helpers, so the reader and writer need code changes.
2. **Meta's internal build.** Meta compiles Velox against its own internal copy of Arrow, which was on 18 as of February 2026. Velox has to keep compiling against that version.
3. **Downstream projects.** Gluten builds Velox against Gluten's own Arrow. Gluten is on 18 and has asked Velox to keep supporting older Arrow for a few releases. Presto builds Arrow through Velox's setup scripts for its Arrow Flight connector, so its Flight code has to move to the new version too.
4. **Docker images and CI.** The Velox dev images install Arrow through the same setup scripts, so they need to be rebuilt and retested. The Arrow 21 attempt was reverted because of build breaks of this kind; those specific breaks have since been fixed though.
So just changing the pinned version won't work and it requires work across communities.
We are pushing this forward with the Velox community. Even so, it will take some time before Velox moves to Arrow 25.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org