You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "rahulsmahadev (via GitHub)" <gi...@apache.org> on 2026/06/10 00:04:14 UTC

[PR] feat(parquet): support reading list columns as Arrow large_list [iceberg-cpp]

rahulsmahadev opened a new pull request, #714:
URL: https://github.com/apache/iceberg-cpp/pull/714

   ## Summary
   
   Implements the remaining `LargeListArray` gaps from #502, following the design suggested by @wgtmac in #513:
   
   1. `ValidateParquetSchemaEvolution` now accepts `LARGE_LIST` wherever `LIST` is accepted, so schema projection works when the Arrow reader presents 64-bit offset list types.
   2. A new reader property `read.arrow.use-large-list` (default: `false`) configures the Parquet reader via `ArrowReaderProperties::set_list_type(::arrow::Type::LARGE_LIST)` to decode list columns as `large_list`.
   
   Since `ToArrowSchema` builds the reader's output schema with 32-bit lists (and `ProjectRecordBatch` dispatches on the output schema type), enabling the property also rewrites list fields in the output Arrow schema to `large_list` so the projection layer takes the `ProjectLargeListArray` path added in #502. The rewrite is local to the Parquet reader to avoid changing the `ToArrowSchema` signature used across writers and manifest readers.
   
   Closes #513
   
   ## Changes
   
   - `src/iceberg/parquet/parquet_schema_util.cc`: accept `LARGE_LIST` for `TypeId::kList` in schema evolution validation.
   - `src/iceberg/file_reader.h`: add `ReaderProperties::kArrowUseLargeList` (`read.arrow.use-large-list`, default `false`), following the `kBatchSize` pattern.
   - `src/iceberg/parquet/parquet_reader.cc`: set `ArrowReaderProperties::set_list_type` when the property is enabled, and align the output Arrow schema (lists nested in structs/maps included) with the large_list arrays produced by the reader.
   
   ## Test plan
   
   - `ParquetSchemaProjectionTest.ValidateSchemaEvolutionAllowsLargeList`: `large_list` Arrow type validates against an Iceberg `ListType`.
   - `ParquetSchemaProjectionTest.ProjectLargeListType`: projection over a `SchemaManifest` built with `set_list_type(LARGE_LIST)` (the same path `BuildProjection` uses in the reader).
   - `ParquetReaderTest.ReadListType`: default behavior unchanged — list columns read as 32-bit offset `list`.
   - `ParquetReaderTest.ReadListAsLargeList`: with `read.arrow.use-large-list=true`, the output schema exposes `large_list` and values round-trip correctly (verified via array slices since JSON parsing creates regular `ListArray`).
   
   Note: my local environment lacks a C++23 toolchain (cmake 3.16/gcc 10), so I could not build locally; relying on CI to verify. All Arrow APIs used (`set_list_type`, `large_list(field)`, `MapType(key_field, item_field, keys_sorted)`, `Field::WithType`) were checked against the pinned Arrow 24.0.0 headers.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] feat(parquet): support reading list columns as Arrow large_list [iceberg-cpp]

Posted by "rahulsmahadev (via GitHub)" <gi...@apache.org>.
rahulsmahadev commented on PR #714:
URL: https://github.com/apache/iceberg-cpp/pull/714#issuecomment-4665840212

   @wgtmac this implements the design you suggested in #513 — would appreciate a review when you get a chance.
   
   One note beyond the two items in the issue: the reader's output Arrow schema (built by `ToArrowSchema`, which uses 32-bit lists) also needed to be aligned so `ProjectRecordBatch` dispatches to the large_list path from #502. I kept that rewrite local to the Parquet reader to avoid changing the `ToArrowSchema` signature, but happy to take a different approach if you'd prefer.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] feat(parquet): support reading list columns as Arrow large_list [iceberg-cpp]

Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on PR #714:
URL: https://github.com/apache/iceberg-cpp/pull/714#issuecomment-4703598652

   Thanks for improving this, @rahulsmahadev! I think the long-term shape is to make this output-schema driven, ideally allowing callers to choose `list` vs `large_list` per projected column. Arrow’s Parquet reader only exposes a global `set_list_type` today, so keeping a global flag for this PR seems reasonable.
   
   One API concern: since this option lives in generic `ReaderProperties`, it reads as format-independent. Either other readers such as Avro should honor it too, or the option should be clearly scoped/documented as Parquet-only.
   
   One thing that is worth noting: when `read.arrow.use-large-list` is enabled, the reader always rewrites the output schema to `large_list`, but Arrow ignores `set_list_type(LARGE_LIST)` when the Parquet file contains serialized `ARROW:schema` metadata. In that case the actual batch can still contain `ListArray`, while `ProjectRecordBatch` dispatches as `LargeListArray`, causing a bad cast/crash. Please base the output schema on the actual reader schema or add a guard, and cover this with an `ARROW:schema` regression test.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] feat(parquet): support reading list columns as Arrow large_list [iceberg-cpp]

Posted by "rahulsmahadev (via GitHub)" <gi...@apache.org>.
rahulsmahadev commented on PR #714:
URL: https://github.com/apache/iceberg-cpp/pull/714#issuecomment-5320724290

   @wgtmac have addressed comments, let me know if this PR is good 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] feat(parquet): support reading list columns as Arrow large_list [iceberg-cpp]

Posted by "kamcheungting-db (via GitHub)" <gi...@apache.org>.
kamcheungting-db commented on code in PR #714:
URL: https://github.com/apache/iceberg-cpp/pull/714#discussion_r3632804173


##########
src/iceberg/parquet/parquet_reader.cc:
##########
@@ -84,6 +85,41 @@ class EmptyRecordBatchReader : public ::arrow::RecordBatchReader {
   }
 };
 
+std::shared_ptr<::arrow::Field> UseLargeListField(

Review Comment:
   add comment: 
   ```suggestion
   // forward declaration to unblock cycle dependence.
   std::shared_ptr<::arrow::Field> UseLargeListField(
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] feat(parquet): support reading list columns as Arrow large_list [iceberg-cpp]

Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on code in PR #714:
URL: https://github.com/apache/iceberg-cpp/pull/714#discussion_r3831443300


##########
src/iceberg/parquet/parquet_reader.cc:
##########
@@ -250,6 +322,24 @@ class ParquetReader::Impl {
           reader_->GetRecordBatchReader(row_group_indices, column_indices));
     }
 
+    // Build the output Arrow schema from the projected Iceberg schema. This schema is the
+    // target of ProjectRecordBatch, so it must describe the projected schema rather than
+    // the schema of the file.
+    ArrowSchema arrow_schema;
+    ICEBERG_RETURN_UNEXPECTED(ToArrowSchema(*read_schema_, &arrow_schema));
+    ICEBERG_ARROW_ASSIGN_OR_RETURN(context_->output_arrow_schema_,
+                                   ::arrow::ImportSchema(&arrow_schema));
+
+    if (use_large_list_ && ProducesLargeList(*context_->record_batch_reader_)) {
+      // Align the output schema with the large_list arrays produced by the Parquet
+      // reader. Note that Arrow ignores the requested list type when the file carries
+      // serialized ARROW:schema metadata, in which case the reader keeps producing plain
+      // list arrays and the output schema must keep describing them as such.
+      context_->output_arrow_schema_ =
+          ::arrow::schema(UseLargeListFields(context_->output_arrow_schema_->fields()),

Review Comment:
   This rewrite is global: one `large_list` makes every projected list a `large_list`. Stored Arrow schemas can contain both types, so please map each projected field recursively from the corresponding reader field instead of using `ProducesLargeList()` as a single flag. For missing or default fields, use the configured output preference.



##########
src/iceberg/parquet/parquet_reader.cc:
##########
@@ -250,6 +322,24 @@ class ParquetReader::Impl {
           reader_->GetRecordBatchReader(row_group_indices, column_indices));
     }
 
+    // Build the output Arrow schema from the projected Iceberg schema. This schema is the
+    // target of ProjectRecordBatch, so it must describe the projected schema rather than
+    // the schema of the file.
+    ArrowSchema arrow_schema;
+    ICEBERG_RETURN_UNEXPECTED(ToArrowSchema(*read_schema_, &arrow_schema));
+    ICEBERG_ARROW_ASSIGN_OR_RETURN(context_->output_arrow_schema_,
+                                   ::arrow::ImportSchema(&arrow_schema));
+
+    if (use_large_list_ && ProducesLargeList(*context_->record_batch_reader_)) {

Review Comment:
   Please align the output schema with the reader schema even when this option is false. Arrow restores a stored `large_list` regardless of `set_list_type`, so this path currently leaves the output as `list` and casts a `LargeListArray` as `ListArray`.



##########
src/iceberg/test/parquet_test.cc:
##########
@@ -267,6 +293,38 @@ class ParquetReaderTest : public TempFileTestBase {
                                         .data_sequence_number = data_sequence_number});
   }
 
+  // Writes a list parquet file through parquet::arrow::WriteTable, which serializes the
+  // Arrow schema of the table into the ARROW:schema key value metadata of the file.
+  void CreateListParquetFileWithArrowSchema() {
+    const std::string kParquetFieldIdKey = "PARQUET:field_id";
+    auto arrow_schema = ::arrow::schema(
+        {::arrow::field("id", ::arrow::int32(), /*nullable=*/false,
+                        ::arrow::KeyValueMetadata::Make({kParquetFieldIdKey}, {"1"})),
+         ::arrow::field(
+             "numbers",
+             ::arrow::list(::arrow::field(
+                 "element", ::arrow::int32(), /*nullable=*/true,
+                 ::arrow::KeyValueMetadata::Make({kParquetFieldIdKey}, {"101"}))),
+             /*nullable=*/true,
+             ::arrow::KeyValueMetadata::Make({kParquetFieldIdKey}, {"2"}))});
+    auto batch =
+        ::arrow::RecordBatch::FromStructArray(
+            ::arrow::json::ArrayFromJSONString(::arrow::struct_(arrow_schema->fields()),
+                                               R"([[1, [1, 2]], [2, [3]]])")
+                .ValueOrDie())
+            .ValueOrDie();
+    auto table = ::arrow::Table::FromRecordBatches(arrow_schema, {batch}).ValueOrDie();
+
+    auto io = internal::checked_cast<arrow::ArrowFileSystemFileIO&>(*file_io_);
+    auto outfile = io.fs()->OpenOutputStream(temp_parquet_file_).ValueOrDie();
+
+    // write a single row group so that one batch holds every row
+    ASSERT_TRUE(::parquet::arrow::WriteTable(*table, ::arrow::default_memory_pool(),

Review Comment:
   `WriteTable` does not store `ARROW:schema` by default. Please pass `ArrowWriterProperties::Builder().store_schema()->build()`, assert that the footer key exists, and cover a stored `large_list` with the default reader option plus a mixed `list`/`large_list` schema.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] feat(parquet): support reading list columns as Arrow large_list [iceberg-cpp]

Posted by "kamcheungting-db (via GitHub)" <gi...@apache.org>.
kamcheungting-db commented on code in PR #714:
URL: https://github.com/apache/iceberg-cpp/pull/714#discussion_r3632869339


##########
src/iceberg/parquet/parquet_reader.cc:
##########
@@ -214,6 +254,17 @@ class ParquetReader::Impl {
     ICEBERG_RETURN_UNEXPECTED(ToArrowSchema(*read_schema_, &arrow_schema));
     ICEBERG_ARROW_ASSIGN_OR_RETURN(context_->output_arrow_schema_,
                                    ::arrow::ImportSchema(&arrow_schema));
+    if (use_large_list_) {
+      // Align the output schema with the large_list arrays produced by the
+      // Parquet reader when kArrowUseLargeList is enabled.
+      ::arrow::FieldVector fields;
+      fields.reserve(context_->output_arrow_schema_->fields().size());
+      for (const auto& field : context_->output_arrow_schema_->fields()) {
+        fields.push_back(UseLargeListField(field));
+      }

Review Comment:
   this is duplicated logic.
   let's add a method.
   ```
   ::arrow::FieldVector UseLargeListFields(const ::arrow::FieldVector& fields) {
     ::arrow::FieldVector rewritten;
     rewritten.reserve(fields.size());
     for (const auto& field : fields) {
       rewritten.push_back(UseLargeListField(field));
     }
     return rewritten;
   }
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] feat(parquet): support reading list columns as Arrow large_list [iceberg-cpp]

Posted by "rahulsmahadev (via GitHub)" <gi...@apache.org>.
rahulsmahadev commented on PR #714:
URL: https://github.com/apache/iceberg-cpp/pull/714#issuecomment-5188240949

   @wgtmac can you help review


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] feat(parquet): support reading list columns as Arrow large_list [iceberg-cpp]

Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac merged PR #714:
URL: https://github.com/apache/iceberg-cpp/pull/714


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org