You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "callmepandey (via GitHub)" <gi...@apache.org> on 2026/01/09 20:19:40 UTC

[I] Add support for Arrow LargeListArray in Parquet data projection [iceberg-cpp]

callmepandey opened a new issue, #502:
URL: https://github.com/apache/iceberg-cpp/issues/502

   ## Summary
   
   The `ProjectRecordBatch` function in `parquet_data_util.cc` only supports `::arrow::ListArray` (32-bit offsets) but not `::arrow::LargeListArray` (64-bit offsets). This limitation is marked with a FIXME comment at line 151.
   
   ## Problem
   
   Arrow's `LargeListArray` uses 64-bit offsets instead of 32-bit, allowing it to handle lists with more than 2^31-1 total child elements. Currently, attempting to project a `LargeListArray` would fail with an error like:
   ```
   Expected list type, got: large_list<...>
   ```
   
   ## Proposed Solution
   
   1. **Add templated `ProjectListArrayImpl<>` function** - Generic implementation that works with both `ListArray` and `LargeListArray`
   
   2. **Add `ProjectLargeListArray` wrapper** - Calls the template with `LargeListArray` and `LargeListType`
   
   3. **Update `ProjectNestedArray`** - Handle both `::arrow::Type::LIST` and `::arrow::Type::LARGE_LIST` in the `TypeId::kList` case
   
   4. **Add test case** - Verify `LargeListArray` projection works correctly
   
   ## Files to Change
   
   - `src/iceberg/parquet/parquet_data_util.cc`
   - `src/iceberg/test/parquet_data_test.cc`
   
   ## References
   
   - FIXME comment: `src/iceberg/parquet/parquet_data_util.cc:151`
   - Arrow LargeListArray docs: https://arrow.apache.org/docs/cpp/api/array.html


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

Re: [I] Add support for Arrow LargeListArray in Parquet data projection [iceberg-cpp]

Posted by "callmepandey (via GitHub)" <gi...@apache.org>.
callmepandey commented on issue #502:
URL: https://github.com/apache/iceberg-cpp/issues/502#issuecomment-3730460096

   Hi @wgtmac, could you please take a look at this proposal? I'd like to work on this improvement if it sounds reasonable to you.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

Re: [I] Add support for Arrow LargeListArray in Parquet data projection [iceberg-cpp]

Posted by "callmepandey (via GitHub)" <gi...@apache.org>.
callmepandey commented on issue #502:
URL: https://github.com/apache/iceberg-cpp/issues/502#issuecomment-3750626094

   @wgtmac PR #510 is ready for review. All CI checks are passing ✅


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

Re: [I] Add support for Arrow LargeListArray in Parquet data projection [iceberg-cpp]

Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac closed issue #502: Add support for Arrow LargeListArray in Parquet data projection
URL: https://github.com/apache/iceberg-cpp/issues/502


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

Re: [I] Add support for Arrow LargeListArray in Parquet data projection [iceberg-cpp]

Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on issue #502:
URL: https://github.com/apache/iceberg-cpp/issues/502#issuecomment-3736917759

   Thanks for your interesting on this! I think it makes sense.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org