You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "hedger9487 (via GitHub)" <gi...@apache.org> on 2026/08/24 20:36:28 UTC
[PR] Transforms: Support dictionary-encoded PyArrow arrays in partition transforms (#3633) [iceberg-python]
hedger9487 opened a new pull request, #3841:
URL: https://github.com/apache/iceberg-python/pull/3841
### Description
Fixes #3633.
When PyArrow tables containing dictionary-encoded columns (`pa.DictionaryArray`) are passed to Iceberg partition transforms (such as `BucketTransform`, `TruncateTransform`, or time transforms), `_pyiceberg_transform_wrapper` forwards the `DictionaryArray` directly to `pyiceberg_core.transform`, raising `ValueError: Feature Unsupported => Unsupported data type for bucket transform: Dictionary(Int64, Utf8)`.
This PR updates `_pyiceberg_transform_wrapper` to normalize dictionary-encoded arrays via `arr.dictionary_decode()` before invoking `transform_func`, allowing all partition transforms to transparently handle dictionary-encoded PyArrow arrays and chunked arrays.
### Testing
- Added `test_pyarrow_transforms_dictionary_encoded` in `tests/test_transforms.py` covering `DictionaryArray` and `ChunkedArray` on `BucketTransform` and `TruncateTransform`.
- All 284 transform tests pass locally.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Transforms: Support dictionary-encoded PyArrow arrays in partition transforms (#3633) [iceberg-python]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #3841:
URL: https://github.com/apache/iceberg-python/pull/3841#discussion_r3847293272
##########
tests/test_transforms.py:
##########
@@ -1710,3 +1710,21 @@ def test_calling_pyarrow_transform_without_pyiceberg_core_installed_correctly_ra
with pytest.raises(NotInstalledError):
transform.pyarrow_transform(StringType())
+
+
+def test_pyarrow_transforms_dictionary_encoded() -> None:
+ dict_arr = pa.DictionaryArray.from_arrays(pa.array([0, 1, 0, None]), pa.array(["foo", "bar"]))
+ raw_arr = pa.array(["foo", "bar", "foo", None])
+ bucket_transform = BucketTransform(num_buckets=10)
+ expected_bucket = bucket_transform.pyarrow_transform(StringType())(raw_arr)
+ assert bucket_transform.pyarrow_transform(StringType())(dict_arr) == expected_bucket
Review Comment:
_pyiceberg_transform_wrapper now normalizes dictionary-encoded arrays for all PyArrow transforms, including Year/Month/Day which also exercise the `expected_type` casting path. The added regression test covers Bucket/Truncate, but does not cover a time transform (or any transform using `expected_type`), so the new behavior is still untested for that wrapper branch.
##########
tests/test_transforms.py:
##########
@@ -1710,3 +1710,21 @@ def test_calling_pyarrow_transform_without_pyiceberg_core_installed_correctly_ra
with pytest.raises(NotInstalledError):
transform.pyarrow_transform(StringType())
+
+
+def test_pyarrow_transforms_dictionary_encoded() -> None:
+ dict_arr = pa.DictionaryArray.from_arrays(pa.array([0, 1, 0, None]), pa.array(["foo", "bar"]))
+ raw_arr = pa.array(["foo", "bar", "foo", None])
+ bucket_transform = BucketTransform(num_buckets=10)
+ expected_bucket = bucket_transform.pyarrow_transform(StringType())(raw_arr)
+ assert bucket_transform.pyarrow_transform(StringType())(dict_arr) == expected_bucket
+
+ chunked_dict = pa.chunked_array([dict_arr, dict_arr])
+ expected_chunked = pa.chunked_array([expected_bucket, expected_bucket])
+ assert bucket_transform.pyarrow_transform(StringType())(chunked_dict) == expected_chunked
+
+ truncate_transform = TruncateTransform(width=3)
+ dict_truncate_arr = pa.DictionaryArray.from_arrays(pa.array([0, 1, 0]), pa.array(["developer", "iceberg"]))
Review Comment:
The new dictionary-encoded test uses default index types, which currently yields Dictionary(Int64, Utf8) in practice. The reported failure was Dictionary(Int32, Utf8); specifying the indices array type explicitly (for both dict arrays in this test) ensures the regression test matches the real-world case and covers both index widths deterministically.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Transforms: Support dictionary-encoded PyArrow arrays in partition transforms (#3633) [iceberg-python]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #3841:
URL: https://github.com/apache/iceberg-python/pull/3841#issuecomment-5805249361
This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that's incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Transforms: Support dictionary-encoded PyArrow arrays in partition transforms (#3633) [iceberg-python]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #3841:
URL: https://github.com/apache/iceberg-python/pull/3841#issuecomment-5922342124
This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Transforms: Support dictionary-encoded PyArrow arrays in partition transforms (#3633) [iceberg-python]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #3841: Transforms: Support dictionary-encoded PyArrow arrays in partition transforms (#3633)
URL: https://github.com/apache/iceberg-python/pull/3841
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org