You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "PyRSA (via GitHub)" <gi...@apache.org> on 2026/07/12 10:09:35 UTC
[PR] [python] Introduce 'nested_update' aggregator function [paimon]
PyRSA opened a new pull request, #8577:
URL: https://github.com/apache/paimon/pull/8577
### Purpose
Introduce the `nested_update` field aggregator for pypaimon.
This change adds support for the nested_update aggregation function in the Python reader, aligning its behavior with the Java implementation. It supports nested-key based updates, optional sequence-field based conflict resolution, configurable nested-key null handling, and count limiting.
### Dependency
This PR depends on #8561 and reuses several shared utility methods introduced there. Please review and merge #8561 first.
### Tests
- `python -m pytest pypaimon/tests/test_field_aggregators.py::FieldNestedUpdateAggTest -q`
- `git diff --check`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [python] Introduce 'nested_update' aggregator function [paimon]
Posted by "PyRSA (via GitHub)" <gi...@apache.org>.
PyRSA commented on PR #8577:
URL: https://github.com/apache/paimon/pull/8577#issuecomment-4988698219
> Please resolve conflicts.
Resolved the merge conflicts and updated the branch. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [python] Introduce 'nested_update' aggregator function [paimon]
Posted by "PyRSA (via GitHub)" <gi...@apache.org>.
PyRSA commented on PR #8577:
URL: https://github.com/apache/paimon/pull/8577#issuecomment-4981411049
> `nested_update` assumes that the elements of `ARRAY<ROW>` are `InternalRow`; however, the PyArrow→Polars reader actually outputs a `list[dict]`, and calling it directly results in an `AttributeError: ‘dict’ object has no attribute ‘get_field’`.
Thanks for pointing this out.
I updated the implementation to support both `InternalRow` and `dict` representations of nested rows.
Instead of accessing fields directly through `InternalRow.get_field()`, the aggregator now uses a common `FieldProjection` helper to extract configured fields from either row representation. This keeps the aggregation logic unchanged while making it compatible with the PyArrow → Polars read path.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [python] Introduce 'nested_update' aggregator function [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi merged PR #8577:
URL: https://github.com/apache/paimon/pull/8577
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [python] Introduce 'nested_update' aggregator function [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on PR #8577:
URL: https://github.com/apache/paimon/pull/8577#issuecomment-4988663489
Please resolve conflicts.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [python] Introduce 'nested_update' aggregator function [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on PR #8577:
URL: https://github.com/apache/paimon/pull/8577#issuecomment-4953811357
- `RowType.get_field_index()` returns `DataField.id`, but the `ProjectedRow` of nested key/sequence uses it as an array index.
- The actual nested field ID is a global ID and does not guarantee continuity from 0.
- Reproducing locally using field IDs `10, 20`: Both generate `mapping=[10]`, then directly throw an `IndexError: Position 10 is out of bounds for row arity 2`.
- The existing tests use `0, 1, 2`, which obscures the issue. The ordinal field should be returned, and non-sequential ID tests should be added.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [python] Introduce 'nested_update' aggregator function [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on PR #8577:
URL: https://github.com/apache/paimon/pull/8577#issuecomment-4975332455
`nested_update` assumes that the elements of `ARRAY<ROW>` are `InternalRow`; however, the PyArrow→Polars reader actually outputs a `list[dict]`, and calling it directly results in an `AttributeError: ‘dict’ object has no attribute ‘get_field’`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [python] Introduce 'nested_update' aggregator function [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on PR #8577:
URL: https://github.com/apache/paimon/pull/8577#issuecomment-4988661644
+1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [python] Introduce 'nested_update' aggregator function [paimon]
Posted by "PyRSA (via GitHub)" <gi...@apache.org>.
PyRSA commented on PR #8577:
URL: https://github.com/apache/paimon/pull/8577#issuecomment-4954617339
> * `RowType.get_field_index()` returns `DataField.id`, but the `ProjectedRow` of nested key/sequence uses it as an array index.
> * The actual nested field ID is a global ID and does not guarantee continuity from 0.
> * Reproducing locally using field IDs `10, 20`: Both generate `mapping=[10]`, then directly throw an `IndexError: Position 10 is out of bounds for row arity 2`.
> * The existing tests use `0, 1, 2`, which obscures the issue. The ordinal field should be returned, and non-sequential ID tests should be added.
Good catch, thanks! I've fixed `RowType.get_field_index()` to return the ordinal field position instead of `DataField.id`, and added a regression test using non-sequential field IDs to cover this case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org