You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "TheR1sing3un (via GitHub)" <gi...@apache.org> on 2026/05/25 04:14:06 UTC

[PR] [python] Implement aggregation merge engine in pypaimon [paimon]

TheR1sing3un opened a new pull request, #7952:
URL: https://github.com/apache/paimon/pull/7952

   ### Purpose
   
   Port Java's `AggregateMergeFunction` to pypaimon, following the shape of #7745.
   
   Ships the `FieldAggregator` framework, 9 value aggregators (`sum` / `max` / `min` / `last_value` / `last_non_null_value` / `first_value` / `first_non_null_value` / `bool_or` / `bool_and`) plus the `primary_key` placeholder, the `AggregateMergeFunction` wired into `MergeFileSplitRead._build_merge_function`, and a `merge_engine_support` guard that rejects retract opt-ins, sequence fields and out-of-scope aggregator identifiers (`collect` / `nested_update` / `theta_sketch` / `roaring_bitmap_*` / ...).
   
   Retract handling (DELETE / UPDATE_BEFORE) and the remaining 14 Java aggregators are intentionally deferred to follow-up PRs, mirroring #7745's scoping.
   
   ### Tests
   
   - Unit: `test_field_aggregator_registry.py`, `test_field_aggregators.py`, `test_aggregation_merge_function.py`
   - E2E: `test_aggregation_e2e.py` covers positive merges (sum across commits, mixed aggregators, null semantics, disjoint keys, default aggregator) plus the unsupported-option guard.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [python] Implement aggregation merge engine in pypaimon [paimon]

Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on PR #7952:
URL: https://github.com/apache/paimon/pull/7952#issuecomment-4592163769

   +1


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [python] Implement aggregation merge engine in pypaimon [paimon]

Posted by "TheR1sing3un (via GitHub)" <gi...@apache.org>.
TheR1sing3un commented on PR #7952:
URL: https://github.com/apache/paimon/pull/7952#issuecomment-4591049572

   @JingsongLi @leaves12138 Hi, any suggestions? Lets land it! Thanks!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [python] Implement aggregation merge engine in pypaimon [paimon]

Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi merged PR #7952:
URL: https://github.com/apache/paimon/pull/7952


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [python] Implement aggregation merge engine in pypaimon [paimon]

Posted by "TheR1sing3un (via GitHub)" <gi...@apache.org>.
TheR1sing3un commented on code in PR #7952:
URL: https://github.com/apache/paimon/pull/7952#discussion_r3309372607


##########
paimon-python/pypaimon/read/split_read.py:
##########
@@ -650,6 +652,17 @@ def _build_merge_function(self):
                 value_arity=self.value_arity,
                 nullables=[f.type.nullable for f in self.value_fields],
             )
+        if engine == MergeEngine.AGGREGATE:
+            field_aggregators = build_field_aggregators(
+                self.value_fields,
+                self.trimmed_primary_key,
+                self.table.options,
+            )
+            return AggregateMergeFunction(
+                key_arity=len(self.trimmed_primary_key),
+                value_arity=self.value_arity,
+                field_aggregators=field_aggregators,
+            )

Review Comment:
   done



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [python] Implement aggregation merge engine in pypaimon [paimon]

Posted by "leaves12138 (via GitHub)" <gi...@apache.org>.
leaves12138 commented on code in PR #7952:
URL: https://github.com/apache/paimon/pull/7952#discussion_r3308125695


##########
paimon-python/pypaimon/read/split_read.py:
##########
@@ -650,6 +652,17 @@ def _build_merge_function(self):
                 value_arity=self.value_arity,
                 nullables=[f.type.nullable for f in self.value_fields],
             )
+        if engine == MergeEngine.AGGREGATE:
+            field_aggregators = build_field_aggregators(
+                self.value_fields,
+                self.trimmed_primary_key,
+                self.table.options,
+            )
+            return AggregateMergeFunction(
+                key_arity=len(self.trimmed_primary_key),
+                value_arity=self.value_arity,
+                field_aggregators=field_aggregators,
+            )

Review Comment:
   This should use the full table primary key list, not `trimmed_primary_key`. `read_type`/`value_fields` still contains partition columns, while `trimmed_primary_key` removes partition columns from the heap key. If a primary key column is also a partition column and a table-wide default aggregator is configured, that PK value is treated as a normal value field and gets aggregated.
   
   Minimal repro on this PR:
   
   ```python
   schema = pa.schema([
       pa.field('p', pa.int64(), nullable=False),
       pa.field('id', pa.int64(), nullable=False),
       pa.field('v', pa.int64()),
   ])
   Schema.from_pyarrow_schema(
       schema,
       primary_keys=['p', 'id'],
       partition_keys=['p'],
       options={
           'bucket': '1',
           'merge-engine': 'aggregation',
           'fields.default-aggregate-function': 'sum',
       },
   )
   # write {'p': 1, 'id': 1, 'v': 10}, then {'p': 1, 'id': 1, 'v': 20}
   ```
   
   Expected read result is `{'p': 1, 'id': 1, 'v': 30}`, but pypaimon currently returns `{'p': 2, 'id': 1, 'v': 30}` because field `p` falls through to the default `sum` aggregator. Java builds `AggregateMergeFunction` with `tableSchema.primaryKeys()` for the same reason. Please pass `self.table.primary_keys` here and add an e2e test covering a partition key that is also part of the primary key with `fields.default-aggregate-function=sum`.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [python] Implement aggregation merge engine in pypaimon [paimon]

Posted by "TheR1sing3un (via GitHub)" <gi...@apache.org>.
TheR1sing3un commented on PR #7952:
URL: https://github.com/apache/paimon/pull/7952#issuecomment-4553474130

   @JingsongLi @leaves12138 Thanks for your review, please review again! 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org