You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@fluss.apache.org by "Kaixuan-Duan (via GitHub)" <gi...@apache.org> on 2026/08/27 14:50:26 UTC
[I] Support sequence groups in the default merge engine [fluss]
Kaixuan-Duan opened a new issue, #4129:
URL: https://github.com/apache/fluss/issues/4129
### Search before asking
- [x] I searched in the [issues](https://github.com/apache/fluss/issues) and found nothing similar.
### Description
## Motivation
### What we have today
A primary key table without `table.merge-engine` keeps the last written row of the same primary key. It supports partial updates, where a writer only lists some columns and the rest keep their stored values:
```sql
CREATE TABLE orders (
order_id BIGINT,
pay_status STRING,
pay_time BIGINT,
ship_status STRING,
ship_time BIGINT,
PRIMARY KEY (order_id) NOT ENFORCED
);
-- the payment stream writes its own columns
INSERT INTO orders (order_id, pay_status, pay_time) SELECT ... FROM payments;
-- the shipping stream writes its own columns
INSERT INTO orders (order_id, ship_status, ship_time) SELECT ... FROM shipments;
```
Within one such column set the last write always wins, whether or not it is actually the newest. There is no way to tell Fluss that a write should be dropped because the row already holds a newer value.
The `versioned` merge engine does arbitrate by a version column, but it versions the **whole row** with a single `table.merge-engine.versioned.ver-column`. Two streams updating disjoint columns of the same row cannot each carry their own version, so `versioned` does not fit the case above.
### Why this matters
Two problems show up as soon as more than one stream writes the same row:
1. **Out-of-order writes silently overwrite newer data.** A retried or delayed record from the payment stream overwrites the payment columns even though the stored ones are newer. Today the only way to avoid this is to order the writes outside Fluss.
2. **Independent streams interfere through a shared version.** With `versioned`, a write from the shipping stream carrying a smaller version is rejected as a whole, even though its own columns are newer than what is stored. Each stream needs to be arbitrated on its own.
Paimon solves both with sequence groups on its `partial-update` merge engine, and the same mechanism fits Fluss's default merge engine, which already supports partial updates.
## Solution
Introduce `fields.<sequence-column>.sequence-group`, matching Paimon's DDL so that users can move between the two systems without rewriting their tables:
```sql
CREATE TABLE orders (
order_id BIGINT,
pay_status STRING,
pay_time BIGINT,
ship_status STRING,
ship_time BIGINT,
PRIMARY KEY (order_id) NOT ENFORCED
) WITH (
'fields.pay_time.sequence-group' = 'pay_status',
'fields.ship_time.sequence-group' = 'ship_status'
);
```
Each group is arbitrated on its own, so within a single write one group may advance while another does not:
```sql
INSERT INTO orders VALUES (1, 'paid', 100, 'shipped', 100);
-- pay_time moves forward while ship_time falls behind:
-- only the payment columns take the incoming values
INSERT INTO orders VALUES (1, 'refunded', 200, 'lost', 99);
SELECT * FROM orders;
-- output 1, refunded, 200, shipped, 100
```
### Semantics
Following Paimon's behaviour without aggregate functions, a sequence group acts as a version filter per group:
- The group takes the incoming values when its sequence columns are **not older** than the stored ones. Equal sequences advance, so that a replayed record still refreshes the group.
- A group whose incoming sequence columns are **all NULL** carries no order information and is skipped.
- NULL orders before every value, so a stored NULL is the oldest sequence.
- A sequence column is arbitrated by the very group it orders, keeping its value in step with the columns it protects.
- Naming several sequence columns (`fields.<c1>,<c2>.sequence-group`) declares a composite sequence key, compared in the declared order until one differs.
Sequence groups apply to both full-row and partial-column writes.
### Scope
Only the default merge engine. A sequence group declared together with `first_row`, `versioned` or `aggregation` is rejected at table creation, since none of them consults it while merging. Support for the `aggregation` merge engine, where the sequence column becomes an ordering key rather than a version filter, can be proposed separately.
### Rejected at table creation
- a log table, or any table with a merge engine
- a sequence column that does not exist in the schema
- a sequence column whose type cannot order a group; the accepted types are `INT`, `BIGINT`, `TIMESTAMP` and `TIMESTAMP_LTZ`, the same as the version column of the `versioned` merge engine
- a primary key column put into a group or used as a sequence column, since it holds the same value in both rows being merged
- a sequence column put into another group, since it reports the order of its own group
- the same column declared by more than one group
### Implementation sketch
- **Store the relation on the protected columns.** `Schema.Column` gains a `sequenceColumns` attribute, so a column carries the sequence columns ordering it. The DDL is keyed the other way round, so the option is inverted while converting the Flink table. This mirrors how `fields.<col>.agg` is already stored, and is required because `fields.*` keys cannot be registered as Fluss table properties.
- **Arbitrate in the merger.** A new `SequenceGroups` resolves the declared groups into field positions once per schema, and answers, for a given stored and incoming row, which fields may take the incoming values. The default merger consults it for a full-row write, and `PartialUpdater` for a partial-column write.
- **Leave the existing paths untouched.** A table without sequence groups keeps its current merger, and therefore also the fast path that skips reading the stored row. Recovering by overwriting an already decided value keeps bypassing the arbitration, since such a write restores an earlier state on purpose.
### Not covered
`DELETE` is not arbitrated. A Fluss delete record carries the primary key alone, so it holds no sequence values for any group to compare against the stored row, and it keeps removing the whole row as it does today. Paimon's `partial-update.remove-record-on-delete` and `partial-update.remove-record-on-sequence-group` therefore have no counterpart here. Arbitrating a delete would first require the delete record to carry the sequence columns, which is a change to the write protocol and should be proposed separately.
## Anything else?
- Paimon's sequence group documentation: https://paimon.apache.org/docs/master/primary-key-table/merge-engine/partial-update/#sequence-group
### Willingness to contribute
- [x] I'm willing to submit a PR!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org