You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "TheR1sing3un (via GitHub)" <gi...@apache.org> on 2026/05/16 12:45:46 UTC
[PR] [python] Extend commit protocol for compaction (DataIncrement/CompactIncrement) [paimon]
TheR1sing3un opened a new pull request, #7873:
URL: https://github.com/apache/paimon/pull/7873
## Summary
Lay the protocol-level groundwork for upcoming compaction work in pypaimon by aligning `CommitMessage` with Java's `CommitMessageImpl` shape and adding a JSON-safe wire format for cross-process transport.
**No observable behavior change today** — read / write / commit paths keep producing the same snapshots. This is foundation for the follow-up PRs that ship the compaction module, append-only compaction, PK LSM compaction, and Ray distributed executor.
Split from #7771 — originally bundled all of Phase 1-4 into one ~5000-line PR; this is the first of 6 smaller, focused PRs.
## What's in this PR
**Structural changes**
- New `DataIncrement` (write side) and `CompactIncrement` (compaction side) value objects, direct ports of `org.apache.paimon.io.DataIncrement` and `CompactIncrement`. Each holds `(new_files, deleted_files, changelog_files, new_index_files, deleted_index_files)` so future deletion-vector / changelog work has an unambiguous slot.
- `CommitMessage` refactored to `(partition, bucket, total_buckets, data_increment, compact_increment, check_from_snapshot)`. Convenience properties (`new_files`, `compact_before`, `compact_after`, …) preserve read-site ergonomics.
- `FileStoreCommit` emits ADD entries for `compact_after`, DELETE entries for `compact_before`, and auto-selects `commit_kind=COMPACT` when a message carries only compact increments. A dedicated `commit_compact()` helper enforces COMPACT-only semantics with no row-id assignment.
- `FileStoreWrite` / `TableUpdate` construct `CommitMessage` via `DataIncrement` on the existing write path — no behavior change for current callers.
**DataFileMeta serialization**
- `to_dict` / `from_dict` round-trip with tagged-value encoding for `bytes`, `Decimal`, `datetime`, `date`, `time`, and `Timestamp` so file metas can ship through JSON-only transports (Ray task payloads later).
- Public `encode_value` / `decode_value` helpers reused by `CommitMessage.partition` (DATE / DECIMAL / bytes / Timestamp partitions).
- Tolerates manifest-side `BinaryRow` (lazy-decoded) and pyarrow Array-like `null_counts` so round-tripping a freshly-produced file meta doesn't fail.
**CommitMessageSerializer**
- VERSION=1 wire format covering full `DataIncrement` + `CompactIncrement` shape (including `IndexFileMeta` identity fields). `dv_ranges` / `global_index_meta` will be wired up alongside deletion-vector phases.
## Test plan
- [x] New `commit_message_serializer_test`: round-trip `CommitMessage` with `DataIncrement` / `CompactIncrement` / index files / non-JSON-native partition tuples (DATE, Decimal, bytes, Timestamp); IndexFileMeta round-trip; unknown-version rejection.
- [x] New `file_store_commit_compact_test`: protocol-level coverage of `compact_before` → DELETE entry, `compact_after` → ADD entry, auto-COMPACT kind selection. Full e2e lands when the compactor lands (PR2+).
- [x] Existing `file_store_commit_test` / `partition_predicate_test` / `table_commit_test` updated to construct `CommitMessage` via `DataIncrement` instead of the legacy `new_files=` kwarg (signature change adapter, no production behavior).
## Note: no consumers yet
`CompactIncrement` / `commit_compact` / `CommitMessageSerializer` introduced here have no callers in this PR — they're the foundation. The next 5 split PRs build on top:
1. **This PR** — commit protocol foundation
2. Append-only compaction module + LocalExecutor
3. MergeFunction abstraction (read-path refactor)
4. Levels + UniversalCompaction strategy
5. MergeTree compaction integration (PK e2e)
6. Ray distributed executor
Each follow-up will land separately for incremental review.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [python] Extend commit protocol for compaction (DataIncrement/CompactIncrement) [paimon]
Posted by "TheR1sing3un (via GitHub)" <gi...@apache.org>.
TheR1sing3un commented on code in PR #7873:
URL: https://github.com/apache/paimon/pull/7873#discussion_r3465237293
##########
paimon-python/pypaimon/write/table_update_by_row_id.py:
##########
@@ -382,7 +383,7 @@ def _write_group(self, partition: GenericRow, first_row_id: int,
CommitMessage(
partition=partition_tuple,
bucket=0,
- new_files=new_files,
+ data_increment=DataIncrement(new_files=new_files),
Review Comment:
fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [python] Extend commit protocol for compaction (DataIncrement/CompactIncrement) [paimon]
Posted by "TheR1sing3un (via GitHub)" <gi...@apache.org>.
TheR1sing3un commented on PR #7873:
URL: https://github.com/apache/paimon/pull/7873#issuecomment-4467057114
Ready for review, 1st pr of the entire compaction feature.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [python] Extend commit protocol for compaction (DataIncrement/CompactIncrement) [paimon]
Posted by "TheR1sing3un (via GitHub)" <gi...@apache.org>.
TheR1sing3un commented on PR #7873:
URL: https://github.com/apache/paimon/pull/7873#issuecomment-4893472875
> Thank you for your continuous updates, but I still feel concerned about this ability because Compaction is a very inefficient task in Python. I would prefer Compaction to be completed by the Rust layer.
Agreed! Lets close it,thanks for your kindly reviews
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [python] Extend commit protocol for compaction (DataIncrement/CompactIncrement) [paimon]
Posted by "TheR1sing3un (via GitHub)" <gi...@apache.org>.
TheR1sing3un commented on code in PR #7873:
URL: https://github.com/apache/paimon/pull/7873#discussion_r3459010518
##########
paimon-python/pypaimon/write/file_store_commit.py:
##########
@@ -189,6 +191,98 @@ def commit(self, commit_messages: List[CommitMessage], commit_identifier: int):
allow_rollback=allow_rollback,
index_deletes=index_deletes)
+ def commit_compact(self, commit_messages: List[CommitMessage], commit_identifier: int):
+ """Commit compaction-only messages.
+
+ Each message must carry no new_files; compact_before → DELETE entries,
+ compact_after → ADD entries; snapshot kind = COMPACT. Skips row-id
+ assignment and conflict detection (a compaction never produces new rows).
+ """
+ if not commit_messages:
+ return
+
+ for msg in commit_messages:
+ if msg.new_files:
+ raise ValueError(
+ "commit_compact rejects messages with new_files; use commit() instead."
+ )
+
+ logger.info(
+ "Ready to commit compact to table %s, number of commit messages: %d",
+ self.table.identifier,
+ len(commit_messages),
+ )
+ commit_entries = self._build_commit_entries(commit_messages)
+ if not commit_entries:
+ return
+
+ logger.info("Finished collecting compact changes: %d entries", len(commit_entries))
+
+ self._try_commit(
Review Comment:
fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [python] Extend commit protocol for compaction (DataIncrement/CompactIncrement) [paimon]
Posted by "TheR1sing3un (via GitHub)" <gi...@apache.org>.
TheR1sing3un commented on code in PR #7873:
URL: https://github.com/apache/paimon/pull/7873#discussion_r3523446409
##########
paimon-python/pypaimon/write/file_store_commit.py:
##########
@@ -114,10 +114,22 @@ def __init__(self, snapshot_commit: SnapshotCommit, table, commit_user: str,
self.rollback = CommitRollback(table_rollback) if table_rollback is not None else None
def commit(self, commit_messages: List[CommitMessage], commit_identifier: int):
- """Commit the given commit messages in normal append mode."""
+ """Commit write-side messages (data_increment only).
+
+ Compaction results (compact_before / compact_after) must be committed
+ through commit_compact() instead — they produce a separate snapshot kind
+ and skip row-id assignment.
+ """
if not commit_messages:
return
+ for msg in commit_messages:
+ if not msg.compact_increment.is_empty():
+ raise ValueError(
+ "commit() rejects messages carrying compact_increment; "
+ "use commit_compact() for compaction results."
Review Comment:
fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [python] Extend commit protocol for compaction (DataIncrement/CompactIncrement) [paimon]
Posted by "TheR1sing3un (via GitHub)" <gi...@apache.org>.
TheR1sing3un commented on PR #7873:
URL: https://github.com/apache/paimon/pull/7873#issuecomment-4638041005
> I think the new `CommitMessageSerializer` drops one existing part of `CommitMessage`.
>
> `CommitMessage` has `index_deletes`, and `is_empty()` treats it as meaningful content. The global-index update path can build messages that only contain `index_deletes` (`build_index_delete_msgs(...)`). But `CommitMessageSerializer.to_dict/from_dict` only round-trips partition, bucket, total_buckets, data/compact increments, and `check_from_snapshot`; it does not serialize `index_deletes`.
>
> If such a message is shipped through this serializer, the deserialized message will lose the index manifest deletions and the commit can leave stale global-index entries behind. Could we include `index_deletes` in the wire format and add a round-trip test for a message whose only content is `index_deletes`?
Nice catch, fixed~
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [python] Extend commit protocol for compaction (DataIncrement/CompactIncrement) [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #7873:
URL: https://github.com/apache/paimon/pull/7873#discussion_r3485047833
##########
paimon-python/pypaimon/write/file_store_commit.py:
##########
@@ -189,6 +191,118 @@ def commit(self, commit_messages: List[CommitMessage], commit_identifier: int):
allow_rollback=allow_rollback,
index_deletes=index_deletes)
+ def commit_compact(self, commit_messages: List[CommitMessage], commit_identifier: int):
+ """Commit compaction-only messages.
+
+ Each message must carry no data increment; compact_before → DELETE
+ entries, compact_after → ADD entries; snapshot kind = COMPACT. Skips
+ row-id assignment and conflict detection (a compaction never produces
+ new rows).
+ """
+ if not commit_messages:
+ return
+
+ for msg in commit_messages:
+ if not msg.data_increment.is_empty():
+ raise ValueError(
+ "commit_compact rejects messages carrying a data increment; "
+ "use commit() instead."
+ )
+
+ logger.info(
+ "Ready to commit compact to table %s, number of commit messages: %d",
+ self.table.identifier,
+ len(commit_messages),
+ )
+ commit_entries = self._build_commit_entries(commit_messages)
+
+ # Forward index manifest deletions (e.g. global-index updates) carried by
+ # the compact messages, mirroring commit(). Unlike commit() we do not run
+ # the new_files-driven apply_global_index_update_action derivation, since a
+ # compaction produces no new_files; we only ship deletes the caller set
+ # explicitly. A message whose only content is index_deletes still needs to
+ # commit, so the empty-entries short-circuit must account for them.
+ index_deletes = []
+ for msg in commit_messages:
+ index_deletes.extend(msg.index_deletes)
+
+ if not commit_entries and not index_deletes:
+ return
+
+ logger.info("Finished collecting compact changes: %d entries", len(commit_entries))
+
+ self._try_commit(
+ commit_kind="COMPACT",
+ commit_identifier=commit_identifier,
+ commit_entries_plan=lambda snapshot: commit_entries,
+ detect_conflicts=False,
+ allow_rollback=False,
+ index_deletes=index_deletes,
+ )
+
+ def _build_commit_entries(self, commit_messages: List[CommitMessage]) -> List[ManifestEntry]:
+ # new_files / compact_before / compact_after become delta manifest entries
+ # here; data_increment.changelog_files is handled separately by
+ # _collect_changelog_entries (written to the changelog manifest, not the
+ # delta). Reject the remaining unwired slots loudly so a future writer
+ # that starts filling them cannot silently lose data at commit time.
+ #
+ # Entry ordering for a message that carries both kinds is
+ # ADD(new_files) -> DELETE(compact_before) -> ADD(compact_after). Today
+ # commit()/commit_compact() enforce that a single message never mixes
+ # data and compact increments, so the mixed case is unreachable; a future
+ # change that relaxes that validation must keep this ordering in mind.
+ for msg in commit_messages:
+ di = msg.data_increment
+ if (di.deleted_files
+ or di.new_index_files or di.deleted_index_files):
+ raise NotImplementedError(
+ "FileStoreCommit does not yet handle DataIncrement.deleted_files / "
+ "new_index_files / deleted_index_files; these slots "
+ "will be wired in by the feature that first needs each one."
+ )
+ ci = msg.compact_increment
+ if (ci.changelog_files or ci.new_index_files or ci.deleted_index_files):
+ raise NotImplementedError(
+ "FileStoreCommit does not yet handle CompactIncrement.changelog_files / "
+ "new_index_files / deleted_index_files; these slots will be wired in by "
+ "the feature that first needs each one."
+ )
+
+ entries: List[ManifestEntry] = []
+ for msg in commit_messages:
+ partition = GenericRow(list(msg.partition), self.table.partition_keys_fields)
+ # Prefer the message's total_buckets (captured when the plan was
+ # built) over the current table value, so a plan that survived a
+ # bucket rescale is not silently rewritten with the new count.
+ total_buckets = msg.total_buckets if msg.total_buckets is not None \
Review Comment:
This fixes `total_buckets` for the delta manifest entries, but `_collect_changelog_entries()` still writes changelog manifest entries with `self.table.total_buckets`. For a write plan prepared before a bucket-count change, the data entries now keep `msg.total_buckets` while the changelog entries are silently rewritten with the current table count, so the same commit can publish inconsistent bucket metadata. Please apply the same `msg.total_buckets if msg.total_buckets is not None else self.table.total_buckets` logic in `_collect_changelog_entries()` (and ideally cover a message with both `new_files` and `changelog_files`).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [python] Extend commit protocol for compaction (DataIncrement/CompactIncrement) [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #7873:
URL: https://github.com/apache/paimon/pull/7873#discussion_r3458872721
##########
paimon-python/pypaimon/write/file_store_commit.py:
##########
@@ -189,6 +191,98 @@ def commit(self, commit_messages: List[CommitMessage], commit_identifier: int):
allow_rollback=allow_rollback,
index_deletes=index_deletes)
+ def commit_compact(self, commit_messages: List[CommitMessage], commit_identifier: int):
+ """Commit compaction-only messages.
+
+ Each message must carry no new_files; compact_before → DELETE entries,
+ compact_after → ADD entries; snapshot kind = COMPACT. Skips row-id
+ assignment and conflict detection (a compaction never produces new rows).
+ """
+ if not commit_messages:
+ return
+
+ for msg in commit_messages:
+ if msg.new_files:
+ raise ValueError(
+ "commit_compact rejects messages with new_files; use commit() instead."
+ )
+
+ logger.info(
+ "Ready to commit compact to table %s, number of commit messages: %d",
+ self.table.identifier,
+ len(commit_messages),
+ )
+ commit_entries = self._build_commit_entries(commit_messages)
+ if not commit_entries:
+ return
+
+ logger.info("Finished collecting compact changes: %d entries", len(commit_entries))
+
+ self._try_commit(
Review Comment:
`commit_compact()` drops `CommitMessage.index_deletes`. The normal `commit()` path collects these deletions and passes them into `_try_commit`, and the serializer now explicitly preserves `index_deletes` because a message can be non-empty only due to index manifest deletions. Here, however, compact messages are neither rejected when `msg.index_deletes` is set nor forwarded to `_try_commit`, so any compaction task that emits index manifest deletions would silently leave the old index manifest entries behind. Could we either collect/pass `index_deletes` here like `commit()` does, or explicitly reject them until compact-side index deletion is wired, with a regression test?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [python] Extend commit protocol for compaction (DataIncrement/CompactIncrement) [paimon]
Posted by "TheR1sing3un (via GitHub)" <gi...@apache.org>.
TheR1sing3un closed pull request #7873: [python] Extend commit protocol for compaction (DataIncrement/CompactIncrement)
URL: https://github.com/apache/paimon/pull/7873
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [python] Extend commit protocol for compaction (DataIncrement/CompactIncrement) [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #7873:
URL: https://github.com/apache/paimon/pull/7873#discussion_r3485047083
##########
paimon-python/pypaimon/write/file_store_commit.py:
##########
@@ -189,6 +191,118 @@ def commit(self, commit_messages: List[CommitMessage], commit_identifier: int):
allow_rollback=allow_rollback,
index_deletes=index_deletes)
+ def commit_compact(self, commit_messages: List[CommitMessage], commit_identifier: int):
+ """Commit compaction-only messages.
+
+ Each message must carry no data increment; compact_before → DELETE
+ entries, compact_after → ADD entries; snapshot kind = COMPACT. Skips
Review Comment:
This says COMPACT skips row-id assignment, but `commit_compact()` still calls `_try_commit()` with `commit_kind="COMPACT"`, and `_try_commit()` applies `_assign_snapshot_id()` / `_assign_row_tracking_meta()` whenever row tracking is enabled. That means a `compact_after` file with no `first_row_id` is treated like a new append file, gets a fresh row-id range from `latest_snapshot.next_row_id`, and the COMPACT snapshot advances `next_row_id`. Compaction should preserve the logical row ids of the compacted input files, otherwise row-id based updates/conflict checks can no longer locate those rows after Python commits the compaction. Please gate the row-tracking assignment to write/overwrite commits (or explicitly skip it for COMPACT) and add a row-tracking compact test that verifies `compact_after.first_row_id` and `next_row_id` are not rewritten.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [python] Extend commit protocol for compaction (DataIncrement/CompactIncrement) [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #7873:
URL: https://github.com/apache/paimon/pull/7873#discussion_r3460597817
##########
paimon-python/pypaimon/write/file_store_commit.py:
##########
@@ -189,6 +191,116 @@ def commit(self, commit_messages: List[CommitMessage], commit_identifier: int):
allow_rollback=allow_rollback,
index_deletes=index_deletes)
+ def commit_compact(self, commit_messages: List[CommitMessage], commit_identifier: int):
+ """Commit compaction-only messages.
+
+ Each message must carry no new_files; compact_before → DELETE entries,
+ compact_after → ADD entries; snapshot kind = COMPACT. Skips row-id
+ assignment and conflict detection (a compaction never produces new rows).
+ """
+ if not commit_messages:
+ return
+
+ for msg in commit_messages:
+ if msg.new_files:
Review Comment:
`commit_compact()` should reject any non-empty `data_increment`, not only `new_files`. Today a message such as `CommitMessage(..., data_increment=DataIncrement(changelog_files=[...]))` passes this check, `_build_commit_entries()` deliberately produces no delta entries for data-side changelog files, and the `if not commit_entries and not index_deletes` branch below returns without committing or raising. That silently drops a non-empty commit message. Please change this guard to `if not msg.data_increment.is_empty(): raise ...` (or explicitly route data-side changelog entries through `commit()`), and add a regression test for a data-increment changelog-only compact message.
##########
paimon-python/pypaimon/write/file_store_commit.py:
##########
@@ -189,6 +191,116 @@ def commit(self, commit_messages: List[CommitMessage], commit_identifier: int):
allow_rollback=allow_rollback,
index_deletes=index_deletes)
+ def commit_compact(self, commit_messages: List[CommitMessage], commit_identifier: int):
+ """Commit compaction-only messages.
+
+ Each message must carry no new_files; compact_before → DELETE entries,
+ compact_after → ADD entries; snapshot kind = COMPACT. Skips row-id
Review Comment:
Once `commit_compact()` accepts `compact_after` files, the abort path needs to clean them as well. `FileStoreCommit.abort()` still iterates only over `message.new_files + message.changelog_files`, so a failed/cancelled compaction would leave the rewritten `compact_after` files (and any future `compact_changelog_files`) behind, unlike the Java abort path which includes `compactIncrement().compactAfter()` and compact changelog files. Please extend `abort()` and add a regression test for aborting a compact message.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [python] Extend commit protocol for compaction (DataIncrement/CompactIncrement) [paimon]
Posted by "TheR1sing3un (via GitHub)" <gi...@apache.org>.
TheR1sing3un commented on code in PR #7873:
URL: https://github.com/apache/paimon/pull/7873#discussion_r3523458894
##########
paimon-python/pypaimon/write/file_store_commit.py:
##########
@@ -124,33 +124,62 @@ def __init__(self, snapshot_commit: SnapshotCommit, table, commit_user: str,
self.rollback = CommitRollback(table_rollback) if table_rollback is not None else None
def commit(self, commit_messages: List[CommitMessage], commit_identifier: int):
- """Commit the given commit messages in normal append mode."""
+ """Commit the given messages, splitting write and compaction results.
+
+ Each CommitMessage may carry a data_increment (normal write), a
+ compact_increment (compaction result), or both. Data increments feed an
+ APPEND (or OVERWRITE) snapshot; compact increments feed a separate
+ COMPACT snapshot, so a single commit() call produces up to two
+ snapshots, mirroring Java's FileStoreCommitImpl. Compaction never
+ assigns row-ids or runs conflict detection.
+ """
if not commit_messages:
return
- # Extract the minimum check_from_snapshot from commit messages
- valid_snapshots = [msg.check_from_snapshot for msg in commit_messages
- if msg.check_from_snapshot != -1]
- if valid_snapshots:
- self.conflict_detection._row_id_check_from_snapshot = min(valid_snapshots)
+ self._reject_unwired_slots(commit_messages)
logger.info(
"Ready to commit to table %s, number of commit messages: %d",
self.table.identifier,
len(commit_messages),
)
- commit_entries = self._collect_manifest_entries(commit_messages)
+
+ # index_adds / index_deletes ride with the snapshot their payload
+ # belongs to: messages carrying a compact increment go to the COMPACT
+ # snapshot, everything else (writes and pure index updates) goes to the
+ # APPEND snapshot. Attributing them to exactly one side keeps the same
+ # index manifest change from being committed twice.
+ data_index_adds, data_index_deletes = [], []
+ compact_index_adds, compact_index_deletes = [], []
+ for msg in commit_messages:
+ if msg.compact_increment.is_empty():
+ data_index_adds.extend(msg.index_adds)
+ data_index_deletes.extend(msg.index_deletes)
+ else:
+ compact_index_adds.extend(msg.index_adds)
+ compact_index_deletes.extend(msg.index_deletes)
+
+ # APPEND/OVERWRITE snapshot first, COMPACT second, mirroring Java's
+ # FileStoreCommitImpl.commit ordering.
+ self._commit_data(commit_messages, commit_identifier,
+ data_index_adds, data_index_deletes)
+ self._commit_compact(commit_messages, commit_identifier,
+ compact_index_adds, compact_index_deletes)
+
+ def _commit_data(self, commit_messages: List[CommitMessage], commit_identifier: int,
+ index_adds: List, index_deletes: List) -> None:
+ # Extract the minimum check_from_snapshot from commit messages
+ valid_snapshots = [msg.check_from_snapshot for msg in commit_messages
Review Comment:
fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [python] Extend commit protocol for compaction (DataIncrement/CompactIncrement) [paimon]
Posted by "TheR1sing3un (via GitHub)" <gi...@apache.org>.
TheR1sing3un commented on PR #7873:
URL: https://github.com/apache/paimon/pull/7873#issuecomment-4727826854
@JingsongLi @XiaoHongbo-Hope Hi, could me help to review it? Hope to land it!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [python] Extend commit protocol for compaction (DataIncrement/CompactIncrement) [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #7873:
URL: https://github.com/apache/paimon/pull/7873#discussion_r3503453708
##########
paimon-python/pypaimon/write/file_store_commit.py:
##########
@@ -114,10 +114,22 @@ def __init__(self, snapshot_commit: SnapshotCommit, table, commit_user: str,
self.rollback = CommitRollback(table_rollback) if table_rollback is not None else None
def commit(self, commit_messages: List[CommitMessage], commit_identifier: int):
- """Commit the given commit messages in normal append mode."""
+ """Commit write-side messages (data_increment only).
+
+ Compaction results (compact_before / compact_after) must be committed
+ through commit_compact() instead — they produce a separate snapshot kind
+ and skip row-id assignment.
+ """
if not commit_messages:
return
+ for msg in commit_messages:
+ if not msg.compact_increment.is_empty():
+ raise ValueError(
+ "commit() rejects messages carrying compact_increment; "
+ "use commit_compact() for compaction results."
Review Comment:
This new path is not reachable through the public table commit API yet. `WriteBuilder.new_commit()` returns `BatchTableCommit` / `StreamTableCommit`, and both `commit()` implementations still delegate to `FileStoreCommit.commit(...)`; with this guard, any caller that receives a `CommitMessage` carrying `compact_increment` through the normal writer/driver flow will just get this `ValueError`. Please expose a matching `commit_compact(...)` on `BatchTableCommit` and `StreamTableCommit` (or make the public `commit()` dispatch compact-only messages to `FileStoreCommit.commit_compact`) so the new compact protocol can actually be committed without reaching into `table_commit.file_store_commit` internals.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [python] Extend commit protocol for compaction (DataIncrement/CompactIncrement) [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on PR #7873:
URL: https://github.com/apache/paimon/pull/7873#issuecomment-4637974650
I think the new `CommitMessageSerializer` drops one existing part of `CommitMessage`.
`CommitMessage` has `index_deletes`, and `is_empty()` treats it as meaningful content. The global-index update path can build messages that only contain `index_deletes` (`build_index_delete_msgs(...)`). But `CommitMessageSerializer.to_dict/from_dict` only round-trips partition, bucket, total_buckets, data/compact increments, and `check_from_snapshot`; it does not serialize `index_deletes`.
If such a message is shipped through this serializer, the deserialized message will lose the index manifest deletions and the commit can leave stale global-index entries behind. Could we include `index_deletes` in the wire format and add a round-trip test for a message whose only content is `index_deletes`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [python] Extend commit protocol for compaction (DataIncrement/CompactIncrement) [paimon]
Posted by "TheR1sing3un (via GitHub)" <gi...@apache.org>.
TheR1sing3un commented on code in PR #7873:
URL: https://github.com/apache/paimon/pull/7873#discussion_r3485144099
##########
paimon-python/pypaimon/write/file_store_commit.py:
##########
@@ -189,6 +191,118 @@ def commit(self, commit_messages: List[CommitMessage], commit_identifier: int):
allow_rollback=allow_rollback,
index_deletes=index_deletes)
+ def commit_compact(self, commit_messages: List[CommitMessage], commit_identifier: int):
+ """Commit compaction-only messages.
+
+ Each message must carry no data increment; compact_before → DELETE
+ entries, compact_after → ADD entries; snapshot kind = COMPACT. Skips
+ row-id assignment and conflict detection (a compaction never produces
+ new rows).
+ """
+ if not commit_messages:
+ return
+
+ for msg in commit_messages:
+ if not msg.data_increment.is_empty():
+ raise ValueError(
+ "commit_compact rejects messages carrying a data increment; "
+ "use commit() instead."
+ )
+
+ logger.info(
+ "Ready to commit compact to table %s, number of commit messages: %d",
+ self.table.identifier,
+ len(commit_messages),
+ )
+ commit_entries = self._build_commit_entries(commit_messages)
+
+ # Forward index manifest deletions (e.g. global-index updates) carried by
+ # the compact messages, mirroring commit(). Unlike commit() we do not run
+ # the new_files-driven apply_global_index_update_action derivation, since a
+ # compaction produces no new_files; we only ship deletes the caller set
+ # explicitly. A message whose only content is index_deletes still needs to
+ # commit, so the empty-entries short-circuit must account for them.
+ index_deletes = []
+ for msg in commit_messages:
+ index_deletes.extend(msg.index_deletes)
+
+ if not commit_entries and not index_deletes:
+ return
+
+ logger.info("Finished collecting compact changes: %d entries", len(commit_entries))
+
+ self._try_commit(
+ commit_kind="COMPACT",
+ commit_identifier=commit_identifier,
+ commit_entries_plan=lambda snapshot: commit_entries,
+ detect_conflicts=False,
+ allow_rollback=False,
+ index_deletes=index_deletes,
+ )
+
+ def _build_commit_entries(self, commit_messages: List[CommitMessage]) -> List[ManifestEntry]:
+ # new_files / compact_before / compact_after become delta manifest entries
+ # here; data_increment.changelog_files is handled separately by
+ # _collect_changelog_entries (written to the changelog manifest, not the
+ # delta). Reject the remaining unwired slots loudly so a future writer
+ # that starts filling them cannot silently lose data at commit time.
+ #
+ # Entry ordering for a message that carries both kinds is
+ # ADD(new_files) -> DELETE(compact_before) -> ADD(compact_after). Today
+ # commit()/commit_compact() enforce that a single message never mixes
+ # data and compact increments, so the mixed case is unreachable; a future
+ # change that relaxes that validation must keep this ordering in mind.
+ for msg in commit_messages:
+ di = msg.data_increment
+ if (di.deleted_files
+ or di.new_index_files or di.deleted_index_files):
+ raise NotImplementedError(
+ "FileStoreCommit does not yet handle DataIncrement.deleted_files / "
+ "new_index_files / deleted_index_files; these slots "
+ "will be wired in by the feature that first needs each one."
+ )
+ ci = msg.compact_increment
+ if (ci.changelog_files or ci.new_index_files or ci.deleted_index_files):
+ raise NotImplementedError(
+ "FileStoreCommit does not yet handle CompactIncrement.changelog_files / "
+ "new_index_files / deleted_index_files; these slots will be wired in by "
+ "the feature that first needs each one."
+ )
+
+ entries: List[ManifestEntry] = []
+ for msg in commit_messages:
+ partition = GenericRow(list(msg.partition), self.table.partition_keys_fields)
+ # Prefer the message's total_buckets (captured when the plan was
+ # built) over the current table value, so a plan that survived a
+ # bucket rescale is not silently rewritten with the new count.
+ total_buckets = msg.total_buckets if msg.total_buckets is not None \
Review Comment:
fixed
##########
paimon-python/pypaimon/write/file_store_commit.py:
##########
@@ -189,6 +191,118 @@ def commit(self, commit_messages: List[CommitMessage], commit_identifier: int):
allow_rollback=allow_rollback,
index_deletes=index_deletes)
+ def commit_compact(self, commit_messages: List[CommitMessage], commit_identifier: int):
+ """Commit compaction-only messages.
+
+ Each message must carry no data increment; compact_before → DELETE
+ entries, compact_after → ADD entries; snapshot kind = COMPACT. Skips
Review Comment:
fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [python] Extend commit protocol for compaction (DataIncrement/CompactIncrement) [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #7873:
URL: https://github.com/apache/paimon/pull/7873#discussion_r3519691967
##########
paimon-python/pypaimon/write/file_store_commit.py:
##########
@@ -124,33 +124,62 @@ def __init__(self, snapshot_commit: SnapshotCommit, table, commit_user: str,
self.rollback = CommitRollback(table_rollback) if table_rollback is not None else None
def commit(self, commit_messages: List[CommitMessage], commit_identifier: int):
- """Commit the given commit messages in normal append mode."""
+ """Commit the given messages, splitting write and compaction results.
+
+ Each CommitMessage may carry a data_increment (normal write), a
+ compact_increment (compaction result), or both. Data increments feed an
+ APPEND (or OVERWRITE) snapshot; compact increments feed a separate
+ COMPACT snapshot, so a single commit() call produces up to two
+ snapshots, mirroring Java's FileStoreCommitImpl. Compaction never
+ assigns row-ids or runs conflict detection.
+ """
if not commit_messages:
return
- # Extract the minimum check_from_snapshot from commit messages
- valid_snapshots = [msg.check_from_snapshot for msg in commit_messages
- if msg.check_from_snapshot != -1]
- if valid_snapshots:
- self.conflict_detection._row_id_check_from_snapshot = min(valid_snapshots)
+ self._reject_unwired_slots(commit_messages)
logger.info(
"Ready to commit to table %s, number of commit messages: %d",
self.table.identifier,
len(commit_messages),
)
- commit_entries = self._collect_manifest_entries(commit_messages)
+
+ # index_adds / index_deletes ride with the snapshot their payload
+ # belongs to: messages carrying a compact increment go to the COMPACT
+ # snapshot, everything else (writes and pure index updates) goes to the
+ # APPEND snapshot. Attributing them to exactly one side keeps the same
+ # index manifest change from being committed twice.
+ data_index_adds, data_index_deletes = [], []
+ compact_index_adds, compact_index_deletes = [], []
+ for msg in commit_messages:
+ if msg.compact_increment.is_empty():
+ data_index_adds.extend(msg.index_adds)
+ data_index_deletes.extend(msg.index_deletes)
+ else:
+ compact_index_adds.extend(msg.index_adds)
+ compact_index_deletes.extend(msg.index_deletes)
+
+ # APPEND/OVERWRITE snapshot first, COMPACT second, mirroring Java's
+ # FileStoreCommitImpl.commit ordering.
+ self._commit_data(commit_messages, commit_identifier,
+ data_index_adds, data_index_deletes)
+ self._commit_compact(commit_messages, commit_identifier,
+ compact_index_adds, compact_index_deletes)
+
+ def _commit_data(self, commit_messages: List[CommitMessage], commit_identifier: int,
+ index_adds: List, index_deletes: List) -> None:
+ # Extract the minimum check_from_snapshot from commit messages
+ valid_snapshots = [msg.check_from_snapshot for msg in commit_messages
Review Comment:
`check_from_snapshot` is collected from every message, including compaction-only messages, and the value is stored on the reusable `ConflictDetection` instance. A compact message can already round-trip this field through `CommitMessageSerializer`, but compaction is supposed to skip row-id conflict detection. After such a compact-only commit returns at the data side, `_row_id_check_from_snapshot` remains set; the next normal `StreamTableCommit` using the same `FileStoreCommit` will run `detect_conflicts=True`/`allow_rollback=True` even if its own data messages have `check_from_snapshot=-1`. Please filter this to messages that actually contribute data-side entries, or reset the row-id check state around each commit phase so compact metadata cannot leak into later APPEND/OVERWRITE commits.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [python] Extend commit protocol for compaction (DataIncrement/CompactIncrement) [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #7873:
URL: https://github.com/apache/paimon/pull/7873#discussion_r3464931944
##########
paimon-python/pypaimon/write/table_update_by_row_id.py:
##########
@@ -382,7 +383,7 @@ def _write_group(self, partition: GenericRow, first_row_id: int,
CommitMessage(
partition=partition_tuple,
bucket=0,
- new_files=new_files,
+ data_increment=DataIncrement(new_files=new_files),
Review Comment:
This constructor is the one write-side path that still does not carry the planned bucket count into the new protocol. `FileStoreWrite.prepare_commit()` and `ShardTableUpdator.prepare_commit()` both set `total_buckets=self.table.total_buckets`, and `_build_commit_entries()` now relies on `msg.total_buckets` to avoid rewriting a stale plan with the table value observed at commit time. Row-id updates created here will fall back to the current table total, so the new protection is not applied to this path. Please set `total_buckets` here as well (and ideally add a small regression alongside the existing total_buckets test).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [python] Extend commit protocol for compaction (DataIncrement/CompactIncrement) [paimon]
Posted by "TheR1sing3un (via GitHub)" <gi...@apache.org>.
TheR1sing3un commented on code in PR #7873:
URL: https://github.com/apache/paimon/pull/7873#discussion_r3461088682
##########
paimon-python/pypaimon/write/file_store_commit.py:
##########
@@ -189,6 +191,116 @@ def commit(self, commit_messages: List[CommitMessage], commit_identifier: int):
allow_rollback=allow_rollback,
index_deletes=index_deletes)
+ def commit_compact(self, commit_messages: List[CommitMessage], commit_identifier: int):
+ """Commit compaction-only messages.
+
+ Each message must carry no new_files; compact_before → DELETE entries,
+ compact_after → ADD entries; snapshot kind = COMPACT. Skips row-id
Review Comment:
fixed
##########
paimon-python/pypaimon/write/file_store_commit.py:
##########
@@ -189,6 +191,116 @@ def commit(self, commit_messages: List[CommitMessage], commit_identifier: int):
allow_rollback=allow_rollback,
index_deletes=index_deletes)
+ def commit_compact(self, commit_messages: List[CommitMessage], commit_identifier: int):
+ """Commit compaction-only messages.
+
+ Each message must carry no new_files; compact_before → DELETE entries,
+ compact_after → ADD entries; snapshot kind = COMPACT. Skips row-id
+ assignment and conflict detection (a compaction never produces new rows).
+ """
+ if not commit_messages:
+ return
+
+ for msg in commit_messages:
+ if msg.new_files:
Review Comment:
fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [python] Extend commit protocol for compaction (DataIncrement/CompactIncrement) [paimon]
Posted by "TheR1sing3un (via GitHub)" <gi...@apache.org>.
TheR1sing3un commented on code in PR #7873:
URL: https://github.com/apache/paimon/pull/7873#discussion_r3461089019
##########
paimon-python/pypaimon/write/file_store_commit.py:
##########
@@ -189,6 +191,116 @@ def commit(self, commit_messages: List[CommitMessage], commit_identifier: int):
allow_rollback=allow_rollback,
index_deletes=index_deletes)
+ def commit_compact(self, commit_messages: List[CommitMessage], commit_identifier: int):
+ """Commit compaction-only messages.
+
+ Each message must carry no new_files; compact_before → DELETE entries,
+ compact_after → ADD entries; snapshot kind = COMPACT. Skips row-id
+ assignment and conflict detection (a compaction never produces new rows).
+ """
+ if not commit_messages:
+ return
+
+ for msg in commit_messages:
+ if msg.new_files:
Review Comment:
fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org