You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "XiaoHongbo-Hope (via GitHub)" <gi...@apache.org> on 2026/07/23 07:56:24 UTC
[PR] [python][ray] Incrementally commit update_by_row_id file groups [paimon]
XiaoHongbo-Hope opened a new pull request, #8826:
URL: https://github.com/apache/paimon/pull/8826
### Purpose
- Add an opt-in `max_groups_per_commit` parameter to `update_by_row_id`.
- Deliver completed `_FIRST_ROW_ID` file groups to the driver as Ray workers finish, and commit each bounded window through a reusable `StreamTableCommit`.
- Keep the existing single atomic commit behavior when the option is unset.
- Preserve the original base snapshot on every commit message so conflict detection still covers the full read-to-commit window.
- Document that incremental mode may leave earlier windows committed if a later window fails.
### Tests
- `pytest -q pypaimon/tests/ray_update_by_row_id_test.py pypaimon/tests/table_merge_into_test.py pypaimon/tests/table_update_test.py` (138 passed, 16 subtests passed)
- Multi-worker Ray coverage with 8 input blocks, 4 shuffle partitions, 8 file groups, and 3 groups per commit.
- `flake8 --config=dev/cfg.ini` on changed Python files.
- Python 3.6 `py_compile` compatibility check.
- `git diff --check`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [python][ray] Incrementally commit update_by_row_id file groups [paimon]
Posted by "XiaoHongbo-Hope (via GitHub)" <gi...@apache.org>.
XiaoHongbo-Hope closed pull request #8826: [python][ray] Incrementally commit update_by_row_id file groups
URL: https://github.com/apache/paimon/pull/8826
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [python][ray] Incrementally commit update_by_row_id file groups [paimon]
Posted by "XiaoHongbo-Hope (via GitHub)" <gi...@apache.org>.
XiaoHongbo-Hope commented on code in PR #8826:
URL: https://github.com/apache/paimon/pull/8826#discussion_r3674998474
##########
paimon-python/pypaimon/ray/data_evolution_merge_join.py:
##########
@@ -598,14 +627,34 @@ def _apply_group(group: pa.Table) -> pa.Table:
all_msgs: list = []
num_updated = 0
action_row_ids = []
+ group_error = None
for batch in msgs_ds.iter_batches(batch_format="pyarrow"):
- for blob in batch.column("msgs_blob").to_pylist():
- all_msgs.extend(pickle.loads(blob))
- for n in batch.column("n_updated").to_pylist():
+ message_blobs = batch.column("msgs_blob").to_pylist()
+ updated_counts = batch.column("n_updated").to_pylist()
+ errors = batch.column("error").to_pylist()
+ row_id_blobs = (
+ batch.column("row_ids_blob").to_pylist()
+ if collect_row_ids else [None] * len(message_blobs)
+ )
+ for blob, n, row_ids_blob, error in zip(
+ message_blobs, updated_counts, row_id_blobs, errors):
+ if error is not None:
+ if group_error is None:
+ group_error = error
+ continue
+ group_msgs = pickle.loads(blob)
+ group_row_ids = (
+ pickle.loads(row_ids_blob) if collect_row_ids else []
+ )
+ if on_group_result is None:
+ all_msgs.extend(group_msgs)
+ else:
+ on_group_result(group_msgs, n, group_row_ids)
Review Comment:
> When `on_group_result` raises (for example, the first incremental commit hits a deterministic `CommitConflictError`), this loop exits immediately even though Ray may already have materialized several group results in the same batch. Only the messages passed to the failing callback are known and aborted; the remaining blobs are dropped, so their staged files are left orphaned. I reproduced this with three file groups, `num_partitions=1`, and `max_groups_per_commit=1`: forcing `FileStoreCommit.commit` to raise `CommitConflictError` left two new Parquet files after the call returned. Please drain and abort every uncommitted group result, or restructure the flow so a commit failure cannot abandon buffered outputs, and add a regression test for this case.
Fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [python][ray] Incrementally commit update_by_row_id file groups [paimon]
Posted by "XiaoHongbo-Hope (via GitHub)" <gi...@apache.org>.
XiaoHongbo-Hope commented on PR #8826:
URL: https://github.com/apache/paimon/pull/8826#issuecomment-5155005774
@JingsongLi Could you please help review again? The scope and following PRs are described in the PR description.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [python][ray] Add file-group-aligned row-id range processing [paimon]
Posted by "XiaoHongbo-Hope (via GitHub)" <gi...@apache.org>.
XiaoHongbo-Hope commented on PR #8826:
URL: https://github.com/apache/paimon/pull/8826#issuecomment-5214160484
> I think this can be simplified further by keeping row-id range planning as a passive planning primitive, instead of making `RowIdRangeContext` own the read/update/skip sequence and commit lifecycle.
>
> The existing row-id update path already carries the information needed by the normal commit conflict detector:
>
> * `CommitMessage.check_from_snapshot` identifies the snapshot from which the update was planned.
> * The new files in the message carry `first_row_id`, `row_count`, and `write_cols`.
> * `FileStoreCommit` can therefore derive row-range × write-column conflicts and detect missing or replaced anchor files from the ordinary commit messages.
>
> The important requirement is that the writer must build its `CommitMessage` from the same pinned snapshot/file information used by the range planner. It must not re-plan against the latest snapshot after the user has read an older pinned snapshot. This is a writer/planner concern; it does not require `commit()` to accept additional range arguments.
>
> A simpler API shape would be:
>
> ```python
> with plan_row_id_ranges(...) as plan:
> for row_range in plan:
> source = build_read(
> snapshot_id=row_range.snapshot_id,
> row_id_range=row_range,
> )
> updates = user_process(source)
> update_by_row_id_from_plan(row_range, updates, update_cols)
> ```
>
> Here, `RowIdRange` can be immutable metadata (plus an opaque internal planning token if needed for efficient file selection). The user controls read and processing, while the update writer creates the same normal `CommitMessage` that existing row-id updates already use.
>
> The current `protected_columns` and context-side read tracking introduce a stronger semantic contract. For example:
>
> ```
> read text -> compute embedding -> write embedding
> ```
>
> If another writer changes `text`, the existing row-id update semantics allow the `embedding` write because the actual write columns are disjoint. Tracking `text` through the context changes this into a transform/read-dependency conflict. That may be useful, but it is a new consistency contract rather than something required by row-id range commits themselves. If we need that contract, I think it should be specified and justified independently, and the dependency columns could be supplied explicitly rather than inferred through a stateful context.
>
> If we retain the existing row-id update conflict semantics for the first version, we should be able to remove most of the operation-specific machinery:
>
> * context-side `_read_columns`, `_updated`, `skip()`, and sequence state;
> * the operation checkpoint and exact-snapshot return path;
> * `protect_from_external_rewrites` state added to the generic commit layer.
>
> The range planner, pinned-snapshot tag, file-group-aligned packing, and normal `CommitMessage` conflict detection should be sufficient. This keeps the abstraction focused: planning returns ranges, reading reads a pinned range, and committing commits ordinary row-id update messages.
Thanks for your very kind and detail review and explanation, which helps me a lot. Will check more carefully in my future PR too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [python][ray] Support row-count-based commits for large backfills [paimon]
Posted by "XiaoHongbo-Hope (via GitHub)" <gi...@apache.org>.
XiaoHongbo-Hope closed pull request #8826: [python][ray] Support row-count-based commits for large backfills
URL: https://github.com/apache/paimon/pull/8826
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [python][ray] Incrementally commit update_by_row_id file groups [paimon]
Posted by "QuakeWang (via GitHub)" <gi...@apache.org>.
QuakeWang commented on code in PR #8826:
URL: https://github.com/apache/paimon/pull/8826#discussion_r3652877718
##########
paimon-python/pypaimon/ray/data_evolution_merge_join.py:
##########
@@ -598,14 +606,31 @@ def _apply_group(group: pa.Table) -> pa.Table:
all_msgs: list = []
num_updated = 0
action_row_ids = []
- for batch in msgs_ds.iter_batches(batch_format="pyarrow"):
- for blob in batch.column("msgs_blob").to_pylist():
- all_msgs.extend(pickle.loads(blob))
- for n in batch.column("n_updated").to_pylist():
+ iter_batches_kwargs = {"batch_format": "pyarrow"}
+ if on_group_result is not None:
+ # Yield each group immediately.
Review Comment:
`batch_size=1` does not yield each group immediately; it only splits blocks after `map_groups` emits them. Ray buffers multiple group results within the same task, so if a later group fails, earlier completed groups never reach `on_group_result` and are not committed. This reproduces with `num_partitions=1`, three groups, and `max_groups_per_commit=1`: when the third group fails, the snapshot does not advance.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [python][ray] Add file-group-aligned row-id range processing [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on PR #8826:
URL: https://github.com/apache/paimon/pull/8826#issuecomment-5215202460
Can we just introduce `process_row_id_ranges`? Can you just provide examples for the other APIs? Do we not need to encapsulate them for now?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [python][ray] Batch row-id update pipelines by row count [paimon]
Posted by "XiaoHongbo-Hope (via GitHub)" <gi...@apache.org>.
XiaoHongbo-Hope commented on PR #8826:
URL: https://github.com/apache/paimon/pull/8826#issuecomment-5205155073
> Maybe a `update_by_transform`? It seems users don't need to obtain the RowID?
Updated with `update_by_transform ` and add a `filter` parameter, could you please take a look at.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [python][ray] Incrementally commit update_by_row_id file groups [paimon]
Posted by "XiaoHongbo-Hope (via GitHub)" <gi...@apache.org>.
XiaoHongbo-Hope commented on PR #8826:
URL: https://github.com/apache/paimon/pull/8826#issuecomment-5179583721
Closing because the target workflow now only needs resumable incremental write_paimon. Incremental update_by_row_id is no longer part of the required solution.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [python][ray] Incrementally commit update_by_row_id file groups [paimon]
Posted by "Zouxxyy (via GitHub)" <gi...@apache.org>.
Zouxxyy commented on code in PR #8826:
URL: https://github.com/apache/paimon/pull/8826#discussion_r3655887919
##########
paimon-python/pypaimon/ray/data_evolution_merge_join.py:
##########
@@ -481,7 +537,10 @@ def distributed_update_apply(
)
sorted_first_row_ids = list(planner.first_row_ids)
if not sorted_first_row_ids:
- return [], 0, []
+ # No target file groups (e.g. an existing but empty snapshot). Match the
+ # 4-tuple contract so callers that unpack (msgs, num, row_ids, err) --
+ # including merge_into's NOT MATCHED insert path -- don't crash.
+ return [], 0, [], None
Review Comment:
Returning a 4-tuple fixes the unpacking crash, but it does not make an existing-but-empty snapshot follow the empty-target path. `_build_datasets` still passes `target_empty=base_snapshot is None`; for an empty snapshot, `base_snapshot` is non-null, so the NOT MATCHED insert path builds a left-anti join against a zero-block target. The new test fails on Python 3.10 and 3.11 with `ArrowInvalid`. Please also treat `base_snapshot.total_record_count == 0` as empty, and skip the matched branches, so this regression passes the supported CI matrix.
##########
paimon-python/pypaimon/write/file_store_commit.py:
##########
@@ -612,26 +759,84 @@ def _try_commit_once(self, retry_result: Optional[RetryResult], commit_kind: str
logger.warning(f"Exception occurs when preparing snapshot: {e}", exc_info=True)
raise RuntimeError(f"Failed to prepare snapshot: {e}")
- # Use SnapshotCommit for atomic commit
+ def clean_up_rejected_commit():
+ try:
+ self._clean_up_reuse_tmp_manifests(
+ delta_manifest_list,
+ changelog_manifest_list_name,
+ new_index_manifest,
+ )
+ self._clean_up_no_reuse_tmp_manifests(
+ base_manifest_list, merge_new_files)
+ except Exception:
+ logger.warning(
+ "Failed to clean up rejected commit manifests.",
+ exc_info=True,
+ )
+
+ # Mirror a ``with`` block but drive it manually so only commit() decides
+ # the outcome. __enter__ and commit() share one capture, so an __enter__
+ # failure flows through the classification below instead of bypassing it.
+ # __exit__ gets the real commit exception triple (for extension rollback)
+ # but its own failure is only logged -- never allowed to override the
+ # commit outcome (a lost/landed commit must not be misclassified).
+ success = None
+ commit_exc = None
+ entered = False
try:
- with self.snapshot_commit:
- success = self.snapshot_commit.commit(snapshot_data, statistics)
- if not success:
- commit_time_s = (int(time.time() * 1000) - start_millis) / 1000
+ self.snapshot_commit.__enter__()
+ entered = True
+ success = self.snapshot_commit.commit(snapshot_data, statistics)
+ except Exception as e:
+ commit_exc = e
+ finally:
+ # Never call __exit__ without a successful __enter__ (mirrors
+ # ``with``); the outcome is still classified from commit_exc below.
+ if entered:
+ exit_exc_info = (
+ (type(commit_exc), commit_exc, commit_exc.__traceback__)
+ if commit_exc is not None else (None, None, None)
+ )
+ try:
+ self.snapshot_commit.__exit__(*exit_exc_info)
+ except Exception:
logger.warning(
- "Atomic commit failed for snapshot #%d by user %s "
- "with identifier %s and kind %s after %.0f seconds. Try again.",
- new_snapshot_id,
- self.commit_user,
- commit_identifier,
- commit_kind,
- commit_time_s,
+ "Failed to close snapshot commit; ignoring because it "
+ "must not override the commit outcome.",
+ exc_info=True,
)
- return RetryResult(latest_snapshot, None, base_data_files=base_data_files)
- except Exception as e:
+
+ if commit_exc is not None:
+ if _is_deterministic_atomic_commit_failure(commit_exc):
+ logger.warning(
+ "Atomic commit was rejected deterministically; do not retry.",
+ exc_info=commit_exc,
+ )
+ clean_up_rejected_commit()
+ raise commit_exc
Review Comment:
Once this branch classifies the rejection as deterministic, the staged `CommitMessage` files are safe to abort. Rethrowing the raw exception loses that signal: `TableCommit._commit` only aborts on `CommitConflictError`, while `merge_into` and `RayDataSink` stop aborting after commit starts, so 400/403/404 rejections can leave data, changelog, and index files orphaned. Please propagate a safe-to-abort rejection type or marker to the layer that owns the messages, while preserving the original cause, and add a caller-level test showing that no manual `abort()` is required.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [python][ray] Incrementally commit update_by_row_id file groups [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on PR #8826:
URL: https://github.com/apache/paimon/pull/8826#issuecomment-5119240596
> Incremental commits are not atomic across the whole operation. Ordinary exceptions
raised while applying a group are reported as results so other groups can finish;
completed groups are flushed before the error is raised.
> Ray task retry options in `ray_remote_args` do not retry group exceptions reported
as results, so a transient group failure can leave a partial update. Worker loss and
other task-level failures which bypass Python exception handling may still lose
results buffered by that task.
If this is indeed the case, as stated in your documentation, then the implications are significant—far beyond what a simple description like “max_groups_per_commit” can capture. This is a major semantic change that has been overlooked.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [python][ray] Add file-group-aligned row-id range processing [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on PR #8826:
URL: https://github.com/apache/paimon/pull/8826#issuecomment-5213515512
I think this can be simplified further by keeping row-id range planning as a passive planning primitive, instead of making `RowIdRangeContext` own the read/update/skip sequence and commit lifecycle.
The existing row-id update path already carries the information needed by the normal commit conflict detector:
- `CommitMessage.check_from_snapshot` identifies the snapshot from which the update was planned.
- The new files in the message carry `first_row_id`, `row_count`, and `write_cols`.
- `FileStoreCommit` can therefore derive row-range × write-column conflicts and detect missing or replaced anchor files from the ordinary commit messages.
The important requirement is that the writer must build its `CommitMessage` from the same pinned snapshot/file information used by the range planner. It must not re-plan against the latest snapshot after the user has read an older pinned snapshot. This is a writer/planner concern; it does not require `commit()` to accept additional range arguments.
A simpler API shape would be:
```python
with plan_row_id_ranges(...) as plan:
for row_range in plan:
source = build_read(
snapshot_id=row_range.snapshot_id,
row_id_range=row_range,
)
updates = user_process(source)
update_by_row_id_from_plan(row_range, updates, update_cols)
```
Here, `RowIdRange` can be immutable metadata (plus an opaque internal planning token if needed for efficient file selection). The user controls read and processing, while the update writer creates the same normal `CommitMessage` that existing row-id updates already use.
The current `protected_columns` and context-side read tracking introduce a stronger semantic contract. For example:
```text
read text -> compute embedding -> write embedding
```
If another writer changes `text`, the existing row-id update semantics allow the `embedding` write because the actual write columns are disjoint. Tracking `text` through the context changes this into a transform/read-dependency conflict. That may be useful, but it is a new consistency contract rather than something required by row-id range commits themselves. If we need that contract, I think it should be specified and justified independently, and the dependency columns could be supplied explicitly rather than inferred through a stateful context.
If we retain the existing row-id update conflict semantics for the first version, we should be able to remove most of the operation-specific machinery:
- context-side `_read_columns`, `_updated`, `skip()`, and sequence state;
- the operation checkpoint and exact-snapshot return path;
- `protect_from_external_rewrites` state added to the generic commit layer.
The range planner, pinned-snapshot tag, file-group-aligned packing, and normal `CommitMessage` conflict detection should be sufficient. This keeps the abstraction focused: planning returns ranges, reading reads a pinned range, and committing commits ordinary row-id update messages.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [python][ray] Incrementally commit update_by_row_id file groups [paimon]
Posted by "XiaoHongbo-Hope (via GitHub)" <gi...@apache.org>.
XiaoHongbo-Hope commented on code in PR #8826:
URL: https://github.com/apache/paimon/pull/8826#discussion_r3656742524
##########
paimon-python/pypaimon/write/file_store_commit.py:
##########
@@ -612,26 +759,84 @@ def _try_commit_once(self, retry_result: Optional[RetryResult], commit_kind: str
logger.warning(f"Exception occurs when preparing snapshot: {e}", exc_info=True)
raise RuntimeError(f"Failed to prepare snapshot: {e}")
- # Use SnapshotCommit for atomic commit
+ def clean_up_rejected_commit():
+ try:
+ self._clean_up_reuse_tmp_manifests(
+ delta_manifest_list,
+ changelog_manifest_list_name,
+ new_index_manifest,
+ )
+ self._clean_up_no_reuse_tmp_manifests(
+ base_manifest_list, merge_new_files)
+ except Exception:
+ logger.warning(
+ "Failed to clean up rejected commit manifests.",
+ exc_info=True,
+ )
+
+ # Mirror a ``with`` block but drive it manually so only commit() decides
+ # the outcome. __enter__ and commit() share one capture, so an __enter__
+ # failure flows through the classification below instead of bypassing it.
+ # __exit__ gets the real commit exception triple (for extension rollback)
+ # but its own failure is only logged -- never allowed to override the
+ # commit outcome (a lost/landed commit must not be misclassified).
+ success = None
+ commit_exc = None
+ entered = False
try:
- with self.snapshot_commit:
- success = self.snapshot_commit.commit(snapshot_data, statistics)
- if not success:
- commit_time_s = (int(time.time() * 1000) - start_millis) / 1000
+ self.snapshot_commit.__enter__()
+ entered = True
+ success = self.snapshot_commit.commit(snapshot_data, statistics)
+ except Exception as e:
+ commit_exc = e
+ finally:
+ # Never call __exit__ without a successful __enter__ (mirrors
+ # ``with``); the outcome is still classified from commit_exc below.
+ if entered:
+ exit_exc_info = (
+ (type(commit_exc), commit_exc, commit_exc.__traceback__)
+ if commit_exc is not None else (None, None, None)
+ )
+ try:
+ self.snapshot_commit.__exit__(*exit_exc_info)
+ except Exception:
logger.warning(
- "Atomic commit failed for snapshot #%d by user %s "
- "with identifier %s and kind %s after %.0f seconds. Try again.",
- new_snapshot_id,
- self.commit_user,
- commit_identifier,
- commit_kind,
- commit_time_s,
+ "Failed to close snapshot commit; ignoring because it "
+ "must not override the commit outcome.",
+ exc_info=True,
)
- return RetryResult(latest_snapshot, None, base_data_files=base_data_files)
- except Exception as e:
+
+ if commit_exc is not None:
+ if _is_deterministic_atomic_commit_failure(commit_exc):
+ logger.warning(
+ "Atomic commit was rejected deterministically; do not retry.",
+ exc_info=commit_exc,
+ )
+ clean_up_rejected_commit()
+ raise commit_exc
Review Comment:
> Once this branch classifies the rejection as deterministic, the staged `CommitMessage` files are safe to abort. Rethrowing the raw exception loses that signal: `TableCommit._commit` only aborts on `CommitConflictError`, while `merge_into` and `RayDataSink` stop aborting after commit starts, so 400/403/404 rejections can leave data, changelog, and index files orphaned. Please propagate a safe-to-abort rejection type or marker to the layer that owns the messages, while preserving the original cause, and add a caller-level test showing that no manual `abort()` is required.
Fixed
##########
paimon-python/pypaimon/ray/data_evolution_merge_join.py:
##########
@@ -481,7 +537,10 @@ def distributed_update_apply(
)
sorted_first_row_ids = list(planner.first_row_ids)
if not sorted_first_row_ids:
- return [], 0, []
+ # No target file groups (e.g. an existing but empty snapshot). Match the
+ # 4-tuple contract so callers that unpack (msgs, num, row_ids, err) --
+ # including merge_into's NOT MATCHED insert path -- don't crash.
+ return [], 0, [], None
Review Comment:
> Returning a 4-tuple fixes the unpacking crash, but it does not make an existing-but-empty snapshot follow the empty-target path. `_build_datasets` still passes `target_empty=base_snapshot is None`; for an empty snapshot, `base_snapshot` is non-null, so the NOT MATCHED insert path builds a left-anti join against a zero-block target. The new test fails on Python 3.10 and 3.11 with `ArrowInvalid`. Please also treat `base_snapshot.total_record_count == 0` as empty, and skip the matched branches, so this regression passes the supported CI matrix.
Fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [python][ray] Batch row-id update pipelines by row count [paimon]
Posted by "leaves12138 (via GitHub)" <gi...@apache.org>.
leaves12138 commented on code in PR #8826:
URL: https://github.com/apache/paimon/pull/8826#discussion_r3729372480
##########
paimon-python/pypaimon/ray/data_evolution_merge_join.py:
##########
@@ -503,112 +765,248 @@ def distributed_update_apply(
precomputed_info_ref = ray.put(files_info)
frid_col = "_FIRST_ROW_ID"
+ range_col = "_ROW_ID_RANGE_START"
captured_sorted = sorted_first_row_ids
captured_sorted_arr = np.asarray(captured_sorted, dtype=np.int64)
valid_ranges = planner.valid_row_id_ranges
range_starts = np.asarray([r.from_ for r in valid_ranges], dtype=np.int64)
range_ends = np.asarray([r.to for r in valid_ranges], dtype=np.int64)
+ row_id_range_starts = (
+ _row_id_range_starts(files_info, rows_per_range)
+ if rows_per_range is not None else None
+ )
def _assign_frid(batch: pa.Table) -> pa.Table:
if batch.num_rows == 0:
- return batch.append_column(
- frid_col, pa.array([], type=pa.int64())
- )
+ result = batch.append_column(
+ frid_col, pa.array([], type=pa.int64()))
+ if row_id_range_starts is None:
+ return result
+ return result.append_column(
+ range_col, pa.array([], type=pa.int64()))
rid_col = batch.column(row_id_name)
if rid_col.null_count:
raise ValueError(
"_ROW_ID is null; planner snapshot is stale "
"or matched rows come from a different table."
)
rids = rid_col.to_numpy(zero_copy_only=False)
- # Check each row_id belongs to a valid range (vectorized).
in_range = np.zeros(len(rids), dtype=bool)
- for s, e in zip(range_starts, range_ends):
- in_range |= (rids >= s) & (rids <= e)
+ for start, end in zip(range_starts, range_ends):
+ in_range |= (rids >= start) & (rids <= end)
if not in_range.all():
bad = rids[~in_range][0]
raise ValueError(
f"_ROW_ID {bad} does not belong to any valid range "
f"{[f'[{r.from_}, {r.to}]' for r in valid_ranges]}; "
- f"planner snapshot is stale or matched rows come "
- f"from a different table."
+ "planner snapshot is stale or row ids come from another table."
)
- idx = np.searchsorted(
- captured_sorted_arr, rids, side="right"
- ) - 1
- frids = captured_sorted_arr[idx]
- return batch.append_column(
- frid_col, pa.array(frids, type=pa.int64())
+ indexes = np.searchsorted(
+ captured_sorted_arr, rids, side="right") - 1
+ result = batch.append_column(
+ frid_col,
+ pa.array(captured_sorted_arr[indexes], type=pa.int64()))
+ if row_id_range_starts is None:
+ return result
+ return result.append_column(
+ range_col,
+ pa.array(row_id_range_starts[indexes], type=pa.int64()),
)
- map_kwargs = _map_kwargs(ray_remote_args)
- with_frid = update_ds.map_batches(_assign_frid, **map_kwargs)
+ worker_map_kwargs = _map_kwargs(ray_remote_args)
+ if transform is not None:
+ with_frid = ray.data.from_arrow(pa.table({
+ frid_col: pa.array(captured_sorted, type=pa.int64()),
+ range_col: pa.array(row_id_range_starts, type=pa.int64()),
+ }))
+ else:
+ with_frid = update_ds.map_batches(
+ _assign_frid, **worker_map_kwargs)
captured_table = table
captured_cols = cols
+ capture_group_errors = on_group_result is not None
def _apply_group(group: pa.Table) -> pa.Table:
- if group.num_rows == 0:
- return pa.Table.from_pydict({
- "msgs_blob": pa.array([], type=pa.binary()),
- "n_updated": pa.array([], type=pa.int64()),
- "row_ids_blob": pa.array([], type=pa.binary()),
- })
-
- if (
- pc.count_distinct(group.column(row_id_name)).as_py()
- != group.num_rows
- ):
- raise ValueError(
- "MERGE matched multiple source rows to the same "
- "target _ROW_ID. Deduplicate the source before "
- "merging."
+ worker = None
+ try:
+ if group.num_rows == 0:
+ return pa.Table.from_pydict({
+ "msgs_blob": pa.array([], type=pa.binary()),
+ "n_updated": pa.array([], type=pa.int64()),
+ "row_ids_blob": pa.array([], type=pa.binary()),
+ "error": pa.array([], type=pa.string()),
+ })
+
+ if (pc.count_distinct(group.column(row_id_name)).as_py()
+ != group.num_rows):
+ raise ValueError(
+ "MERGE matched multiple source rows to the same "
+ "target _ROW_ID. Deduplicate the source before merging.")
+
+ routing_columns = [frid_col]
+ if row_id_range_starts is not None:
+ routing_columns.append(range_col)
+ for_update = group.drop_columns(routing_columns)
+ row_ids = (
+ for_update.column(row_id_name).to_pylist()
+ if collect_row_ids else []
+ )
+ worker = TableUpdateByRowId(
+ captured_table,
+ "_merge_into_shard_" + uuid.uuid4().hex[:8],
+ BATCH_COMMIT_IDENTIFIER,
+ _precomputed_files_info=ray.get(precomputed_info_ref),
+ )
+ if capture_group_errors:
+ return _write_group_result(
+ captured_table, worker, for_update, captured_cols, row_ids)
+ messages = worker.update_columns(for_update, captured_cols)
+ return _group_result(messages, for_update.num_rows, row_ids)
+ except Exception as error:
+ if capture_group_errors:
+ return _failed_group_result(captured_table, worker, error)
+ raise
+
+ group_col = frid_col
+ num_groups = len(captured_sorted)
+ if row_id_range_starts is not None:
+ group_col = range_col
+ num_groups = len(set(row_id_range_starts.tolist()))
+ group_partitions = max(1, min(num_groups, num_partitions))
+ grouped = with_frid.groupby(
+ group_col, num_partitions=group_partitions)
+ if transform is None:
+ msgs_ds = grouped.map_groups(_apply_group, **worker_map_kwargs)
+ else:
+ retry_options = ray_remote_args or {}
+ worker_options = {
+ "table": scan_table,
Review Comment:
This prevents the documented new-column backfill use case. `scan_table` is pinned to `base_snapshot_id`, so after adding `embedding` its schema still lacks that column. `update_by_transform(..., update_cols=["embedding"])` then fails in `TableUpdateByRowId.update_columns` with `ValueError: Column embedding not found in table schema`. Please use separate tables: the pinned table for reading, and the latest-schema `table` for writing, while keeping the precomputed base-snapshot file metadata for row-id routing/conflict checks.
##########
paimon-python/pypaimon/ray/update_by_transform.py:
##########
@@ -0,0 +1,304 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""Distributed transform updates on Ray for data-evolution tables."""
+
+import logging
+import uuid
+from typing import Any, Callable, Dict, List, Optional, Union
+
+from pypaimon.ray.data_evolution_merge_into import (
+ _reraise_inner,
+ _require_ray_join,
+ _resolve_num_partitions,
+)
+from pypaimon.ray.data_evolution_merge_join import (
+ GroupApplyError,
+ distributed_update_apply,
+)
+from pypaimon.ray.data_evolution_merge_transform import build_update_schema
+from pypaimon.ray.update_by_row_id import (
+ _abort_pending_update_messages,
+ _blob_col_names,
+)
+
+__all__ = ["update_by_transform"]
+
+logger = logging.getLogger(__name__)
+
+
+def _positive_int(name, value):
+ if (isinstance(value, bool)
+ or not isinstance(value, int)
+ or value <= 0):
+ raise ValueError("{} must be a positive integer.".format(name))
+
+
+def _prepare_transform_update(
+ target,
+ catalog_options,
+ read_projection,
+ transform,
+ update_cols,
+ rows_per_commit,
+ transform_filter,
+ transform_batch_size):
+ from pypaimon.catalog.catalog_factory import CatalogFactory
+ from pypaimon.common.where_parser import parse_where_clause
+ from pypaimon.schema.data_types import PyarrowFieldParser
+ from pypaimon.table.special_fields import SpecialFields
+
+ _positive_int("rows_per_commit", rows_per_commit)
+ _positive_int("transform_batch_size", transform_batch_size)
+ if not callable(transform):
+ raise ValueError("transform must be callable.")
+ if not read_projection:
+ raise ValueError("read_projection must be non-empty.")
+ if not update_cols:
+ raise ValueError("update_cols must be non-empty.")
+ if (transform_filter is not None
+ and not isinstance(transform_filter, str)
+ and not callable(transform_filter)):
+ raise ValueError("filter must be a SQL expression string or callable.")
+
+ update_cols = list(dict.fromkeys(update_cols))
+ read_cols = list(dict.fromkeys(read_projection))
+ table = CatalogFactory.create(catalog_options).get_table(target)
+ if table.is_primary_key_table:
+ raise ValueError(
+ "update_by_transform requires a non-primary-key table.")
+ if not table.options.data_evolution_enabled():
+ raise ValueError(
+ f"update_by_transform requires 'data-evolution.enabled'='true' "
+ f"on '{target}'.")
+ if not table.options.row_tracking_enabled():
+ raise ValueError(
+ f"update_by_transform requires 'row-tracking.enabled'='true' "
+ f"on '{target}'.")
+ if table.options.deletion_vectors_enabled():
+ raise ValueError(
+ "update_by_transform does not support deletion-vectors-enabled "
+ f"tables yet: '{target}'.")
+
+ rid = SpecialFields.ROW_ID.name
+ if rid in read_cols:
+ raise ValueError(
+ "update_by_transform keeps _ROW_ID internal; remove it from "
+ "read_projection.")
+ unknown = [col for col in read_cols if col not in table.field_names]
+ if unknown:
+ raise ValueError(
+ f"read column {unknown[0]!r} is not in target '{target}'.")
+ blob_cols = _blob_col_names(table)
+ partition_keys = set(table.partition_keys or [])
+ for col in update_cols:
+ if col not in table.field_names:
+ raise ValueError(
+ f"update column {col!r} is not in target '{target}'.")
+ if col in blob_cols:
+ raise ValueError(
+ f"update_by_transform cannot update blob column {col!r}.")
+ if col in partition_keys:
+ raise ValueError(
+ f"update_by_transform cannot update partition column {col!r}.")
+
+ target_pa = PyarrowFieldParser.from_paimon_schema(
+ table.table_schema.fields)
+ update_schema = build_update_schema(target_pa, update_cols, rid)
+ predicate = (
+ parse_where_clause(transform_filter, table.table_schema.fields)
+ if isinstance(transform_filter, str) else None
+ )
+ filter_fn = transform_filter if callable(transform_filter) else None
+ return table, read_cols, update_cols, update_schema, predicate, filter_fn
+
+
+def update_by_transform(
+ target: str,
+ catalog_options: Dict[str, str],
+ *,
+ read_projection: List[str],
+ transform: Callable,
+ update_cols: List[str],
+ rows_per_commit: int,
+ filter: Optional[Union[str, Callable]] = None,
+ num_partitions: Optional[int] = None,
+ ray_remote_args: Optional[Dict[str, Any]] = None,
+ transform_batch_size: int = 1024,
+) -> Dict[str, int]:
+ """Transform matching rows and commit continuous row-id ranges.
+
+ The transform receives ``read_projection`` and returns ``update_cols``
+ with the same row count and order. ``filter`` defaults to the full target.
+ Row ids remain internal to this operation.
+
+ Returns ``{"num_updated": <rows>}``.
+ """
+ _require_ray_join()
+ (table, read_cols, update_cols, update_schema,
+ predicate, filter_fn) = _prepare_transform_update(
+ target,
+ catalog_options,
+ read_projection,
+ transform,
+ update_cols,
+ rows_per_commit,
+ filter,
+ transform_batch_size,
+ )
+ num_partitions = _resolve_num_partitions(num_partitions)
+ base = table.snapshot_manager().get_latest_snapshot()
+ if base is None or base.total_record_count == 0:
+ return {"num_updated": 0}
+
+ retention_tag = "pypaimon-transform-update-{}".format(uuid.uuid4().hex)
+ table.create_tag(retention_tag, snapshot_id=base.id, time_retained="30d")
+ committer = _IncrementalUpdateCommitter(
+ table, base, table.table_schema.id)
+ try:
+ _, num_updated, _ = distributed_update_apply(
+ None,
+ table,
+ update_cols,
+ num_partitions=num_partitions,
+ ray_remote_args=ray_remote_args,
+ base_snapshot_id=base.id,
+ on_group_result=committer.add_range,
+ rows_per_range=rows_per_commit,
+ read_projection=read_cols,
+ transform=transform,
+ transform_filter=filter_fn,
+ transform_predicate=predicate,
+ transform_update_schema=update_schema,
+ transform_batch_size=transform_batch_size,
+ )
+ committer.finish()
+ except GroupApplyError:
+ committer.finish()
+ raise
+ except Exception as error:
+ if committer.failed:
+ raise
+ _reraise_inner(error)
+ raise
+ finally:
+ committer.close()
+ try:
+ table.delete_tag(retention_tag)
+ except Exception as error:
+ logger.warning(
+ "Failed to delete transform retention tag %s: %s",
+ retention_tag,
+ error,
+ exc_info=error,
+ )
+ return {"num_updated": num_updated}
+
+
+class _IncrementalUpdateCommitter:
+
+ def __init__(self, table, base_snapshot=None, planned_schema_id=None):
+ self._table = table
+ self._table_commit = None
+ self._snapshot_callback = None
+ self._commit_user = None
+ self._checkpoint_snapshot = base_snapshot
+ self._planned_schema_id = planned_schema_id
+ self._next_commit_identifier = 1
+ self._deferred_commit_error = None
+
+ @property
+ def failed(self) -> bool:
+ return self._deferred_commit_error is not None
+
+ def add_range(self, commit_messages, _num_updated, _row_ids) -> None:
+ if self.failed:
+ _abort_pending_update_messages(self._table, commit_messages)
+ return
+ try:
+ self._commit(commit_messages)
+ except Exception as error:
+ self._deferred_commit_error = error
+
+ def finish(self) -> None:
+ if self.failed:
+ raise self._deferred_commit_error
+
+ def _commit(self, messages) -> None:
+ if not messages:
+ return
+ if self._table_commit is None:
+ builder = self._table.new_stream_write_builder()
+ if self._checkpoint_snapshot is not None:
+ self._commit_user = builder.commit_user
+ self._table_commit = builder.new_commit()
+ self._snapshot_callback = _SnapshotCallback()
+ self._table_commit.add_commit_callback(self._snapshot_callback)
+
+ identifier = self._next_commit_identifier
+ if self._checkpoint_snapshot is not None:
+ self._table_commit.protect_from_external_rewrites(
Review Comment:
The external-rewrite guard does not protect transform dependencies. Conflict detection only sees the staged `update_cols`, so an overlapping concurrent row-id update to a `read_projection` or filter column is accepted. Repro: read `name="a"`, derive `age=len(name)`, concurrently update `name` to `"long"` before this commit; the operation succeeds with `name="long", age=1` instead of `age=4`. Please include read/filter dependencies in the overlapping-row conflict set, or conservatively reject all concurrent row updates during an incremental transform.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [python][ray] Support batched transform updates [paimon]
Posted by "XiaoHongbo-Hope (via GitHub)" <gi...@apache.org>.
XiaoHongbo-Hope commented on PR #8826:
URL: https://github.com/apache/paimon/pull/8826#issuecomment-5205903208
Addressed the latest correctness review in 7cb26cd46:
- Snapshot pinning now uses `copy_without_time_travel()`. Workers read the pinned snapshot and write with the planned current schema, so a newly added column can be backfilled before any new data snapshot exists.
- `read_projection` and SQL filter fields are protected as read dependencies. Concurrent updates to those fields on overlapping row ranges now fail conflict detection.
Added real-Ray regressions for the new-column case and both read/filter dependency conflicts. Focused tests and lint pass.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [python][ray] Batch row-id update pipelines by row count [paimon]
Posted by "leaves12138 (via GitHub)" <gi...@apache.org>.
leaves12138 commented on code in PR #8826:
URL: https://github.com/apache/paimon/pull/8826#discussion_r3729372480
##########
paimon-python/pypaimon/ray/data_evolution_merge_join.py:
##########
@@ -503,112 +765,248 @@ def distributed_update_apply(
precomputed_info_ref = ray.put(files_info)
frid_col = "_FIRST_ROW_ID"
+ range_col = "_ROW_ID_RANGE_START"
captured_sorted = sorted_first_row_ids
captured_sorted_arr = np.asarray(captured_sorted, dtype=np.int64)
valid_ranges = planner.valid_row_id_ranges
range_starts = np.asarray([r.from_ for r in valid_ranges], dtype=np.int64)
range_ends = np.asarray([r.to for r in valid_ranges], dtype=np.int64)
+ row_id_range_starts = (
+ _row_id_range_starts(files_info, rows_per_range)
+ if rows_per_range is not None else None
+ )
def _assign_frid(batch: pa.Table) -> pa.Table:
if batch.num_rows == 0:
- return batch.append_column(
- frid_col, pa.array([], type=pa.int64())
- )
+ result = batch.append_column(
+ frid_col, pa.array([], type=pa.int64()))
+ if row_id_range_starts is None:
+ return result
+ return result.append_column(
+ range_col, pa.array([], type=pa.int64()))
rid_col = batch.column(row_id_name)
if rid_col.null_count:
raise ValueError(
"_ROW_ID is null; planner snapshot is stale "
"or matched rows come from a different table."
)
rids = rid_col.to_numpy(zero_copy_only=False)
- # Check each row_id belongs to a valid range (vectorized).
in_range = np.zeros(len(rids), dtype=bool)
- for s, e in zip(range_starts, range_ends):
- in_range |= (rids >= s) & (rids <= e)
+ for start, end in zip(range_starts, range_ends):
+ in_range |= (rids >= start) & (rids <= end)
if not in_range.all():
bad = rids[~in_range][0]
raise ValueError(
f"_ROW_ID {bad} does not belong to any valid range "
f"{[f'[{r.from_}, {r.to}]' for r in valid_ranges]}; "
- f"planner snapshot is stale or matched rows come "
- f"from a different table."
+ "planner snapshot is stale or row ids come from another table."
)
- idx = np.searchsorted(
- captured_sorted_arr, rids, side="right"
- ) - 1
- frids = captured_sorted_arr[idx]
- return batch.append_column(
- frid_col, pa.array(frids, type=pa.int64())
+ indexes = np.searchsorted(
+ captured_sorted_arr, rids, side="right") - 1
+ result = batch.append_column(
+ frid_col,
+ pa.array(captured_sorted_arr[indexes], type=pa.int64()))
+ if row_id_range_starts is None:
+ return result
+ return result.append_column(
+ range_col,
+ pa.array(row_id_range_starts[indexes], type=pa.int64()),
)
- map_kwargs = _map_kwargs(ray_remote_args)
- with_frid = update_ds.map_batches(_assign_frid, **map_kwargs)
+ worker_map_kwargs = _map_kwargs(ray_remote_args)
+ if transform is not None:
+ with_frid = ray.data.from_arrow(pa.table({
+ frid_col: pa.array(captured_sorted, type=pa.int64()),
+ range_col: pa.array(row_id_range_starts, type=pa.int64()),
+ }))
+ else:
+ with_frid = update_ds.map_batches(
+ _assign_frid, **worker_map_kwargs)
captured_table = table
captured_cols = cols
+ capture_group_errors = on_group_result is not None
def _apply_group(group: pa.Table) -> pa.Table:
- if group.num_rows == 0:
- return pa.Table.from_pydict({
- "msgs_blob": pa.array([], type=pa.binary()),
- "n_updated": pa.array([], type=pa.int64()),
- "row_ids_blob": pa.array([], type=pa.binary()),
- })
-
- if (
- pc.count_distinct(group.column(row_id_name)).as_py()
- != group.num_rows
- ):
- raise ValueError(
- "MERGE matched multiple source rows to the same "
- "target _ROW_ID. Deduplicate the source before "
- "merging."
+ worker = None
+ try:
+ if group.num_rows == 0:
+ return pa.Table.from_pydict({
+ "msgs_blob": pa.array([], type=pa.binary()),
+ "n_updated": pa.array([], type=pa.int64()),
+ "row_ids_blob": pa.array([], type=pa.binary()),
+ "error": pa.array([], type=pa.string()),
+ })
+
+ if (pc.count_distinct(group.column(row_id_name)).as_py()
+ != group.num_rows):
+ raise ValueError(
+ "MERGE matched multiple source rows to the same "
+ "target _ROW_ID. Deduplicate the source before merging.")
+
+ routing_columns = [frid_col]
+ if row_id_range_starts is not None:
+ routing_columns.append(range_col)
+ for_update = group.drop_columns(routing_columns)
+ row_ids = (
+ for_update.column(row_id_name).to_pylist()
+ if collect_row_ids else []
+ )
+ worker = TableUpdateByRowId(
+ captured_table,
+ "_merge_into_shard_" + uuid.uuid4().hex[:8],
+ BATCH_COMMIT_IDENTIFIER,
+ _precomputed_files_info=ray.get(precomputed_info_ref),
+ )
+ if capture_group_errors:
+ return _write_group_result(
+ captured_table, worker, for_update, captured_cols, row_ids)
+ messages = worker.update_columns(for_update, captured_cols)
+ return _group_result(messages, for_update.num_rows, row_ids)
+ except Exception as error:
+ if capture_group_errors:
+ return _failed_group_result(captured_table, worker, error)
+ raise
+
+ group_col = frid_col
+ num_groups = len(captured_sorted)
+ if row_id_range_starts is not None:
+ group_col = range_col
+ num_groups = len(set(row_id_range_starts.tolist()))
+ group_partitions = max(1, min(num_groups, num_partitions))
+ grouped = with_frid.groupby(
+ group_col, num_partitions=group_partitions)
+ if transform is None:
+ msgs_ds = grouped.map_groups(_apply_group, **worker_map_kwargs)
+ else:
+ retry_options = ray_remote_args or {}
+ worker_options = {
+ "table": scan_table,
Review Comment:
This prevents the documented new-column backfill use case. `scan_table` is pinned to `base_snapshot_id`, so after adding `embedding` its schema still lacks that column. `update_by_transform(..., update_cols=["embedding"])` then fails in `TableUpdateByRowId.update_columns` with `ValueError: Column embedding not found in table schema`. Please use separate tables: the pinned table for reading, and the latest-schema `table` for writing, while keeping the precomputed base-snapshot file metadata for row-id routing/conflict checks.
##########
paimon-python/pypaimon/ray/update_by_transform.py:
##########
@@ -0,0 +1,304 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""Distributed transform updates on Ray for data-evolution tables."""
+
+import logging
+import uuid
+from typing import Any, Callable, Dict, List, Optional, Union
+
+from pypaimon.ray.data_evolution_merge_into import (
+ _reraise_inner,
+ _require_ray_join,
+ _resolve_num_partitions,
+)
+from pypaimon.ray.data_evolution_merge_join import (
+ GroupApplyError,
+ distributed_update_apply,
+)
+from pypaimon.ray.data_evolution_merge_transform import build_update_schema
+from pypaimon.ray.update_by_row_id import (
+ _abort_pending_update_messages,
+ _blob_col_names,
+)
+
+__all__ = ["update_by_transform"]
+
+logger = logging.getLogger(__name__)
+
+
+def _positive_int(name, value):
+ if (isinstance(value, bool)
+ or not isinstance(value, int)
+ or value <= 0):
+ raise ValueError("{} must be a positive integer.".format(name))
+
+
+def _prepare_transform_update(
+ target,
+ catalog_options,
+ read_projection,
+ transform,
+ update_cols,
+ rows_per_commit,
+ transform_filter,
+ transform_batch_size):
+ from pypaimon.catalog.catalog_factory import CatalogFactory
+ from pypaimon.common.where_parser import parse_where_clause
+ from pypaimon.schema.data_types import PyarrowFieldParser
+ from pypaimon.table.special_fields import SpecialFields
+
+ _positive_int("rows_per_commit", rows_per_commit)
+ _positive_int("transform_batch_size", transform_batch_size)
+ if not callable(transform):
+ raise ValueError("transform must be callable.")
+ if not read_projection:
+ raise ValueError("read_projection must be non-empty.")
+ if not update_cols:
+ raise ValueError("update_cols must be non-empty.")
+ if (transform_filter is not None
+ and not isinstance(transform_filter, str)
+ and not callable(transform_filter)):
+ raise ValueError("filter must be a SQL expression string or callable.")
+
+ update_cols = list(dict.fromkeys(update_cols))
+ read_cols = list(dict.fromkeys(read_projection))
+ table = CatalogFactory.create(catalog_options).get_table(target)
+ if table.is_primary_key_table:
+ raise ValueError(
+ "update_by_transform requires a non-primary-key table.")
+ if not table.options.data_evolution_enabled():
+ raise ValueError(
+ f"update_by_transform requires 'data-evolution.enabled'='true' "
+ f"on '{target}'.")
+ if not table.options.row_tracking_enabled():
+ raise ValueError(
+ f"update_by_transform requires 'row-tracking.enabled'='true' "
+ f"on '{target}'.")
+ if table.options.deletion_vectors_enabled():
+ raise ValueError(
+ "update_by_transform does not support deletion-vectors-enabled "
+ f"tables yet: '{target}'.")
+
+ rid = SpecialFields.ROW_ID.name
+ if rid in read_cols:
+ raise ValueError(
+ "update_by_transform keeps _ROW_ID internal; remove it from "
+ "read_projection.")
+ unknown = [col for col in read_cols if col not in table.field_names]
+ if unknown:
+ raise ValueError(
+ f"read column {unknown[0]!r} is not in target '{target}'.")
+ blob_cols = _blob_col_names(table)
+ partition_keys = set(table.partition_keys or [])
+ for col in update_cols:
+ if col not in table.field_names:
+ raise ValueError(
+ f"update column {col!r} is not in target '{target}'.")
+ if col in blob_cols:
+ raise ValueError(
+ f"update_by_transform cannot update blob column {col!r}.")
+ if col in partition_keys:
+ raise ValueError(
+ f"update_by_transform cannot update partition column {col!r}.")
+
+ target_pa = PyarrowFieldParser.from_paimon_schema(
+ table.table_schema.fields)
+ update_schema = build_update_schema(target_pa, update_cols, rid)
+ predicate = (
+ parse_where_clause(transform_filter, table.table_schema.fields)
+ if isinstance(transform_filter, str) else None
+ )
+ filter_fn = transform_filter if callable(transform_filter) else None
+ return table, read_cols, update_cols, update_schema, predicate, filter_fn
+
+
+def update_by_transform(
+ target: str,
+ catalog_options: Dict[str, str],
+ *,
+ read_projection: List[str],
+ transform: Callable,
+ update_cols: List[str],
+ rows_per_commit: int,
+ filter: Optional[Union[str, Callable]] = None,
+ num_partitions: Optional[int] = None,
+ ray_remote_args: Optional[Dict[str, Any]] = None,
+ transform_batch_size: int = 1024,
+) -> Dict[str, int]:
+ """Transform matching rows and commit continuous row-id ranges.
+
+ The transform receives ``read_projection`` and returns ``update_cols``
+ with the same row count and order. ``filter`` defaults to the full target.
+ Row ids remain internal to this operation.
+
+ Returns ``{"num_updated": <rows>}``.
+ """
+ _require_ray_join()
+ (table, read_cols, update_cols, update_schema,
+ predicate, filter_fn) = _prepare_transform_update(
+ target,
+ catalog_options,
+ read_projection,
+ transform,
+ update_cols,
+ rows_per_commit,
+ filter,
+ transform_batch_size,
+ )
+ num_partitions = _resolve_num_partitions(num_partitions)
+ base = table.snapshot_manager().get_latest_snapshot()
+ if base is None or base.total_record_count == 0:
+ return {"num_updated": 0}
+
+ retention_tag = "pypaimon-transform-update-{}".format(uuid.uuid4().hex)
+ table.create_tag(retention_tag, snapshot_id=base.id, time_retained="30d")
+ committer = _IncrementalUpdateCommitter(
+ table, base, table.table_schema.id)
+ try:
+ _, num_updated, _ = distributed_update_apply(
+ None,
+ table,
+ update_cols,
+ num_partitions=num_partitions,
+ ray_remote_args=ray_remote_args,
+ base_snapshot_id=base.id,
+ on_group_result=committer.add_range,
+ rows_per_range=rows_per_commit,
+ read_projection=read_cols,
+ transform=transform,
+ transform_filter=filter_fn,
+ transform_predicate=predicate,
+ transform_update_schema=update_schema,
+ transform_batch_size=transform_batch_size,
+ )
+ committer.finish()
+ except GroupApplyError:
+ committer.finish()
+ raise
+ except Exception as error:
+ if committer.failed:
+ raise
+ _reraise_inner(error)
+ raise
+ finally:
+ committer.close()
+ try:
+ table.delete_tag(retention_tag)
+ except Exception as error:
+ logger.warning(
+ "Failed to delete transform retention tag %s: %s",
+ retention_tag,
+ error,
+ exc_info=error,
+ )
+ return {"num_updated": num_updated}
+
+
+class _IncrementalUpdateCommitter:
+
+ def __init__(self, table, base_snapshot=None, planned_schema_id=None):
+ self._table = table
+ self._table_commit = None
+ self._snapshot_callback = None
+ self._commit_user = None
+ self._checkpoint_snapshot = base_snapshot
+ self._planned_schema_id = planned_schema_id
+ self._next_commit_identifier = 1
+ self._deferred_commit_error = None
+
+ @property
+ def failed(self) -> bool:
+ return self._deferred_commit_error is not None
+
+ def add_range(self, commit_messages, _num_updated, _row_ids) -> None:
+ if self.failed:
+ _abort_pending_update_messages(self._table, commit_messages)
+ return
+ try:
+ self._commit(commit_messages)
+ except Exception as error:
+ self._deferred_commit_error = error
+
+ def finish(self) -> None:
+ if self.failed:
+ raise self._deferred_commit_error
+
+ def _commit(self, messages) -> None:
+ if not messages:
+ return
+ if self._table_commit is None:
+ builder = self._table.new_stream_write_builder()
+ if self._checkpoint_snapshot is not None:
+ self._commit_user = builder.commit_user
+ self._table_commit = builder.new_commit()
+ self._snapshot_callback = _SnapshotCallback()
+ self._table_commit.add_commit_callback(self._snapshot_callback)
+
+ identifier = self._next_commit_identifier
+ if self._checkpoint_snapshot is not None:
+ self._table_commit.protect_from_external_rewrites(
Review Comment:
The external-rewrite guard does not protect transform dependencies. Conflict detection only sees the staged `update_cols`, so an overlapping concurrent row-id update to a `read_projection` or filter column is accepted. Repro: read `name="a"`, derive `age=len(name)`, concurrently update `name` to `"long"` before this commit; the operation succeeds with `name="long", age=1` instead of `age=4`. Please include read/filter dependencies in the overlapping-row conflict set, or conservatively reject all concurrent row updates during an incremental transform.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [python][ray] Support row-count-based commits for large backfills [paimon]
Posted by "XiaoHongbo-Hope (via GitHub)" <gi...@apache.org>.
XiaoHongbo-Hope commented on PR #8826:
URL: https://github.com/apache/paimon/pull/8826#issuecomment-5225293212
close
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [python][ray] Incrementally commit update_by_row_id file groups [paimon]
Posted by "XiaoHongbo-Hope (via GitHub)" <gi...@apache.org>.
XiaoHongbo-Hope commented on PR #8826:
URL: https://github.com/apache/paimon/pull/8826#issuecomment-5111439705
> Just looking at the changes alone, it's too complicated; we need a simpler, more straightforward approach.
Got your point, just left core change now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [python][ray] Incrementally commit update_by_row_id file groups [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi closed pull request #8826: [python][ray] Incrementally commit update_by_row_id file groups
URL: https://github.com/apache/paimon/pull/8826
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [python][ray] Incrementally commit update_by_row_id file groups [paimon]
Posted by "XiaoHongbo-Hope (via GitHub)" <gi...@apache.org>.
XiaoHongbo-Hope commented on PR #8826:
URL: https://github.com/apache/paimon/pull/8826#issuecomment-5126281580
> > Incremental commits are not atomic across the whole operation. Ordinary exceptions
> > raised while applying a group are reported as results so other groups can finish;
> > completed groups are flushed before the error is raised.
> > Ray task retry options in `ray_remote_args` do not retry group exceptions reported
> > as results, so a transient group failure can leave a partial update. Worker loss and
> > other task-level failures which bypass Python exception handling may still lose
> > results buffered by that task.
>
> If this is indeed the case, as stated in your documentation, then the implications are significant—far beyond what a simple description like “max_groups_per_commit” can capture. This is a major semantic change that has been overlooked.
Updated, can you help check whether this looks good.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [python][ray] Incrementally commit update_by_row_id file groups [paimon]
Posted by "leaves12138 (via GitHub)" <gi...@apache.org>.
leaves12138 commented on code in PR #8826:
URL: https://github.com/apache/paimon/pull/8826#discussion_r3674303381
##########
paimon-python/pypaimon/ray/data_evolution_merge_join.py:
##########
@@ -598,14 +627,34 @@ def _apply_group(group: pa.Table) -> pa.Table:
all_msgs: list = []
num_updated = 0
action_row_ids = []
+ group_error = None
for batch in msgs_ds.iter_batches(batch_format="pyarrow"):
- for blob in batch.column("msgs_blob").to_pylist():
- all_msgs.extend(pickle.loads(blob))
- for n in batch.column("n_updated").to_pylist():
+ message_blobs = batch.column("msgs_blob").to_pylist()
+ updated_counts = batch.column("n_updated").to_pylist()
+ errors = batch.column("error").to_pylist()
+ row_id_blobs = (
+ batch.column("row_ids_blob").to_pylist()
+ if collect_row_ids else [None] * len(message_blobs)
+ )
+ for blob, n, row_ids_blob, error in zip(
+ message_blobs, updated_counts, row_id_blobs, errors):
+ if error is not None:
+ if group_error is None:
+ group_error = error
+ continue
+ group_msgs = pickle.loads(blob)
+ group_row_ids = (
+ pickle.loads(row_ids_blob) if collect_row_ids else []
+ )
+ if on_group_result is None:
+ all_msgs.extend(group_msgs)
+ else:
+ on_group_result(group_msgs, n, group_row_ids)
Review Comment:
When `on_group_result` raises (for example, the first incremental commit hits a deterministic `CommitConflictError`), this loop exits immediately even though Ray may already have materialized several group results in the same batch. Only the messages passed to the failing callback are known and aborted; the remaining blobs are dropped, so their staged files are left orphaned. I reproduced this with three file groups, `num_partitions=1`, and `max_groups_per_commit=1`: forcing `FileStoreCommit.commit` to raise `CommitConflictError` left two new Parquet files after the call returned. Please drain and abort every uncommitted group result, or restructure the flow so a commit failure cannot abandon buffered outputs, and add a regression test for this case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [python][ray] Incrementally commit update_by_row_id file groups [paimon]
Posted by "XiaoHongbo-Hope (via GitHub)" <gi...@apache.org>.
XiaoHongbo-Hope commented on code in PR #8826:
URL: https://github.com/apache/paimon/pull/8826#discussion_r3655507211
##########
paimon-python/pypaimon/ray/data_evolution_merge_join.py:
##########
@@ -598,14 +606,31 @@ def _apply_group(group: pa.Table) -> pa.Table:
all_msgs: list = []
num_updated = 0
action_row_ids = []
- for batch in msgs_ds.iter_batches(batch_format="pyarrow"):
- for blob in batch.column("msgs_blob").to_pylist():
- all_msgs.extend(pickle.loads(blob))
- for n in batch.column("n_updated").to_pylist():
+ iter_batches_kwargs = {"batch_format": "pyarrow"}
+ if on_group_result is not None:
+ # Yield each group immediately.
Review Comment:
> `batch_size=1` does not yield each group immediately; it only splits blocks after `map_groups` emits them. Ray buffers multiple group results within the same task, so if a later group fails, earlier completed groups never reach `on_group_result` and are not committed. This reproduces with `num_partitions=1`, three groups, and `max_groups_per_commit=1`: when the third group fails, the snapshot does not advance.
Fixed as of ac420260e: _apply_group returns each group’s failure as data instead of
raising, so completed groups in the same task are still committed before the error is
surfaced.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [python][ray] Support batched transform updates [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on PR #8826:
URL: https://github.com/apache/paimon/pull/8826#issuecomment-5207066215
I think this can be simplified by moving the abstraction one level down: instead of introducing a specialized `update_by_transform` pipeline, provide a generic, file-group-aligned row-id range processing primitive.
The ranges can be planned once from a pinned base snapshot and packed to approximately `target_rows_per_range`, but boundaries should always align with complete file groups (not individual physical files, because one logical row-id range may be represented by a base file plus multiple data-evolution delta/blob/vector files).
A possible low-level API is:
```python
with plan_row_id_ranges(
target="db.table",
catalog_options=options,
target_rows_per_range=1_000_000,
) as ranges:
for range_ctx in ranges:
source = range_ctx.read(
projection=["text"],
filter="language = 'en'",
)
updates = my_distributed_process(source)
range_ctx.update_by_row_id(
updates,
update_cols=["embedding"],
)
```
`RowIdRangeContext` could expose the stable public information and operations, while keeping file metadata internal:
```python
class RowIdRangeContext:
snapshot_id
range_start
range_end
estimated_rows
sequence_number
def read(...): ...
def update_by_row_id(...): ...
```
A convenience wrapper could execute the same ranges sequentially:
```python
def process(ctx):
source = ctx.read(["text"])
updates = build_embeddings_distributed(source)
ctx.update_by_row_id(updates, ["embedding"])
process_row_id_ranges(
target="db.table",
catalog_options=options,
target_rows_per_range=1_000_000,
processor=process,
)
```
Here “sequentially” means multiple range execution/commit rounds within the same Ray job and cluster, not submitting a brand-new Ray job per range. The processor can construct its own Ray Dataset DAG, actor pool, GPU inference, or even use another distributed service.
This model seems substantially simpler:
- one range follows a clear `read -> user processing -> write -> commit` lifecycle;
- only one range has uncommitted staged files at a time;
- earlier committed ranges naturally remain visible after a later failure;
- no cross-range `groupby/map_groups`, result draining, error-as-data protocol, or `on_group_result` callback is needed;
- commit ordering is deterministic;
- users control distributed transform execution instead of the API owning a second scheduling layer.
If arbitrary distributed processing may reorder rows, the result needs a stable association key. We either need to require row-count/order preservation, or pass through an opaque internal row token. It is not possible to allow arbitrary shuffle/repartition while also completely removing row identity from the contract.
The commit layer still needs to protect correctness when a range reads a pinned old snapshot: read/filter/output columns must participate in conflict detection if concurrent non-conflicting writes are allowed. However, the exact committed snapshot could be returned from an internal `commit(...)` result (including the duplicate-after-uncertain-retry case), rather than adding an operation-specific callback/state hook to the generic `TableCommit`.
Naming-wise, I suggest:
- `plan_row_id_ranges` for the low-level iterable planner;
- `process_row_id_ranges` for the managed sequential callback API;
- `target_rows_per_range` rather than `rows_per_commit`, because a generic processor does not necessarily commit.
Then embedding backfill becomes one use of this primitive instead of defining the primitive around transform updates.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [python][ray] Add file-group-aligned row-id range processing [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on PR #8826:
URL: https://github.com/apache/paimon/pull/8826#issuecomment-5215212605
I don't understand why we need to introduce methods like RowIdRange, plan_row_id_ranges, read_row_id_range, and update_by_row_id_from_plan.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org