[D] Paimon's function about compaction and partial-update [incubator-xtable]

["yuzhangqin55-bit (via GitHub)" <gi...@apache.org>]

GitHub user yuzhangqin55-bit created a discussion: Paimon's function about compaction and partial-update

In my business scenarios,I need to use 8 tasks write to the same table, and I also need only a few columns keep the first non-null value, other be covered.
Now,I set table as:
WITH (
  'bucket' = '40',
  'path' = 's3://yj-datalake-test/paimon/qinyuzhang.db/Mt5HistoryTradeRecordModel',
  'write-mode' = 'WRITE_ONLY',
  'merge-engine' = 'partial-update',
  'bucket-key' = 'b_id,s_id'
)
and run 8 tasks to write,but it prints logs as:
2025-10-09 05:33:13,865 WARN  org.apache.flink.runtime.taskmanager.Task                    [] - Writer : Mt5HistoryTradeRecordModel -> Global Committer : Mt5HistoryTradeRecordModel -> end: Writer (1/1)#677 (70647f047568adcfd35f0ed8f64fd97d_f6dc7f4d2283f4605b127b9364e21148_0_677) switched from RUNNING to FAILED with failure cause:
java.lang.RuntimeException: File deletion conflicts detected! Give up committing.

Don't panic!
Conflicts during commits are normal and this failure is intended to resolve the conflicts.
Conflicts are mainly caused by the following scenarios:
1. Multiple jobs are writing into the same partition at the same time, or you use STATEMENT SET to execute multiple INSERT statements into the same Paimon table.
   You'll probably see different base commit user and current commit user below.
   You can use https://paimon.apache.org/docs/master/maintenance/dedicated-compaction#dedicated-compaction-job to support multiple writing.
2. You're recovering from an old savepoint, or you're creating multiple jobs from a savepoint.
   The job will fail continuously in this scenario to protect metadata from corruption.
   You can either recover from the latest savepoint, or you can revert the table to the snapshot corresponding to the old savepoint.

Base commit user is: d0d36272-9541-497b-8710-0ad75295d42b; Current commit user is: d0d36272-9541-497b-8710-0ad75295d42b
So table does not save all data.
How should I set up the table to eliminate the problem and retain all the data?
Help me thanks a lot.

GitHub link: https://github.com/apache/incubator-xtable/discussions/751

----
This is an automatically sent email for commits@xtable.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@xtable.apache.org

Re: [D] Paimon's function about compaction and partial-update [incubator-xtable]

["skyyyy955 (via GitHub) via dev" <de...@xtable.apache.org>]

GitHub user skyyyy955 added a comment to the discussion: Paimon's function about compaction and partial-update

The compaction option should be `'write-only' = 'true'`. If these are eight separate jobs writing to the same partition, set this for all writers and use a dedicated compaction job. See the [Paimon 1.2 docs](https://paimon.apache.org/docs/1.2/maintenance/dedicated-compaction/).

Your log also shows the same base and current commit user. Were any jobs restored from an old savepoint, or started from the same savepoint? I'd check that before assuming compaction is the only cause.

For the column values, plain `partial-update` keeps the latest non-null value. Keeping the first one needs additional configuration; the docs describe [per-column aggregation](https://paimon.apache.org/docs/1.2/primary-key-table/merge-engine/partial-update/#aggregation-for-partial-update).

Could you share your Paimon/Flink versions, full table DDL and catalog configuration? Also, does "first" mean the first value received, or the earliest by event time?

GitHub link: https://github.com/apache/incubator-xtable/discussions/751#discussioncomment-19313980

----
This is an automatically sent email for dev@xtable.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@xtable.apache.org