You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "MrTaozui (via GitHub)" <gi...@apache.org> on 2023/11/23 03:56:23 UTC
[I] [Bug] Partial Update table delete error. [incubator-paimon]
MrTaozui opened a new issue, #2374:
URL: https://github.com/apache/incubator-paimon/issues/2374
### Search before asking
- [X] I searched in the [issues](https://github.com/apache/incubator-paimon/issues) and found nothing similar.
### Paimon version
0.5
master
### Compute Engine
flink
### Minimal reproduce step
there is my example
```java
mysql table
CREATE TABLE `mr_1` (
`id` int NOT NULL AUTO_INCREMENT,
`c_1` varchar(64) DEFAULT NULL,
`c_2` varchar(64) DEFAULT NULL,
PRIMARY KEY (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=1 DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_0900_ai_ci;
CREATE TABLE `mr_2` (
`id` int NOT NULL AUTO_INCREMENT,
`c_3` varchar(64) DEFAULT NULL,
`c_4` varchar(64) DEFAULT NULL,
PRIMARY KEY (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=1 DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_0900_ai_ci;
paimon table
CREATE TABLE `paimon`.`crm`.`mr_1` (
`id` INT ,
`c_1` STRING,
`c_2` STRING,
PRIMARY KEY (`id`) NOT ENFORCED
) WITH (
'changelog-producer' = 'input',
'snapshot.time-retained' = '72h'
);
CREATE TABLE `paimon`.`crm`.`mr_2` (
`id` INT ,
`c_3` STRING,
`c_4` STRING,
` PRIMARY KEY (`id`) NOT ENFORCED
) WITH (
'changelog-producer' = 'input',
'snapshot.time-retained' = '72h'
);
CREATE TABLE `paimon`.`crm`.`mr_all` (
`id` INT ,
`c_1` STRING,
`c_2` STRING,
`c_3` STRING,
`c_4` STRING,
PRIMARY KEY (`id`) NOT ENFORCED
) WITH (
'changelog-producer' = 'lookup',
'merge-engine' = 'partial-update',
'partial-update.ignore-delete' = 'true'
);
mysql cdc table :
CREATE TABLE `hive`.`crm`.`ods_mysql_mr_1` (
`id` INT NOT NULL,
`c_1` VARCHAR(2147483647),
`c_2` VARCHAR(2147483647),
PRIMARY KEY (`id`) NOT ENFORCED
) WITH (
'server-time-zone' = 'UTC',
'hostname' = 'xxxxx',
'password' = '******',
'connector' = 'mysql-cdc',
'port' = '3306',
'database-name' = 'test',
'connection.pool.size' = '3',
'table-name' = 'mr_1',
'debezium.snapshot.locking.mode' = 'none',
'username' = 'root'
);
CREATE TABLE `hive`.`crm`.`ods_mysql_mr_2` (
`id` INT NOT NULL,
`c_3` VARCHAR(2147483647),
`c_4` VARCHAR(2147483647),
CONSTRAINT `PK_3386` PRIMARY KEY (`id`) NOT ENFORCED
) WITH (
'server-time-zone' = 'UTC',
'hostname' = 'xxxx',
'password' = '******',
'connector' = 'mysql-cdc',
'port' = '3306',
'database-name' = 'test',
'connection.pool.size' = '3',
'table-name' = 'mr_2',
'debezium.snapshot.locking.mode' = 'none',
'username' = 'root'
);
flink sql job
insert into paimon.crm.mr_1
select * from `hive`.`crm`.`ods_mysql_mr_1`
insert into paimon.crm.mr_2
select * from `hive`.`crm`.`ods_mysql_mr_2`
------merge-job------
SET
execution.checkpointing.max-concurrent-checkpoints = 1;
SET
table.exec.sink.upsert-materialize = 'NONE';
insert into
paimon.crm.mr_all2 (id, c_1, c_2, c_3, c_4)
select
id,
c_1,
c_2,
cast(null as string) as c_3,
cast(null as string) as c_4
from
paimon.crm.mr_1
union all
select
id,
cast(null as string) c_1,
cast(null as string) c_2,
c_3,
c_4
from
paimon.crm.mr_2
```
when i delete the record for mysql. the merge-job has excepetion:
Caused by: java.io.IOException: java.util.concurrent.ExecutionException: java.lang.IllegalArgumentException
at org.apache.paimon.flink.sink.StoreSinkWriteImpl.prepareCommit(StoreSinkWriteImpl.java:134)
at org.apache.paimon.flink.sink.TableWriteOperator.prepareCommit(TableWriteOperator.java:118)
at org.apache.paimon.flink.sink.RowDataStoreWriteOperator.prepareCommit(RowDataStoreWriteOperator.java:183)
at org.apache.paimon.flink.sink.PrepareCommitOperator.emitCommittables(PrepareCommitOperator.java:110)
at org.apache.paimon.flink.sink.PrepareCommitOperator.prepareSnapshotPreBarrier(PrepareCommitOperator.java:90)
at org.apache.flink.streaming.runtime.tasks.RegularOperatorChain.prepareSnapshotPreBarrier(RegularOperatorChain.java:89)
at org.apache.flink.streaming.runtime.tasks.SubtaskCheckpointCoordinatorImpl.checkpointState(SubtaskCheckpointCoordinatorImpl.java:334)
at org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$performCheckpoint$13(StreamTask.java:1281)
at org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.runThrowing(StreamTaskActionExecutor.java:50)
at org.apache.flink.streaming.runtime.tasks.StreamTask.performCheckpoint(StreamTask.java:1269)
at org.apache.flink.streaming.runtime.tasks.StreamTask.triggerCheckpointOnBarrier(StreamTask.java:1226)
... 22 more
Caused by: java.util.concurrent.ExecutionException: java.lang.IllegalArgumentException
at java.base/java.util.concurrent.FutureTask.report(Unknown Source)
at java.base/java.util.concurrent.FutureTask.get(Unknown Source)
at org.apache.paimon.compact.CompactFutureManager.obtainCompactResult(CompactFutureManager.java:67)
at org.apache.paimon.compact.CompactFutureManager.innerGetCompactionResult(CompactFutureManager.java:53)
at org.apache.paimon.mergetree.compact.MergeTreeCompactManager.getCompactionResult(MergeTreeCompactManager.java:183)
at org.apache.paimon.mergetree.MergeTreeWriter.trySyncLatestCompaction(MergeTreeWriter.java:284)
at org.apache.paimon.mergetree.MergeTreeWriter.prepareCommit(MergeTreeWriter.java:230)
at org.apache.paimon.operation.AbstractFileStoreWrite.prepareCommit(AbstractFileStoreWrite.java:178)
at org.apache.paimon.table.sink.TableWriteImpl.prepareCommit(TableWriteImpl.java:151)
at org.apache.paimon.flink.sink.StoreSinkWriteImpl.prepareCommit(StoreSinkWriteImpl.java:129)
... 32 more
Caused by: java.lang.IllegalArgumentException
at org.apache.paimon.utils.Preconditions.checkArgument(Preconditions.java:111)
at org.apache.paimon.mergetree.compact.LookupChangelogMergeFunctionWrapper.getResult(LookupChangelogMergeFunctionWrapper.java:89)
at org.apache.paimon.mergetree.compact.LookupChangelogMergeFunctionWrapper.getResult(LookupChangelogMergeFunctionWrapper.java:45)
at org.apache.paimon.mergetree.compact.SortMergeReaderWithLoserTree$SortMergeIterator.merge(SortMergeReaderWithLoserTree.java:94)
at org.apache.paimon.mergetree.compact.SortMergeReaderWithLoserTree$SortMergeIterator.next(SortMergeReaderWithLoserTree.java:80)
at org.apache.paimon.reader.RecordReaderIterator.advanceIfNeeded(RecordReaderIterator.java:74)
at org.apache.paimon.reader.RecordReaderIterator.hasNext(RecordReaderIterator.java:53)
at org.apache.paimon.mergetree.compact.ChangelogMergeTreeRewriter.rewriteChangelogCompaction(ChangelogMergeTreeRewriter.java:98)
at org.apache.paimon.mergetree.compact.ChangelogMergeTreeRewriter.rewrite(ChangelogMergeTreeRewriter.java:69)
at org.apache.paimon.mergetree.compact.MergeTreeCompactTask.rewrite(MergeTreeCompactTask.java:128)
at org.apache.paimon.mergetree.compact.MergeTreeCompactTask.doCompact(MergeTreeCompactTask.java:92)
at org.apache.paimon.compact.CompactTask.call(CompactTask.java:37)
at org.apache.paimon.compact.CompactTask.call(CompactTask.java:30)
at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
... 1 more
### What doesn't meet your expectations?
Where the bug occurred ？
LookupChangelogMergeFunctionWrapper :
PartialUpdateMergeFunction:
### Anything else?
_No response_
### Are you willing to submit a PR?
- [X] I'm willing to submit a PR!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] Partial Update table delete error. [incubator-paimon]
Posted by "leaves12138 (via GitHub)" <gi...@apache.org>.
leaves12138 closed issue #2374: [Bug] Partial Update table delete error.
URL: https://github.com/apache/incubator-paimon/issues/2374
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] Partial Update table delete error. [incubator-paimon]
Posted by "MrTaozui (via GitHub)" <gi...@apache.org>.
MrTaozui commented on issue #2374:
URL: https://github.com/apache/incubator-paimon/issues/2374#issuecomment-1823794408
pr: https://github.com/apache/incubator-paimon/pull/2375
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org