You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "www2388258980 (via GitHub)" <gi...@apache.org> on 2023/04/13 03:05:28 UTC
[GitHub] [incubator-paimon] www2388258980 opened a new issue, #891: [Bug] for streaming queries, 'merge-engine' = 'aggregation' dot't produce the correctness of the query result
www2388258980 opened a new issue, #891:
URL: https://github.com/apache/incubator-paimon/issues/891
### Search before asking
- [X] I searched in the [issues](https://github.com/apache/incubator-paimon/issues) and found nothing similar.
### Paimon version
paimon0.4(paimon-flink-1.16-0.4-20230331.003039-13.jar)
### Compute Engine
flink1.16.0
### Minimal reproduce step
**table schema**
ods_ticket_fees ingest data from pgsql(flink cdc,cdc -> ods_ticket_fees )
`
create table ods_ticket_fees(
id int,
ticket_id int,
`type` string,
`data` string,
total decimal(11,2),
primary key(id) not enforced
)
with (
'changelog-producer' = 'input',
'snapshot.time-retained' = '24h'
);
create table dwa_ticket_fees(
ticket_id int,
labor_cost decimal(12,2) ,
accessory_cost decimal(12,2) ,
consumable_cost decimal(12,2),
primary key(ticket_id) not enforced
)
with (
'merge-engine' = 'aggregation',
'changelog-producer' = 'full-compaction',
'snapshot.time-retained' = '24h',
'fields.labor_cost.aggregate-function' = 'sum',
'fields.accessory_cost.aggregate-function' = 'sum',
'fields.consumable_cost.aggregate-function' = 'sum'
);
SQL A:
insert into dwa_ticket_fees(ticket_id,labor_cost,accessory_cost,consumable_cost)
select a.ticket_id,
case when type='fee' then total else 0 end labor_cost,
case when type='accessory' then total else 0 end accessory_cost,
case when type='material' then total else 0 end consumable_cost
from ods_ticket_fees a;
SQL B:
select a.ticket_id,
sum(case when type='fee' then total else 0 end) labor_cost,
sum(case when type='accessory' then total else 0 end) accessory_cost,
sum(case when type='material' then total else 0 end) consumable_cost
from catalog_paimon.ods_medatc_fts.src_public_ticket_fees a
group by ticket_id;
`
**Preparin data int Postgres** \\
`
INSERT INTO ticket_fees(id, ticket_id, "type", "data", total, )
VALUES(40, 1352, 'fee', '平板', 1.00);
INSERT INTO ticket_fees(id, ticket_id, "type", "data", total, )
VALUES(1, 1352, 'accessory', '电脑', 1111.00);
`
** Result** \\
SQL A and SQL B are equal. for updates,SQL A and SQL B should equal(Only sum supports retraction (UPDATE_BEFORE and DELETE), others aggregate functions do not support retraction) and ods_ticket_fees produce the correctness of changelog(-U,+U). However,the value of SQL A keeps adding up.
**update sql**
`
update ticket_fees set "data"='平板1' where id=40;
update ticket_fees set "total"=2 where id=40;
`
### What doesn't meet your expectations?
aggragation model produce right result
### Anything else?
_No response_
### Are you willing to submit a PR?
- [ ] I'm willing to submit a PR!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-paimon] www2388258980 commented on issue #891: [Bug] for streaming queries, 'merge-engine' = 'aggregation' dot't produce the correctness of the query result
Posted by "www2388258980 (via GitHub)" <gi...@apache.org>.
www2388258980 commented on issue #891:
URL: https://github.com/apache/incubator-paimon/issues/891#issuecomment-1506706767
cdc注入paimon,需要设置'table.exec.sink.upsert-materialize' = 'NONE',才能够保证聚合模型产生正确的结果
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-paimon] www2388258980 commented on issue #891: [Bug] for streaming queries, 'merge-engine' = 'aggregation' dot't produce the correctness of the query result
Posted by "www2388258980 (via GitHub)" <gi...@apache.org>.
www2388258980 commented on issue #891:
URL: https://github.com/apache/incubator-paimon/issues/891#issuecomment-1506410794
create table cdc_ticket_fees(
id int,
ticket_id int,
`type` string,
`data` string,
total decimal(11,2),
primary key(id) not enforced
)
with (
'connector' = 'postgres-cdc',
'hostname' = '',
'port' = '5432',
'username' = '',
'password' = '',
'database-name' = 'ibupro_cloud',
'schema-name' = 'public',
'table-name' = 'ticket_fees',
'slot.name' = 'ticket_fees',
'decoding.plugin.name' = 'wal2json_streaming',
'changelog-mode' = 'upsert'
);
insert into ods_ticket_fees
select *
from cdc_ticket_fees;
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-paimon] www2388258980 closed issue #891: [Bug] for streaming queries, 'merge-engine' = 'aggregation' dot't produce the correctness of the query result
Posted by "www2388258980 (via GitHub)" <gi...@apache.org>.
www2388258980 closed issue #891: [Bug] for streaming queries, 'merge-engine' = 'aggregation' dot't produce the correctness of the query result
URL: https://github.com/apache/incubator-paimon/issues/891
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org