You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "Dsrong (via GitHub)" <gi...@apache.org> on 2023/06/16 03:48:44 UTC
[GitHub] [incubator-paimon] Dsrong opened a new issue, #1387: [Bug] paimon table data loss
Dsrong opened a new issue, #1387:
URL: https://github.com/apache/incubator-paimon/issues/1387
### Search before asking
- [X] I searched in the [issues](https://github.com/apache/incubator-paimon/issues) and found nothing similar.
### Paimon version
0.5-20230609.002253-37
### Compute Engine
flink
### Minimal reproduce step
--paimon ddl
CREATE CATALOG paimon WITH (
'type' = 'paimon',
'metastore' = 'hive',
'uri' = 'thrift://172.30.199.208:9083',
'warehouse' = 'hdfs:////user/hive/warehouse'
);
use catalog paimon;
CREATE TABLE IF NOT EXISTS test.ods_ddpt_ddpt_tbllmlsgetmbillsum(
instid string
,orgid string
......
,linestatus string
,event_time TIMESTAMP(3)
,yt string
,PRIMARY KEY (instid,yt) NOT ENFORCED
)PARTITIONED BY (yt)
WITH (
'sequence.field' = 'event_time'
);
--flink ddl
CREATE CATALOG realtime WITH (
'type' = 'hive',
'default-database' = 'default',
'hive-conf-dir' = '/opt/app/hive-3.1.3/conf/'
);
USE CATALOG realtime;
CREATE TABLE IF NOT EXISTS flink.ddpt_ddpt_tbllmlsgetmbillsum (
INSTID STRING
,ORGID STRING
,TLBLSID STRING
....
,LINESTATUS STRING
,EVENT_TIME TIMESTAMP(3) METADATA FROM 'value.source.timestamp' VIRTUAL
,PRIMARY KEY (INSTID) NOT ENFORCED
)
WITH (
'connector' = 'kafka',
'topic' = 'ddpt_ddpt_tbllmlsgetmbillsum_cdc',
'properties.bootstrap.servers' = 'dpprd1-kafka01.sunwoda.com:9092',
'properties.group.id' = 'ddpt_ddpt_tbllmlsgetmbillsum_cdc',
'scan.startup.mode' = 'earliest-offset',
'format' = 'debezium-json'
);
--insert sql
SET 'pipeline.name' = 'flink_ods_ddpt_ddpt_tbllmlsgetmbillsum_paimon';
SET 'state.backend'='rocksdb';
SET 'state.backend.incremental'='true';
SET 'state.checkpoints.dir' = 'hdfs:////flink/checkpoint/flink_ods_ddpt_tbllmlsgetmbillsum_paimon';
SET 'state.checkpoints.num-retained'='2';
SET 'execution.checkpointing.interval'='5min';
SET 'execution.checkpointing.max-concurrent-checkpoints'='1';
SET 'execution.checkpointing.min-pause'='300ms';
SET 'execution.checkpointing.mode'='EXACTLY_ONCE';
SET 'execution.checkpointing.timeout'='10min';
SET 'parallelism.default' = '1';
SET 'pipeline.auto-watermark-interval' = '200';
SET 'pipeline.max-parallelism' = '1';
SET 'restart-strategy' = 'fixed-delay';
SET 'restart-strategy.fixed-delay.attempts'='3';
SET 'restart-strategy.fixed-delay.delay'='10';
SET 'execution.checkpointing.externalized-checkpoint-retention'='RETAIN_ON_CANCELLATION';
SET 'table.exec.sink.upsert-materialize'='none';
INSERT INTO paimon.test.ods_ddpt_ddpt_tbllmlsgetmbillsum
SELECT
INSTID
,ORGID
,TLBLSID
.....
,EVENT_TIME
,CAST(from_unixtime(CREATEDATE/1000-8*60*60,'yyyy') AS STRING)
FROM realtime.flink.ddpt_ddpt_tbllmlsgetmbillsum
;
### What doesn't meet your expectations?
1. flink table
select INSTID
,from_unixtime(CREATEDATE/1000-8*60*60,'yyyy-MM-dd HH:mm:ss') as CREATEDATE
,from_unixtime(MAINTAINDATE/1000-8*60*60,'yyyy-MM-dd HH:mm:ss') as maintaindate
,TIMESTAMPADD(HOUR,-8,EVENT_TIME) AS EVENT_TIME
from flink.ddpt_ddpt_tbllmlsgetmbillsum
where INSTID='24957541'
--result
+----+--------------------------------+--------------------------------+--------------------------------+-------------------------+
| op | INSTID | CREATEDATE | maintaindate | EVENT_TIME |
+----+--------------------------------+--------------------------------+--------------------------------+-------------------------+
| -U | 24957541 | 2023-06-15 10:00:40 | 2023-06-15 10:00:40 | 2023-06-15 14:09:45.000 |
| +U | 24957541 | 2023-06-15 10:00:40 | 2023-06-15 14:08:41 | 2023-06-15 14:09:45.000 |
| +I | 24957541 | 2023-06-15 10:00:40 | 2023-06-15 10:00:40 | 2023-06-15 10:01:51.000 |
| -U | 24957541 | 2023-06-15 10:00:40 | 2023-06-15 14:08:41 | 2023-06-15 14:18:19.000 |
| +U | 24957541 | 2023-06-15 10:00:40 | 2023-06-15 14:17:15 | 2023-06-15 14:18:19.000 |
2. oracle table
SELECT INSTID,ORGID,TLBLSID,TLBLSCODE,CUSER,CREATEUSER,CREATEDATE,MAINTAINUSER,MAINTAINDATE from ddpt.tbllmlsgetmbillsum
where instid=24957541
--result
INSTID ORGID TLBLSID TLBLSCODE CUSER CREATEUSER CREATEDATE MAINTAINUSER MAINTAINDATE
24957541 985 11863168 HZM202306151047 130822021 2023-6-15 10:00 2112280007 2023-6-15 14:17
3. paimon table
select * from test.ods_ddpt_ddpt_tbllmlsgetmbillsum
where instid='24957541'
--result
none
### Anything else?
_No response_
### Are you willing to submit a PR?
- [ ] I'm willing to submit a PR!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-paimon] Dsrong commented on issue #1387: [Question] Write Hive table to paimon table
Posted by "Dsrong (via GitHub)" <gi...@apache.org>.
Dsrong commented on issue #1387:
URL: https://github.com/apache/incubator-paimon/issues/1387#issuecomment-1598232193
yes, it‘s successfully write the data to paimon。
this case is not write data to paimon table from hive table.
just insert into paimon from kafka by kafka。
when i use flink-cdc init mode to capture oracle data to kafka topic.
then insert into paimon ,some data also loss.
bug compare with flink-doris-connector, use same kafka topic ,doris data is ok.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-paimon] JingsongLi closed issue #1387: [Question] Write Hive table to paimon table
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi closed issue #1387: [Question] Write Hive table to paimon table
URL: https://github.com/apache/incubator-paimon/issues/1387
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-paimon] JingsongLi commented on issue #1387: [Question] Write Hive table to paimon table
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on issue #1387:
URL: https://github.com/apache/incubator-paimon/issues/1387#issuecomment-1598610450
The conclusion is that 'sequence.field' = 'event_time', the values of -U and +U are the same
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-paimon] JingsongLi commented on issue #1387: [Bug] paimon table data loss
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on issue #1387:
URL: https://github.com/apache/incubator-paimon/issues/1387#issuecomment-1596451206
Can you provide the UI of flink? Did you actually write the data to paimon? Did it successful?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org