You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "lppsuixn (via GitHub)" <gi...@apache.org> on 2024/04/26 14:23:55 UTC
[I] [Bug] first-row engine Batch Incremental count not equal changelogRecordCount [paimon]
lppsuixn opened a new issue, #3276:
URL: https://github.com/apache/paimon/issues/3276
### Search before asking
- [X] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.
### Paimon version
0.7
### Compute Engine
flink
### Minimal reproduce step
Between snapshot-12 and snapshot-14 has 408859167 changelog records,But batch Incremental count is 1545129599.
```sql
SELECT count(1) FROM my_catalog_paimon.db.table /*+ OPTIONS('incremental-between' = '12,14') */ limit 10;
```
And scan.file-creation result is correct.
```
SELECT count(1) FROM my_catalog_paimon.db.table /*+ OPTIONS('scan.file-creation-time-millis' = '1714119900000','scan.infer-parallelism.max'='500') */
```
snapshot-12
```json
{
"version" : 3,
"id" : 12,
"schemaId" : 0,
"baseManifestList" : "manifest-list-a0cd3036-c7a9-4d97-9b3b-4fe117a2806e-2",
"deltaManifestList" : "manifest-list-a0cd3036-c7a9-4d97-9b3b-4fe117a2806e-3",
"changelogManifestList" : "manifest-list-a0cd3036-c7a9-4d97-9b3b-4fe117a2806e-4",
"commitUser" : "bb69057c-5a2b-4c8d-bf88-ebea7b373764",
"commitIdentifier" : 9223372036854775807,
"commitKind" : "COMPACT",
"timeMillis" : 1714115951112,
"logOffsets" : { },
"totalRecordCount" : 949445145099,
"deltaRecordCount" : -4538585,
"changelogRecordCount" : 42198489518,
"watermark" : -9223372036854775808
}
```
snapshot-14
```
{
"version" : 3,
"id" : 14,
"schemaId" : 0,
"baseManifestList" : "manifest-list-bf44c64a-80e6-410d-b43c-2d65b8c03823-2",
"deltaManifestList" : "manifest-list-bf44c64a-80e6-410d-b43c-2d65b8c03823-3",
"changelogManifestList" : "manifest-list-bf44c64a-80e6-410d-b43c-2d65b8c03823-4",
"commitUser" : "b457cb4b-2a32-4ca7-be21-f50328894121",
"commitIdentifier" : 9223372036854775807,
"commitKind" : "COMPACT",
"timeMillis" : 1714124210161,
"logOffsets" : { },
"totalRecordCount" : 949854004266,
"deltaRecordCount" : -1136270432,
"changelogRecordCount" : 408859167,
"watermark" : -9223372036854775808
}
```
### What doesn't meet your expectations?
Batch Incremental count equal changelogRecordCount
### Anything else?
```
CREATE TABLE my_catalog_paimon.db.table(
a bigint,
b bigint,
c string,
d string,
e string,
f string,
g string,
hstring,
PRIMARY KEY (a,b) NOT ENFORCED ) WITH (
'merge-engine'='first-row',
'changelog-producer' = 'lookup',
'bucket' = '5000'
);
```
### Are you willing to submit a PR?
- [ ] I'm willing to submit a PR!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] first-row engine Batch Incremental count not equal changelogRecordCount [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on issue #3276:
URL: https://github.com/apache/paimon/issues/3276#issuecomment-2084869101
You can use `incremental-between-scan-mode = changelog` to fix this.
I will change the default behavior.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] first-row engine Batch Incremental count not equal changelogRecordCount [paimon]
Posted by "leaves12138 (via GitHub)" <gi...@apache.org>.
leaves12138 closed issue #3276: [Bug] first-row engine Batch Incremental count not equal changelogRecordCount
URL: https://github.com/apache/paimon/issues/3276
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org