You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "liming30 (via GitHub)" <gi...@apache.org> on 2024/08/01 09:11:10 UTC
[I] [Feature] When the MergeEngine of the dimension table is aggregation / partial-update, there is no need to forcibly enable changelog-producer. [paimon]
liming30 opened a new issue, #3868:
URL: https://github.com/apache/paimon/issues/3868
### Search before asking
- [X] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.
### Motivation
Currently, two built-in compaction strategies are provided, but they are mainly set automatically based on `MergeEngine` / `ChangelogProducer` / `DeletionVectors`. In the scenario of using dimension tables, if the `MergeEngine` of the dimension table is `aggregation`/`partial-update`, we have to set `CHANGELOG_PRODUCER` to `lookup`. But when using `PrimaryKeyPartialLookupTable`, changelog is useless.
Therefore, I hope to add a compact-strategy configuration, so that `CHANGELOG_PRODUCER` can be enabled only when necessary.
At the same time, for `FullCacheLookupTable`, even if the user does not enable `CHANGELOG_PRODUCER`, we can also obtain the changelog through `IncrementalDiffSplitRead`.
### Solution
I would like to do it in two parts:
1. add the compact-strategy configuration, so that other types of `MergeEngine` can be compacted quickly without writing changelog.
2. adjust the refresh strategy of `FullCacheLookupTable` to support streaming updates of other types of `MergeEngine` without enable CHANGELOG_PRODUCER.
### Anything else?
_No response_
### Are you willing to submit a PR?
- [X] I'm willing to submit a PR!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature] When the MergeEngine of the dimension table is aggregation / partial-update, there is no need to forcibly enable changelog-producer. [paimon]
Posted by "liming30 (via GitHub)" <gi...@apache.org>.
liming30 commented on issue #3868:
URL: https://github.com/apache/paimon/issues/3868#issuecomment-2271149634
> What case do you want to solve? Lookup Join for partial-update table without changelog-producer? If this is your requirement, can we just modify Flink LookupJoin Function?
@JingsongLi the dim table usually has no streaming consumption jobs, so generating a changelog is useless. In most cases, we will ensure that the primary key of the dim table is the same as the key of the `lookup-join`, so we will use `PrimaryKeyPartialLookupTable` to perform the lookup.
By adding a `prefer-compaction-strategy`, I think most of the code can be reused. If we perform the merge operation in the `LookupJoin Function`, this will result in a merge overhead in each job.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature] When the MergeEngine of the dimension table is aggregation / partial-update, there is no need to forcibly enable changelog-producer. [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi closed issue #3868: [Feature] When the MergeEngine of the dimension table is aggregation / partial-update, there is no need to forcibly enable changelog-producer.
URL: https://github.com/apache/paimon/issues/3868
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature] When the MergeEngine of the dimension table is aggregation / partial-update, there is no need to forcibly enable changelog-producer. [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on issue #3868:
URL: https://github.com/apache/paimon/issues/3868#issuecomment-2325838126
Hi @liming30 , so what you want to support is just for partial-update&agg table with lookup but without changelog?
Why? The most cost is in lookup, the cost of changelog is not so high.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature] When the MergeEngine of the dimension table is aggregation / partial-update, there is no need to forcibly enable changelog-producer. [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on issue #3868:
URL: https://github.com/apache/paimon/issues/3868#issuecomment-2270868912
What case do you want to solve? Lookup Join for partial-update table without changelog-producer?
If this is your requirement, can we just modify Flink LookupJoin Function?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature] When the MergeEngine of the dimension table is aggregation / partial-update, there is no need to forcibly enable changelog-producer. [paimon]
Posted by "liming30 (via GitHub)" <gi...@apache.org>.
liming30 commented on issue #3868:
URL: https://github.com/apache/paimon/issues/3868#issuecomment-2325857253
As an issue following #3905 , dim tables do not require streaming consumption in most cases, so there is no need to generate changelog files to reduce write IO.
When the primary key of the lookup join is exactly the same as the primary key of the table, we can use `PrimaryKeyPartialLookupTable` without reading the changelog file. When the primary key of the lookup join is inconsistent with the primary key of the table, we can use `FullCacheLookupTable` based on the diff generated by `compaction`, so I hope to relax the restrictions on `FullCacheLookupTable` for dim tables.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org