You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@fluss.apache.org by "platinumhamburg (via GitHub)" <gi...@apache.org> on 2025/11/08 07:51:08 UTC
[PR] Support Aggregation Merge Engine for Primary Key Tables [fluss]
platinumhamburg opened a new pull request, #1946:
URL: https://github.com/apache/fluss/pull/1946
<!--
*Thank you very much for contributing to Fluss - we are happy that you want to help us improve Fluss. To help the community review your contribution in the best possible way, please go through the checklist below, which will get the contribution into a shape in which it can be best reviewed.*
## Contribution Checklist
- Make sure that the pull request corresponds to a [GitHub issue](https://github.com/apache/fluss/issues). Exceptions are made for typos in JavaDoc or documentation files, which need no issue.
- Name the pull request in the format "[component] Title of the pull request", where *[component]* should be replaced by the name of the component being changed. Typically, this corresponds to the component label assigned to the issue (e.g., [kv], [log], [client], [flink]). Skip *[component]* if you are unsure about which is the best component.
- Fill out the template below to describe the changes contributed by the pull request. That will give reviewers the context they need to do the review.
- Make sure that the change passes the automated tests, i.e., `mvn clean verify` passes.
- Each pull request should address only one issue, not mix up code from multiple issues.
**(The sections below can be removed for hotfixes or typos)**
-->
### Purpose
<!-- Linking this pull request to the issue -->
Linked issue: close #212
<!-- What is the purpose of the change -->
### Brief change log
<!-- Please describe the changes made in this pull request and explain how they address the issue -->
### Tests
<!-- List UT and IT cases to verify this change -->
### API and Format
<!-- Does this change affect API or storage format -->
### Documentation
<!-- Does this change introduce a new feature -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support Aggregation Merge Engine for Primary Key Tables [fluss]
Posted by "platinumhamburg (via GitHub)" <gi...@apache.org>.
platinumhamburg commented on PR #1946:
URL: https://github.com/apache/fluss/pull/1946#issuecomment-3506180702
> @platinumhamburg This is 🔥 I’m curious though, doesn’t the aggregation merge engine, require some kind of transactions mechanism to be introduced first? If a job fails and restarts, does it ensure deterministic results?
Thanks @polyzos, your concern is absolutely correct. Currently, the KV engine does not have distributed transaction capabilities. The KV transaction commit relies solely on the local High Watermark, and even its idempotency mechanism is only based on the LogTablet's idempotency manager. Therefore, the KV engine currently only provides AT LEAST ONCE semantics and cannot provide Exactly Once semantics. This is not an issue introduced by the Aggregation Merge Engine itself, but aggregation operations are particularly sensitive to this limitation.
After careful evaluation, I agree that the reliability of the Aggregation Merge Engine does depend on distributed transactions. However, I still believe it can be treated as a relatively independent functional unit. The distributed transaction support for the KV engine is a broader topic that probably deserves a separate discussion or issue to address comprehensively.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support Aggregation Merge Engine for Primary Key Tables [fluss]
Posted by "platinumhamburg (via GitHub)" <gi...@apache.org>.
platinumhamburg closed pull request #1946: Support Aggregation Merge Engine for Primary Key Tables
URL: https://github.com/apache/fluss/pull/1946
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support Aggregation Merge Engine for Primary Key Tables [fluss]
Posted by "platinumhamburg (via GitHub)" <gi...@apache.org>.
platinumhamburg commented on PR #1946:
URL: https://github.com/apache/fluss/pull/1946#issuecomment-3509111813
@polyzos In my current consideration, I believe the consistency issue should be addressed separately in two domains:
- For Streaming Computation Domain: I think we should expose WriterState checkpoint/restore interfaces to the upper-layer compute engines. Engines like Flink can leverage these interfaces to implement state persistence, allowing Fluss to utilize the existing idempotence manager to ensure idempotent data writes.
- For Batch Processing Domain: The problem becomes significantly more complex, as each batch processing task execution will create independent Writers, and the idempotence manager cannot guarantee computational consistency. Imagine a batch processing task that fails over midway through writing data - the aggregation state of all related primary keys will be completely corrupted. To ensure consistency in the batch processing domain, we will need:
- Explicit BeginTransaction (PrepareTransaction is optional) / CommitTransaction / AbortTransaction interfaces exposed to compute engines
- True transactional visibility control for single Buckets (not through transactional buffers like PrewriteBuffer, but complete transaction control based on MvccValueEncoder/MvccValueDecoder + TransactionStateManager)
- Compute engines coordinating calls to Bucket local transaction managers through a global transaction manager to complete transactional writes in an orderly manner.
In summary, for streaming computation consistency, we should be able to achieve the required consistency guarantees with relatively minor modifications based on our current infrastructure. However, for batch processing, we will need to introduce a major architectural extension.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support Aggregation Merge Engine for Primary Key Tables [fluss]
Posted by "polyzos (via GitHub)" <gi...@apache.org>.
polyzos commented on PR #1946:
URL: https://github.com/apache/fluss/pull/1946#issuecomment-3506124610
@platinumhamburg This is 🔥
I’m curious though, doesn’t the aggregation merge engine, require some kind of transactions mechanism to be introduced first?
If a job fails and restarts, does it ensure deterministic results?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support Aggregation Merge Engine for Primary Key Tables [fluss]
Posted by "platinumhamburg (via GitHub)" <gi...@apache.org>.
platinumhamburg commented on PR #1946:
URL: https://github.com/apache/fluss/pull/1946#issuecomment-3509249146
> @polyzos In my current consideration, I believe the consistency issue should be addressed separately in two domains:
>
> * For Streaming Computation Domain: I think we should expose WriterState checkpoint/restore interfaces to the upper-layer compute engines. Engines like Flink can leverage these interfaces to implement state persistence, allowing Fluss to utilize the existing idempotence manager to ensure idempotent data writes.
> * For Batch Processing Domain: The problem becomes significantly more complex, as each batch processing task execution will create independent Writers, and the idempotence manager cannot guarantee computational consistency. Imagine a batch processing task that fails over midway through writing data - the aggregation state of all related primary keys will be completely corrupted. To ensure consistency in the batch processing domain, we will need:
>
> * Explicit BeginTransaction (PrepareTransaction is optional) / CommitTransaction / AbortTransaction interfaces exposed to compute engines
> * True transactional visibility control for single Buckets (not through transactional buffers like PrewriteBuffer, but complete transaction control based on MvccValueEncoder/MvccValueDecoder + TransactionStateManager)
> * Compute engines coordinating calls to Bucket local transaction managers through a global transaction manager to complete transactional writes in an orderly manner.
>
> In summary, for streaming computation consistency, we should be able to achieve the required consistency guarantees with relatively minor modifications based on our current infrastructure. However, for batch processing, we will need to introduce a major architectural extension.
I actually overestimated the functionality of WriterState. Currently, the idempotency manager of WriterIdState cannot actually ensure streaming data consistency either, as its state recording is currently quite limited. Therefore, regardless of streaming or batch processing, the related mechanisms need to be reimplemented uniformly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support Aggregation Merge Engine for Primary Key Tables [fluss]
Posted by "polyzos (via GitHub)" <gi...@apache.org>.
polyzos commented on PR #1946:
URL: https://github.com/apache/fluss/pull/1946#issuecomment-3506625037
@platinumhamburg i share similar thoughts.
What I’m thinking is the best way to approach this until transactions are introduced.
I’m thinking that maybe it makes sense to combine it with a versioned merge engine table upstream to handle any potential deduplication that might be introduced.
WDYT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support Aggregation Merge Engine for Primary Key Tables [fluss]
Posted by "platinumhamburg (via GitHub)" <gi...@apache.org>.
platinumhamburg commented on PR #1946:
URL: https://github.com/apache/fluss/pull/1946#issuecomment-3691002014
This PR will be closed and split into several smaller PRs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org