You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@fluss.apache.org by "luoyuxia (via GitHub)" <gi...@apache.org> on 2025/09/24 01:52:24 UTC
[PR] [doc] Add streaming union read part for paimon document [fluss]
luoyuxia opened a new pull request, #1747:
URL: https://github.com/apache/fluss/pull/1747
<!--
*Thank you very much for contributing to Fluss - we are happy that you want to help us improve Fluss. To help the community review your contribution in the best possible way, please go through the checklist below, which will get the contribution into a shape in which it can be best reviewed.*
## Contribution Checklist
- Make sure that the pull request corresponds to a [GitHub issue](https://github.com/apache/fluss/issues). Exceptions are made for typos in JavaDoc or documentation files, which need no issue.
- Name the pull request in the format "[component] Title of the pull request", where *[component]* should be replaced by the name of the component being changed. Typically, this corresponds to the component label assigned to the issue (e.g., [kv], [log], [client], [flink]). Skip *[component]* if you are unsure about which is the best component.
- Fill out the template below to describe the changes contributed by the pull request. That will give reviewers the context they need to do the review.
- Make sure that the change passes the automated tests, i.e., `mvn clean verify` passes.
- Each pull request should address only one issue, not mix up code from multiple issues.
**(The sections below can be removed for hotfixes or typos)**
-->
### Purpose
<!-- Linking this pull request to the issue -->
Linked issue: close #1659
<!-- What is the purpose of the change -->
### Brief change log
<!-- Please describe the changes made in this pull request and explain how they address the issue -->
### Tests
<!-- List UT and IT cases to verify this change -->
### API and Format
<!-- Does this change affect API or storage format -->
### Documentation
<!-- Does this change introduce a new feature -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [doc] Add streaming union read part for paimon document [fluss]
Posted by "luoyuxia (via GitHub)" <gi...@apache.org>.
luoyuxia merged PR #1747:
URL: https://github.com/apache/fluss/pull/1747
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [doc] Add streaming union read part for paimon document [fluss]
Posted by "luoyuxia (via GitHub)" <gi...@apache.org>.
luoyuxia commented on PR #1747:
URL: https://github.com/apache/fluss/pull/1747#issuecomment-3326161523
cc @beryllw @caozhen1937
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [doc] Add streaming union read part for paimon document [fluss]
Posted by "luoyuxia (via GitHub)" <gi...@apache.org>.
luoyuxia commented on code in PR #1747:
URL: https://github.com/apache/fluss/pull/1747#discussion_r2374038713
##########
website/docs/streaming-lakehouse/integrate-data-lakes/paimon.md:
##########
@@ -92,14 +96,29 @@ For further information, refer to Paimon’s [SQL Query documentation](https://p
#### Union Read of Data in Fluss and Paimon
+##### Prerequisites
+Download the [fluss-lake-paimon-$FLUSS_VERSION$.jar](https://repo1.maven.org/maven2/org/apache/fluss/fluss-lake-paimon/$FLUSS_VERSION$/fluss-lake-paimon-$FLUSS_VERSION$.jar), and place it into `${FLINK_HOME}/lib`.
+
Review Comment:
Yes, we 'll deploy these jars. All jar download link is refer to `https://repo1.maven.org/maven2/org/apache/fluss/` in the doc
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [doc] Add streaming union read part for paimon document [fluss]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #1747:
URL: https://github.com/apache/fluss/pull/1747#discussion_r2373849802
##########
website/docs/streaming-lakehouse/integrate-data-lakes/paimon.md:
##########
@@ -92,14 +96,29 @@ For further information, refer to Paimon’s [SQL Query documentation](https://p
#### Union Read of Data in Fluss and Paimon
+##### Prerequisites
+Download the [fluss-lake-paimon-$FLUSS_VERSION$.jar](https://repo1.maven.org/maven2/org/apache/fluss/fluss-lake-paimon/$FLUSS_VERSION$/fluss-lake-paimon-$FLUSS_VERSION$.jar), and place it into `${FLINK_HOME}/lib`.
+
+##### Union Read
To read the full dataset, which includes both Fluss (fresh) and Paimon (historical) data, simply query the table without any suffix. The following example illustrates this:
```sql title="Flink SQL"
-- Query will union data from Fluss and Paimon
SELECT SUM(order_count) AS total_orders FROM ads_nation_purchase_power;
```
+It can run both in batch and streaming mode, with Paimon as historical data and Fluss as fresh data:
+- In batch mode
+
+ The query may run slower than reading only from Paimon because it needs to merge rows from both Paimon and Fluss. However, it returns the most up-to-date results. Multiple executions of the query may produce different outputs due to continuous data ingestion.
+
+- In streaming mode
+
+ Flink first reads the latest snapshot of Paimon tierd via tiering service, then switches to Fluss starting from the log offset aligned with that snapshot, ensuring exactly-once semantics.
Review Comment:
There's a typo in 'tierd' which should be 'tiered'.
```suggestion
Flink first reads the latest snapshot of Paimon tiered via tiering service, then switches to Fluss starting from the log offset aligned with that snapshot, ensuring exactly-once semantics.
```
##########
website/docs/streaming-lakehouse/integrate-data-lakes/paimon.md:
##########
@@ -92,14 +96,29 @@ For further information, refer to Paimon’s [SQL Query documentation](https://p
#### Union Read of Data in Fluss and Paimon
+##### Prerequisites
+Download the [fluss-lake-paimon-$FLUSS_VERSION$.jar](https://repo1.maven.org/maven2/org/apache/fluss/fluss-lake-paimon/$FLUSS_VERSION$/fluss-lake-paimon-$FLUSS_VERSION$.jar), and place it into `${FLINK_HOME}/lib`.
+
+##### Union Read
To read the full dataset, which includes both Fluss (fresh) and Paimon (historical) data, simply query the table without any suffix. The following example illustrates this:
```sql title="Flink SQL"
-- Query will union data from Fluss and Paimon
SELECT SUM(order_count) AS total_orders FROM ads_nation_purchase_power;
```
+It can run both in batch and streaming mode, with Paimon as historical data and Fluss as fresh data:
+- In batch mode
+
+ The query may run slower than reading only from Paimon because it needs to merge rows from both Paimon and Fluss. However, it returns the most up-to-date results. Multiple executions of the query may produce different outputs due to continuous data ingestion.
+
+- In streaming mode
+
+ Flink first reads the latest snapshot of Paimon tierd via tiering service, then switches to Fluss starting from the log offset aligned with that snapshot, ensuring exactly-once semantics.
+ This design enabled Fluss to store only a small portion of the dataset in Fluss cluster, reducing costs, while Paimon as the source of complete historical data when needed.
Review Comment:
There's a grammatical error - 'enabled' should be 'enables' for present tense consistency. Also, there's an extra space before 'as' and the sentence is missing 'serves' before 'as the source'.
```suggestion
This design enables Fluss to store only a small portion of the dataset in the Fluss cluster, reducing costs, while Paimon serves as the source of complete historical data when needed.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [doc] Add streaming union read part for paimon document [fluss]
Posted by "beryllw (via GitHub)" <gi...@apache.org>.
beryllw commented on code in PR #1747:
URL: https://github.com/apache/fluss/pull/1747#discussion_r2373863509
##########
website/docs/streaming-lakehouse/integrate-data-lakes/paimon.md:
##########
@@ -92,14 +96,29 @@ For further information, refer to Paimon’s [SQL Query documentation](https://p
#### Union Read of Data in Fluss and Paimon
+##### Prerequisites
+Download the [fluss-lake-paimon-$FLUSS_VERSION$.jar](https://repo1.maven.org/maven2/org/apache/fluss/fluss-lake-paimon/$FLUSS_VERSION$/fluss-lake-paimon-$FLUSS_VERSION$.jar), and place it into `${FLINK_HOME}/lib`.
+
Review Comment:
Just to confirm: We'll deploy these JARs after releasing version 0.8.0, right? (Reference link: https://repo1.maven.org/maven2/org/apache/fluss/) Also, why aren't we using this link instead: https://github.com/apache/fluss/releases/download/v0.7.0/?
<img width="683" height="265" alt="Image" src="https://github.com/user-attachments/assets/4eeec2e3-23a2-4bb2-a152-f4812458d6b0" />
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [doc] Add streaming union read part for paimon document [fluss]
Posted by "beryllw (via GitHub)" <gi...@apache.org>.
beryllw commented on code in PR #1747:
URL: https://github.com/apache/fluss/pull/1747#discussion_r2373875825
##########
website/docs/streaming-lakehouse/integrate-data-lakes/paimon.md:
##########
@@ -92,14 +96,29 @@ For further information, refer to Paimon’s [SQL Query documentation](https://p
#### Union Read of Data in Fluss and Paimon
+##### Prerequisites
+Download the [fluss-lake-paimon-$FLUSS_VERSION$.jar](https://repo1.maven.org/maven2/org/apache/fluss/fluss-lake-paimon/$FLUSS_VERSION$/fluss-lake-paimon-$FLUSS_VERSION$.jar), and place it into `${FLINK_HOME}/lib`.
+
+##### Union Read
To read the full dataset, which includes both Fluss (fresh) and Paimon (historical) data, simply query the table without any suffix. The following example illustrates this:
```sql title="Flink SQL"
-- Query will union data from Fluss and Paimon
SELECT SUM(order_count) AS total_orders FROM ads_nation_purchase_power;
```
+It can run both in batch and streaming mode, with Paimon as historical data and Fluss as fresh data:
Review Comment:
It supports both batch and streaming modes, using Paimon for historical data and Fluss for fresh data:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [doc] Add streaming union read part for paimon document [fluss]
Posted by "beryllw (via GitHub)" <gi...@apache.org>.
beryllw commented on PR #1747:
URL: https://github.com/apache/fluss/pull/1747#issuecomment-3326207189
LGTM!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [doc] Add streaming union read part for paimon document [fluss]
Posted by "beryllw (via GitHub)" <gi...@apache.org>.
beryllw commented on code in PR #1747:
URL: https://github.com/apache/fluss/pull/1747#discussion_r2373878065
##########
website/docs/streaming-lakehouse/integrate-data-lakes/paimon.md:
##########
@@ -92,14 +96,29 @@ For further information, refer to Paimon’s [SQL Query documentation](https://p
#### Union Read of Data in Fluss and Paimon
+##### Prerequisites
+Download the [fluss-lake-paimon-$FLUSS_VERSION$.jar](https://repo1.maven.org/maven2/org/apache/fluss/fluss-lake-paimon/$FLUSS_VERSION$/fluss-lake-paimon-$FLUSS_VERSION$.jar), and place it into `${FLINK_HOME}/lib`.
+
+##### Union Read
To read the full dataset, which includes both Fluss (fresh) and Paimon (historical) data, simply query the table without any suffix. The following example illustrates this:
```sql title="Flink SQL"
-- Query will union data from Fluss and Paimon
SELECT SUM(order_count) AS total_orders FROM ads_nation_purchase_power;
```
+It can run both in batch and streaming mode, with Paimon as historical data and Fluss as fresh data:
+- In batch mode
+
+ The query may run slower than reading only from Paimon because it needs to merge rows from both Paimon and Fluss. However, it returns the most up-to-date results. Multiple executions of the query may produce different outputs due to continuous data ingestion.
+
+- In streaming mode
+
+ Flink first reads the latest snapshot of Paimon tiered via tiering service, then switches to Fluss starting from the log offset aligned with that snapshot, ensuring exactly-once semantics.
Review Comment:
Flink first reads the latest Paimon snapshot (tiered via the tiering service),
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [doc] Add streaming union read part for paimon document [fluss]
Posted by "caozhen1937 (via GitHub)" <gi...@apache.org>.
caozhen1937 commented on code in PR #1747:
URL: https://github.com/apache/fluss/pull/1747#discussion_r2373929388
##########
website/docs/streaming-lakehouse/integrate-data-lakes/paimon.md:
##########
@@ -92,14 +96,29 @@ For further information, refer to Paimon’s [SQL Query documentation](https://p
#### Union Read of Data in Fluss and Paimon
+##### Prerequisites
+Download the [fluss-lake-paimon-$FLUSS_VERSION$.jar](https://repo1.maven.org/maven2/org/apache/fluss/fluss-lake-paimon/$FLUSS_VERSION$/fluss-lake-paimon-$FLUSS_VERSION$.jar), and place it into `${FLINK_HOME}/lib`.
+
+##### Union Read
To read the full dataset, which includes both Fluss (fresh) and Paimon (historical) data, simply query the table without any suffix. The following example illustrates this:
```sql title="Flink SQL"
Review Comment:
```suggestion
```sql title="Flink SQL"
-- Set execution mode to streaming or batch
SET 'execution.runtime-mode' = 'streaming';
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org