You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@fluss.apache.org by "zuston (via GitHub)" <gi...@apache.org> on 2026/08/18 08:44:13 UTC
[PR] [docs] Add more details for historical partition lookup [fluss]
zuston opened a new pull request, #4030:
URL: https://github.com/apache/fluss/pull/4030
### Purpose
Add more details for historical partition lookup
### Brief change log
<!-- Please describe the changes made in this pull request and explain how they address the issue -->
### Tests
<!-- List UT and IT cases to verify this change -->
### API and Format
<!-- Does this change affect API or storage format -->
### Documentation
<!-- Does this change introduce a new feature -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [docs] Add more details for historical partition lookup [fluss]
Posted by "luoyuxia (via GitHub)" <gi...@apache.org>.
luoyuxia commented on code in PR #4030:
URL: https://github.com/apache/fluss/pull/4030#discussion_r4058841300
##########
website/docs/streaming-lakehouse/datalake-formats/paimon.md:
##########
@@ -122,6 +122,25 @@ CREATE TABLE fluss_order_with_lake (
For example, you can specify the Paimon property `file.format` to change the file format of the Paimon table, or set `deletion-vectors.enabled` to enable or disable deletion vectors for the Paimon table.
+### Historical Partition Lookup Setup
Review Comment:
Now, it's not only lookup, but also write
##########
website/docs/streaming-lakehouse/datalake-formats/paimon.md:
##########
@@ -122,6 +122,25 @@ CREATE TABLE fluss_order_with_lake (
For example, you can specify the Paimon property `file.format` to change the file format of the Paimon table, or set `deletion-vectors.enabled` to enable or disable deletion vectors for the Paimon table.
+### Historical Partition Lookup Setup
+
+Historical partition lookups run on TabletServers. Configure `datalake.format` and the related
+`datalake.paimon.*` options on every TabletServer, using the same Paimon catalog and warehouse as
+the tiering service, for example:
+
+```yaml title="server.yaml"
+datalake.enabled: true
+datalake.format: paimon
+datalake.paimon.metastore: filesystem
+datalake.paimon.warehouse: /path/to/paimon/warehouse
+```
+
+The Fluss binary distribution already includes the Paimon connector and `paimon-bundle` in
Review Comment:
Actually, Paimon connector is not included.
##########
website/docs/table-design/data-distribution/partitioning.md:
##########
@@ -71,6 +71,9 @@ In this case, when automatic partitioning occurs (Fluss will periodically operat
### Historical Partition Access
+Before using historical partition lookup, complete the
Review Comment:
```suggestion
Before using historical partition lookup, make sure the [Paimon server-side setup](../../streaming-lakehouse/datalake-formats/paimon.md#historical-partition-lookup-setup) is completed.
```
##########
website/docs/streaming-lakehouse/datalake-formats/paimon.md:
##########
@@ -122,6 +122,25 @@ CREATE TABLE fluss_order_with_lake (
For example, you can specify the Paimon property `file.format` to change the file format of the Paimon table, or set `deletion-vectors.enabled` to enable or disable deletion vectors for the Paimon table.
+### Historical Partition Lookup Setup
Review Comment:
Also, maybe it'll be better to put it after `## Configure Paimon as LakeHouse Storage`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [docs] Add more details for historical partition lookup [fluss]
Posted by "zuston (via GitHub)" <gi...@apache.org>.
zuston commented on code in PR #4030:
URL: https://github.com/apache/fluss/pull/4030#discussion_r3809798079
##########
website/docs/engine-flink/lookups.md:
##########
@@ -289,6 +289,20 @@ partitioning enabled and exactly one partition key. When enabled, the Coordinato
retains the `__historical__` system partition used to route lookups to Paimon. Disabling the option
removes that system partition.
+Historical partition lookups run on TabletServers. Configure `datalake.format` and the related
+`datalake.<format>.*` options on every TabletServer, for example:
+
+```yaml title="server.yaml"
+datalake.enabled: true
+datalake.format: paimon
+datalake.paimon.metastore: filesystem
+datalake.paimon.warehouse: /path/to/paimon/warehouse
+```
+
+Also make the Paimon connector, `paimon-bundle`, and required catalog or storage JARs available on
Review Comment:
> Just curious about do we really need to mentention about it since it's already in fluss distribution. But It reminds me that we did need to put paimon oss jar into ${FLUSS_HOME}/plugins/paimon/.
Yes. from my side, without this historical lookup feature, the tablet server don't need to reserve the lake related jars. and the data lake sepcified options also needn't in tablet server server.yml . that means maybe we should mention this in this section.
> Also, i feel like this page may not a right place to put this. Maybe https://fluss.apache.org/docs/next/install-deploy/deploying-streaming-lakehouse/ or https://fluss.apache.org/docs/next/streaming-lakehouse/datalake-formats/paimon/. But we can add a link in this page for how to make historical partition lookups works
Do you mean the whole section of `Historical Partition Lookup` need to be moved into the `streaming-lakehouse/datalake-formats/paimon` ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [docs] Add more details for historical partition lookup [fluss]
Posted by "luoyuxia (via GitHub)" <gi...@apache.org>.
luoyuxia commented on code in PR #4030:
URL: https://github.com/apache/fluss/pull/4030#discussion_r3921681468
##########
website/docs/engine-flink/lookups.md:
##########
@@ -289,6 +289,20 @@ partitioning enabled and exactly one partition key. When enabled, the Coordinato
retains the `__historical__` system partition used to route lookups to Paimon. Disabling the option
removes that system partition.
+Historical partition lookups run on TabletServers. Configure `datalake.format` and the related
+`datalake.<format>.*` options on every TabletServer, for example:
+
+```yaml title="server.yaml"
+datalake.enabled: true
+datalake.format: paimon
+datalake.paimon.metastore: filesystem
+datalake.paimon.warehouse: /path/to/paimon/warehouse
+```
+
+Also make the Paimon connector, `paimon-bundle`, and required catalog or storage JARs available on
Review Comment:
I mean only the server-side setup part, rather than the entire Historical Partition Lookup section:
Historical partition lookups run on TabletServers. Configure `datalake.format` and the related
`datalake.<format>.*` options on every TabletServer, for example:
[configuration and dependency setup]
We can move this setup guidance to the Paimon data lake format page, and add a link to it as a prerequisite at the beginning of the
Historical Partition Lookup section here. The lookup-specific behavior and examples can remain on this page.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [docs] Add more details for historical partition lookup [fluss]
Posted by "luoyuxia (via GitHub)" <gi...@apache.org>.
luoyuxia commented on code in PR #4030:
URL: https://github.com/apache/fluss/pull/4030#discussion_r3803772451
##########
website/docs/engine-flink/lookups.md:
##########
@@ -289,6 +289,20 @@ partitioning enabled and exactly one partition key. When enabled, the Coordinato
retains the `__historical__` system partition used to route lookups to Paimon. Disabling the option
removes that system partition.
+Historical partition lookups run on TabletServers. Configure `datalake.format` and the related
+`datalake.<format>.*` options on every TabletServer, for example:
+
+```yaml title="server.yaml"
+datalake.enabled: true
+datalake.format: paimon
+datalake.paimon.metastore: filesystem
+datalake.paimon.warehouse: /path/to/paimon/warehouse
+```
+
+Also make the Paimon connector, `paimon-bundle`, and required catalog or storage JARs available on
Review Comment:
Just curious about do we really need to mentention about it since it's already in fluss distribution. But It reminds me that we did need to put paimon oss jar into `${FLUSS_HOME}/plugins/paimon/`.
Also, i feel like this page may not a right place to put this. Maybe `https://fluss.apache.org/docs/next/install-deploy/deploying-streaming-lakehouse/` or `https://fluss.apache.org/docs/next/streaming-lakehouse/datalake-formats/paimon/`. But we can add a link in this page for how to make historical partition lookups works
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [docs] Add more details for historical partition lookup [fluss]
Posted by "luoyuxia (via GitHub)" <gi...@apache.org>.
luoyuxia merged PR #4030:
URL: https://github.com/apache/fluss/pull/4030
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [docs] Add more details for historical partition lookup [fluss]
Posted by "zuston (via GitHub)" <gi...@apache.org>.
zuston commented on PR #4030:
URL: https://github.com/apache/fluss/pull/4030#issuecomment-5756976297
> @zuston Could you please also back port to release-1.0
sure!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org