You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@fluss.apache.org by "luoyuxia (via GitHub)" <gi...@apache.org> on 2026/07/10 02:35:56 UTC

[PR] [lookup] Support historical partition lookup for lake tables [fluss]

luoyuxia opened a new pull request, #3630:
URL: https://github.com/apache/fluss/pull/3630

   ### Purpose
   
   Linked issue: N/A (FIP-28 follow-up)
   
   This PR adds historical partition lookup support for datalake-enabled primary key tables. It lets clients resolve an expired Fluss partition to the corresponding lake partition and serve point lookups from lake storage when the partition is no longer available in Fluss.
   
   ### Brief change log
   
   - Add historical partition metadata utilities and client-side lookup plumbing for requests carrying partition names.
   - Add lake lookup SPI plus the Paimon implementation, including key conversion, row encoding, schema evolution padding, and lookup cache temp context.
   - Route historical lookup requests through TabletServer/ReplicaManager to lake lookup instead of normal KV lookup.
   - Add dedicated historical lookup admission control with `netty.server.max-queued-historical-requests` and a retriable throttle error.
   - Add client-side backoff for historical lookup throttling without blocking normal lookup dispatch.
   
   ### Tests
   
   - `./mvnw test -pl fluss-rpc -am -Dtest=ApiErrorTest -DfailIfNoTests=false`
   - `./mvnw test -pl fluss-client -am -Dtest=LookupSenderTest -DfailIfNoTests=false`
   - `./mvnw test -pl fluss-server -am -Dtest=HistoricalLakeLookupManagerTest -DfailIfNoTests=false`
   - `./mvnw spotless:check -pl fluss-common,fluss-rpc,fluss-client,fluss-server`
   
   Full affected-module `verify` was started locally, but the environment hit unrelated local-state issues: disk usage was above the server disk-lock threshold and stale `target/test-classes` contained a removed `LakeStorageUtilsTest` class.
   
   ### API and Format
   
   - Adds `LakeTableLookuper`/lookup context APIs for lake lookup implementations.
   - Adds `partition_name` to lookup RPC bucket requests for historical lookup routing.
   - Adds `HISTORICAL_LOOKUP_THROTTLED` RPC error mapping.
   - Adds `netty.server.max-queued-historical-requests` server config.
   
   ### Documentation
   
   - Adds the FIP-28 historical partition lookup design documentation in English and Chinese.
   
   ### Generative AI disclosure
   
   Yes. Generated with OpenAI Codex and reviewed by the human developer.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [lookup] Support historical partition lookup for lake tables [fluss]

Posted by "luoyuxia (via GitHub)" <gi...@apache.org>.
luoyuxia commented on PR #3630:
URL: https://github.com/apache/fluss/pull/3630#issuecomment-4941968270

   > Thanks for proposing this. @luoyuxia I’m also exploring this area with Fluss + Paimon.
   > 
   > Regarding the proposed client fallback to server-side historical partition lookup, I’m a bit concerned that this could be risky. The lake lookup operation can be much slower than a local RocksDB lookup, especially when Paimon is backed by HDFS with poor performance. If this lookup is executed on the tablet server side, it may slow down Netty worker threads and affect the latency of other RPC requests.
   > 
   > For the case without historical data ingestion, I think this could potentially be handled by direct client-side lake lookup instead. On the Fluss side, we could consider introducing a Flink hybrid lake lookup async function for this purpose. For the case with historical data ingestion, client-side lookup could still be used as a fallback when the historical partition + key is not found on the tablet server side.
   > 
   > Overall, I would prefer using client-side PK lookup where possible, to avoid overloading tablet servers during historical data ingestion bursts.
   
   Thanks for sharing your thoughts. First, I’d like to clarify that the expensive Paimon lookup does not run on the Netty worker thread. It is handed off to the TabletServer IO executor through CompletableFuture.supplyAsync. Its concurrency is bounded by a semaphore, and excess requests receive a retriable throttling error with client-side backoff.
   
   I agree that historical lookup still consumes additional TabletServer CPU, memory, network IO, and local disk for the Paimon lookup cache. This is a resource trade-off, even though it does not block the Netty event loop.
   
   An important motivation for server-side historical lookup is historical data ingestion. We want to support writing into historical partitions while still producing the correct changelog. The server-side write path therefore needs to retrieve the previous value from historical storage before applying a write. Some server-accessible historical lookup infrastructure is required independently of the client fallback in this PR; the client lookup path reuses the same infrastructure.
   
   Server-side lookup also keeps the generic Fluss lookup API independent of a particular lake implementation. Clients do not need to manage Paimon/Hadoop dependencies, lake credentials, catalog configuration, key encoding, snapshot selection, and fallback semantics.
   
   A Flink hybrid lake lookup async function would be useful for Flink workloads, but Fluss also has Java, Rust, CPP, Python clients. Requiring every client to implement its own lake lookup and fallback logic would duplicate behavior and would be particularly difficult for Rust and Python clients when the lake implementation is JVM-based.
   
   I think the two approaches can coexist, similar to Paimon’s local lookup and remote query service models. A remote lookup service can provide generic lookup capability for historical ingestion and all Fluss clients. The current PR runs this capability on TabletServers, but it could potentially be moved to a dedicated service later for better resource isolation. Client-side local lookup can then be introduced as an optional optimization where practical, with Flink async lookup as one possible first integration.
   
   For this PR, my preference is to keep the bounded asynchronous server-side path because it also supports historical ingestion semantics, and explore client-side lookup as an optional strategy in a follow-up.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [lookup] Support historical partition lookup for lake tables [fluss]

Posted by "wuchong (via GitHub)" <gi...@apache.org>.
wuchong commented on code in PR #3630:
URL: https://github.com/apache/fluss/pull/3630#discussion_r3644093281


##########
fluss-client/src/main/java/org/apache/fluss/client/lookup/LookupClient.java:
##########
@@ -100,7 +101,18 @@ public CompletableFuture<byte[]> lookup(
             TableBucket tableBucket,
             byte[] keyBytes,
             boolean insertIfNotExists) {
-        LookupQuery lookup = new LookupQuery(tablePath, tableBucket, keyBytes, insertIfNotExists);

Review Comment:
   This method is unused and can be removed. We should keep the interface simple to prevent misuse, as excessive method overloading increases complexity and the risk of errors.



##########
fluss-client/src/main/java/org/apache/fluss/client/lookup/LookupQuery.java:
##########


Review Comment:
   Add `@VisibleForTesting` annotatiton. 



##########
fluss-client/src/main/java/org/apache/fluss/client/lookup/LookupQuery.java:
##########
@@ -39,7 +41,16 @@ public class LookupQuery extends AbstractLookupQuery<byte[]> {
 
     LookupQuery(

Review Comment:
   This constructor can be removed. 



##########
fluss-server/src/main/java/org/apache/fluss/server/coordinator/CoordinatorService.java:
##########
@@ -742,17 +744,38 @@ public CompletableFuture<CreatePartitionResponse> createPartition(
 
         // first, validate the partition spec, and get resolved partition spec.
         PartitionSpec partitionSpec = getPartitionSpec(request.getPartitionSpec());
-        validatePartitionSpec(tablePath, tableInfo.getPartitionKeys(), partitionSpec, true);
-
-        // second, check whether the partition is out-of-date.
-        validateAutoPartitionTime(
-                partitionSpec,
-                tableInfo.getPartitionKeys(),
-                tableInfo.getTableConfig().getAutoPartitionStrategy());
+        ResolvedPartitionSpec partitionToCreate;
+        if (isHistoricalPartitionCreate(tableInfo, partitionSpec)) {
+            // Historical system partitions are lookup metadata, so creating one requires the same
+            // permission as reading the table instead of writing table data.
+            authorizeTable(OperationType.READ, tablePath);

Review Comment:
   Routing historical creation through this public `createPartition` RPC under `READ` authorization gives a read-only principal a DDL capability: a client can create persistent historical system partitions without the normal `WRITE` authorization, potentially consuming partition and replica quota. Please move ownership of this lifecycle to the coordinator instead. Introduce a dynamically alterable table-level historical-partition option, defaulting to `false`; when enabled, the coordinator/auto-partition manager should create and retain the required `__historical__` partition(s), and when disabled it should remove them. The lookup client can then assume that the historical routing partition already exists and only refresh/resolve metadata, which closes the READ-to-DDL privilege leak and simplifies the client path.



##########
fluss-client/src/main/java/org/apache/fluss/client/lookup/PrimaryKeyLookuper.java:
##########
@@ -115,19 +146,111 @@ public CompletableFuture<LookupResult> lookup(InternalRow lookupKey) {
                                 tableInfo.getTablePath(),
                                 metadataUpdater);
             } catch (PartitionNotExistException e) {
-                return CompletableFuture.completedFuture(new LookupResult(Collections.emptyList()));
+                return mayFallbackToHistoricalLookup(
+                        bucketId, pkBytes, originalPartitionName, lookupKey);
             }
         }
 
-        int bucketId = bucketingFunction.bucketing(bkBytes, numBuckets);
         TableBucket tableBucket = new TableBucket(tableInfo.getTableId(), partitionId, bucketId);
+        return lookupBucket(tableBucket, pkBytes, insertIfNotExists, null, lookupKey);
+    }
+
+    /** Falls back to historical lookup if the lookup key belongs to a historical partition. */
+    private CompletableFuture<LookupResult> mayFallbackToHistoricalLookup(
+            int bucketId, byte[] keyBytes, String originalPartitionName, InternalRow lookupKey) {
+        // The lookuper keeps tableInfo from creation time, but retention can be changed later.
+        // Fetch the latest table metadata before classifying a missing partition as historical and
+        // verify that the table path still refers to the same table.
+        return admin.getTableInfo(tableInfo.getTablePath())

Review Comment:
   We can fail fast if the table does not enable historical partitions, eliminating the need to fetch table metadata again. Instead, we can directly send the historical lookup request to the server, which will verify whether the partition has expired. This approach is more reliable as it avoids potential clock skew issues between the client and the server.



##########
fluss-client/src/main/java/org/apache/fluss/client/lookup/LookupSender.java:
##########
@@ -236,7 +254,47 @@ private void sendLookupRequest(
                                         acks,
                                         maxRequestTimeoutMs),
                                 tableId,
-                                lookupsByBucket));
+                                lookupsByBucket);
+                    }
+                });
+    }
+
+    /**
+     * Packs lookup batches into RPC request groups so each request has one lookup kind and response
+     * dispatch can still use {@link TableBucket} as the lookup key.
+     *
+     * <p>Normal and historical lookups are sent in separate RPCs. Historical lookups can have
+     * multiple batches with the same target {@link TableBucket} but different original partition
+     * names. The request carries that original partition name, but the response does not. If those
+     * batches were sent in one RPC, two response buckets would have the same {@link TableBucket}
+     * and the client could not map each response back to the right batch.

Review Comment:
   Can we add `original_partition_name` to the `PbLookupRespForBucket`? So that we can recognize the response buckets for each original partition. This can simplify much the client logic and improve the throughput. 



##########
fluss-common/src/main/java/org/apache/fluss/config/ConfigOptions.java:
##########
@@ -157,6 +157,13 @@ public class ConfigOptions {
                     .withDescription(
                             "The default size of the write buffer for writing the local files to remote file systems.");
 
+    public static final ConfigOption<String> IO_TMP_DIR =
+            key("io.tmpdir")

Review Comment:
   `IO_TMP_DIR` is only consumed by the server-side historical lake lookup path, while Fluss already scopes comparable options under `server.*` (for example, `server.io-pool.size`) and uses `client.scanner.io.tmpdir` for the client-side temporary directory. Exposing this new server-only setting as the generic `io.tmpdir` makes the public configuration namespace inconsistent and can cause it to be mistaken for a process-wide option. Maybe `server.io.tmpdir` is better here. 



##########
fluss-common/src/main/java/org/apache/fluss/lake/lakestorage/LakeTableLookuper.java:
##########
@@ -0,0 +1,92 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.fluss.lake.lakestorage;
+
+import org.apache.fluss.annotation.PublicEvolving;
+import org.apache.fluss.metadata.ResolvedPartitionSpec;
+import org.apache.fluss.types.RowType;
+
+import javax.annotation.Nullable;
+
+import static org.apache.fluss.utils.Preconditions.checkNotNull;
+
+/**
+ * A table-level point lookuper for data stored in lake storage.
+ *
+ * @since 1.0
+ */
+@PublicEvolving
+public interface LakeTableLookuper extends AutoCloseable {
+
+    /**
+     * Looks up one key from the lake table.

Review Comment:
   We should add description that the implementation should be thread-safe. 



##########
fluss-server/src/main/java/org/apache/fluss/server/tablet/TabletService.java:
##########
@@ -292,30 +296,59 @@ public CompletableFuture<PutKvResponse> putKv(PutKvRequest request) {
 
     @Override
     public CompletableFuture<LookupResponse> lookup(LookupRequest request) {
-        Map<TableBucket, List<byte[]>> lookupData = toLookupData(request);
         Map<TableBucket, LookupResultForBucket> errorResponseMap = new HashMap<>();
         CompletableFuture<LookupResponse> response = new CompletableFuture<>();
 
         if (request.hasInsertIfNotExists() && request.isInsertIfNotExists()) {
             authorizeTable(WRITE, request.getTableId());
+            if (hasHistoricalLookup(request)) {
+                throw new InvalidTableException(
+                        "Lookup with insertIfNotExists is not supported for "
+                                + "historical partition lookup.");
+            }
+            Map<TableBucket, List<byte[]>> normalLookupData = toLookupData(request);
             replicaManager.lookups(
                     request.isInsertIfNotExists(),
                     request.getTimeoutMs(),
                     request.getAcks(),
-                    lookupData,
+                    normalLookupData,
                     currentSession().getApiVersion(),
                     value -> response.complete(makeLookupResponse(value, errorResponseMap)));
         } else {
-            Map<TableBucket, List<byte[]>> interesting =
-                    authorizeRequestData(
-                            READ, lookupData, errorResponseMap, LookupResultForBucket::new);
-            if (interesting.isEmpty()) {
-                return CompletableFuture.completedFuture(makeLookupResponse(errorResponseMap));
+            boolean historicalLookupRequest = hasHistoricalLookup(request);
+            if (historicalLookupRequest) {

Review Comment:
   IIUC, we don't allow hybrid normal lookup and historical lookup in a single RPC, currently, it is guaranteed by client. But server should add this hard check, otherwise, normal lookup will also be routed to historical lookup and returns wrong result. 



##########
fluss-client/src/main/java/org/apache/fluss/client/lookup/PrimaryKeyLookuper.java:
##########
@@ -53,6 +61,16 @@ class PrimaryKeyLookuper extends AbstractLookuper implements Lookuper {
     private final BucketingFunction bucketingFunction;
     private final int numBuckets;
     private final boolean insertIfNotExists;
+    private final Admin admin;
+    private final HistoricalPartitionResolver historicalPartitionResolver;
+
+    /**
+     * Partition names already confirmed as expired and eligible for historical lookup.
+     *
+     * <p>Expired partitions cannot be recreated or written in the current lifecycle, so this
+     * decision remains valid for the lifetime of the lookuper.
+     */
+    private final Set<String> confirmedHistoricalPartitions;

Review Comment:
   Add a TODO for this, in the future, we can introduce a dedicate `list_historical_partitions` or `list_all_partitions`, to initialize the set at the beginning, to avoid too many exception which slows down the bootstrap perfromance. 



##########
fluss-lake/fluss-lake-paimon/src/main/java/org/apache/fluss/lake/paimon/lookup/PaimonLakeTableLookuper.java:
##########
@@ -0,0 +1,421 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.fluss.lake.paimon.lookup;
+
+import org.apache.fluss.config.ConfigOptions;
+import org.apache.fluss.config.Configuration;
+import org.apache.fluss.config.TableConfig;
+import org.apache.fluss.exception.KvStorageException;
+import org.apache.fluss.lake.lakestorage.LakeTableLookuper;
+import org.apache.fluss.lake.paimon.utils.PaimonPartitionBucket;
+import org.apache.fluss.lake.paimon.utils.PaimonRowAsFlussRow;
+import org.apache.fluss.metadata.ResolvedPartitionSpec;
+import org.apache.fluss.metadata.TablePath;
+import org.apache.fluss.row.BinaryRow;
+import org.apache.fluss.row.InternalRow;
+import org.apache.fluss.row.decode.CompactedKeyDecoder;
+import org.apache.fluss.row.encode.RowEncoder;
+import org.apache.fluss.row.encode.ValueEncoder;
+import org.apache.fluss.row.encode.paimon.PaimonKeyEncoder;
+import org.apache.fluss.types.RowType;
+import org.apache.fluss.utils.IOUtils;
+
+import org.apache.paimon.CoreOptions;
+import org.apache.paimon.catalog.Catalog;
+import org.apache.paimon.catalog.CatalogContext;
+import org.apache.paimon.catalog.CatalogFactory;
+import org.apache.paimon.disk.IOManager;
+import org.apache.paimon.io.DataFileMeta;
+import org.apache.paimon.memory.MemorySegment;
+import org.apache.paimon.options.MemorySize;
+import org.apache.paimon.options.Options;
+import org.apache.paimon.table.FileStoreTable;
+import org.apache.paimon.table.query.LocalTableQuery;
+import org.apache.paimon.table.sink.RowPartitionKeyExtractor;
+import org.apache.paimon.table.source.DataSplit;
+import org.apache.paimon.table.source.InnerTableScan;
+import org.apache.paimon.table.source.Split;
+import org.apache.paimon.types.DataField;
+
+import javax.annotation.Nullable;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Set;
+
+import static org.apache.fluss.config.ConfigOptions.KV_FORMAT_VERSION_2;
+import static org.apache.fluss.lake.paimon.PaimonLakeCatalog.SYSTEM_COLUMNS;
+import static org.apache.fluss.lake.paimon.utils.PaimonConversions.toPaimon;
+import static org.apache.fluss.lake.paimon.utils.PaimonConversions.toPaimonPartition;
+import static org.apache.fluss.utils.Preconditions.checkNotNull;
+
+/**
+ * Paimon implementation of {@link LakeTableLookuper} for primary-key tables.
+ *
+ * <p>The catalog, table, local query, and I/O manager are initialized lazily on the first lookup.
+ * For each partition and bucket, the lookuper scans the latest Paimon snapshot once and registers
+ * its data files with {@link LocalTableQuery}. Paimon then creates local lookup files lazily as
+ * individual remote data files are queried.
+ *
+ * <p>A cached partition-bucket file set can become stale when Paimon compaction replaces its data
+ * files and snapshot expiration physically deletes the old files. Because {@code FileIO}
+ * implementations may represent a missing file with different {@link IOException} types, the first
+ * lookup I/O failure closes the cached query state, reopens the table from the latest snapshot, and
+ * retries once.
+ *
+ * <p>Lookup and close operations are synchronized because they share mutable Paimon query, local
+ * cache, and value-encoding state.
+ */
+public class PaimonLakeTableLookuper implements LakeTableLookuper {
+
+    // Each TabletServer caches at most ten table lookupers, bounding their retained lookup cache
+    // capacity to 20GB. See HistoricalLakeLookupManager for the follow-up to make this configurable
+    // and use a global Paimon IOManager limit.
+    private static final String LOOKUP_CACHE_MAX_DISK_SIZE = "2gb";
+    private static final MemorySize LOOKUP_CACHE_MAX_DISK_MEMORY_SIZE =
+            MemorySize.parse(LOOKUP_CACHE_MAX_DISK_SIZE);
+
+    private final Configuration paimonConfig;
+    private final TablePath tablePath;
+    private final String ioTmpDir;
+    private final TableConfig tableConfig;
+
+    private final Set<PaimonPartitionBucket> initializedBuckets;
+
+    private @Nullable Catalog catalog;
+    private @Nullable FileStoreTable fileStoreTable;
+    private @Nullable IOManager ioManager;
+    private @Nullable LocalTableQuery localTableQuery;
+    private @Nullable RowPartitionKeyExtractor partitionKeyExtractor;
+    private int primaryKeyFieldCount;
+
+    // Both encoders are initialized only for a kv-format-v2 table whose bucket key differs from
+    // its physical primary key. They remain null when the incoming Fluss key already uses Paimon's
+    // BinaryRow encoding and no conversion is needed.
+    private @Nullable CompactedKeyDecoder compactedKeyDecoder;
+    private @Nullable PaimonKeyEncoder paimonKeyEncoder;
+    private boolean hasCachedValueEncoder;
+    private short cachedValueSchemaId;
+    private @Nullable RowEncoder cachedValueRowEncoder;
+    private @Nullable InternalRow.FieldGetter[] cachedValueFieldGetters;
+    private boolean closed;
+
+    public PaimonLakeTableLookuper(
+            Configuration paimonConfig, TablePath tablePath, TableConfig tableConfig) {
+        this(paimonConfig, tablePath, ConfigOptions.IO_TMP_DIR.defaultValue(), tableConfig);
+    }
+
+    public PaimonLakeTableLookuper(
+            Configuration paimonConfig,
+            TablePath tablePath,
+            String ioTmpDir,
+            TableConfig tableConfig) {
+        this.paimonConfig = checkNotNull(paimonConfig, "paimonConfig must not be null.");
+        this.tablePath = checkNotNull(tablePath, "tablePath must not be null.");
+        this.ioTmpDir = checkNotNull(ioTmpDir, "ioTmpDir must not be null.");
+        this.tableConfig = checkNotNull(tableConfig, "tableConfig must not be null.");
+        this.initializedBuckets = new HashSet<>();
+    }
+
+    @Override
+    public synchronized @Nullable byte[] lookup(byte[] key, LookupContext context)
+            throws Exception {
+        checkNotNull(key, "key must not be null.");
+        checkNotNull(context, "context must not be null.");
+        checkNotClosed();
+        ensureInitialized(context.valueRowType());
+
+        org.apache.paimon.data.BinaryRow partition =
+                convertPartition(context.partitionSpec(), context.valueRowType());
+        org.apache.paimon.data.BinaryRow keyRow = toPaimonLookupKey(key);
+        initializeFilesIfNeeded(partition, context.bucketId());
+
+        org.apache.paimon.data.InternalRow paimonRow =
+                lookupWithFileRefresh(
+                        partition, context.bucketId(), keyRow, context.valueRowType());
+        if (paimonRow == null) {
+            return null;
+        }
+        return encodeValue(paimonRow, context.schemaId(), context.valueRowType());
+    }
+
+    @Override
+    public synchronized void close() {
+        if (closed) {
+            return;
+        }
+        closed = true;
+        IOUtils.closeQuietly(cachedValueRowEncoder, "Fluss value row encoder");
+        IOUtils.closeQuietly(localTableQuery, "Paimon local table query");
+        IOUtils.closeQuietly(ioManager, "Paimon lookup IO manager");
+        IOUtils.closeQuietly(catalog, "Paimon catalog");
+        initializedBuckets.clear();
+    }
+
+    private void checkNotClosed() {
+        if (closed) {
+            throw new IllegalStateException("Paimon lake table lookuper has been closed.");
+        }
+    }
+
+    private void ensureInitialized(RowType valueRowType) throws Exception {
+        if (localTableQuery != null) {
+            return;
+        }
+
+        catalog =
+                CatalogFactory.createCatalog(
+                        CatalogContext.create(Options.fromMap(paimonConfig.toMap())));
+        fileStoreTable =
+                withLookupCacheOptions((FileStoreTable) catalog.getTable(toPaimon(tablePath)));
+        if (fileStoreTable.primaryKeys().isEmpty()) {
+            throw new UnsupportedOperationException(
+                    "Point lookup is only supported for primary-key Paimon tables.");
+        }
+        ioManager = createIOManager(ioTmpDir);
+        localTableQuery = createLocalTableQuery();
+        partitionKeyExtractor = new RowPartitionKeyExtractor(fileStoreTable.schema());
+        List<String> trimmedPrimaryKeys = fileStoreTable.schema().trimmedPrimaryKeys();
+        primaryKeyFieldCount = trimmedPrimaryKeys.size();
+        initializeLookupKeyConverter(valueRowType, trimmedPrimaryKeys);
+    }
+
+    private void initializeLookupKeyConverter(
+            RowType valueRowType, List<String> trimmedPrimaryKeys) {
+        // Legacy/v1 tables and v2 tables with a default bucket key already encode Fluss lookup
+        // keys with Paimon's key encoder. Only v2 tables with a non-default bucket key use the
+        // compacted key encoding and need conversion before querying Paimon.
+        if (tableConfig.getKvFormatVersion().orElse(1) != KV_FORMAT_VERSION_2
+                || fileStoreTable().schema().bucketKeys().equals(trimmedPrimaryKeys)) {
+            return;
+        }
+
+        // Kv-format-v2 tables with a non-default bucket key store Fluss keys using the compacted
+        // encoding to support prefix lookup. Paimon's LocalTableQuery expects its own BinaryRow
+        // encoding, so convert the key at the lake lookup boundary.
+        compactedKeyDecoder =
+                CompactedKeyDecoder.createKeyDecoder(valueRowType, trimmedPrimaryKeys);
+        RowType keyRowType = valueRowType.project(trimmedPrimaryKeys);
+        paimonKeyEncoder = new PaimonKeyEncoder(keyRowType, trimmedPrimaryKeys);
+    }
+
+    private LocalTableQuery createLocalTableQuery() {
+        return fileStoreTable()
+                .newLocalTableQuery()
+                .withValueProjection(businessFieldProjection(fileStoreTable()))
+                .withIOManager(checkNotNull(ioManager, "ioManager must be initialized."));
+    }
+
+    private FileStoreTable withLookupCacheOptions(FileStoreTable table) {
+        String key = CoreOptions.LOOKUP_CACHE_MAX_DISK_SIZE.key();
+        String configuredMaxDiskSize = table.options().get(key);
+        if (configuredMaxDiskSize != null
+                && MemorySize.parse(configuredMaxDiskSize)
+                                .compareTo(LOOKUP_CACHE_MAX_DISK_MEMORY_SIZE)
+                        <= 0) {
+            return table;
+        }
+        return table.copy(Collections.singletonMap(key, LOOKUP_CACHE_MAX_DISK_SIZE));
+    }
+
+    private static IOManager createIOManager(String ioTmpDir) {
+        return IOManager.create(ioTmpDir);
+    }
+
+    private static int[] businessFieldProjection(FileStoreTable fileStoreTable) {
+        List<DataField> fields = fileStoreTable.schema().logicalRowType().getFields();
+        List<Integer> projectedFields = new ArrayList<>();
+        for (int i = 0; i < fields.size(); i++) {
+            if (!SYSTEM_COLUMNS.containsKey(fields.get(i).name())) {
+                projectedFields.add(i);
+            }
+        }
+
+        int[] projection = new int[projectedFields.size()];
+        for (int i = 0; i < projectedFields.size(); i++) {
+            projection[i] = projectedFields.get(i);
+        }
+        return projection;
+    }
+
+    private org.apache.paimon.data.BinaryRow convertPartition(
+            ResolvedPartitionSpec partitionSpec, RowType valueRowType) {
+        return toPaimonPartition(
+                partitionSpec,
+                valueRowType,
+                fileStoreTable().schema().logicalRowType(),
+                partitionKeyExtractor()::partition);
+    }
+
+    private org.apache.paimon.data.BinaryRow toPaimonLookupKey(byte[] key) {
+        byte[] paimonKey = key;
+        if (compactedKeyDecoder != null) {
+            // A non-null decoder means the Fluss lookup key uses compacted encoding. Decode it
+            // first, then re-encode it as the Paimon BinaryRow expected by LocalTableQuery.
+            InternalRow decodedKey = compactedKeyDecoder.decodeKey(key);
+            paimonKey =
+                    checkNotNull(paimonKeyEncoder, "Paimon key encoder must be initialized.")
+                            .encodeKey(decodedKey);
+        }
+
+        org.apache.paimon.data.BinaryRow keyRow =
+                new org.apache.paimon.data.BinaryRow(primaryKeyFieldCount);
+        keyRow.pointTo(MemorySegment.wrap(paimonKey), 0, paimonKey.length);
+        return keyRow;
+    }
+
+    private void initializeFilesIfNeeded(org.apache.paimon.data.BinaryRow partition, int bucketId) {
+        PaimonPartitionBucket partitionBucket = new PaimonPartitionBucket(partition, bucketId);

Review Comment:
   `RowPartitionKeyExtractor` uses a generated projection whose default is `reusedOutRecord=true`, so `convertPartition` returns the same mutable `BinaryRow` instance on every lookup. This line retains that row in `initializedBuckets`, and `refreshFiles` also retains it as a key in `LocalTableQuery.tableView`; converting the next partition mutates keys that are already inside hash-based collections. In particular, two partitions with the same `BinaryRow.hashCode()` make the second partition appear initialized and reuse the first partition file view, which can return a row from the wrong lake partition or produce a false miss. Please copy the projected row before it escapes (for example, `partitionKeyExtractor().partition(...).copy()`) and add a two-partition hash-collision regression test.



##########
fluss-server/src/main/java/org/apache/fluss/server/replica/ReplicaManager.java:
##########
@@ -237,6 +246,47 @@ public ReplicaManager(
             ExecutorService ioExecutor,
             LocalDiskManager localDiskManager)
             throws IOException {
+        this(
+                conf,
+                scheduler,
+                logManager,
+                kvManager,
+                zkClient,
+                serverId,
+                metadataCache,
+                rpcClient,
+                coordinatorGateway,
+                completedKvSnapshotCommitter,
+                fatalErrorHandler,
+                serverMetricGroup,
+                userMetrics,
+                scannerManager,
+                clock,
+                ioExecutor,
+                localDiskManager,
+                null);
+    }
+
+    public ReplicaManager(

Review Comment:
   Please avoid introducing too many constructors, especially public constructors. Otherwise, it's easy to use the wrong constructor. 



##########
fluss-server/src/main/java/org/apache/fluss/server/replica/HistoricalLakeLookupManager.java:
##########
@@ -0,0 +1,403 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.fluss.server.replica;
+
+import org.apache.fluss.annotation.VisibleForTesting;
+import org.apache.fluss.config.ConfigOptions;
+import org.apache.fluss.config.Configuration;
+import org.apache.fluss.config.TableConfig;
+import org.apache.fluss.exception.FlussRuntimeException;
+import org.apache.fluss.exception.HistoricalLookupThrottledException;
+import org.apache.fluss.exception.InvalidPartitionException;
+import org.apache.fluss.exception.LakeStorageNotConfiguredException;
+import org.apache.fluss.lake.lakestorage.LakeStorage;
+import org.apache.fluss.lake.lakestorage.LakeStoragePlugin;
+import org.apache.fluss.lake.lakestorage.LakeStoragePluginSetUp;
+import org.apache.fluss.lake.lakestorage.LakeTableLookuper;
+import org.apache.fluss.metadata.DataLakeFormat;
+import org.apache.fluss.metadata.ResolvedPartitionSpec;
+import org.apache.fluss.metadata.SchemaInfo;
+import org.apache.fluss.metadata.TableBucket;
+import org.apache.fluss.metadata.TableInfo;
+import org.apache.fluss.metadata.TablePath;
+import org.apache.fluss.plugin.PluginManager;
+import org.apache.fluss.rpc.entity.LookupResultForBucket;
+import org.apache.fluss.rpc.protocol.ApiError;
+import org.apache.fluss.server.entity.LookupDataForBucket;
+import org.apache.fluss.utils.FileUtils;
+import org.apache.fluss.utils.IOUtils;
+import org.apache.fluss.utils.concurrent.Scheduler;
+
+import com.github.benmanes.caffeine.cache.Cache;
+import com.github.benmanes.caffeine.cache.Caffeine;
+import com.github.benmanes.caffeine.cache.RemovalCause;
+import com.github.benmanes.caffeine.cache.Ticker;
+
+import javax.annotation.Nullable;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Semaphore;
+
+import static org.apache.fluss.server.utils.LakeStorageUtils.extractLakeProperties;
+import static org.apache.fluss.utils.Preconditions.checkArgument;
+import static org.apache.fluss.utils.Preconditions.checkNotNull;
+
+/**
+ * Handles server-side point lookup for historical partitions stored in lake storage.
+ *
+ * <p>Accepted requests run on the TabletServer I/O executor. A semaphore bounds the total number of
+ * accepted historical lookup tasks so slow lake storage cannot create an unbounded request backlog.
+ *
+ * <p>Creating a lake table lookuper may initialize catalog, table, and query state and allocate
+ * local lookup files, so lookupers are cached and reused. The cache is keyed by table ID rather
+ * than table path to prevent a deleted and recreated table from reusing the old table's lookuper. A
+ * cached lookuper is replaced when its schema ID no longer matches the requested table schema.
+ *
+ * <p>A lookuper is closed when replaced, explicitly invalidated by a replica lifecycle event,
+ * evicted after the cache reaches ten tables, the manager shuts down, or after three hours without
+ * access. Caffeine expiration is scheduled on the shared TabletServer scheduler, allowing idle
+ * resources to be released even if no subsequent lookup accesses the cache.
+ */
+class HistoricalLakeLookupManager implements AutoCloseable {
+
+    private static final String PAIMON_LOOKUP_DIR_NAME = "paimon-lookup";
+    private static final String LOOKUPER_CACHE_EXPIRATION_TASK_NAME =
+            "historical-lookuper-cache-expiration";
+    private static final Duration LOOKUPER_CACHE_EXPIRATION = Duration.ofHours(3);
+    private static final int MAX_CACHED_LOOKUPERS = 10;
+
+    // TODO: MAX_CACHED_LOOKUPERS and the 2GB per-table limit configured by
+    // PaimonLakeTableLookuper through Paimon's "lookup.cache-max-disk-size" option are hard-coded.
+    // Make them configurable, and prefer a Paimon IOManager-level global disk limit shared by all
+    // table lookupers because fixed per-table limits can underutilize cache for hot tables while
+    // reserving too much for cold tables.
+
+    private final Configuration conf;
+    private final @Nullable PluginManager pluginManager;
+    private final ExecutorService ioExecutor;
+    private final int serverId;
+    private final Semaphore lookupPermits;
+    private final Cache<Long, CachedLakeTableLookuper> lakeTableLookupers;
+    private @Nullable String paimonLookupTempDir;
+
+    HistoricalLakeLookupManager(
+            Configuration conf,
+            @Nullable PluginManager pluginManager,
+            ExecutorService ioExecutor,
+            int serverId,
+            Scheduler scheduler) {
+        this(
+                conf,
+                pluginManager,
+                ioExecutor,
+                serverId,
+                Ticker.systemTicker(),
+                createCacheScheduler(scheduler));
+    }
+
+    @VisibleForTesting
+    HistoricalLakeLookupManager(
+            Configuration conf,
+            @Nullable PluginManager pluginManager,
+            ExecutorService ioExecutor,
+            int serverId,
+            Ticker ticker,
+            com.github.benmanes.caffeine.cache.Scheduler cacheScheduler) {
+        this.conf = checkNotNull(conf, "conf must not be null.");
+        this.pluginManager = pluginManager;
+        this.ioExecutor = checkNotNull(ioExecutor, "ioExecutor must not be null.");
+        this.serverId = serverId;
+        this.lakeTableLookupers =
+                Caffeine.newBuilder()
+                        .expireAfterAccess(LOOKUPER_CACHE_EXPIRATION)
+                        .maximumSize(MAX_CACHED_LOOKUPERS)
+                        .ticker(checkNotNull(ticker, "ticker must not be null."))
+                        .scheduler(checkNotNull(cacheScheduler, "cacheScheduler must not be null."))
+                        .executor(Runnable::run)
+                        .removalListener(
+                                (Long ignored,
+                                        CachedLakeTableLookuper cachedLookuper,
+                                        RemovalCause ignoredCause) -> {
+                                    if (cachedLookuper != null) {
+                                        cachedLookuper.invalidate();
+                                    }
+                                })
+                        .build();
+        int maxQueuedHistoricalRequests =
+                conf.get(ConfigOptions.NETTY_SERVER_MAX_QUEUED_HISTORICAL_REQUESTS);
+        checkArgument(
+                maxQueuedHistoricalRequests > 0,
+                "%s must be greater than 0.",
+                ConfigOptions.NETTY_SERVER_MAX_QUEUED_HISTORICAL_REQUESTS.key());
+        this.lookupPermits = new Semaphore(maxQueuedHistoricalRequests);
+    }
+
+    private static com.github.benmanes.caffeine.cache.Scheduler createCacheScheduler(
+            Scheduler scheduler) {
+        checkNotNull(scheduler, "scheduler must not be null.");
+        // Schedule expiration maintenance so idle lookupers are closed even if no more lookups
+        // arrive.
+        return (executor, command, delay, timeUnit) ->
+                scheduler.scheduleOnce(
+                        LOOKUPER_CACHE_EXPIRATION_TASK_NAME,
+                        () -> executor.execute(command),
+                        timeUnit.toMillis(delay));
+    }
+
+    CompletableFuture<LookupResultForBucket> lookup(
+            LookupDataForBucket lookupData, TableInfo tableInfo, SchemaInfo schemaInfo) {
+        TableBucket tableBucket = lookupData.tableBucket();
+        if (!lookupPermits.tryAcquire()) {
+            return CompletableFuture.completedFuture(
+                    new LookupResultForBucket(
+                            tableBucket,
+                            ApiError.fromThrowable(
+                                    new HistoricalLookupThrottledException(
+                                            "Historical lookup is throttled for "
+                                                    + tableBucket
+                                                    + "."))));
+        }
+
+        CompletableFuture<LookupResultForBucket> future;
+        try {
+            future =
+                    CompletableFuture.supplyAsync(
+                            () -> lookupInternal(lookupData, tableInfo, schemaInfo), ioExecutor);

Review Comment:
   `ioExecutor` is the shared TabletServer I/O pool (10 threads by default), while `PaimonLakeTableLookuper.lookup` synchronizes the entire table lookuper. For a hot table, one worker performs the lake I/O and the remaining accepted lookups can occupy other shared workers waiting on the same monitor; with the historical admission limit defaulting to 50, this can exhaust the pool and starve snapshot transfer, remote-log work, and other server I/O. Please isolate historical lookup on a bounded dedicated executor or reserve capacity in the shared pool. 
   
   Longer term, an event-pool model with table-ID affinity could route all lookup events for the same table to one worker thread. This preserves per-table serialization without wasting multiple threads on synchronized waits.
   



##########
fluss-server/src/main/java/org/apache/fluss/server/utils/ServerRpcMessageUtils.java:
##########
@@ -1138,6 +1139,50 @@ public static Map<TableBucket, List<byte[]>> toLookupData(LookupRequest lookupRe
         return lookupEntryData;
     }
 
+    /**
+     * Converts lookup bucket requests for historical partition lookup.
+     *
+     * <p>Unlike normal lookup data, historical lookup must keep the original partition name carried
+     * by each bucket request. The name is later used to resolve the original partition in lake
+     * storage, while the {@link TableBucket} points to the historical system partition.
+     */
+    public static Map<TableBucket, LookupDataForBucket> toHistoricalLookupData(
+            LookupRequest lookupRequest) {
+        long tableId = lookupRequest.getTableId();
+        Map<TableBucket, LookupDataForBucket> lookupEntryData = new HashMap<>();
+        for (PbLookupReqForBucket lookupReqForBucket : lookupRequest.getBucketsReqsList()) {
+            TableBucket tb =
+                    new TableBucket(
+                            tableId,
+                            lookupReqForBucket.hasPartitionId()
+                                    ? lookupReqForBucket.getPartitionId()
+                                    : null,
+                            lookupReqForBucket.getBucketId());
+            List<byte[]> keys = new ArrayList<>(lookupReqForBucket.getKeysCount());
+            for (int i = 0; i < lookupReqForBucket.getKeysCount(); i++) {
+                keys.add(lookupReqForBucket.getKeyAt(i));
+            }
+            LookupDataForBucket lookupData =
+                    new LookupDataForBucket(
+                            tb,
+                            keys,
+                            lookupReqForBucket.hasOriginalPartitionName()
+                                    ? lookupReqForBucket.getOriginalPartitionName()
+                                    : null);
+            LookupDataForBucket previous = lookupEntryData.putIfAbsent(tb, lookupData);
+            if (previous != null) {
+                throw new IllegalArgumentException(
+                        String.format(
+                                "Duplicate historical lookup requests for table bucket %s with "

Review Comment:
   We should allow different original partition for the same table bucket on a single RPC, this is very important for the performance on production. 



##########
fluss-server/src/main/java/org/apache/fluss/server/replica/ReplicaManager.java:
##########
@@ -775,6 +873,52 @@ public void lookups(
         lookups(false, null, null, entriesPerBucket, apiVersion, responseCallback);
     }
 
+    /** Lookup historical lake data for requests that carry original partition names. */
+    public void historicalLookups(
+            List<LookupDataForBucket> lookupData,
+            Consumer<Map<TableBucket, LookupResultForBucket>> responseCallback) {
+        if (lookupData.isEmpty()) {
+            responseCallback.accept(Collections.emptyMap());
+            return;
+        }
+
+        Map<TableBucket, LookupResultForBucket> result = new ConcurrentHashMap<>();
+        AtomicInteger remainingLookups = new AtomicInteger(lookupData.size());
+        for (LookupDataForBucket data : lookupData) {
+            CompletableFuture<LookupResultForBucket> lookupFuture;
+            try {
+                Replica replica = getReplicaOrException(data.tableBucket());

Review Comment:
   Should throw an exception if the replica is not a priamry key table. 



##########
fluss-server/src/main/java/org/apache/fluss/server/replica/ReplicaManager.java:
##########
@@ -979,6 +1129,9 @@ public void stopReplicas(
 
                     for (StopReplicaData data : stopReplicaDataList) {
                         TableBucket tb = data.getTableBucket();
+                        long tableId = tb.getTableId();
+                        ioExecutor.execute(
+                                () -> historicalLakeLookupManager.invalidateTableLookuper(tableId));

Review Comment:
   `StopReplica` does not imply that the table was dropped: it is also sent for replica-offline transitions, reassignments, and expired-partition deletion. This code invalidates the table-scoped lookuper once per bucket, so a normal rebalance or a multi-bucket partition expiration can repeatedly discard Paimon's catalog, `LocalTableQuery`, `IOManager`, and local lookup cache, forcing later historical lookups through expensive cold initialization. A newly expired partition does not need table-wide invalidation because the lookuper initializes each `(partition, bucket)` lazily, and schema changes are already handled by the `schemaId` check.
   
   Please remove this invalidation from `stopReplicas` and trigger it only when `UpdateMetadata` applies an actual table-deletion tombstone, deduplicated by table ID, for example:
   
   ```java
   Set<Long> deletedTableIds = metadataCache.updateClusterMetadata(clusterMetadata);
   updateReplicaTableConfig(clusterMetadata);
   
   if (!deletedTableIds.isEmpty()) {
       ioExecutor.execute(
               () ->
                       deletedTableIds.forEach(
                               historicalLakeLookupManager::invalidateTableLookuper));
   }
   ```
   
   Filtering on `deleteRemote` is not sufficient because expired-partition deletion also sets it.



##########
fluss-client/src/main/java/org/apache/fluss/client/lookup/LookupSender.java:
##########


Review Comment:
   @luoyuxia This doesn't match the FIP-28 design, could you add a TODO comment to explain this? I'm afraid we may need to introduce this configuration sooner or later. 



##########
fluss-client/src/main/java/org/apache/fluss/client/lookup/LookupSender.java:
##########
@@ -445,6 +504,18 @@ private void reEnqueueLookup(AbstractLookupQuery<?> lookup) {
         lookupQueue.reEnqueue(lookup);
     }
 
+    private long prepareRetry(AbstractLookupQuery<?> lookup, Exception exception) {

Review Comment:
   Add a TODO above this, this will introduce out-of-order for the lookup queries. We will see whether we need to handle this in the future. 



##########
fluss-server/src/main/java/org/apache/fluss/server/coordinator/AutoPartitionManager.java:
##########
@@ -541,13 +542,14 @@ private void dropPartitions(
                 currentPartitions.headMap(lastRetainPartitionTime).entrySet().iterator();
         while (iterator.hasNext()) {
             Map.Entry<String, Set<String>> entry = iterator.next();
-            dropPartitions(tablePath, partitionKeys, iterator, entry);
+            dropPartitions(tablePath, partitionKeys, autoPartitionStrategy, iterator, entry);

Review Comment:
   If we introduce historical partition enable option on table, we can simple check whether the table enabled this option, and check the time partition value (`entry.key()`) is equal to `__historical__`, then we can skip to drop this partition. This can avoid to change the logic of `dropPartitions`. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [lookup] Support historical partition lookup for lake tables [fluss]

Posted by "luoyuxia (via GitHub)" <gi...@apache.org>.
luoyuxia commented on PR #3630:
URL: https://github.com/apache/fluss/pull/3630#issuecomment-5018034458

   @wuchong Could you please help have a final review?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [lookup] Support historical partition lookup for lake tables [fluss]

Posted by "luoyuxia (via GitHub)" <gi...@apache.org>.
luoyuxia commented on code in PR #3630:
URL: https://github.com/apache/fluss/pull/3630#discussion_r3601523117


##########
fluss-lake/fluss-lake-paimon/src/main/java/org/apache/fluss/lake/paimon/lookup/PaimonLakeTableLookuper.java:
##########
@@ -0,0 +1,359 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.fluss.lake.paimon.lookup;
+
+import org.apache.fluss.config.ConfigOptions;
+import org.apache.fluss.config.Configuration;
+import org.apache.fluss.exception.KvStorageException;
+import org.apache.fluss.lake.lakestorage.LakeTableLookuper;
+import org.apache.fluss.lake.paimon.utils.PaimonPartitionBucket;
+import org.apache.fluss.lake.paimon.utils.PaimonRowAsFlussRow;
+import org.apache.fluss.metadata.ResolvedPartitionSpec;
+import org.apache.fluss.metadata.TablePath;
+import org.apache.fluss.row.BinaryRow;
+import org.apache.fluss.row.InternalRow;
+import org.apache.fluss.row.encode.CompactedRowEncoder;
+import org.apache.fluss.row.encode.ValueEncoder;
+import org.apache.fluss.types.DataType;
+import org.apache.fluss.types.RowType;
+import org.apache.fluss.utils.IOUtils;
+
+import org.apache.paimon.CoreOptions;
+import org.apache.paimon.catalog.Catalog;
+import org.apache.paimon.catalog.CatalogContext;
+import org.apache.paimon.catalog.CatalogFactory;
+import org.apache.paimon.disk.IOManager;
+import org.apache.paimon.io.DataFileMeta;
+import org.apache.paimon.memory.MemorySegment;
+import org.apache.paimon.options.Options;
+import org.apache.paimon.table.FileStoreTable;
+import org.apache.paimon.table.query.LocalTableQuery;
+import org.apache.paimon.table.sink.RowPartitionKeyExtractor;
+import org.apache.paimon.table.source.DataSplit;
+import org.apache.paimon.table.source.InnerTableScan;
+import org.apache.paimon.table.source.Split;
+import org.apache.paimon.types.DataField;
+
+import javax.annotation.Nullable;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Set;
+
+import static org.apache.fluss.lake.paimon.PaimonLakeCatalog.SYSTEM_COLUMNS;
+import static org.apache.fluss.lake.paimon.utils.PaimonConversions.toPaimon;
+import static org.apache.fluss.lake.paimon.utils.PaimonConversions.toPaimonPartition;
+import static org.apache.fluss.utils.Preconditions.checkNotNull;
+
+/**
+ * Paimon implementation of {@link LakeTableLookuper} for primary-key tables.
+ *
+ * <p>The catalog, table, local query, and I/O manager are initialized lazily on the first lookup.
+ * For each partition and bucket, the lookuper scans the latest Paimon snapshot once and registers
+ * its data files with {@link LocalTableQuery}. Paimon then creates local lookup files lazily as
+ * individual remote data files are queried.
+ *
+ * <p>A cached partition-bucket file set can become stale when Paimon compaction replaces its data
+ * files and snapshot expiration physically deletes the old files. Because {@code FileIO}
+ * implementations may represent a missing file with different {@link IOException} types, the first
+ * lookup I/O failure closes the cached query state, reopens the table from the latest snapshot, and
+ * retries once.
+ *
+ * <p>Lookup and close operations are synchronized because they share mutable Paimon query, local
+ * cache, and value-encoding state.
+ */
+public class PaimonLakeTableLookuper implements LakeTableLookuper {
+
+    // Hard-code a conservative 1GB default for now to avoid unbounded lookup cache growth.
+    private static final String DEFAULT_LOOKUP_CACHE_MAX_DISK_SIZE = "1gb";
+
+    private final Configuration paimonConfig;
+    private final TablePath tablePath;
+    private final String ioTmpDir;
+
+    private final Set<PaimonPartitionBucket> initializedBuckets;
+
+    private @Nullable Catalog catalog;
+    private @Nullable FileStoreTable fileStoreTable;
+    private @Nullable IOManager ioManager;
+    private @Nullable LocalTableQuery localTableQuery;
+    private @Nullable RowPartitionKeyExtractor partitionKeyExtractor;
+    private int primaryKeyFieldCount;
+    private boolean hasCachedValueEncoder;
+    private short cachedValueSchemaId;
+    private @Nullable CompactedRowEncoder cachedValueRowEncoder;
+    private @Nullable InternalRow.FieldGetter[] cachedValueFieldGetters;
+    private boolean closed;
+
+    public PaimonLakeTableLookuper(Configuration paimonConfig, TablePath tablePath) {
+        this(paimonConfig, tablePath, ConfigOptions.IO_TMP_DIR.defaultValue());
+    }
+
+    public PaimonLakeTableLookuper(
+            Configuration paimonConfig, TablePath tablePath, String ioTmpDir) {
+        this.paimonConfig = checkNotNull(paimonConfig, "paimonConfig must not be null.");
+        this.tablePath = checkNotNull(tablePath, "tablePath must not be null.");
+        this.ioTmpDir = checkNotNull(ioTmpDir, "ioTmpDir must not be null.");
+        this.initializedBuckets = new HashSet<>();
+    }
+
+    @Override
+    public synchronized @Nullable byte[] lookup(byte[] key, LookupContext context)
+            throws Exception {
+        checkNotNull(key, "key must not be null.");
+        checkNotNull(context, "context must not be null.");
+        checkNotClosed();
+        ensureInitialized();
+
+        org.apache.paimon.data.BinaryRow partition =
+                convertPartition(context.partitionSpec(), context.valueRowType());
+        org.apache.paimon.data.BinaryRow keyRow = wrapLookupKey(key);
+        initializeFilesIfNeeded(partition, context.bucketId());
+
+        org.apache.paimon.data.InternalRow paimonRow =
+                lookupWithFileRefresh(partition, context.bucketId(), keyRow);
+        if (paimonRow == null) {
+            return null;
+        }
+        return encodeValue(paimonRow, context.schemaId(), context.valueRowType());
+    }
+
+    @Override
+    public synchronized void close() {
+        if (closed) {
+            return;
+        }
+        closed = true;
+        IOUtils.closeQuietly(cachedValueRowEncoder, "Fluss value row encoder");
+        IOUtils.closeQuietly(localTableQuery, "Paimon local table query");
+        IOUtils.closeQuietly(ioManager, "Paimon lookup IO manager");
+        IOUtils.closeQuietly(catalog, "Paimon catalog");
+        initializedBuckets.clear();
+    }
+
+    private void checkNotClosed() {
+        if (closed) {
+            throw new IllegalStateException("Paimon lake table lookuper has been closed.");
+        }
+    }
+
+    private void ensureInitialized() throws Exception {
+        if (localTableQuery != null) {
+            return;
+        }
+
+        catalog =
+                CatalogFactory.createCatalog(
+                        CatalogContext.create(Options.fromMap(paimonConfig.toMap())));
+        fileStoreTable =
+                withLookupCacheOptions((FileStoreTable) catalog.getTable(toPaimon(tablePath)));
+        if (fileStoreTable.primaryKeys().isEmpty()) {
+            throw new UnsupportedOperationException(
+                    "Point lookup is only supported for primary-key Paimon tables.");
+        }
+        ioManager = createIOManager(ioTmpDir);
+        localTableQuery = createLocalTableQuery();
+        partitionKeyExtractor = new RowPartitionKeyExtractor(fileStoreTable.schema());
+        primaryKeyFieldCount = fileStoreTable.primaryKeys().size();
+    }
+
+    private LocalTableQuery createLocalTableQuery() {
+        return fileStoreTable()
+                .newLocalTableQuery()
+                .withValueProjection(businessFieldProjection(fileStoreTable()))
+                .withIOManager(checkNotNull(ioManager, "ioManager must be initialized."));
+    }
+
+    private FileStoreTable withLookupCacheOptions(FileStoreTable table) {
+        String key = CoreOptions.LOOKUP_CACHE_MAX_DISK_SIZE.key();
+        if (table.options().containsKey(key)) {
+            return table;
+        }
+        return table.copy(Collections.singletonMap(key, DEFAULT_LOOKUP_CACHE_MAX_DISK_SIZE));
+    }
+
+    private static IOManager createIOManager(String ioTmpDir) {
+        return IOManager.create(ioTmpDir);
+    }
+
+    private static int[] businessFieldProjection(FileStoreTable fileStoreTable) {
+        List<DataField> fields = fileStoreTable.schema().logicalRowType().getFields();
+        List<Integer> projectedFields = new ArrayList<>();
+        for (int i = 0; i < fields.size(); i++) {
+            if (!SYSTEM_COLUMNS.containsKey(fields.get(i).name())) {
+                projectedFields.add(i);
+            }
+        }
+
+        int[] projection = new int[projectedFields.size()];
+        for (int i = 0; i < projectedFields.size(); i++) {
+            projection[i] = projectedFields.get(i);
+        }
+        return projection;
+    }
+
+    private org.apache.paimon.data.BinaryRow convertPartition(
+            ResolvedPartitionSpec partitionSpec, RowType valueRowType) {
+        return toPaimonPartition(
+                partitionSpec,
+                valueRowType,
+                fileStoreTable().schema().logicalRowType(),
+                partitionKeyExtractor()::partition);
+    }
+
+    private org.apache.paimon.data.BinaryRow wrapLookupKey(byte[] key) {
+        org.apache.paimon.data.BinaryRow keyRow =
+                new org.apache.paimon.data.BinaryRow(primaryKeyFieldCount);
+        keyRow.pointTo(MemorySegment.wrap(key), 0, key.length);
+        return keyRow;
+    }
+
+    private void initializeFilesIfNeeded(org.apache.paimon.data.BinaryRow partition, int bucketId) {
+        PaimonPartitionBucket partitionBucket = new PaimonPartitionBucket(partition, bucketId);
+        if (initializedBuckets.contains(partitionBucket)) {

Review Comment:
   Historical lookup is triggered only after the original Fluss partition has expired and been dropped.
   Since this pr does not support writes to expired partitions, so no new rows are expected and initializing the file set once is sufficient.
   I also add a todo to explan it.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [lookup] Support historical partition lookup for lake tables [fluss]

Posted by "beryllw (via GitHub)" <gi...@apache.org>.
beryllw commented on code in PR #3630:
URL: https://github.com/apache/fluss/pull/3630#discussion_r3609704039


##########
fluss-server/src/main/java/org/apache/fluss/server/utils/ServerRpcMessageUtils.java:
##########
@@ -1138,6 +1139,41 @@ public static Map<TableBucket, List<byte[]>> toLookupData(LookupRequest lookupRe
         return lookupEntryData;
     }
 
+    /**
+     * Converts lookup bucket requests for historical partition lookup.
+     *
+     * <p>Unlike normal lookup data, historical lookup must keep the original partition name carried
+     * by each bucket request. The name is later used to resolve the original partition in lake
+     * storage, while the {@link TableBucket} points to the historical system partition.
+     */
+    public static Map<TableBucket, LookupDataForBucket> toHistoricalLookupData(
+            LookupRequest lookupRequest) {
+        long tableId = lookupRequest.getTableId();
+        Map<TableBucket, LookupDataForBucket> lookupEntryData = new HashMap<>();
+        for (PbLookupReqForBucket lookupReqForBucket : lookupRequest.getBucketsReqsList()) {
+            TableBucket tb =
+                    new TableBucket(
+                            tableId,
+                            lookupReqForBucket.hasPartitionId()
+                                    ? lookupReqForBucket.getPartitionId()
+                                    : null,
+                            lookupReqForBucket.getBucketId());
+            List<byte[]> keys = new ArrayList<>(lookupReqForBucket.getKeysCount());
+            for (int i = 0; i < lookupReqForBucket.getKeysCount(); i++) {
+                keys.add(lookupReqForBucket.getKeyAt(i));
+            }
+            lookupEntryData.put(

Review Comment:
   nit: in toHistoricalLookupData, two buckets with the same TableBucket but different originalPartitionName silently overwrite each other. Might be safer to fail fast on a duplicate here — the Java client already splits these, so this would just guard against other clients.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [lookup] Support historical partition lookup for lake tables [fluss]

Posted by "wuchong (via GitHub)" <gi...@apache.org>.
wuchong commented on code in PR #3630:
URL: https://github.com/apache/fluss/pull/3630#discussion_r3680878009


##########
fluss-server/src/main/java/org/apache/fluss/server/replica/HistoricalLakeLookupManager.java:
##########
@@ -0,0 +1,472 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.fluss.server.replica;
+
+import org.apache.fluss.annotation.VisibleForTesting;
+import org.apache.fluss.config.ConfigOptions;
+import org.apache.fluss.config.Configuration;
+import org.apache.fluss.config.TableConfig;
+import org.apache.fluss.exception.FlussRuntimeException;
+import org.apache.fluss.exception.HistoricalPartitionThrottledException;
+import org.apache.fluss.exception.InvalidPartitionException;
+import org.apache.fluss.exception.LakeStorageNotConfiguredException;
+import org.apache.fluss.lake.lakestorage.LakeStorage;
+import org.apache.fluss.lake.lakestorage.LakeStoragePlugin;
+import org.apache.fluss.lake.lakestorage.LakeStoragePluginSetUp;
+import org.apache.fluss.lake.lakestorage.LakeTableLookuper;
+import org.apache.fluss.metadata.DataLakeFormat;
+import org.apache.fluss.metadata.ResolvedPartitionSpec;
+import org.apache.fluss.metadata.SchemaInfo;
+import org.apache.fluss.metadata.TableBucket;
+import org.apache.fluss.metadata.TableInfo;
+import org.apache.fluss.metadata.TablePath;
+import org.apache.fluss.plugin.PluginManager;
+import org.apache.fluss.rpc.entity.LookupResultForBucket;
+import org.apache.fluss.rpc.protocol.ApiError;
+import org.apache.fluss.server.entity.LookupDataForBucket;
+import org.apache.fluss.utils.ExecutorUtils;
+import org.apache.fluss.utils.FileUtils;
+import org.apache.fluss.utils.IOUtils;
+import org.apache.fluss.utils.concurrent.ExecutorThreadFactory;
+import org.apache.fluss.utils.concurrent.Scheduler;
+
+import com.github.benmanes.caffeine.cache.Cache;
+import com.github.benmanes.caffeine.cache.Caffeine;
+import com.github.benmanes.caffeine.cache.RemovalCause;
+import com.github.benmanes.caffeine.cache.Ticker;
+
+import javax.annotation.Nullable;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.ArrayBlockingQueue;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Semaphore;
+import java.util.concurrent.ThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
+
+import static org.apache.fluss.server.utils.LakeStorageUtils.extractLakeProperties;
+import static org.apache.fluss.utils.Preconditions.checkArgument;
+import static org.apache.fluss.utils.Preconditions.checkNotNull;
+
+/**
+ * Handles server-side point lookup for historical partitions stored in lake storage.
+ *
+ * <p>Accepted requests run on a dedicated, bounded executor that is created lazily and releases
+ * idle threads. A semaphore bounds the total number of accepted historical lookup tasks so slow
+ * lake storage cannot create an unbounded request backlog.
+ *
+ * <p>Creating a lake table lookuper may initialize catalog, table, and query state and allocate
+ * local lookup files, so lookupers are cached and reused. The cache is keyed by table ID rather
+ * than table path to prevent a deleted and recreated table from reusing the old table's lookuper. A
+ * cached lookuper is replaced when its schema ID no longer matches the requested table schema.
+ *
+ * <p>A lookuper is closed when replaced, explicitly invalidated by a replica lifecycle event,
+ * evicted after the cache reaches ten tables, the manager shuts down, or after three hours without
+ * access. Caffeine expiration is scheduled on the shared TabletServer scheduler, allowing idle
+ * resources to be released even if no subsequent lookup accesses the cache.
+ */
+class HistoricalLakeLookupManager implements AutoCloseable {
+
+    private static final String PAIMON_LOOKUP_DIR_NAME = "paimon-lookup";
+    private static final String LOOKUPER_CACHE_EXPIRATION_TASK_NAME =
+            "historical-lookuper-cache-expiration";
+    private static final Duration LOOKUPER_CACHE_EXPIRATION = Duration.ofHours(3);
+    private static final Duration HISTORICAL_PARTITION_THREAD_KEEP_ALIVE = Duration.ofSeconds(60);
+    private static final Duration HISTORICAL_PARTITION_EXECUTOR_SHUTDOWN_TIMEOUT =
+            Duration.ofSeconds(10);
+    private static final int MAX_CACHED_LOOKUPERS = 10;
+    private static final String HISTORICAL_PARTITION_THREAD_NAME_PREFIX = "historical-partition-io";
+
+    // TODO: MAX_CACHED_LOOKUPERS and the 2GB per-table limit configured by
+    // PaimonLakeTableLookuper through Paimon's "lookup.cache-max-disk-size" option are hard-coded.
+    // Make them configurable, and prefer a Paimon IOManager-level global disk limit shared by all
+    // table lookupers because fixed per-table limits can underutilize cache for hot tables while
+    // reserving too much for cold tables.
+
+    private final Configuration conf;
+    private final @Nullable PluginManager pluginManager;
+    private final int serverId;
+    private final Semaphore lookupPermits;
+    // Upper bound on running and queued historical lookups; also used as the executor queue size.
+    private final int maxQueuedHistoricalRequests;
+    // Accepted lookup futures tracked so close() can cancel tasks left after executor shutdown.
+    private final Set<CompletableFuture<LookupResultForBucket>> pendingLookups;
+    private final Cache<Long, CachedLakeTableLookuper> lakeTableLookupers;
+    // Dedicated historical partition executor, created lazily on the first lookup.
+    private @Nullable ExecutorService historicalPartitionExecutor;
+    private boolean closed;
+    private @Nullable String paimonLookupTempDir;
+
+    HistoricalLakeLookupManager(
+            Configuration conf,
+            @Nullable PluginManager pluginManager,
+            int serverId,
+            Scheduler scheduler) {
+        this(
+                conf,
+                pluginManager,
+                null,
+                serverId,
+                Ticker.systemTicker(),
+                createCacheScheduler(scheduler));
+    }
+
+    @VisibleForTesting
+    HistoricalLakeLookupManager(
+            Configuration conf,
+            @Nullable PluginManager pluginManager,
+            @Nullable ExecutorService historicalPartitionExecutor,
+            int serverId,
+            Ticker ticker,
+            com.github.benmanes.caffeine.cache.Scheduler cacheScheduler) {
+        this.conf = checkNotNull(conf, "conf must not be null.");
+        this.pluginManager = pluginManager;
+        this.historicalPartitionExecutor = historicalPartitionExecutor;
+        this.serverId = serverId;
+        this.lakeTableLookupers =
+                Caffeine.newBuilder()
+                        .expireAfterAccess(LOOKUPER_CACHE_EXPIRATION)
+                        .maximumSize(MAX_CACHED_LOOKUPERS)
+                        .ticker(checkNotNull(ticker, "ticker must not be null."))
+                        .scheduler(checkNotNull(cacheScheduler, "cacheScheduler must not be null."))
+                        .executor(Runnable::run)
+                        .removalListener(
+                                (Long ignored,
+                                        CachedLakeTableLookuper cachedLookuper,
+                                        RemovalCause ignoredCause) -> {
+                                    if (cachedLookuper != null) {
+                                        cachedLookuper.invalidate();
+                                    }
+                                })
+                        .build();
+        this.maxQueuedHistoricalRequests =
+                conf.get(ConfigOptions.NETTY_SERVER_MAX_QUEUED_HISTORICAL_REQUESTS);
+        checkArgument(
+                maxQueuedHistoricalRequests > 0,
+                "%s must be greater than 0.",
+                ConfigOptions.NETTY_SERVER_MAX_QUEUED_HISTORICAL_REQUESTS.key());
+        this.lookupPermits = new Semaphore(maxQueuedHistoricalRequests);
+        this.pendingLookups = ConcurrentHashMap.newKeySet();
+    }
+
+    private static com.github.benmanes.caffeine.cache.Scheduler createCacheScheduler(
+            Scheduler scheduler) {
+        checkNotNull(scheduler, "scheduler must not be null.");
+        // Schedule expiration maintenance so idle lookupers are closed even if no more lookups
+        // arrive.
+        return (executor, command, delay, timeUnit) ->
+                scheduler.scheduleOnce(
+                        LOOKUPER_CACHE_EXPIRATION_TASK_NAME,
+                        () -> executor.execute(command),
+                        timeUnit.toMillis(delay));
+    }
+
+    CompletableFuture<LookupResultForBucket> lookup(
+            LookupDataForBucket lookupData, TableInfo tableInfo, SchemaInfo schemaInfo) {
+        TableBucket tableBucket = lookupData.tableBucket();
+        if (!lookupPermits.tryAcquire()) {
+            return CompletableFuture.completedFuture(
+                    new LookupResultForBucket(
+                            tableBucket,
+                            null,
+                            lookupData.originalPartitionName(),
+                            ApiError.fromThrowable(
+                                    new HistoricalPartitionThrottledException(
+                                            "Historical lookup is throttled for "
+                                                    + tableBucket
+                                                    + "."))));
+        }
+
+        CompletableFuture<LookupResultForBucket> future;
+        try {
+            future = submitLookup(lookupData, tableInfo, schemaInfo);
+        } catch (RuntimeException e) {
+            lookupPermits.release();
+            throw e;
+        }
+        future.whenComplete(
+                (ignored, error) -> {
+                    pendingLookups.remove(future);
+                    lookupPermits.release();
+                });
+        return future;
+    }
+
+    @Override
+    public void close() {
+        ExecutorService executor;
+        synchronized (this) {
+            closed = true;
+            executor = historicalPartitionExecutor;
+        }
+        if (executor != null) {
+            ExecutorUtils.gracefulShutdown(
+                    HISTORICAL_PARTITION_EXECUTOR_SHUTDOWN_TIMEOUT.toMillis(),
+                    TimeUnit.MILLISECONDS,
+                    executor);
+        }
+        pendingLookups.forEach(future -> future.cancel(true));
+        lakeTableLookupers.invalidateAll();
+        lakeTableLookupers.cleanUp();
+    }
+
+    private synchronized CompletableFuture<LookupResultForBucket> submitLookup(
+            LookupDataForBucket lookupData, TableInfo tableInfo, SchemaInfo schemaInfo) {
+        CompletableFuture<LookupResultForBucket> future =
+                CompletableFuture.supplyAsync(
+                        () -> lookupInternal(lookupData, tableInfo, schemaInfo),
+                        getOrCreateHistoricalPartitionExecutor());
+        pendingLookups.add(future);
+        return future;
+    }
+
+    private synchronized ExecutorService getOrCreateHistoricalPartitionExecutor() {

Review Comment:
   We don't need to lazily initialize `ThreadPoolExecutor`, because the `ThreadPoolExecutor` itself doens't have much overhead, all the threads are started lazily. This can simplify and remove the lock on each submit path. 



##########
fluss-server/src/main/java/org/apache/fluss/server/replica/HistoricalLakeLookupManager.java:
##########
@@ -0,0 +1,472 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.fluss.server.replica;
+
+import org.apache.fluss.annotation.VisibleForTesting;
+import org.apache.fluss.config.ConfigOptions;
+import org.apache.fluss.config.Configuration;
+import org.apache.fluss.config.TableConfig;
+import org.apache.fluss.exception.FlussRuntimeException;
+import org.apache.fluss.exception.HistoricalPartitionThrottledException;
+import org.apache.fluss.exception.InvalidPartitionException;
+import org.apache.fluss.exception.LakeStorageNotConfiguredException;
+import org.apache.fluss.lake.lakestorage.LakeStorage;
+import org.apache.fluss.lake.lakestorage.LakeStoragePlugin;
+import org.apache.fluss.lake.lakestorage.LakeStoragePluginSetUp;
+import org.apache.fluss.lake.lakestorage.LakeTableLookuper;
+import org.apache.fluss.metadata.DataLakeFormat;
+import org.apache.fluss.metadata.ResolvedPartitionSpec;
+import org.apache.fluss.metadata.SchemaInfo;
+import org.apache.fluss.metadata.TableBucket;
+import org.apache.fluss.metadata.TableInfo;
+import org.apache.fluss.metadata.TablePath;
+import org.apache.fluss.plugin.PluginManager;
+import org.apache.fluss.rpc.entity.LookupResultForBucket;
+import org.apache.fluss.rpc.protocol.ApiError;
+import org.apache.fluss.server.entity.LookupDataForBucket;
+import org.apache.fluss.utils.ExecutorUtils;
+import org.apache.fluss.utils.FileUtils;
+import org.apache.fluss.utils.IOUtils;
+import org.apache.fluss.utils.concurrent.ExecutorThreadFactory;
+import org.apache.fluss.utils.concurrent.Scheduler;
+
+import com.github.benmanes.caffeine.cache.Cache;
+import com.github.benmanes.caffeine.cache.Caffeine;
+import com.github.benmanes.caffeine.cache.RemovalCause;
+import com.github.benmanes.caffeine.cache.Ticker;
+
+import javax.annotation.Nullable;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.ArrayBlockingQueue;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Semaphore;
+import java.util.concurrent.ThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
+
+import static org.apache.fluss.server.utils.LakeStorageUtils.extractLakeProperties;
+import static org.apache.fluss.utils.Preconditions.checkArgument;
+import static org.apache.fluss.utils.Preconditions.checkNotNull;
+
+/**
+ * Handles server-side point lookup for historical partitions stored in lake storage.
+ *
+ * <p>Accepted requests run on a dedicated, bounded executor that is created lazily and releases
+ * idle threads. A semaphore bounds the total number of accepted historical lookup tasks so slow
+ * lake storage cannot create an unbounded request backlog.
+ *
+ * <p>Creating a lake table lookuper may initialize catalog, table, and query state and allocate
+ * local lookup files, so lookupers are cached and reused. The cache is keyed by table ID rather
+ * than table path to prevent a deleted and recreated table from reusing the old table's lookuper. A
+ * cached lookuper is replaced when its schema ID no longer matches the requested table schema.
+ *
+ * <p>A lookuper is closed when replaced, explicitly invalidated by a replica lifecycle event,
+ * evicted after the cache reaches ten tables, the manager shuts down, or after three hours without
+ * access. Caffeine expiration is scheduled on the shared TabletServer scheduler, allowing idle
+ * resources to be released even if no subsequent lookup accesses the cache.
+ */
+class HistoricalLakeLookupManager implements AutoCloseable {
+
+    private static final String PAIMON_LOOKUP_DIR_NAME = "paimon-lookup";
+    private static final String LOOKUPER_CACHE_EXPIRATION_TASK_NAME =
+            "historical-lookuper-cache-expiration";
+    private static final Duration LOOKUPER_CACHE_EXPIRATION = Duration.ofHours(3);
+    private static final Duration HISTORICAL_PARTITION_THREAD_KEEP_ALIVE = Duration.ofSeconds(60);

Review Comment:
   Can be a little longer, such as 10min, to allow the flink job restart period. 



##########
fluss-client/src/main/java/org/apache/fluss/client/lookup/PrimaryKeyLookuper.java:
##########
@@ -115,19 +139,107 @@ public CompletableFuture<LookupResult> lookup(InternalRow lookupKey) {
                                 tableInfo.getTablePath(),
                                 metadataUpdater);
             } catch (PartitionNotExistException e) {
-                return CompletableFuture.completedFuture(new LookupResult(Collections.emptyList()));
+                return mayFallbackToHistoricalLookup(
+                        bucketId, pkBytes, originalPartitionName, lookupKey);
             }
         }
 
-        int bucketId = bucketingFunction.bucketing(bkBytes, numBuckets);
         TableBucket tableBucket = new TableBucket(tableInfo.getTableId(), partitionId, bucketId);
+        return lookupBucket(tableBucket, pkBytes, insertIfNotExists, null, lookupKey);
+    }
+
+    /**
+     * Falls back to historical lookup when the normal partition is missing and fallback is enabled.
+     */
+    private CompletableFuture<LookupResult> mayFallbackToHistoricalLookup(
+            int bucketId, byte[] keyBytes, String originalPartitionName, InternalRow lookupKey) {
+        // Clear the stale normal-partition route before deciding whether to fall back so that a
+        // partition created later can be discovered by the next lookup.
+        metadataUpdater.invalidPhysicalTableBucketAndPartitionMeta(
+                Collections.singleton(
+                        PhysicalTablePath.of(tableInfo.getTablePath(), originalPartitionName)));
+        if (!tableInfo.getTableConfig().isHistoricalPartitionEnabled()) {
+            return CompletableFuture.completedFuture(new LookupResult(Collections.emptyList()));
+        }
+        // A missing current or future partition may still be created by auto-partitioning. Do not
+        // cache it as historical, otherwise this lookuper would keep bypassing the normal partition
+        // after it is created.
+        if (!isPastAutoPartition(
+                originalPartitionName,
+                tableInfo.getTableConfig().getAutoPartitionStrategy(),
+                Instant.now())) {
+            return CompletableFuture.completedFuture(new LookupResult(Collections.emptyList()));
+        }
+        confirmedHistoricalPartitions.add(originalPartitionName);
+        return historicalLookup(bucketId, keyBytes, originalPartitionName, lookupKey);
+    }
+
+    private CompletableFuture<LookupResult> historicalLookup(
+            int bucketId, byte[] keyBytes, String originalPartitionName, InternalRow lookupKey) {
+        if (insertIfNotExists) {
+            return completedExceptionally(
+                    new UnsupportedOperationException(
+                            "Lookup with insertIfNotExists is not supported for historical partition lookup."));
+        }
+        PhysicalTablePath historicalPartitionPath =
+                PhysicalTablePath.of(tableInfo.getTablePath(), HISTORICAL_PARTITION_VALUE);
+        try {
+            if (!metadataUpdater.checkAndUpdatePartitionMetadata(historicalPartitionPath)) {
+                throw new PartitionNotExistException(
+                        "Historical partition " + historicalPartitionPath + " does not exist.");
+            }
+            Long historicalPartitionId =
+                    metadataUpdater.getPartitionIdOrElseThrow(historicalPartitionPath);
+            TableBucket tableBucket =
+                    new TableBucket(tableInfo.getTableId(), historicalPartitionId, bucketId);
+            return lookupBucket(tableBucket, keyBytes, false, originalPartitionName, lookupKey);
+        } catch (Throwable t) {
+            return completedExceptionally(t);
+        }
+    }
+
+    private CompletableFuture<LookupResult> lookupBucket(
+            TableBucket tableBucket,
+            byte[] keyBytes,
+            boolean insertIfNotExists,
+            @Nullable String originalPartitionName,
+            InternalRow lookupKey) {
         CompletableFuture<LookupResult> lookupFuture = new CompletableFuture<>();
         lookupClient
-                .lookup(tableInfo.getTablePath(), tableBucket, pkBytes, insertIfNotExists)
+                .lookup(
+                        tableInfo.getTablePath(),
+                        tableBucket,
+                        keyBytes,
+                        insertIfNotExists,
+                        originalPartitionName)
                 .whenComplete(
                         (result, error) -> {
                             if (error != null) {
-                                lookupFuture.completeExceptionally(error);
+                                // A historical lookup already carries the original partition name.
+                                // Propagate its failure instead of falling back again.
+                                if (!(error instanceof PartitionNotExistException)
+                                        || originalPartitionName != null) {
+                                    lookupFuture.completeExceptionally(error);
+                                    return;
+                                }
+
+                                // The cached normal partition was deleted. Re-evaluate the routing
+                                // using the lookup key.
+                                String partitionName = partitionGetter.getPartition(lookupKey);

Review Comment:
   Capture the original partition name before entering the asynchronous callback instead of reading `lookupKey` here. `FlinkAsyncLookupFunction` reuses a single `FlinkAsFlussRow` and mutates it with `replace()` for each invocation, so while request A is waiting for its RPC, request B can replace the backing row. If A then gets `PartitionNotExistException`, this callback combines A’s `keyBytes` with B’s partition name and may query the wrong historical partition or return a false miss. Please pass the immutable partition name computed during initial routing (we should use an explicit boolean flag to indicate historical lookup rather than relying on the nullable `original_partition_name` ) through the callback, and cover this interleaving with a deterministic test.



##########
fluss-server/src/main/java/org/apache/fluss/server/replica/HistoricalLakeLookupManager.java:
##########
@@ -0,0 +1,472 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.fluss.server.replica;
+
+import org.apache.fluss.annotation.VisibleForTesting;
+import org.apache.fluss.config.ConfigOptions;
+import org.apache.fluss.config.Configuration;
+import org.apache.fluss.config.TableConfig;
+import org.apache.fluss.exception.FlussRuntimeException;
+import org.apache.fluss.exception.HistoricalPartitionThrottledException;
+import org.apache.fluss.exception.InvalidPartitionException;
+import org.apache.fluss.exception.LakeStorageNotConfiguredException;
+import org.apache.fluss.lake.lakestorage.LakeStorage;
+import org.apache.fluss.lake.lakestorage.LakeStoragePlugin;
+import org.apache.fluss.lake.lakestorage.LakeStoragePluginSetUp;
+import org.apache.fluss.lake.lakestorage.LakeTableLookuper;
+import org.apache.fluss.metadata.DataLakeFormat;
+import org.apache.fluss.metadata.ResolvedPartitionSpec;
+import org.apache.fluss.metadata.SchemaInfo;
+import org.apache.fluss.metadata.TableBucket;
+import org.apache.fluss.metadata.TableInfo;
+import org.apache.fluss.metadata.TablePath;
+import org.apache.fluss.plugin.PluginManager;
+import org.apache.fluss.rpc.entity.LookupResultForBucket;
+import org.apache.fluss.rpc.protocol.ApiError;
+import org.apache.fluss.server.entity.LookupDataForBucket;
+import org.apache.fluss.utils.ExecutorUtils;
+import org.apache.fluss.utils.FileUtils;
+import org.apache.fluss.utils.IOUtils;
+import org.apache.fluss.utils.concurrent.ExecutorThreadFactory;
+import org.apache.fluss.utils.concurrent.Scheduler;
+
+import com.github.benmanes.caffeine.cache.Cache;
+import com.github.benmanes.caffeine.cache.Caffeine;
+import com.github.benmanes.caffeine.cache.RemovalCause;
+import com.github.benmanes.caffeine.cache.Ticker;
+
+import javax.annotation.Nullable;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.ArrayBlockingQueue;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Semaphore;
+import java.util.concurrent.ThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
+
+import static org.apache.fluss.server.utils.LakeStorageUtils.extractLakeProperties;
+import static org.apache.fluss.utils.Preconditions.checkArgument;
+import static org.apache.fluss.utils.Preconditions.checkNotNull;
+
+/**
+ * Handles server-side point lookup for historical partitions stored in lake storage.
+ *
+ * <p>Accepted requests run on a dedicated, bounded executor that is created lazily and releases
+ * idle threads. A semaphore bounds the total number of accepted historical lookup tasks so slow
+ * lake storage cannot create an unbounded request backlog.
+ *
+ * <p>Creating a lake table lookuper may initialize catalog, table, and query state and allocate
+ * local lookup files, so lookupers are cached and reused. The cache is keyed by table ID rather
+ * than table path to prevent a deleted and recreated table from reusing the old table's lookuper. A
+ * cached lookuper is replaced when its schema ID no longer matches the requested table schema.
+ *
+ * <p>A lookuper is closed when replaced, explicitly invalidated by a replica lifecycle event,
+ * evicted after the cache reaches ten tables, the manager shuts down, or after three hours without
+ * access. Caffeine expiration is scheduled on the shared TabletServer scheduler, allowing idle
+ * resources to be released even if no subsequent lookup accesses the cache.
+ */
+class HistoricalLakeLookupManager implements AutoCloseable {
+
+    private static final String PAIMON_LOOKUP_DIR_NAME = "paimon-lookup";
+    private static final String LOOKUPER_CACHE_EXPIRATION_TASK_NAME =
+            "historical-lookuper-cache-expiration";
+    private static final Duration LOOKUPER_CACHE_EXPIRATION = Duration.ofHours(3);
+    private static final Duration HISTORICAL_PARTITION_THREAD_KEEP_ALIVE = Duration.ofSeconds(60);
+    private static final Duration HISTORICAL_PARTITION_EXECUTOR_SHUTDOWN_TIMEOUT =
+            Duration.ofSeconds(10);
+    private static final int MAX_CACHED_LOOKUPERS = 10;
+    private static final String HISTORICAL_PARTITION_THREAD_NAME_PREFIX = "historical-partition-io";
+
+    // TODO: MAX_CACHED_LOOKUPERS and the 2GB per-table limit configured by
+    // PaimonLakeTableLookuper through Paimon's "lookup.cache-max-disk-size" option are hard-coded.
+    // Make them configurable, and prefer a Paimon IOManager-level global disk limit shared by all
+    // table lookupers because fixed per-table limits can underutilize cache for hot tables while
+    // reserving too much for cold tables.
+
+    private final Configuration conf;
+    private final @Nullable PluginManager pluginManager;
+    private final int serverId;
+    private final Semaphore lookupPermits;
+    // Upper bound on running and queued historical lookups; also used as the executor queue size.
+    private final int maxQueuedHistoricalRequests;
+    // Accepted lookup futures tracked so close() can cancel tasks left after executor shutdown.
+    private final Set<CompletableFuture<LookupResultForBucket>> pendingLookups;
+    private final Cache<Long, CachedLakeTableLookuper> lakeTableLookupers;
+    // Dedicated historical partition executor, created lazily on the first lookup.
+    private @Nullable ExecutorService historicalPartitionExecutor;
+    private boolean closed;
+    private @Nullable String paimonLookupTempDir;
+
+    HistoricalLakeLookupManager(
+            Configuration conf,
+            @Nullable PluginManager pluginManager,
+            int serverId,
+            Scheduler scheduler) {
+        this(
+                conf,
+                pluginManager,
+                null,
+                serverId,
+                Ticker.systemTicker(),
+                createCacheScheduler(scheduler));
+    }
+
+    @VisibleForTesting
+    HistoricalLakeLookupManager(
+            Configuration conf,
+            @Nullable PluginManager pluginManager,
+            @Nullable ExecutorService historicalPartitionExecutor,
+            int serverId,
+            Ticker ticker,
+            com.github.benmanes.caffeine.cache.Scheduler cacheScheduler) {
+        this.conf = checkNotNull(conf, "conf must not be null.");
+        this.pluginManager = pluginManager;
+        this.historicalPartitionExecutor = historicalPartitionExecutor;
+        this.serverId = serverId;
+        this.lakeTableLookupers =
+                Caffeine.newBuilder()
+                        .expireAfterAccess(LOOKUPER_CACHE_EXPIRATION)
+                        .maximumSize(MAX_CACHED_LOOKUPERS)
+                        .ticker(checkNotNull(ticker, "ticker must not be null."))
+                        .scheduler(checkNotNull(cacheScheduler, "cacheScheduler must not be null."))
+                        .executor(Runnable::run)
+                        .removalListener(
+                                (Long ignored,
+                                        CachedLakeTableLookuper cachedLookuper,
+                                        RemovalCause ignoredCause) -> {
+                                    if (cachedLookuper != null) {
+                                        cachedLookuper.invalidate();
+                                    }
+                                })
+                        .build();
+        this.maxQueuedHistoricalRequests =
+                conf.get(ConfigOptions.NETTY_SERVER_MAX_QUEUED_HISTORICAL_REQUESTS);
+        checkArgument(
+                maxQueuedHistoricalRequests > 0,
+                "%s must be greater than 0.",
+                ConfigOptions.NETTY_SERVER_MAX_QUEUED_HISTORICAL_REQUESTS.key());
+        this.lookupPermits = new Semaphore(maxQueuedHistoricalRequests);
+        this.pendingLookups = ConcurrentHashMap.newKeySet();
+    }
+
+    private static com.github.benmanes.caffeine.cache.Scheduler createCacheScheduler(
+            Scheduler scheduler) {
+        checkNotNull(scheduler, "scheduler must not be null.");
+        // Schedule expiration maintenance so idle lookupers are closed even if no more lookups
+        // arrive.
+        return (executor, command, delay, timeUnit) ->
+                scheduler.scheduleOnce(
+                        LOOKUPER_CACHE_EXPIRATION_TASK_NAME,
+                        () -> executor.execute(command),
+                        timeUnit.toMillis(delay));
+    }
+
+    CompletableFuture<LookupResultForBucket> lookup(
+            LookupDataForBucket lookupData, TableInfo tableInfo, SchemaInfo schemaInfo) {
+        TableBucket tableBucket = lookupData.tableBucket();
+        if (!lookupPermits.tryAcquire()) {
+            return CompletableFuture.completedFuture(
+                    new LookupResultForBucket(
+                            tableBucket,
+                            null,
+                            lookupData.originalPartitionName(),
+                            ApiError.fromThrowable(
+                                    new HistoricalPartitionThrottledException(
+                                            "Historical lookup is throttled for "
+                                                    + tableBucket
+                                                    + "."))));
+        }
+
+        CompletableFuture<LookupResultForBucket> future;
+        try {
+            future = submitLookup(lookupData, tableInfo, schemaInfo);
+        } catch (RuntimeException e) {
+            lookupPermits.release();
+            throw e;
+        }
+        future.whenComplete(
+                (ignored, error) -> {
+                    pendingLookups.remove(future);
+                    lookupPermits.release();
+                });
+        return future;
+    }
+
+    @Override
+    public void close() {
+        ExecutorService executor;
+        synchronized (this) {
+            closed = true;
+            executor = historicalPartitionExecutor;
+        }
+        if (executor != null) {
+            ExecutorUtils.gracefulShutdown(
+                    HISTORICAL_PARTITION_EXECUTOR_SHUTDOWN_TIMEOUT.toMillis(),
+                    TimeUnit.MILLISECONDS,
+                    executor);
+        }
+        pendingLookups.forEach(future -> future.cancel(true));
+        lakeTableLookupers.invalidateAll();
+        lakeTableLookupers.cleanUp();
+    }
+
+    private synchronized CompletableFuture<LookupResultForBucket> submitLookup(
+            LookupDataForBucket lookupData, TableInfo tableInfo, SchemaInfo schemaInfo) {
+        CompletableFuture<LookupResultForBucket> future =
+                CompletableFuture.supplyAsync(
+                        () -> lookupInternal(lookupData, tableInfo, schemaInfo),
+                        getOrCreateHistoricalPartitionExecutor());
+        pendingLookups.add(future);
+        return future;
+    }
+
+    private synchronized ExecutorService getOrCreateHistoricalPartitionExecutor() {
+        if (closed) {
+            throw new IllegalStateException("Historical lake lookup manager has been closed.");
+        }
+        if (historicalPartitionExecutor == null) {
+            int maxThreadPoolSize =

Review Comment:
   Validate `maxThreadPoolSize` as greater than zero before constructing the executor. The new integer option accepts zero or negative values, so the server can start successfully and the first historical lookup then throws `IllegalArgumentException` from `ThreadPoolExecutor`, leaving the feature unavailable with a delayed and non-actionable failure. Please fail fast during manager or configuration initialization, similar to `maxQueuedHistoricalRequests`, and add boundary tests.



##########
fluss-server/src/main/java/org/apache/fluss/server/coordinator/AutoPartitionManager.java:
##########
@@ -230,6 +244,62 @@ public void handleAutoPartitionStrategyChange(
         }
     }
 
+    /** Creates the historical system partition if it does not already exist. */
+    void createHistoricalPartition(TableInfo tableInfo) {
+        checkNotClosed();
+        inLock(
+                lock,
+                () -> {
+                    long tableId = tableInfo.getTableId();
+                    TreeMap<String, Set<String>> currentPartitions =
+                            checkNotNull(
+                                    partitionsByTable.get(tableId),
+                                    "Auto partition state does not exist for table " + tableId);
+                    if (!currentPartitions.containsKey(HISTORICAL_PARTITION_VALUE)) {
+                        createPartition(

Review Comment:
   Please make this system-partition lifecycle self-healing instead of a one-shot best-effort transition. `createPartition` catches capacity and other failures without propagating them, so once the table option is persisted as enabled, a transient failure can leave `__historical__` absent with no online retry and historical lookups keep failing until a coordinator restart or another option change. The disable path has the symmetric problem: `MetadataManager.dropPartition` logs and swallows ZooKeeper deletion failures, after which this manager removes the in-memory entry and reports success. 
   
   Could we move historical system partition creation into the synchronous `CoordinatorService` create/alter flow? Validate the additional KV leader capacity first, then create the table and `__historical__` before completing the RPC. Any capacity or metadata failure should be propagated to the caller instead of being swallowed. If the table was already persisted when partition creation failed, make that partial state explicit and tell the user to either drop and recreate the table, or disable and re-enable the historical-partition option after fixing the underlying issue. Startup reconciliation can remain as crash-recovery fallback, but a handled failure must never be reported as a successful create/enable operation.



##########
fluss-lake/fluss-lake-paimon/src/main/java/org/apache/fluss/lake/paimon/lookup/PaimonLakeTableLookuper.java:
##########
@@ -0,0 +1,418 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.fluss.lake.paimon.lookup;
+
+import org.apache.fluss.config.Configuration;
+import org.apache.fluss.config.TableConfig;
+import org.apache.fluss.exception.KvStorageException;
+import org.apache.fluss.lake.lakestorage.LakeTableLookuper;
+import org.apache.fluss.lake.paimon.utils.PaimonPartitionBucket;
+import org.apache.fluss.lake.paimon.utils.PaimonRowAsFlussRow;
+import org.apache.fluss.metadata.ResolvedPartitionSpec;
+import org.apache.fluss.metadata.TablePath;
+import org.apache.fluss.row.BinaryRow;
+import org.apache.fluss.row.InternalRow;
+import org.apache.fluss.row.decode.CompactedKeyDecoder;
+import org.apache.fluss.row.encode.RowEncoder;
+import org.apache.fluss.row.encode.ValueEncoder;
+import org.apache.fluss.row.encode.paimon.PaimonKeyEncoder;
+import org.apache.fluss.types.RowType;
+import org.apache.fluss.utils.IOUtils;
+
+import org.apache.paimon.CoreOptions;
+import org.apache.paimon.catalog.Catalog;
+import org.apache.paimon.catalog.CatalogContext;
+import org.apache.paimon.catalog.CatalogFactory;
+import org.apache.paimon.disk.IOManager;
+import org.apache.paimon.io.DataFileMeta;
+import org.apache.paimon.memory.MemorySegment;
+import org.apache.paimon.options.MemorySize;
+import org.apache.paimon.options.Options;
+import org.apache.paimon.table.FileStoreTable;
+import org.apache.paimon.table.query.LocalTableQuery;
+import org.apache.paimon.table.sink.RowPartitionKeyExtractor;
+import org.apache.paimon.table.source.DataSplit;
+import org.apache.paimon.table.source.InnerTableScan;
+import org.apache.paimon.table.source.Split;
+import org.apache.paimon.types.DataField;
+
+import javax.annotation.Nullable;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Set;
+
+import static org.apache.fluss.config.ConfigOptions.KV_FORMAT_VERSION_2;
+import static org.apache.fluss.lake.paimon.PaimonLakeCatalog.SYSTEM_COLUMNS;
+import static org.apache.fluss.lake.paimon.utils.PaimonConversions.toPaimon;
+import static org.apache.fluss.lake.paimon.utils.PaimonConversions.toPaimonPartition;
+import static org.apache.fluss.utils.Preconditions.checkNotNull;
+
+/**
+ * Paimon implementation of {@link LakeTableLookuper} for primary-key tables.
+ *
+ * <p>The catalog, table, local query, and I/O manager are initialized lazily on the first lookup.
+ * For each partition and bucket, the lookuper scans the latest Paimon snapshot once and registers
+ * its data files with {@link LocalTableQuery}. Paimon then creates local lookup files lazily as
+ * individual remote data files are queried.
+ *
+ * <p>A cached partition-bucket file set can become stale when Paimon compaction replaces its data
+ * files and snapshot expiration physically deletes the old files. Because {@code FileIO}
+ * implementations may represent a missing file with different {@link IOException} types, the first
+ * lookup I/O failure closes the cached query state, reopens the table from the latest snapshot, and
+ * retries once.
+ *
+ * <p>Lookup and close operations are synchronized because they share mutable Paimon query, local
+ * cache, and value-encoding state.
+ */
+public class PaimonLakeTableLookuper implements LakeTableLookuper {
+
+    // Each TabletServer caches at most ten table lookupers, bounding their retained lookup cache
+    // capacity to 20GB. See HistoricalLakeLookupManager for the follow-up to make this configurable
+    // and use a global Paimon IOManager limit.
+    private static final String LOOKUP_CACHE_MAX_DISK_SIZE = "2gb";
+    private static final MemorySize LOOKUP_CACHE_MAX_DISK_MEMORY_SIZE =
+            MemorySize.parse(LOOKUP_CACHE_MAX_DISK_SIZE);
+
+    private final Configuration paimonConfig;
+    private final TablePath tablePath;
+    private final String ioTmpDir;
+    private final TableConfig tableConfig;
+
+    private final Set<PaimonPartitionBucket> initializedBuckets;
+
+    private @Nullable Catalog catalog;
+    private @Nullable FileStoreTable fileStoreTable;
+    private @Nullable IOManager ioManager;
+    private @Nullable LocalTableQuery localTableQuery;
+    private @Nullable RowPartitionKeyExtractor partitionKeyExtractor;
+    private int primaryKeyFieldCount;
+
+    // Both encoders are initialized only for a kv-format-v2 table whose bucket key differs from
+    // its physical primary key. They remain null when the incoming Fluss key already uses Paimon's
+    // BinaryRow encoding and no conversion is needed.
+    private @Nullable CompactedKeyDecoder compactedKeyDecoder;
+    private @Nullable PaimonKeyEncoder paimonKeyEncoder;
+    private boolean hasCachedValueEncoder;
+    private short cachedValueSchemaId;
+    private @Nullable RowEncoder cachedValueRowEncoder;
+    private @Nullable InternalRow.FieldGetter[] cachedValueFieldGetters;
+    private boolean closed;
+
+    public PaimonLakeTableLookuper(
+            Configuration paimonConfig,
+            TablePath tablePath,
+            String ioTmpDir,
+            TableConfig tableConfig) {
+        this.paimonConfig = checkNotNull(paimonConfig, "paimonConfig must not be null.");
+        this.tablePath = checkNotNull(tablePath, "tablePath must not be null.");
+        this.ioTmpDir = checkNotNull(ioTmpDir, "ioTmpDir must not be null.");
+        this.tableConfig = checkNotNull(tableConfig, "tableConfig must not be null.");
+        this.initializedBuckets = new HashSet<>();
+    }
+
+    @Override
+    public synchronized @Nullable byte[] lookup(byte[] key, LookupContext context)
+            throws Exception {
+        checkNotNull(key, "key must not be null.");
+        checkNotNull(context, "context must not be null.");
+        checkNotClosed();
+        ensureInitialized(context.valueRowType());
+
+        org.apache.paimon.data.BinaryRow partition =
+                convertPartition(context.partitionSpec(), context.valueRowType());
+        org.apache.paimon.data.BinaryRow keyRow = toPaimonLookupKey(key);
+        initializeFilesIfNeeded(partition, context.bucketId());
+
+        org.apache.paimon.data.InternalRow paimonRow =
+                lookupWithFileRefresh(
+                        partition, context.bucketId(), keyRow, context.valueRowType());
+        if (paimonRow == null) {
+            return null;
+        }
+        return encodeValue(paimonRow, context.schemaId(), context.valueRowType());
+    }
+
+    @Override
+    public synchronized void close() {
+        if (closed) {
+            return;
+        }
+        closed = true;
+        IOUtils.closeQuietly(cachedValueRowEncoder, "Fluss value row encoder");
+        IOUtils.closeQuietly(localTableQuery, "Paimon local table query");
+        IOUtils.closeQuietly(ioManager, "Paimon lookup IO manager");
+        IOUtils.closeQuietly(catalog, "Paimon catalog");
+        initializedBuckets.clear();
+    }
+
+    private void checkNotClosed() {
+        if (closed) {
+            throw new IllegalStateException("Paimon lake table lookuper has been closed.");
+        }
+    }
+
+    private void ensureInitialized(RowType valueRowType) throws Exception {
+        if (localTableQuery != null) {
+            return;
+        }
+
+        catalog =

Review Comment:
   Initialize these resources transactionally. If any step after assigning `catalog` fails, such as loading the table, validating the primary key, creating the IO manager/query, or initializing the key converter, this object retains partially initialized fields. A later lookup can overwrite the previous `Catalog` or `IOManager` without closing it, causing unbounded resource leaks under repeated failures; if `localTableQuery` was already assigned, the next lookup skips initialization with the remaining fields incomplete. Please build the resources in local variables and publish them only after full success, or close and reset every field on failure, and add a failure-injection test.



##########
fluss-server/src/main/java/org/apache/fluss/server/replica/HistoricalLakeLookupManager.java:
##########
@@ -0,0 +1,472 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.fluss.server.replica;
+
+import org.apache.fluss.annotation.VisibleForTesting;
+import org.apache.fluss.config.ConfigOptions;
+import org.apache.fluss.config.Configuration;
+import org.apache.fluss.config.TableConfig;
+import org.apache.fluss.exception.FlussRuntimeException;
+import org.apache.fluss.exception.HistoricalPartitionThrottledException;
+import org.apache.fluss.exception.InvalidPartitionException;
+import org.apache.fluss.exception.LakeStorageNotConfiguredException;
+import org.apache.fluss.lake.lakestorage.LakeStorage;
+import org.apache.fluss.lake.lakestorage.LakeStoragePlugin;
+import org.apache.fluss.lake.lakestorage.LakeStoragePluginSetUp;
+import org.apache.fluss.lake.lakestorage.LakeTableLookuper;
+import org.apache.fluss.metadata.DataLakeFormat;
+import org.apache.fluss.metadata.ResolvedPartitionSpec;
+import org.apache.fluss.metadata.SchemaInfo;
+import org.apache.fluss.metadata.TableBucket;
+import org.apache.fluss.metadata.TableInfo;
+import org.apache.fluss.metadata.TablePath;
+import org.apache.fluss.plugin.PluginManager;
+import org.apache.fluss.rpc.entity.LookupResultForBucket;
+import org.apache.fluss.rpc.protocol.ApiError;
+import org.apache.fluss.server.entity.LookupDataForBucket;
+import org.apache.fluss.utils.ExecutorUtils;
+import org.apache.fluss.utils.FileUtils;
+import org.apache.fluss.utils.IOUtils;
+import org.apache.fluss.utils.concurrent.ExecutorThreadFactory;
+import org.apache.fluss.utils.concurrent.Scheduler;
+
+import com.github.benmanes.caffeine.cache.Cache;
+import com.github.benmanes.caffeine.cache.Caffeine;
+import com.github.benmanes.caffeine.cache.RemovalCause;
+import com.github.benmanes.caffeine.cache.Ticker;
+
+import javax.annotation.Nullable;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.ArrayBlockingQueue;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Semaphore;
+import java.util.concurrent.ThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
+
+import static org.apache.fluss.server.utils.LakeStorageUtils.extractLakeProperties;
+import static org.apache.fluss.utils.Preconditions.checkArgument;
+import static org.apache.fluss.utils.Preconditions.checkNotNull;
+
+/**
+ * Handles server-side point lookup for historical partitions stored in lake storage.
+ *
+ * <p>Accepted requests run on a dedicated, bounded executor that is created lazily and releases
+ * idle threads. A semaphore bounds the total number of accepted historical lookup tasks so slow
+ * lake storage cannot create an unbounded request backlog.
+ *
+ * <p>Creating a lake table lookuper may initialize catalog, table, and query state and allocate
+ * local lookup files, so lookupers are cached and reused. The cache is keyed by table ID rather
+ * than table path to prevent a deleted and recreated table from reusing the old table's lookuper. A
+ * cached lookuper is replaced when its schema ID no longer matches the requested table schema.
+ *
+ * <p>A lookuper is closed when replaced, explicitly invalidated by a replica lifecycle event,
+ * evicted after the cache reaches ten tables, the manager shuts down, or after three hours without
+ * access. Caffeine expiration is scheduled on the shared TabletServer scheduler, allowing idle
+ * resources to be released even if no subsequent lookup accesses the cache.
+ */
+class HistoricalLakeLookupManager implements AutoCloseable {
+
+    private static final String PAIMON_LOOKUP_DIR_NAME = "paimon-lookup";
+    private static final String LOOKUPER_CACHE_EXPIRATION_TASK_NAME =
+            "historical-lookuper-cache-expiration";
+    private static final Duration LOOKUPER_CACHE_EXPIRATION = Duration.ofHours(3);
+    private static final Duration HISTORICAL_PARTITION_THREAD_KEEP_ALIVE = Duration.ofSeconds(60);
+    private static final Duration HISTORICAL_PARTITION_EXECUTOR_SHUTDOWN_TIMEOUT =
+            Duration.ofSeconds(10);
+    private static final int MAX_CACHED_LOOKUPERS = 10;
+    private static final String HISTORICAL_PARTITION_THREAD_NAME_PREFIX = "historical-partition-io";
+
+    // TODO: MAX_CACHED_LOOKUPERS and the 2GB per-table limit configured by
+    // PaimonLakeTableLookuper through Paimon's "lookup.cache-max-disk-size" option are hard-coded.
+    // Make them configurable, and prefer a Paimon IOManager-level global disk limit shared by all
+    // table lookupers because fixed per-table limits can underutilize cache for hot tables while
+    // reserving too much for cold tables.
+
+    private final Configuration conf;
+    private final @Nullable PluginManager pluginManager;
+    private final int serverId;
+    private final Semaphore lookupPermits;
+    // Upper bound on running and queued historical lookups; also used as the executor queue size.
+    private final int maxQueuedHistoricalRequests;
+    // Accepted lookup futures tracked so close() can cancel tasks left after executor shutdown.
+    private final Set<CompletableFuture<LookupResultForBucket>> pendingLookups;
+    private final Cache<Long, CachedLakeTableLookuper> lakeTableLookupers;
+    // Dedicated historical partition executor, created lazily on the first lookup.
+    private @Nullable ExecutorService historicalPartitionExecutor;
+    private boolean closed;
+    private @Nullable String paimonLookupTempDir;
+
+    HistoricalLakeLookupManager(
+            Configuration conf,
+            @Nullable PluginManager pluginManager,
+            int serverId,
+            Scheduler scheduler) {
+        this(
+                conf,
+                pluginManager,
+                null,
+                serverId,
+                Ticker.systemTicker(),
+                createCacheScheduler(scheduler));
+    }
+
+    @VisibleForTesting
+    HistoricalLakeLookupManager(
+            Configuration conf,
+            @Nullable PluginManager pluginManager,
+            @Nullable ExecutorService historicalPartitionExecutor,
+            int serverId,
+            Ticker ticker,
+            com.github.benmanes.caffeine.cache.Scheduler cacheScheduler) {
+        this.conf = checkNotNull(conf, "conf must not be null.");
+        this.pluginManager = pluginManager;
+        this.historicalPartitionExecutor = historicalPartitionExecutor;
+        this.serverId = serverId;
+        this.lakeTableLookupers =
+                Caffeine.newBuilder()
+                        .expireAfterAccess(LOOKUPER_CACHE_EXPIRATION)
+                        .maximumSize(MAX_CACHED_LOOKUPERS)
+                        .ticker(checkNotNull(ticker, "ticker must not be null."))
+                        .scheduler(checkNotNull(cacheScheduler, "cacheScheduler must not be null."))
+                        .executor(Runnable::run)
+                        .removalListener(
+                                (Long ignored,
+                                        CachedLakeTableLookuper cachedLookuper,
+                                        RemovalCause ignoredCause) -> {
+                                    if (cachedLookuper != null) {
+                                        cachedLookuper.invalidate();
+                                    }
+                                })
+                        .build();
+        this.maxQueuedHistoricalRequests =
+                conf.get(ConfigOptions.NETTY_SERVER_MAX_QUEUED_HISTORICAL_REQUESTS);
+        checkArgument(
+                maxQueuedHistoricalRequests > 0,
+                "%s must be greater than 0.",
+                ConfigOptions.NETTY_SERVER_MAX_QUEUED_HISTORICAL_REQUESTS.key());
+        this.lookupPermits = new Semaphore(maxQueuedHistoricalRequests);
+        this.pendingLookups = ConcurrentHashMap.newKeySet();
+    }
+
+    private static com.github.benmanes.caffeine.cache.Scheduler createCacheScheduler(
+            Scheduler scheduler) {
+        checkNotNull(scheduler, "scheduler must not be null.");
+        // Schedule expiration maintenance so idle lookupers are closed even if no more lookups
+        // arrive.
+        return (executor, command, delay, timeUnit) ->
+                scheduler.scheduleOnce(
+                        LOOKUPER_CACHE_EXPIRATION_TASK_NAME,
+                        () -> executor.execute(command),
+                        timeUnit.toMillis(delay));
+    }
+
+    CompletableFuture<LookupResultForBucket> lookup(
+            LookupDataForBucket lookupData, TableInfo tableInfo, SchemaInfo schemaInfo) {
+        TableBucket tableBucket = lookupData.tableBucket();
+        if (!lookupPermits.tryAcquire()) {
+            return CompletableFuture.completedFuture(
+                    new LookupResultForBucket(
+                            tableBucket,
+                            null,
+                            lookupData.originalPartitionName(),
+                            ApiError.fromThrowable(
+                                    new HistoricalPartitionThrottledException(
+                                            "Historical lookup is throttled for "
+                                                    + tableBucket
+                                                    + "."))));
+        }
+
+        CompletableFuture<LookupResultForBucket> future;
+        try {
+            future = submitLookup(lookupData, tableInfo, schemaInfo);
+        } catch (RuntimeException e) {
+            lookupPermits.release();
+            throw e;
+        }
+        future.whenComplete(
+                (ignored, error) -> {
+                    pendingLookups.remove(future);
+                    lookupPermits.release();
+                });
+        return future;
+    }
+
+    @Override
+    public void close() {
+        ExecutorService executor;
+        synchronized (this) {
+            closed = true;
+            executor = historicalPartitionExecutor;
+        }
+        if (executor != null) {
+            ExecutorUtils.gracefulShutdown(
+                    HISTORICAL_PARTITION_EXECUTOR_SHUTDOWN_TIMEOUT.toMillis(),
+                    TimeUnit.MILLISECONDS,
+                    executor);
+        }
+        pendingLookups.forEach(future -> future.cancel(true));
+        lakeTableLookupers.invalidateAll();
+        lakeTableLookupers.cleanUp();
+    }
+
+    private synchronized CompletableFuture<LookupResultForBucket> submitLookup(
+            LookupDataForBucket lookupData, TableInfo tableInfo, SchemaInfo schemaInfo) {
+        CompletableFuture<LookupResultForBucket> future =
+                CompletableFuture.supplyAsync(
+                        () -> lookupInternal(lookupData, tableInfo, schemaInfo),
+                        getOrCreateHistoricalPartitionExecutor());
+        pendingLookups.add(future);
+        return future;
+    }
+
+    private synchronized ExecutorService getOrCreateHistoricalPartitionExecutor() {
+        if (closed) {
+            throw new IllegalStateException("Historical lake lookup manager has been closed.");
+        }
+        if (historicalPartitionExecutor == null) {
+            int maxThreadPoolSize =
+                    conf.get(ConfigOptions.SERVER_HISTORICAL_PARTITION_THREAD_POOL_MAX_SIZE);
+            ThreadPoolExecutor executor =
+                    new ThreadPoolExecutor(
+                            maxThreadPoolSize,
+                            maxThreadPoolSize,
+                            HISTORICAL_PARTITION_THREAD_KEEP_ALIVE.toMillis(),
+                            TimeUnit.MILLISECONDS,
+                            new ArrayBlockingQueue<>(maxQueuedHistoricalRequests),

Review Comment:
   Since we have a outside `lookupPermits` to constraints the number of queued requests, I think we don't need to limit the queue size of the thread pool? 



##########
fluss-server/src/main/java/org/apache/fluss/server/utils/TableDescriptorValidation.java:
##########
@@ -169,6 +170,26 @@ private static void checkTableLakeFormatMatchesCluster(
         }
     }
 
+    private static void checkHistoricalPartition(
+            TableDescriptor tableDescriptor, Configuration tableConf) {
+        if (!tableConf.get(ConfigOptions.TABLE_DATALAKE_HISTORICAL_PARTITION_ENABLED)) {
+            return;
+        }
+
+        if (!tableDescriptor.hasPrimaryKey()
+                || tableDescriptor.getPartitionKeys().size() != 1
+                || !tableConf.get(ConfigOptions.TABLE_AUTO_PARTITION_ENABLED)
+                || !tableConf.get(ConfigOptions.TABLE_DATALAKE_ENABLED)
+                || tableConf.getOptional(ConfigOptions.TABLE_DATALAKE_FORMAT).orElse(null)
+                        != DataLakeFormat.PAIMON) {
+            throw new InvalidConfigException(
+                    String.format(
+                            "'%s' can only be enabled for auto-partitioned Paimon primary key "
+                                    + "tables with a single partition key.",
+                            ConfigOptions.TABLE_DATALAKE_HISTORICAL_PARTITION_ENABLED.key()));

Review Comment:
   Could we make this validation error more actionable by listing every unmet prerequisite? The current generic message does not tell users whether `table.auto-partition.enabled` or `table.datalake.enabled` is disabled, `table.datalake.format` is missing or not `paimon`, or the table lacks a primary key or exactly one partition key. Reporting all unmet requirements would let users correct the table definition in one pass.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [lookup] Support historical partition lookup for lake tables [fluss]

Posted by "luoyuxia (via GitHub)" <gi...@apache.org>.
luoyuxia commented on code in PR #3630:
URL: https://github.com/apache/fluss/pull/3630#discussion_r3602175472


##########
fluss-server/src/main/java/org/apache/fluss/server/coordinator/AutoPartitionManager.java:
##########
@@ -525,8 +526,15 @@ private void dropPartitions(
                 dropIterator = entry.getValue().iterator();
             }
 
+            boolean shouldRemoveEntry = true;
             while (dropIterator.hasNext()) {
                 String partitionName = dropIterator.next();
+                if (isHistoricalPartitionName(

Review Comment:
   Agreed, this is a real correctness issue. Auto-partition cleanup is currently driven by retention time and does not coordinate with tiering progress. Historical lookup exposes the problem as silently stale results, but it does not introduce the underlying lifecycle gap. 
   
   More generally, the same behavior already affects active consumers: for example, a partition may be deleted while a Flink job is still consuming it, causing the remaining data to become unavailable.
   
   Fixing this requires additional lifecycle coordination rather than a local change here. We need to define and persist a deletion barrier, verify that every bucket has been fully tiered, handle stalled or failed tiering, and recover the pending deletion state after restarts.
   
   This PR only changes historical lookup routing and preserves the historical system partition during retention cleanup. I’ll create a separate issue to track safe partition expiration and link it here.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [lookup] Support historical partition lookup for lake tables [fluss]

Posted by "zuston (via GitHub)" <gi...@apache.org>.
zuston commented on PR #3630:
URL: https://github.com/apache/fluss/pull/3630#issuecomment-4933546333

   Thanks for proposing this. @luoyuxia  I’m also exploring this area with Fluss + Paimon.
   
   Regarding the proposed client fallback to server-side historical partition lookup, I’m a bit concerned that this could be risky. The lake lookup operation can be much slower than a local RocksDB lookup, especially when Paimon is backed by HDFS with poor performance. If this lookup is executed on the tablet server side, it may slow down Netty worker threads and affect the latency of other RPC requests.
   
   For the case without historical data ingestion, I think this could potentially be handled by direct client-side lake lookup instead. On the Fluss side, we could consider introducing a Flink hybrid lake lookup async function for this purpose.
   For the case with historical data ingestion, client-side lookup could still be used as a fallback when the historical partition + key is not found on the tablet server side.
   
   Overall, I would prefer using client-side PK lookup where possible, to avoid overloading tablet servers during historical data ingestion bursts.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [lookup] Support historical partition lookup for lake tables [fluss]

Posted by "luoyuxia (via GitHub)" <gi...@apache.org>.
luoyuxia commented on code in PR #3630:
URL: https://github.com/apache/fluss/pull/3630#discussion_r3601468374


##########
fluss-client/src/main/java/org/apache/fluss/client/lookup/LookupSender.java:
##########


Review Comment:
   historical lookup is expected to be quite little, after offine discussion, we tend to keep the code simple to remove client.lookup.historical-inflight-ratio in current pr



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [lookup] Support historical partition lookup for lake tables [fluss]

Posted by "luoyuxia (via GitHub)" <gi...@apache.org>.
luoyuxia commented on code in PR #3630:
URL: https://github.com/apache/fluss/pull/3630#discussion_r3602297057


##########
fluss-client/src/main/java/org/apache/fluss/client/lookup/PrimaryKeyLookuper.java:
##########
@@ -115,19 +125,89 @@ public CompletableFuture<LookupResult> lookup(InternalRow lookupKey) {
                                 tableInfo.getTablePath(),
                                 metadataUpdater);
             } catch (PartitionNotExistException e) {
-                return CompletableFuture.completedFuture(new LookupResult(Collections.emptyList()));
+                return mayFallbackToHistoricalLookup(
+                        bucketingFunction.bucketing(bkBytes, numBuckets), pkBytes, lookupKey);
             }
         }
 
         int bucketId = bucketingFunction.bucketing(bkBytes, numBuckets);
         TableBucket tableBucket = new TableBucket(tableInfo.getTableId(), partitionId, bucketId);
+        return lookupBucket(tableBucket, pkBytes, insertIfNotExists, null, lookupKey);
+    }
+
+    /** Falls back to historical lookup if the lookup key belongs to a historical partition. */
+    private CompletableFuture<LookupResult> mayFallbackToHistoricalLookup(
+            int bucketId, byte[] keyBytes, InternalRow lookupKey) {
+        String originalPartitionName = partitionGetter.getPartition(lookupKey);
+        if (!isHistoricalLookupCandidatePartition(
+                tableInfo, originalPartitionName, Instant.now())) {

Review Comment:
   This is a valid clock-skew edge case.
   
   In a normally time-synchronized deployment, triggering it requires the partition cleanup to complete during the short interval after the Coordinator crosses a partition boundary but before the client does. A lookup must also target the newly expired time slice during that same interval. The negative decision is not cached, so subsequent lookups start using the historical path as soon as the client crosses the boundary.
   
   Making the decision fully authoritative would require moving it to the Coordinator or exposing Coordinator-derived expiration state through metadata/RPC, which is a broader cross-layer change.
   
   Given the narrow and transient failure window, I’d prefer to track this in a follow-up issue and keep the current PR scoped.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [lookup] Support historical partition lookup for lake tables [fluss]

Posted by "zuston (via GitHub)" <gi...@apache.org>.
zuston commented on PR #3630:
URL: https://github.com/apache/fluss/pull/3630#issuecomment-4943744786

   > I think the two approaches can coexist, similar to Paimon’s local lookup and remote query service models.
   
   +1. 
   
   > For this PR, my preference is to keep the bounded asynchronous server-side path because it also supports historical ingestion semantics, and explore client-side lookup as an optional strategy in a follow-up.
   
   After this PR is merged, let me reuse the lake lookup interface to implement the client side fallback mechanism. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [lookup] Support historical partition lookup for lake tables [fluss]

Posted by "beryllw (via GitHub)" <gi...@apache.org>.
beryllw commented on code in PR #3630:
URL: https://github.com/apache/fluss/pull/3630#discussion_r3609663178


##########
fluss-lake/fluss-lake-paimon/src/main/java/org/apache/fluss/lake/paimon/lookup/PaimonLakeTableLookuper.java:
##########
@@ -0,0 +1,375 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.fluss.lake.paimon.lookup;
+
+import org.apache.fluss.config.ConfigOptions;
+import org.apache.fluss.config.Configuration;
+import org.apache.fluss.exception.KvStorageException;
+import org.apache.fluss.lake.lakestorage.LakeTableLookuper;
+import org.apache.fluss.lake.paimon.utils.PaimonPartitionBucket;
+import org.apache.fluss.lake.paimon.utils.PaimonRowAsFlussRow;
+import org.apache.fluss.metadata.KvFormat;
+import org.apache.fluss.metadata.ResolvedPartitionSpec;
+import org.apache.fluss.metadata.TablePath;
+import org.apache.fluss.row.BinaryRow;
+import org.apache.fluss.row.InternalRow;
+import org.apache.fluss.row.encode.RowEncoder;
+import org.apache.fluss.row.encode.ValueEncoder;
+import org.apache.fluss.types.RowType;
+import org.apache.fluss.utils.IOUtils;
+
+import org.apache.paimon.CoreOptions;
+import org.apache.paimon.catalog.Catalog;
+import org.apache.paimon.catalog.CatalogContext;
+import org.apache.paimon.catalog.CatalogFactory;
+import org.apache.paimon.disk.IOManager;
+import org.apache.paimon.io.DataFileMeta;
+import org.apache.paimon.memory.MemorySegment;
+import org.apache.paimon.options.MemorySize;
+import org.apache.paimon.options.Options;
+import org.apache.paimon.table.FileStoreTable;
+import org.apache.paimon.table.query.LocalTableQuery;
+import org.apache.paimon.table.sink.RowPartitionKeyExtractor;
+import org.apache.paimon.table.source.DataSplit;
+import org.apache.paimon.table.source.InnerTableScan;
+import org.apache.paimon.table.source.Split;
+import org.apache.paimon.types.DataField;
+
+import javax.annotation.Nullable;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Set;
+
+import static org.apache.fluss.lake.paimon.PaimonLakeCatalog.SYSTEM_COLUMNS;
+import static org.apache.fluss.lake.paimon.utils.PaimonConversions.toPaimon;
+import static org.apache.fluss.lake.paimon.utils.PaimonConversions.toPaimonPartition;
+import static org.apache.fluss.utils.Preconditions.checkNotNull;
+
+/**
+ * Paimon implementation of {@link LakeTableLookuper} for primary-key tables.
+ *
+ * <p>The catalog, table, local query, and I/O manager are initialized lazily on the first lookup.
+ * For each partition and bucket, the lookuper scans the latest Paimon snapshot once and registers
+ * its data files with {@link LocalTableQuery}. Paimon then creates local lookup files lazily as
+ * individual remote data files are queried.
+ *
+ * <p>A cached partition-bucket file set can become stale when Paimon compaction replaces its data
+ * files and snapshot expiration physically deletes the old files. Because {@code FileIO}
+ * implementations may represent a missing file with different {@link IOException} types, the first
+ * lookup I/O failure closes the cached query state, reopens the table from the latest snapshot, and
+ * retries once.
+ *
+ * <p>Lookup and close operations are synchronized because they share mutable Paimon query, local
+ * cache, and value-encoding state.
+ */
+public class PaimonLakeTableLookuper implements LakeTableLookuper {
+
+    // Each TabletServer caches at most ten table lookupers, bounding their retained lookup cache
+    // capacity to 20GB. See HistoricalLakeLookupManager for the follow-up to make this configurable
+    // and use a global Paimon IOManager limit.
+    private static final String LOOKUP_CACHE_MAX_DISK_SIZE = "2gb";
+    private static final MemorySize LOOKUP_CACHE_MAX_DISK_MEMORY_SIZE =
+            MemorySize.parse(LOOKUP_CACHE_MAX_DISK_SIZE);
+
+    private final Configuration paimonConfig;
+    private final TablePath tablePath;
+    private final String ioTmpDir;
+    private final KvFormat kvFormat;
+
+    private final Set<PaimonPartitionBucket> initializedBuckets;
+
+    private @Nullable Catalog catalog;
+    private @Nullable FileStoreTable fileStoreTable;
+    private @Nullable IOManager ioManager;
+    private @Nullable LocalTableQuery localTableQuery;
+    private @Nullable RowPartitionKeyExtractor partitionKeyExtractor;
+    private int primaryKeyFieldCount;
+    private boolean hasCachedValueEncoder;
+    private short cachedValueSchemaId;
+    private @Nullable RowEncoder cachedValueRowEncoder;
+    private @Nullable InternalRow.FieldGetter[] cachedValueFieldGetters;
+    private boolean closed;
+
+    public PaimonLakeTableLookuper(
+            Configuration paimonConfig, TablePath tablePath, KvFormat kvFormat) {
+        this(paimonConfig, tablePath, ConfigOptions.IO_TMP_DIR.defaultValue(), kvFormat);
+    }
+
+    public PaimonLakeTableLookuper(
+            Configuration paimonConfig, TablePath tablePath, String ioTmpDir, KvFormat kvFormat) {
+        this.paimonConfig = checkNotNull(paimonConfig, "paimonConfig must not be null.");
+        this.tablePath = checkNotNull(tablePath, "tablePath must not be null.");
+        this.ioTmpDir = checkNotNull(ioTmpDir, "ioTmpDir must not be null.");
+        this.kvFormat = checkNotNull(kvFormat, "kvFormat must not be null.");
+        this.initializedBuckets = new HashSet<>();
+    }
+
+    @Override
+    public synchronized @Nullable byte[] lookup(byte[] key, LookupContext context)
+            throws Exception {
+        checkNotNull(key, "key must not be null.");
+        checkNotNull(context, "context must not be null.");
+        checkNotClosed();
+        ensureInitialized();
+
+        org.apache.paimon.data.BinaryRow partition =
+                convertPartition(context.partitionSpec(), context.valueRowType());
+        org.apache.paimon.data.BinaryRow keyRow = wrapLookupKey(key);
+        initializeFilesIfNeeded(partition, context.bucketId());
+
+        org.apache.paimon.data.InternalRow paimonRow =
+                lookupWithFileRefresh(partition, context.bucketId(), keyRow);
+        if (paimonRow == null) {
+            return null;
+        }
+        return encodeValue(paimonRow, context.schemaId(), context.valueRowType());
+    }
+
+    @Override
+    public synchronized void close() {
+        if (closed) {
+            return;
+        }
+        closed = true;
+        IOUtils.closeQuietly(cachedValueRowEncoder, "Fluss value row encoder");
+        IOUtils.closeQuietly(localTableQuery, "Paimon local table query");
+        IOUtils.closeQuietly(ioManager, "Paimon lookup IO manager");
+        IOUtils.closeQuietly(catalog, "Paimon catalog");
+        initializedBuckets.clear();
+    }
+
+    private void checkNotClosed() {
+        if (closed) {
+            throw new IllegalStateException("Paimon lake table lookuper has been closed.");
+        }
+    }
+
+    private void ensureInitialized() throws Exception {
+        if (localTableQuery != null) {
+            return;
+        }
+
+        catalog =
+                CatalogFactory.createCatalog(
+                        CatalogContext.create(Options.fromMap(paimonConfig.toMap())));
+        fileStoreTable =
+                withLookupCacheOptions((FileStoreTable) catalog.getTable(toPaimon(tablePath)));
+        if (fileStoreTable.primaryKeys().isEmpty()) {
+            throw new UnsupportedOperationException(
+                    "Point lookup is only supported for primary-key Paimon tables.");
+        }
+        ioManager = createIOManager(ioTmpDir);
+        localTableQuery = createLocalTableQuery();
+        partitionKeyExtractor = new RowPartitionKeyExtractor(fileStoreTable.schema());
+        primaryKeyFieldCount = fileStoreTable.primaryKeys().size();
+    }
+
+    private LocalTableQuery createLocalTableQuery() {
+        return fileStoreTable()
+                .newLocalTableQuery()
+                .withValueProjection(businessFieldProjection(fileStoreTable()))
+                .withIOManager(checkNotNull(ioManager, "ioManager must be initialized."));
+    }
+
+    private FileStoreTable withLookupCacheOptions(FileStoreTable table) {
+        String key = CoreOptions.LOOKUP_CACHE_MAX_DISK_SIZE.key();
+        String configuredMaxDiskSize = table.options().get(key);
+        if (configuredMaxDiskSize != null
+                && MemorySize.parse(configuredMaxDiskSize)
+                                .compareTo(LOOKUP_CACHE_MAX_DISK_MEMORY_SIZE)
+                        <= 0) {
+            return table;
+        }
+        return table.copy(Collections.singletonMap(key, LOOKUP_CACHE_MAX_DISK_SIZE));
+    }
+
+    private static IOManager createIOManager(String ioTmpDir) {
+        return IOManager.create(ioTmpDir);
+    }
+
+    private static int[] businessFieldProjection(FileStoreTable fileStoreTable) {
+        List<DataField> fields = fileStoreTable.schema().logicalRowType().getFields();
+        List<Integer> projectedFields = new ArrayList<>();
+        for (int i = 0; i < fields.size(); i++) {
+            if (!SYSTEM_COLUMNS.containsKey(fields.get(i).name())) {
+                projectedFields.add(i);
+            }
+        }
+
+        int[] projection = new int[projectedFields.size()];
+        for (int i = 0; i < projectedFields.size(); i++) {
+            projection[i] = projectedFields.get(i);
+        }
+        return projection;
+    }
+
+    private org.apache.paimon.data.BinaryRow convertPartition(
+            ResolvedPartitionSpec partitionSpec, RowType valueRowType) {
+        return toPaimonPartition(
+                partitionSpec,
+                valueRowType,
+                fileStoreTable().schema().logicalRowType(),
+                partitionKeyExtractor()::partition);
+    }
+
+    private org.apache.paimon.data.BinaryRow wrapLookupKey(byte[] key) {

Review Comment:
   Should we double-check the key encoding for historical lookup on kv-format-v2 tables? For a non-default bucket key, `PrimaryKeyLookuper` sends a Fluss `CompactedKeyEncoder` key, but `PaimonLakeTableLookuper.wrapLookupKey` treats the bytes as a Paimon `BinaryRow` — so it might be worth confirming these two encodings line up (the current ITCase uses a default bucket key, so it wouldn't cover this path).



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [lookup] Support historical partition lookup for lake tables [fluss]

Posted by "wuchong (via GitHub)" <gi...@apache.org>.
wuchong merged PR #3630:
URL: https://github.com/apache/fluss/pull/3630


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [lookup] Support historical partition lookup for lake tables [fluss]

Posted by "luoyuxia (via GitHub)" <gi...@apache.org>.
luoyuxia commented on code in PR #3630:
URL: https://github.com/apache/fluss/pull/3630#discussion_r3601658200


##########
fluss-lake/fluss-lake-paimon/src/main/java/org/apache/fluss/lake/paimon/lookup/PaimonLakeTableLookuper.java:
##########
@@ -0,0 +1,359 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.fluss.lake.paimon.lookup;
+
+import org.apache.fluss.config.ConfigOptions;
+import org.apache.fluss.config.Configuration;
+import org.apache.fluss.exception.KvStorageException;
+import org.apache.fluss.lake.lakestorage.LakeTableLookuper;
+import org.apache.fluss.lake.paimon.utils.PaimonPartitionBucket;
+import org.apache.fluss.lake.paimon.utils.PaimonRowAsFlussRow;
+import org.apache.fluss.metadata.ResolvedPartitionSpec;
+import org.apache.fluss.metadata.TablePath;
+import org.apache.fluss.row.BinaryRow;
+import org.apache.fluss.row.InternalRow;
+import org.apache.fluss.row.encode.CompactedRowEncoder;
+import org.apache.fluss.row.encode.ValueEncoder;
+import org.apache.fluss.types.DataType;
+import org.apache.fluss.types.RowType;
+import org.apache.fluss.utils.IOUtils;
+
+import org.apache.paimon.CoreOptions;
+import org.apache.paimon.catalog.Catalog;
+import org.apache.paimon.catalog.CatalogContext;
+import org.apache.paimon.catalog.CatalogFactory;
+import org.apache.paimon.disk.IOManager;
+import org.apache.paimon.io.DataFileMeta;
+import org.apache.paimon.memory.MemorySegment;
+import org.apache.paimon.options.Options;
+import org.apache.paimon.table.FileStoreTable;
+import org.apache.paimon.table.query.LocalTableQuery;
+import org.apache.paimon.table.sink.RowPartitionKeyExtractor;
+import org.apache.paimon.table.source.DataSplit;
+import org.apache.paimon.table.source.InnerTableScan;
+import org.apache.paimon.table.source.Split;
+import org.apache.paimon.types.DataField;
+
+import javax.annotation.Nullable;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Set;
+
+import static org.apache.fluss.lake.paimon.PaimonLakeCatalog.SYSTEM_COLUMNS;
+import static org.apache.fluss.lake.paimon.utils.PaimonConversions.toPaimon;
+import static org.apache.fluss.lake.paimon.utils.PaimonConversions.toPaimonPartition;
+import static org.apache.fluss.utils.Preconditions.checkNotNull;
+
+/**
+ * Paimon implementation of {@link LakeTableLookuper} for primary-key tables.
+ *
+ * <p>The catalog, table, local query, and I/O manager are initialized lazily on the first lookup.
+ * For each partition and bucket, the lookuper scans the latest Paimon snapshot once and registers
+ * its data files with {@link LocalTableQuery}. Paimon then creates local lookup files lazily as
+ * individual remote data files are queried.
+ *
+ * <p>A cached partition-bucket file set can become stale when Paimon compaction replaces its data
+ * files and snapshot expiration physically deletes the old files. Because {@code FileIO}
+ * implementations may represent a missing file with different {@link IOException} types, the first
+ * lookup I/O failure closes the cached query state, reopens the table from the latest snapshot, and
+ * retries once.
+ *
+ * <p>Lookup and close operations are synchronized because they share mutable Paimon query, local
+ * cache, and value-encoding state.
+ */
+public class PaimonLakeTableLookuper implements LakeTableLookuper {
+
+    // Hard-code a conservative 1GB default for now to avoid unbounded lookup cache growth.
+    private static final String DEFAULT_LOOKUP_CACHE_MAX_DISK_SIZE = "1gb";
+
+    private final Configuration paimonConfig;
+    private final TablePath tablePath;
+    private final String ioTmpDir;
+
+    private final Set<PaimonPartitionBucket> initializedBuckets;
+
+    private @Nullable Catalog catalog;
+    private @Nullable FileStoreTable fileStoreTable;
+    private @Nullable IOManager ioManager;
+    private @Nullable LocalTableQuery localTableQuery;
+    private @Nullable RowPartitionKeyExtractor partitionKeyExtractor;
+    private int primaryKeyFieldCount;
+    private boolean hasCachedValueEncoder;
+    private short cachedValueSchemaId;
+    private @Nullable CompactedRowEncoder cachedValueRowEncoder;
+    private @Nullable InternalRow.FieldGetter[] cachedValueFieldGetters;
+    private boolean closed;
+
+    public PaimonLakeTableLookuper(Configuration paimonConfig, TablePath tablePath) {
+        this(paimonConfig, tablePath, ConfigOptions.IO_TMP_DIR.defaultValue());
+    }
+
+    public PaimonLakeTableLookuper(
+            Configuration paimonConfig, TablePath tablePath, String ioTmpDir) {
+        this.paimonConfig = checkNotNull(paimonConfig, "paimonConfig must not be null.");
+        this.tablePath = checkNotNull(tablePath, "tablePath must not be null.");
+        this.ioTmpDir = checkNotNull(ioTmpDir, "ioTmpDir must not be null.");
+        this.initializedBuckets = new HashSet<>();
+    }
+
+    @Override
+    public synchronized @Nullable byte[] lookup(byte[] key, LookupContext context)

Review Comment:
   Good point. 
   For the initial implementation, I’d prefer to keep using the shared executor. Historical lookup is optional, and introducing a dedicated pool and another configuration option would add resources and configuration surface for clusters that may never use it. We expect the initial lookup traffic to be limited and can tune the existing I/O pool for deployments that need it.
   I suggest revisiting a dedicated, possibly lazily created executor if real workloads show contention



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [lookup] Support historical partition lookup for lake tables [fluss]

Posted by "fresh-borzoni (via GitHub)" <gi...@apache.org>.
fresh-borzoni commented on code in PR #3630:
URL: https://github.com/apache/fluss/pull/3630#discussion_r3599724998


##########
fluss-rpc/src/main/proto/FlussApi.proto:
##########
@@ -946,6 +946,8 @@ message PbLookupReqForBucket {
   optional int64 partition_id = 1;
   required int32 bucket_id = 2;
   repeated bytes keys = 3;
+  // The original partition name for historical lookup. It is unset for normal lookup.
+  optional string original_partition_name = 4;

Review Comment:
   An old tablet server ignores this field and looks up the empty historical rocksdb and returns null instead of an error. During rolling upgrades that's silent misses. Gate on api version instead?



##########
fluss-server/src/main/java/org/apache/fluss/server/replica/ReplicaManager.java:
##########
@@ -978,6 +1124,7 @@ public void stopReplicas(
 
                     for (StopReplicaData data : stopReplicaDataList) {
                         TableBucket tb = data.getTableBucket();
+                        historicalLakeLookupManager.invalidateTableLookuper(tb.getTableId());

Review Comment:
   minor: This closes the lookuper on the calling thread, and close() waits for the lock held by an in-flight lookup. So stopping a replica can block on a slow S3 read. Mb move the close to the ioExecutor?



##########
fluss-client/src/main/java/org/apache/fluss/client/lookup/HistoricalPartitionResolver.java:
##########
@@ -0,0 +1,186 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.fluss.client.lookup;
+
+import org.apache.fluss.annotation.Internal;
+import org.apache.fluss.client.admin.Admin;
+import org.apache.fluss.client.metadata.MetadataUpdater;
+import org.apache.fluss.exception.PartitionNotExistException;
+import org.apache.fluss.metadata.PhysicalTablePath;
+import org.apache.fluss.metadata.ResolvedPartitionSpec;
+import org.apache.fluss.metadata.TableInfo;
+import org.apache.fluss.metadata.TablePath;
+
+import javax.annotation.Nullable;
+
+import java.util.Objects;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+
+import static org.apache.fluss.utils.PartitionUtils.toHistoricalPartitionSpec;
+import static org.apache.fluss.utils.Preconditions.checkNotNull;
+
+/**
+ * Resolves original partitions to their historical system partition ids.
+ *
+ * <p>Historical lookup requests still carry the original partition name, but the lookup RPC must be
+ * sent to the generated historical system partition. This resolver bridges that gap on the client
+ * side.
+ */
+@Internal
+class HistoricalPartitionResolver {
+
+    private final MetadataUpdater metadataUpdater;
+    private final Admin admin;
+    private final ConcurrentHashMap<HistoricalPartitionKey, CompletableFuture<Long>>
+            inflightResolves;
+
+    HistoricalPartitionResolver(MetadataUpdater metadataUpdater, Admin admin) {
+        this.metadataUpdater = checkNotNull(metadataUpdater, "metadataUpdater must not be null.");
+        this.admin = checkNotNull(admin, "admin must not be null.");
+        this.inflightResolves = new ConcurrentHashMap<>();
+    }
+
+    CompletableFuture<Long> resolveHistoricalPartitionId(
+            TableInfo tableInfo, String originalPartitionName) {
+        HistoricalPartitionKey key =
+                new HistoricalPartitionKey(
+                        tableInfo.getTableId(), tableInfo.getTablePath(), originalPartitionName);
+        // Multiple lookups for the same original partition can be issued concurrently. Coalesce
+        // them so only one metadata refresh/create path runs for a given partition.
+        CompletableFuture<Long> result = new CompletableFuture<>();
+        CompletableFuture<Long> previous = inflightResolves.putIfAbsent(key, result);
+        if (previous != null) {
+            return previous;
+        }
+
+        resolveHistoricalPartitionIdInternal(tableInfo, originalPartitionName)
+                .whenComplete(
+                        (partitionId, error) -> {
+                            if (error != null) {
+                                result.completeExceptionally(error);
+                            } else {
+                                result.complete(partitionId);
+                            }
+                            inflightResolves.remove(key, result);
+                        });
+        return result;
+    }
+
+    private CompletableFuture<Long> resolveHistoricalPartitionIdInternal(
+            TableInfo tableInfo, String originalPartitionName) {
+        CompletableFuture<Long> result = new CompletableFuture<>();
+        ResolvedPartitionSpec historicalPartitionSpec;
+        PhysicalTablePath historicalPartitionPath;
+        try {
+            // The server-side historical lookup path is keyed by the historical system partition,
+            // not by the original partition that existed before retention cleanup.
+            historicalPartitionSpec = toHistoricalPartitionSpec(tableInfo, originalPartitionName);
+            historicalPartitionPath =
+                    PhysicalTablePath.of(
+                            tableInfo.getTablePath(), historicalPartitionSpec.getPartitionName());
+
+            // Prefer cached metadata, then refresh once before creating the system partition.
+            Long partitionId = getCachedPartitionId(historicalPartitionPath);
+            if (partitionId != null) {
+                result.complete(partitionId);
+                return result;
+            }
+
+            tryRefreshHistoricalPartition(historicalPartitionPath);
+            partitionId = getCachedPartitionId(historicalPartitionPath);
+            if (partitionId != null) {
+                result.complete(partitionId);
+                return result;
+            }
+        } catch (Throwable t) {
+            result.completeExceptionally(t);
+            return result;
+        }
+
+        // If the historical system partition still does not exist, create it idempotently and
+        // refresh metadata again so the caller can route the lookup request by partition id.
+        admin.createPartition(

Review Comment:
   Creating a partition from the lookup path feels backwards: a read allocates real replicas and rocksdb that may never see a write. Could we route lookups without the partition existing, and leave creation to the write path?



##########
fluss-server/src/main/java/org/apache/fluss/server/coordinator/CoordinatorService.java:
##########
@@ -723,16 +725,36 @@ public CompletableFuture<CreatePartitionResponse> createPartition(
 
         // first, validate the partition spec, and get resolved partition spec.
         PartitionSpec partitionSpec = getPartitionSpec(request.getPartitionSpec());
-        validatePartitionSpec(tablePath, table.partitionKeys, partitionSpec, true);
+        ResolvedPartitionSpec partitionToCreate;
+        if (isHistoricalPartitionCreate(table, partitionSpec)) {
+            // Historical system partitions are lookup metadata, so creating one requires the same
+            // permission as reading the table instead of writing table data.
+            authorizeTable(OperationType.READ, tablePath);

Review Comment:
   Why READ? A read-only client can now create partitions and for multi-key tables the prefix values come from the lookup key, so bad keys create real partitions until max.partition.num is full and auto-partitioning breaks.



##########
fluss-server/src/main/java/org/apache/fluss/server/coordinator/AutoPartitionManager.java:
##########
@@ -525,8 +526,15 @@ private void dropPartitions(
                 dropIterator = entry.getValue().iterator();
             }
 
+            boolean shouldRemoveEntry = true;
             while (dropIterator.hasNext()) {
                 String partitionName = dropIterator.next();
+                if (isHistoricalPartitionName(

Review Comment:
   Related to dropping here, drop doesn't wait for tiering to catch up. If a partition is dropped before its tail is tiered, historical lookups quietly serve older values. Should drop wait for the tiering offset, like local log deletion waits for remote copy?



##########
fluss-lake/fluss-lake-paimon/src/main/java/org/apache/fluss/lake/paimon/lookup/PaimonLakeTableLookuper.java:
##########
@@ -0,0 +1,359 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.fluss.lake.paimon.lookup;
+
+import org.apache.fluss.config.ConfigOptions;
+import org.apache.fluss.config.Configuration;
+import org.apache.fluss.exception.KvStorageException;
+import org.apache.fluss.lake.lakestorage.LakeTableLookuper;
+import org.apache.fluss.lake.paimon.utils.PaimonPartitionBucket;
+import org.apache.fluss.lake.paimon.utils.PaimonRowAsFlussRow;
+import org.apache.fluss.metadata.ResolvedPartitionSpec;
+import org.apache.fluss.metadata.TablePath;
+import org.apache.fluss.row.BinaryRow;
+import org.apache.fluss.row.InternalRow;
+import org.apache.fluss.row.encode.CompactedRowEncoder;
+import org.apache.fluss.row.encode.ValueEncoder;
+import org.apache.fluss.types.DataType;
+import org.apache.fluss.types.RowType;
+import org.apache.fluss.utils.IOUtils;
+
+import org.apache.paimon.CoreOptions;
+import org.apache.paimon.catalog.Catalog;
+import org.apache.paimon.catalog.CatalogContext;
+import org.apache.paimon.catalog.CatalogFactory;
+import org.apache.paimon.disk.IOManager;
+import org.apache.paimon.io.DataFileMeta;
+import org.apache.paimon.memory.MemorySegment;
+import org.apache.paimon.options.Options;
+import org.apache.paimon.table.FileStoreTable;
+import org.apache.paimon.table.query.LocalTableQuery;
+import org.apache.paimon.table.sink.RowPartitionKeyExtractor;
+import org.apache.paimon.table.source.DataSplit;
+import org.apache.paimon.table.source.InnerTableScan;
+import org.apache.paimon.table.source.Split;
+import org.apache.paimon.types.DataField;
+
+import javax.annotation.Nullable;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Set;
+
+import static org.apache.fluss.lake.paimon.PaimonLakeCatalog.SYSTEM_COLUMNS;
+import static org.apache.fluss.lake.paimon.utils.PaimonConversions.toPaimon;
+import static org.apache.fluss.lake.paimon.utils.PaimonConversions.toPaimonPartition;
+import static org.apache.fluss.utils.Preconditions.checkNotNull;
+
+/**
+ * Paimon implementation of {@link LakeTableLookuper} for primary-key tables.
+ *
+ * <p>The catalog, table, local query, and I/O manager are initialized lazily on the first lookup.
+ * For each partition and bucket, the lookuper scans the latest Paimon snapshot once and registers
+ * its data files with {@link LocalTableQuery}. Paimon then creates local lookup files lazily as
+ * individual remote data files are queried.
+ *
+ * <p>A cached partition-bucket file set can become stale when Paimon compaction replaces its data
+ * files and snapshot expiration physically deletes the old files. Because {@code FileIO}
+ * implementations may represent a missing file with different {@link IOException} types, the first
+ * lookup I/O failure closes the cached query state, reopens the table from the latest snapshot, and
+ * retries once.
+ *
+ * <p>Lookup and close operations are synchronized because they share mutable Paimon query, local
+ * cache, and value-encoding state.
+ */
+public class PaimonLakeTableLookuper implements LakeTableLookuper {
+
+    // Hard-code a conservative 1GB default for now to avoid unbounded lookup cache growth.
+    private static final String DEFAULT_LOOKUP_CACHE_MAX_DISK_SIZE = "1gb";
+
+    private final Configuration paimonConfig;
+    private final TablePath tablePath;
+    private final String ioTmpDir;
+
+    private final Set<PaimonPartitionBucket> initializedBuckets;
+
+    private @Nullable Catalog catalog;
+    private @Nullable FileStoreTable fileStoreTable;
+    private @Nullable IOManager ioManager;
+    private @Nullable LocalTableQuery localTableQuery;
+    private @Nullable RowPartitionKeyExtractor partitionKeyExtractor;
+    private int primaryKeyFieldCount;
+    private boolean hasCachedValueEncoder;
+    private short cachedValueSchemaId;
+    private @Nullable CompactedRowEncoder cachedValueRowEncoder;
+    private @Nullable InternalRow.FieldGetter[] cachedValueFieldGetters;
+    private boolean closed;
+
+    public PaimonLakeTableLookuper(Configuration paimonConfig, TablePath tablePath) {
+        this(paimonConfig, tablePath, ConfigOptions.IO_TMP_DIR.defaultValue());
+    }
+
+    public PaimonLakeTableLookuper(
+            Configuration paimonConfig, TablePath tablePath, String ioTmpDir) {
+        this.paimonConfig = checkNotNull(paimonConfig, "paimonConfig must not be null.");
+        this.tablePath = checkNotNull(tablePath, "tablePath must not be null.");
+        this.ioTmpDir = checkNotNull(ioTmpDir, "ioTmpDir must not be null.");
+        this.initializedBuckets = new HashSet<>();
+    }
+
+    @Override
+    public synchronized @Nullable byte[] lookup(byte[] key, LookupContext context)
+            throws Exception {
+        checkNotNull(key, "key must not be null.");
+        checkNotNull(context, "context must not be null.");
+        checkNotClosed();
+        ensureInitialized();
+
+        org.apache.paimon.data.BinaryRow partition =
+                convertPartition(context.partitionSpec(), context.valueRowType());
+        org.apache.paimon.data.BinaryRow keyRow = wrapLookupKey(key);
+        initializeFilesIfNeeded(partition, context.bucketId());
+
+        org.apache.paimon.data.InternalRow paimonRow =
+                lookupWithFileRefresh(partition, context.bucketId(), keyRow);
+        if (paimonRow == null) {
+            return null;
+        }
+        return encodeValue(paimonRow, context.schemaId(), context.valueRowType());
+    }
+
+    @Override
+    public synchronized void close() {
+        if (closed) {
+            return;
+        }
+        closed = true;
+        IOUtils.closeQuietly(cachedValueRowEncoder, "Fluss value row encoder");
+        IOUtils.closeQuietly(localTableQuery, "Paimon local table query");
+        IOUtils.closeQuietly(ioManager, "Paimon lookup IO manager");
+        IOUtils.closeQuietly(catalog, "Paimon catalog");
+        initializedBuckets.clear();
+    }
+
+    private void checkNotClosed() {
+        if (closed) {
+            throw new IllegalStateException("Paimon lake table lookuper has been closed.");
+        }
+    }
+
+    private void ensureInitialized() throws Exception {
+        if (localTableQuery != null) {
+            return;
+        }
+
+        catalog =
+                CatalogFactory.createCatalog(
+                        CatalogContext.create(Options.fromMap(paimonConfig.toMap())));
+        fileStoreTable =
+                withLookupCacheOptions((FileStoreTable) catalog.getTable(toPaimon(tablePath)));
+        if (fileStoreTable.primaryKeys().isEmpty()) {
+            throw new UnsupportedOperationException(
+                    "Point lookup is only supported for primary-key Paimon tables.");
+        }
+        ioManager = createIOManager(ioTmpDir);
+        localTableQuery = createLocalTableQuery();
+        partitionKeyExtractor = new RowPartitionKeyExtractor(fileStoreTable.schema());
+        primaryKeyFieldCount = fileStoreTable.primaryKeys().size();
+    }
+
+    private LocalTableQuery createLocalTableQuery() {
+        return fileStoreTable()
+                .newLocalTableQuery()
+                .withValueProjection(businessFieldProjection(fileStoreTable()))
+                .withIOManager(checkNotNull(ioManager, "ioManager must be initialized."));
+    }
+
+    private FileStoreTable withLookupCacheOptions(FileStoreTable table) {
+        String key = CoreOptions.LOOKUP_CACHE_MAX_DISK_SIZE.key();
+        if (table.options().containsKey(key)) {
+            return table;
+        }
+        return table.copy(Collections.singletonMap(key, DEFAULT_LOOKUP_CACHE_MAX_DISK_SIZE));
+    }
+
+    private static IOManager createIOManager(String ioTmpDir) {
+        return IOManager.create(ioTmpDir);
+    }
+
+    private static int[] businessFieldProjection(FileStoreTable fileStoreTable) {
+        List<DataField> fields = fileStoreTable.schema().logicalRowType().getFields();
+        List<Integer> projectedFields = new ArrayList<>();
+        for (int i = 0; i < fields.size(); i++) {
+            if (!SYSTEM_COLUMNS.containsKey(fields.get(i).name())) {
+                projectedFields.add(i);
+            }
+        }
+
+        int[] projection = new int[projectedFields.size()];
+        for (int i = 0; i < projectedFields.size(); i++) {
+            projection[i] = projectedFields.get(i);
+        }
+        return projection;
+    }
+
+    private org.apache.paimon.data.BinaryRow convertPartition(
+            ResolvedPartitionSpec partitionSpec, RowType valueRowType) {
+        return toPaimonPartition(
+                partitionSpec,
+                valueRowType,
+                fileStoreTable().schema().logicalRowType(),
+                partitionKeyExtractor()::partition);
+    }
+
+    private org.apache.paimon.data.BinaryRow wrapLookupKey(byte[] key) {
+        org.apache.paimon.data.BinaryRow keyRow =
+                new org.apache.paimon.data.BinaryRow(primaryKeyFieldCount);
+        keyRow.pointTo(MemorySegment.wrap(key), 0, key.length);
+        return keyRow;
+    }
+
+    private void initializeFilesIfNeeded(org.apache.paimon.data.BinaryRow partition, int bucketId) {
+        PaimonPartitionBucket partitionBucket = new PaimonPartitionBucket(partition, bucketId);
+        if (initializedBuckets.contains(partitionBucket)) {

Review Comment:
   Looks like we never refresh the file set after the first scan. If tiering commits more data into this bucket later, lookups will miss it and the expiry never fires on a hot partition. 
   
   Should we check the snapshot id and rescan when it changes?



##########
fluss-server/src/main/java/org/apache/fluss/server/replica/HistoricalLakeLookupManager.java:
##########
@@ -0,0 +1,346 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.fluss.server.replica;
+
+import org.apache.fluss.annotation.VisibleForTesting;
+import org.apache.fluss.config.ConfigOptions;
+import org.apache.fluss.config.Configuration;
+import org.apache.fluss.exception.FlussRuntimeException;
+import org.apache.fluss.exception.HistoricalLookupThrottledException;
+import org.apache.fluss.exception.InvalidPartitionException;
+import org.apache.fluss.exception.LakeStorageNotConfiguredException;
+import org.apache.fluss.lake.lakestorage.LakeStorage;
+import org.apache.fluss.lake.lakestorage.LakeStoragePlugin;
+import org.apache.fluss.lake.lakestorage.LakeStoragePluginSetUp;
+import org.apache.fluss.lake.lakestorage.LakeTableLookuper;
+import org.apache.fluss.metadata.DataLakeFormat;
+import org.apache.fluss.metadata.ResolvedPartitionSpec;
+import org.apache.fluss.metadata.TableBucket;
+import org.apache.fluss.metadata.TableInfo;
+import org.apache.fluss.metadata.TablePath;
+import org.apache.fluss.plugin.PluginManager;
+import org.apache.fluss.rpc.entity.LookupResultForBucket;
+import org.apache.fluss.rpc.protocol.ApiError;
+import org.apache.fluss.server.entity.LookupDataForBucket;
+import org.apache.fluss.utils.FileUtils;
+import org.apache.fluss.utils.IOUtils;
+import org.apache.fluss.utils.concurrent.Scheduler;
+
+import com.github.benmanes.caffeine.cache.Cache;
+import com.github.benmanes.caffeine.cache.Caffeine;
+import com.github.benmanes.caffeine.cache.RemovalCause;
+import com.github.benmanes.caffeine.cache.Ticker;
+
+import javax.annotation.Nullable;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Semaphore;
+
+import static org.apache.fluss.server.utils.LakeStorageUtils.extractLakeProperties;
+import static org.apache.fluss.utils.Preconditions.checkArgument;
+import static org.apache.fluss.utils.Preconditions.checkNotNull;
+
+/**
+ * Handles server-side point lookup for historical partitions stored in lake storage.
+ *
+ * <p>Accepted requests run on the TabletServer I/O executor. A semaphore bounds the total number of
+ * accepted historical lookup tasks so slow lake storage cannot create an unbounded request backlog.
+ *
+ * <p>Creating a lake table lookuper may initialize catalog, table, and query state and allocate
+ * local lookup files, so lookupers are cached and reused. The cache is keyed by table ID rather
+ * than table path to prevent a deleted and recreated table from reusing the old table's lookuper. A
+ * cached lookuper is replaced when its schema ID no longer matches the requested table schema.
+ *
+ * <p>A lookuper is closed when replaced, explicitly invalidated by a replica lifecycle event, the
+ * manager shuts down, or after three hours without access. Caffeine expiration is scheduled on the
+ * shared TabletServer scheduler, allowing idle resources to be released even if no subsequent
+ * lookup accesses the cache.
+ */
+class HistoricalLakeLookupManager implements AutoCloseable {
+
+    private static final String PAIMON_LOOKUP_DIR_NAME = "paimon-lookup";
+    private static final String LOOKUPER_CACHE_EXPIRATION_TASK_NAME =
+            "historical-lookuper-cache-expiration";
+    private static final Duration LOOKUPER_CACHE_EXPIRATION = Duration.ofHours(3);
+
+    private final Configuration conf;
+    private final @Nullable PluginManager pluginManager;
+    private final ExecutorService ioExecutor;
+    private final int serverId;
+    private final Semaphore lookupPermits;
+    private final Cache<Long, CachedLakeTableLookuper> lakeTableLookupers;
+    private @Nullable String paimonLookupTempDir;
+
+    HistoricalLakeLookupManager(
+            Configuration conf,
+            @Nullable PluginManager pluginManager,
+            ExecutorService ioExecutor,
+            int serverId,
+            Scheduler scheduler) {
+        this(
+                conf,
+                pluginManager,
+                ioExecutor,
+                serverId,
+                Ticker.systemTicker(),
+                createCacheScheduler(scheduler));
+    }
+
+    @VisibleForTesting
+    HistoricalLakeLookupManager(
+            Configuration conf,
+            @Nullable PluginManager pluginManager,
+            ExecutorService ioExecutor,
+            int serverId,
+            Ticker ticker,
+            com.github.benmanes.caffeine.cache.Scheduler cacheScheduler) {
+        this.conf = checkNotNull(conf, "conf must not be null.");
+        this.pluginManager = pluginManager;
+        this.ioExecutor = checkNotNull(ioExecutor, "ioExecutor must not be null.");
+        this.serverId = serverId;
+        this.lakeTableLookupers =
+                Caffeine.newBuilder()
+                        .expireAfterAccess(LOOKUPER_CACHE_EXPIRATION)
+                        .ticker(checkNotNull(ticker, "ticker must not be null."))
+                        .scheduler(checkNotNull(cacheScheduler, "cacheScheduler must not be null."))
+                        .executor(Runnable::run)
+                        .removalListener(
+                                (Long ignored,
+                                        CachedLakeTableLookuper cachedLookuper,
+                                        RemovalCause ignoredCause) -> {
+                                    if (cachedLookuper != null) {
+                                        closeLookuper(cachedLookuper);
+                                    }
+                                })
+                        .build();
+        int maxQueuedHistoricalRequests =
+                conf.get(ConfigOptions.NETTY_SERVER_MAX_QUEUED_HISTORICAL_REQUESTS);
+        checkArgument(
+                maxQueuedHistoricalRequests > 0,
+                "%s must be greater than 0.",
+                ConfigOptions.NETTY_SERVER_MAX_QUEUED_HISTORICAL_REQUESTS.key());
+        this.lookupPermits = new Semaphore(maxQueuedHistoricalRequests);
+    }
+
+    private static com.github.benmanes.caffeine.cache.Scheduler createCacheScheduler(
+            Scheduler scheduler) {
+        checkNotNull(scheduler, "scheduler must not be null.");
+        // Schedule expiration maintenance so idle lookupers are closed even if no more lookups
+        // arrive.
+        return (executor, command, delay, timeUnit) ->
+                scheduler.scheduleOnce(
+                        LOOKUPER_CACHE_EXPIRATION_TASK_NAME,
+                        () -> executor.execute(command),
+                        timeUnit.toMillis(delay));
+    }
+
+    CompletableFuture<LookupResultForBucket> lookup(
+            LookupDataForBucket lookupData, TableInfo tableInfo) {
+        TableBucket tableBucket = lookupData.tableBucket();
+        if (!lookupPermits.tryAcquire()) {
+            return CompletableFuture.completedFuture(
+                    new LookupResultForBucket(
+                            tableBucket,
+                            ApiError.fromThrowable(
+                                    new HistoricalLookupThrottledException(
+                                            "Historical lookup is throttled for "
+                                                    + tableBucket
+                                                    + "."))));
+        }
+
+        CompletableFuture<LookupResultForBucket> future;
+        try {
+            future =
+                    CompletableFuture.supplyAsync(
+                            () -> lookupInternal(lookupData, tableInfo), ioExecutor);
+        } catch (RuntimeException e) {
+            lookupPermits.release();
+            throw e;
+        }
+        future.whenComplete((ignored, error) -> lookupPermits.release());
+        return future;
+    }
+
+    @Override
+    public void close() {
+        lakeTableLookupers.invalidateAll();
+        lakeTableLookupers.cleanUp();
+    }
+
+    void invalidateTableLookuper(long tableId) {
+        lakeTableLookupers.invalidate(tableId);
+    }
+
+    private LookupResultForBucket lookupInternal(
+            LookupDataForBucket lookupData, TableInfo tableInfo) {
+        TableBucket tableBucket = lookupData.tableBucket();
+        try {
+            LookupContext context = createLookupContext(lookupData, tableInfo);
+            CachedLakeTableLookuper cachedLookuper =
+                    lakeTableLookupers
+                            .asMap()
+                            .compute(
+                                    context.tableId,
+                                    (ignored, currentLookuper) -> {
+                                        if (currentLookuper != null
+                                                && currentLookuper.schemaId == context.schemaId) {
+                                            return currentLookuper;
+                                        }
+                                        LakeTableLookuper newLookuper =
+                                                createLakeTableLookuper(
+                                                        context.tablePath,
+                                                        getOrPreparePaimonLookupTempDir());
+                                        return new CachedLakeTableLookuper(
+                                                context.schemaId, newLookuper);
+                                    });
+            List<byte[]> values = new ArrayList<>(lookupData.keys().size());
+            for (byte[] key : lookupData.keys()) {
+                values.add(cachedLookuper.lookuper.lookup(key, context.lookupContext));
+            }
+            return new LookupResultForBucket(tableBucket, values);
+        } catch (Exception e) {
+            return new LookupResultForBucket(tableBucket, ApiError.fromThrowable(e));
+        }
+    }
+
+    private LookupContext createLookupContext(LookupDataForBucket lookupData, TableInfo tableInfo) {
+        TableBucket tableBucket = lookupData.tableBucket();
+        String originalPartitionName = lookupData.originalPartitionName();
+        if (originalPartitionName == null) {
+            throw new InvalidPartitionException(
+                    "Historical lookup request must carry the original partition name.");
+        }
+
+        TablePath tablePath = tableInfo.getTablePath();
+
+        ResolvedPartitionSpec originalPartitionSpec;
+        try {
+            originalPartitionSpec =
+                    ResolvedPartitionSpec.fromPartitionName(
+                            tableInfo.getPartitionKeys(), originalPartitionName);
+        } catch (RuntimeException e) {
+            throw new InvalidPartitionException(
+                    String.format(
+                            "Invalid original partition name %s for historical lookup on table %s.",
+                            originalPartitionName, tablePath));
+        }
+
+        LakeTableLookuper.LookupContext lookupContext =
+                new LakeTableLookuper.LookupContext(
+                        originalPartitionSpec,
+                        tableBucket.getBucket(),
+                        (short) tableInfo.getSchemaInfo().getSchemaId(),

Review Comment:
   Replica.tableInfo is frozen when the replica is created. After adding a column, won't this keep encoding with the old schema and return null for the new column? 
   Maybe use replica.getSchemaGetter() instead?



##########
fluss-server/src/main/java/org/apache/fluss/server/replica/ReplicaManager.java:
##########
@@ -774,6 +871,49 @@ public void lookups(
         lookups(false, null, null, entriesPerBucket, apiVersion, responseCallback);
     }
 
+    /** Lookup historical lake data for requests that carry original partition names. */
+    public void historicalLookups(

Review Comment:
   This goes straight to lake, but the FIP says buffer -> historical rocksdb -> lake. How will the write-path PR plug that in here?



##########
fluss-client/src/main/java/org/apache/fluss/client/lookup/LookupSender.java:
##########


Review Comment:
   Q: Historical lookups share this semaphore with normal ones, and lake lookups are slow, they can hold all the permits and block normal lookups. The FIP splits this into two semaphores (client.lookup.historical-inflight-ratio, historical capped at ~10%). Was dropping that intentional?



##########
fluss-client/src/main/java/org/apache/fluss/client/lookup/PrimaryKeyLookuper.java:
##########
@@ -115,19 +125,89 @@ public CompletableFuture<LookupResult> lookup(InternalRow lookupKey) {
                                 tableInfo.getTablePath(),
                                 metadataUpdater);
             } catch (PartitionNotExistException e) {
-                return CompletableFuture.completedFuture(new LookupResult(Collections.emptyList()));
+                return mayFallbackToHistoricalLookup(
+                        bucketingFunction.bucketing(bkBytes, numBuckets), pkBytes, lookupKey);
             }
         }
 
         int bucketId = bucketingFunction.bucketing(bkBytes, numBuckets);
         TableBucket tableBucket = new TableBucket(tableInfo.getTableId(), partitionId, bucketId);
+        return lookupBucket(tableBucket, pkBytes, insertIfNotExists, null, lookupKey);
+    }
+
+    /** Falls back to historical lookup if the lookup key belongs to a historical partition. */
+    private CompletableFuture<LookupResult> mayFallbackToHistoricalLookup(
+            int bucketId, byte[] keyBytes, InternalRow lookupKey) {
+        String originalPartitionName = partitionGetter.getPartition(lookupKey);
+        if (!isHistoricalLookupCandidatePartition(
+                tableInfo, originalPartitionName, Instant.now())) {
+            return CompletableFuture.completedFuture(new LookupResult(Collections.emptyList()));
+        }
+        metadataUpdater.invalidPhysicalTableBucketAndPartitionMeta(

Review Comment:
   This happens on every lookup to an expired partition, right? Each one does a failed metadata RPC plus a full Cluster copy here. 
   
   Can we cache the "this partition is expired" decision?



##########
fluss-lake/fluss-lake-paimon/src/main/java/org/apache/fluss/lake/paimon/lookup/PaimonLakeTableLookuper.java:
##########
@@ -0,0 +1,359 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.fluss.lake.paimon.lookup;
+
+import org.apache.fluss.config.ConfigOptions;
+import org.apache.fluss.config.Configuration;
+import org.apache.fluss.exception.KvStorageException;
+import org.apache.fluss.lake.lakestorage.LakeTableLookuper;
+import org.apache.fluss.lake.paimon.utils.PaimonPartitionBucket;
+import org.apache.fluss.lake.paimon.utils.PaimonRowAsFlussRow;
+import org.apache.fluss.metadata.ResolvedPartitionSpec;
+import org.apache.fluss.metadata.TablePath;
+import org.apache.fluss.row.BinaryRow;
+import org.apache.fluss.row.InternalRow;
+import org.apache.fluss.row.encode.CompactedRowEncoder;
+import org.apache.fluss.row.encode.ValueEncoder;
+import org.apache.fluss.types.DataType;
+import org.apache.fluss.types.RowType;
+import org.apache.fluss.utils.IOUtils;
+
+import org.apache.paimon.CoreOptions;
+import org.apache.paimon.catalog.Catalog;
+import org.apache.paimon.catalog.CatalogContext;
+import org.apache.paimon.catalog.CatalogFactory;
+import org.apache.paimon.disk.IOManager;
+import org.apache.paimon.io.DataFileMeta;
+import org.apache.paimon.memory.MemorySegment;
+import org.apache.paimon.options.Options;
+import org.apache.paimon.table.FileStoreTable;
+import org.apache.paimon.table.query.LocalTableQuery;
+import org.apache.paimon.table.sink.RowPartitionKeyExtractor;
+import org.apache.paimon.table.source.DataSplit;
+import org.apache.paimon.table.source.InnerTableScan;
+import org.apache.paimon.table.source.Split;
+import org.apache.paimon.types.DataField;
+
+import javax.annotation.Nullable;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Set;
+
+import static org.apache.fluss.lake.paimon.PaimonLakeCatalog.SYSTEM_COLUMNS;
+import static org.apache.fluss.lake.paimon.utils.PaimonConversions.toPaimon;
+import static org.apache.fluss.lake.paimon.utils.PaimonConversions.toPaimonPartition;
+import static org.apache.fluss.utils.Preconditions.checkNotNull;
+
+/**
+ * Paimon implementation of {@link LakeTableLookuper} for primary-key tables.
+ *
+ * <p>The catalog, table, local query, and I/O manager are initialized lazily on the first lookup.
+ * For each partition and bucket, the lookuper scans the latest Paimon snapshot once and registers
+ * its data files with {@link LocalTableQuery}. Paimon then creates local lookup files lazily as
+ * individual remote data files are queried.
+ *
+ * <p>A cached partition-bucket file set can become stale when Paimon compaction replaces its data
+ * files and snapshot expiration physically deletes the old files. Because {@code FileIO}
+ * implementations may represent a missing file with different {@link IOException} types, the first
+ * lookup I/O failure closes the cached query state, reopens the table from the latest snapshot, and
+ * retries once.
+ *
+ * <p>Lookup and close operations are synchronized because they share mutable Paimon query, local
+ * cache, and value-encoding state.
+ */
+public class PaimonLakeTableLookuper implements LakeTableLookuper {
+
+    // Hard-code a conservative 1GB default for now to avoid unbounded lookup cache growth.
+    private static final String DEFAULT_LOOKUP_CACHE_MAX_DISK_SIZE = "1gb";

Review Comment:
   nit: this is per table, so 20 lake tables can use 20gb of tmp disk. Do we need a global cap?



##########
fluss-lake/fluss-lake-paimon/src/main/java/org/apache/fluss/lake/paimon/lookup/PaimonLakeTableLookuper.java:
##########
@@ -0,0 +1,359 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.fluss.lake.paimon.lookup;
+
+import org.apache.fluss.config.ConfigOptions;
+import org.apache.fluss.config.Configuration;
+import org.apache.fluss.exception.KvStorageException;
+import org.apache.fluss.lake.lakestorage.LakeTableLookuper;
+import org.apache.fluss.lake.paimon.utils.PaimonPartitionBucket;
+import org.apache.fluss.lake.paimon.utils.PaimonRowAsFlussRow;
+import org.apache.fluss.metadata.ResolvedPartitionSpec;
+import org.apache.fluss.metadata.TablePath;
+import org.apache.fluss.row.BinaryRow;
+import org.apache.fluss.row.InternalRow;
+import org.apache.fluss.row.encode.CompactedRowEncoder;
+import org.apache.fluss.row.encode.ValueEncoder;
+import org.apache.fluss.types.DataType;
+import org.apache.fluss.types.RowType;
+import org.apache.fluss.utils.IOUtils;
+
+import org.apache.paimon.CoreOptions;
+import org.apache.paimon.catalog.Catalog;
+import org.apache.paimon.catalog.CatalogContext;
+import org.apache.paimon.catalog.CatalogFactory;
+import org.apache.paimon.disk.IOManager;
+import org.apache.paimon.io.DataFileMeta;
+import org.apache.paimon.memory.MemorySegment;
+import org.apache.paimon.options.Options;
+import org.apache.paimon.table.FileStoreTable;
+import org.apache.paimon.table.query.LocalTableQuery;
+import org.apache.paimon.table.sink.RowPartitionKeyExtractor;
+import org.apache.paimon.table.source.DataSplit;
+import org.apache.paimon.table.source.InnerTableScan;
+import org.apache.paimon.table.source.Split;
+import org.apache.paimon.types.DataField;
+
+import javax.annotation.Nullable;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Set;
+
+import static org.apache.fluss.lake.paimon.PaimonLakeCatalog.SYSTEM_COLUMNS;
+import static org.apache.fluss.lake.paimon.utils.PaimonConversions.toPaimon;
+import static org.apache.fluss.lake.paimon.utils.PaimonConversions.toPaimonPartition;
+import static org.apache.fluss.utils.Preconditions.checkNotNull;
+
+/**
+ * Paimon implementation of {@link LakeTableLookuper} for primary-key tables.
+ *
+ * <p>The catalog, table, local query, and I/O manager are initialized lazily on the first lookup.
+ * For each partition and bucket, the lookuper scans the latest Paimon snapshot once and registers
+ * its data files with {@link LocalTableQuery}. Paimon then creates local lookup files lazily as
+ * individual remote data files are queried.
+ *
+ * <p>A cached partition-bucket file set can become stale when Paimon compaction replaces its data
+ * files and snapshot expiration physically deletes the old files. Because {@code FileIO}
+ * implementations may represent a missing file with different {@link IOException} types, the first
+ * lookup I/O failure closes the cached query state, reopens the table from the latest snapshot, and
+ * retries once.
+ *
+ * <p>Lookup and close operations are synchronized because they share mutable Paimon query, local
+ * cache, and value-encoding state.
+ */
+public class PaimonLakeTableLookuper implements LakeTableLookuper {
+
+    // Hard-code a conservative 1GB default for now to avoid unbounded lookup cache growth.
+    private static final String DEFAULT_LOOKUP_CACHE_MAX_DISK_SIZE = "1gb";
+
+    private final Configuration paimonConfig;
+    private final TablePath tablePath;
+    private final String ioTmpDir;
+
+    private final Set<PaimonPartitionBucket> initializedBuckets;
+
+    private @Nullable Catalog catalog;
+    private @Nullable FileStoreTable fileStoreTable;
+    private @Nullable IOManager ioManager;
+    private @Nullable LocalTableQuery localTableQuery;
+    private @Nullable RowPartitionKeyExtractor partitionKeyExtractor;
+    private int primaryKeyFieldCount;
+    private boolean hasCachedValueEncoder;
+    private short cachedValueSchemaId;
+    private @Nullable CompactedRowEncoder cachedValueRowEncoder;
+    private @Nullable InternalRow.FieldGetter[] cachedValueFieldGetters;
+    private boolean closed;
+
+    public PaimonLakeTableLookuper(Configuration paimonConfig, TablePath tablePath) {
+        this(paimonConfig, tablePath, ConfigOptions.IO_TMP_DIR.defaultValue());
+    }
+
+    public PaimonLakeTableLookuper(
+            Configuration paimonConfig, TablePath tablePath, String ioTmpDir) {
+        this.paimonConfig = checkNotNull(paimonConfig, "paimonConfig must not be null.");
+        this.tablePath = checkNotNull(tablePath, "tablePath must not be null.");
+        this.ioTmpDir = checkNotNull(ioTmpDir, "ioTmpDir must not be null.");
+        this.initializedBuckets = new HashSet<>();
+    }
+
+    @Override
+    public synchronized @Nullable byte[] lookup(byte[] key, LookupContext context)

Review Comment:
   This synchronizes all lake lookups of a table on one lock, and each waiting lookup holds an ioExecutor thread. 
   That pool is size 10? by default and also does snapshot/remote-log transfers. Should historical lookups use their own pool?



##########
fluss-lake/fluss-lake-paimon/src/main/java/org/apache/fluss/lake/paimon/lookup/PaimonLakeTableLookuper.java:
##########
@@ -0,0 +1,359 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.fluss.lake.paimon.lookup;
+
+import org.apache.fluss.config.ConfigOptions;
+import org.apache.fluss.config.Configuration;
+import org.apache.fluss.exception.KvStorageException;
+import org.apache.fluss.lake.lakestorage.LakeTableLookuper;
+import org.apache.fluss.lake.paimon.utils.PaimonPartitionBucket;
+import org.apache.fluss.lake.paimon.utils.PaimonRowAsFlussRow;
+import org.apache.fluss.metadata.ResolvedPartitionSpec;
+import org.apache.fluss.metadata.TablePath;
+import org.apache.fluss.row.BinaryRow;
+import org.apache.fluss.row.InternalRow;
+import org.apache.fluss.row.encode.CompactedRowEncoder;
+import org.apache.fluss.row.encode.ValueEncoder;
+import org.apache.fluss.types.DataType;
+import org.apache.fluss.types.RowType;
+import org.apache.fluss.utils.IOUtils;
+
+import org.apache.paimon.CoreOptions;
+import org.apache.paimon.catalog.Catalog;
+import org.apache.paimon.catalog.CatalogContext;
+import org.apache.paimon.catalog.CatalogFactory;
+import org.apache.paimon.disk.IOManager;
+import org.apache.paimon.io.DataFileMeta;
+import org.apache.paimon.memory.MemorySegment;
+import org.apache.paimon.options.Options;
+import org.apache.paimon.table.FileStoreTable;
+import org.apache.paimon.table.query.LocalTableQuery;
+import org.apache.paimon.table.sink.RowPartitionKeyExtractor;
+import org.apache.paimon.table.source.DataSplit;
+import org.apache.paimon.table.source.InnerTableScan;
+import org.apache.paimon.table.source.Split;
+import org.apache.paimon.types.DataField;
+
+import javax.annotation.Nullable;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Set;
+
+import static org.apache.fluss.lake.paimon.PaimonLakeCatalog.SYSTEM_COLUMNS;
+import static org.apache.fluss.lake.paimon.utils.PaimonConversions.toPaimon;
+import static org.apache.fluss.lake.paimon.utils.PaimonConversions.toPaimonPartition;
+import static org.apache.fluss.utils.Preconditions.checkNotNull;
+
+/**
+ * Paimon implementation of {@link LakeTableLookuper} for primary-key tables.
+ *
+ * <p>The catalog, table, local query, and I/O manager are initialized lazily on the first lookup.
+ * For each partition and bucket, the lookuper scans the latest Paimon snapshot once and registers
+ * its data files with {@link LocalTableQuery}. Paimon then creates local lookup files lazily as
+ * individual remote data files are queried.
+ *
+ * <p>A cached partition-bucket file set can become stale when Paimon compaction replaces its data
+ * files and snapshot expiration physically deletes the old files. Because {@code FileIO}
+ * implementations may represent a missing file with different {@link IOException} types, the first
+ * lookup I/O failure closes the cached query state, reopens the table from the latest snapshot, and
+ * retries once.
+ *
+ * <p>Lookup and close operations are synchronized because they share mutable Paimon query, local
+ * cache, and value-encoding state.
+ */
+public class PaimonLakeTableLookuper implements LakeTableLookuper {
+
+    // Hard-code a conservative 1GB default for now to avoid unbounded lookup cache growth.
+    private static final String DEFAULT_LOOKUP_CACHE_MAX_DISK_SIZE = "1gb";
+
+    private final Configuration paimonConfig;
+    private final TablePath tablePath;
+    private final String ioTmpDir;
+
+    private final Set<PaimonPartitionBucket> initializedBuckets;
+
+    private @Nullable Catalog catalog;
+    private @Nullable FileStoreTable fileStoreTable;
+    private @Nullable IOManager ioManager;
+    private @Nullable LocalTableQuery localTableQuery;
+    private @Nullable RowPartitionKeyExtractor partitionKeyExtractor;
+    private int primaryKeyFieldCount;
+    private boolean hasCachedValueEncoder;
+    private short cachedValueSchemaId;
+    private @Nullable CompactedRowEncoder cachedValueRowEncoder;
+    private @Nullable InternalRow.FieldGetter[] cachedValueFieldGetters;
+    private boolean closed;
+
+    public PaimonLakeTableLookuper(Configuration paimonConfig, TablePath tablePath) {
+        this(paimonConfig, tablePath, ConfigOptions.IO_TMP_DIR.defaultValue());
+    }
+
+    public PaimonLakeTableLookuper(
+            Configuration paimonConfig, TablePath tablePath, String ioTmpDir) {
+        this.paimonConfig = checkNotNull(paimonConfig, "paimonConfig must not be null.");
+        this.tablePath = checkNotNull(tablePath, "tablePath must not be null.");
+        this.ioTmpDir = checkNotNull(ioTmpDir, "ioTmpDir must not be null.");
+        this.initializedBuckets = new HashSet<>();
+    }
+
+    @Override
+    public synchronized @Nullable byte[] lookup(byte[] key, LookupContext context)
+            throws Exception {
+        checkNotNull(key, "key must not be null.");
+        checkNotNull(context, "context must not be null.");
+        checkNotClosed();
+        ensureInitialized();
+
+        org.apache.paimon.data.BinaryRow partition =
+                convertPartition(context.partitionSpec(), context.valueRowType());
+        org.apache.paimon.data.BinaryRow keyRow = wrapLookupKey(key);
+        initializeFilesIfNeeded(partition, context.bucketId());
+
+        org.apache.paimon.data.InternalRow paimonRow =
+                lookupWithFileRefresh(partition, context.bucketId(), keyRow);
+        if (paimonRow == null) {
+            return null;
+        }
+        return encodeValue(paimonRow, context.schemaId(), context.valueRowType());
+    }
+
+    @Override
+    public synchronized void close() {
+        if (closed) {
+            return;
+        }
+        closed = true;
+        IOUtils.closeQuietly(cachedValueRowEncoder, "Fluss value row encoder");
+        IOUtils.closeQuietly(localTableQuery, "Paimon local table query");
+        IOUtils.closeQuietly(ioManager, "Paimon lookup IO manager");
+        IOUtils.closeQuietly(catalog, "Paimon catalog");
+        initializedBuckets.clear();
+    }
+
+    private void checkNotClosed() {
+        if (closed) {
+            throw new IllegalStateException("Paimon lake table lookuper has been closed.");
+        }
+    }
+
+    private void ensureInitialized() throws Exception {
+        if (localTableQuery != null) {
+            return;
+        }
+
+        catalog =
+                CatalogFactory.createCatalog(
+                        CatalogContext.create(Options.fromMap(paimonConfig.toMap())));
+        fileStoreTable =
+                withLookupCacheOptions((FileStoreTable) catalog.getTable(toPaimon(tablePath)));
+        if (fileStoreTable.primaryKeys().isEmpty()) {
+            throw new UnsupportedOperationException(
+                    "Point lookup is only supported for primary-key Paimon tables.");
+        }
+        ioManager = createIOManager(ioTmpDir);
+        localTableQuery = createLocalTableQuery();
+        partitionKeyExtractor = new RowPartitionKeyExtractor(fileStoreTable.schema());
+        primaryKeyFieldCount = fileStoreTable.primaryKeys().size();
+    }
+
+    private LocalTableQuery createLocalTableQuery() {
+        return fileStoreTable()
+                .newLocalTableQuery()
+                .withValueProjection(businessFieldProjection(fileStoreTable()))
+                .withIOManager(checkNotNull(ioManager, "ioManager must be initialized."));
+    }
+
+    private FileStoreTable withLookupCacheOptions(FileStoreTable table) {
+        String key = CoreOptions.LOOKUP_CACHE_MAX_DISK_SIZE.key();
+        if (table.options().containsKey(key)) {
+            return table;
+        }
+        return table.copy(Collections.singletonMap(key, DEFAULT_LOOKUP_CACHE_MAX_DISK_SIZE));
+    }
+
+    private static IOManager createIOManager(String ioTmpDir) {
+        return IOManager.create(ioTmpDir);
+    }
+
+    private static int[] businessFieldProjection(FileStoreTable fileStoreTable) {
+        List<DataField> fields = fileStoreTable.schema().logicalRowType().getFields();
+        List<Integer> projectedFields = new ArrayList<>();
+        for (int i = 0; i < fields.size(); i++) {
+            if (!SYSTEM_COLUMNS.containsKey(fields.get(i).name())) {
+                projectedFields.add(i);
+            }
+        }
+
+        int[] projection = new int[projectedFields.size()];
+        for (int i = 0; i < projectedFields.size(); i++) {
+            projection[i] = projectedFields.get(i);
+        }
+        return projection;
+    }
+
+    private org.apache.paimon.data.BinaryRow convertPartition(
+            ResolvedPartitionSpec partitionSpec, RowType valueRowType) {
+        return toPaimonPartition(
+                partitionSpec,
+                valueRowType,
+                fileStoreTable().schema().logicalRowType(),
+                partitionKeyExtractor()::partition);
+    }
+
+    private org.apache.paimon.data.BinaryRow wrapLookupKey(byte[] key) {
+        org.apache.paimon.data.BinaryRow keyRow =
+                new org.apache.paimon.data.BinaryRow(primaryKeyFieldCount);
+        keyRow.pointTo(MemorySegment.wrap(key), 0, key.length);
+        return keyRow;
+    }
+
+    private void initializeFilesIfNeeded(org.apache.paimon.data.BinaryRow partition, int bucketId) {
+        PaimonPartitionBucket partitionBucket = new PaimonPartitionBucket(partition, bucketId);
+        if (initializedBuckets.contains(partitionBucket)) {
+            return;
+        }
+
+        LinkedHashMap<String, DataFileMeta> beforeFilesByName = new LinkedHashMap<>();
+        LinkedHashMap<String, DataFileMeta> dataFilesByName = new LinkedHashMap<>();
+
+        InnerTableScan tableScan =
+                fileStoreTable()
+                        .newScan()
+                        .withPartitionFilter(Collections.singletonList(partition))
+                        .withBucket(bucketId);
+        for (Split split : tableScan.plan().splits()) {
+            if (!(split instanceof DataSplit)) {
+                continue;
+            }
+            DataSplit dataSplit = (DataSplit) split;
+            addFilesByName(beforeFilesByName, dataSplit.beforeFiles());
+            addFilesByName(dataFilesByName, dataSplit.dataFiles());
+        }
+
+        localTableQuery()
+                .refreshFiles(
+                        partition,
+                        bucketId,
+                        new ArrayList<>(beforeFilesByName.values()),
+                        new ArrayList<>(dataFilesByName.values()));
+        initializedBuckets.add(partitionBucket);
+    }
+
+    private org.apache.paimon.data.InternalRow lookupWithFileRefresh(
+            org.apache.paimon.data.BinaryRow partition,
+            int bucketId,
+            org.apache.paimon.data.InternalRow keyRow)
+            throws Exception {
+        try {
+            return localTableQuery().lookup(partition, bucketId, keyRow);
+        } catch (IOException e) {
+            // FileIO only guarantees IOException and storage plugins may use different exception
+            // types for a missing file. The missing old file after compaction may therefore
+            // surface as any IOException. Refresh and retry only once so persistent I/O failures
+            // do not repeatedly rebuild Paimon lookup state within one request.
+            try {
+                refreshFiles(partition, bucketId);
+                return localTableQuery().lookup(partition, bucketId, keyRow);
+            } catch (IOException retryError) {
+                retryError.addSuppressed(e);
+                // Historical Paimon point lookup is part of the Fluss KV lookup path. Expose a
+                // persistent I/O failure as a retriable KV error so the existing KV RPC retry
+                // semantics can handle it consistently.
+                throw new KvStorageException(
+                        "Failed to lookup historical data from Paimon after refreshing files for "
+                                + tablePath
+                                + ".",
+                        retryError);
+            }
+        }
+    }
+
+    private void refreshFiles(org.apache.paimon.data.BinaryRow partition, int bucketId)
+            throws Exception {
+        IOUtils.closeQuietly(localTableQuery, "Paimon local table query");
+        IOUtils.closeQuietly(ioManager, "Paimon lookup IO manager");
+        IOUtils.closeQuietly(catalog, "Paimon catalog");
+        localTableQuery = null;
+        ioManager = null;
+        catalog = null;
+        fileStoreTable = null;
+        partitionKeyExtractor = null;
+        initializedBuckets.clear();
+        ensureInitialized();
+        initializeFilesIfNeeded(partition, bucketId);
+    }
+
+    private static void addFilesByName(
+            LinkedHashMap<String, DataFileMeta> filesByName, List<DataFileMeta> files) {
+        for (DataFileMeta file : files) {
+            filesByName.put(file.fileName(), file);
+        }
+    }
+
+    private byte[] encodeValue(
+            org.apache.paimon.data.InternalRow paimonRow, short schemaId, RowType valueRowType) {
+        PaimonRowAsFlussRow flussRow = new PaimonRowAsFlussRow(paimonRow);
+        try {
+            ensureValueEncoder(schemaId, valueRowType);
+            CompactedRowEncoder rowEncoder =
+                    checkNotNull(cachedValueRowEncoder, "cachedValueRowEncoder must not be null.");
+            InternalRow.FieldGetter[] fieldGetters =
+                    checkNotNull(
+                            cachedValueFieldGetters, "cachedValueFieldGetters must not be null.");
+
+            rowEncoder.startNewRow();
+            for (int i = 0; i < fieldGetters.length; i++) {
+                rowEncoder.encodeField(i, fieldGetters[i].getFieldOrNull(flussRow));
+            }
+            BinaryRow row = rowEncoder.finishRow();
+            return ValueEncoder.encodeValue(schemaId, row);
+        } catch (Exception e) {
+            throw new RuntimeException("Failed to encode Paimon lookup row as Fluss value.", e);
+        }
+    }
+
+    private void ensureValueEncoder(short schemaId, RowType valueRowType) {
+        if (hasCachedValueEncoder && cachedValueSchemaId == schemaId) {
+            return;
+        }
+
+        IOUtils.closeQuietly(cachedValueRowEncoder, "Fluss value row encoder");
+        DataType[] fieldTypes = valueRowType.getChildren().toArray(new DataType[0]);
+        cachedValueRowEncoder = new CompactedRowEncoder(fieldTypes);

Review Comment:
   nit: The client decodes lookup values with the table's kv format, but here we always encode compacted. 
   What happens for a table with table.kv.format = indexed?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [lookup] Support historical partition lookup for lake tables [fluss]

Posted by "beryllw (via GitHub)" <gi...@apache.org>.
beryllw commented on PR #3630:
URL: https://github.com/apache/fluss/pull/3630#issuecomment-5018196140

   LGTM!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [lookup] Support historical partition lookup for lake tables [fluss]

Posted by "beryllw (via GitHub)" <gi...@apache.org>.
beryllw commented on code in PR #3630:
URL: https://github.com/apache/fluss/pull/3630#discussion_r3609672188


##########
fluss-lake/fluss-lake-paimon/src/test/java/org/apache/fluss/lake/paimon/lookup/HistoricalPartitionLookupITCase.java:
##########
@@ -0,0 +1,218 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.fluss.lake.paimon.lookup;
+
+import org.apache.fluss.client.Connection;
+import org.apache.fluss.client.ConnectionFactory;
+import org.apache.fluss.client.lookup.Lookuper;
+import org.apache.fluss.client.table.Table;
+import org.apache.fluss.config.AutoPartitionTimeUnit;
+import org.apache.fluss.config.ConfigOptions;
+import org.apache.fluss.lake.paimon.testutils.FlinkPaimonTieringTestBase;
+import org.apache.fluss.metadata.PartitionInfo;
+import org.apache.fluss.metadata.PartitionSpec;
+import org.apache.fluss.metadata.Schema;
+import org.apache.fluss.metadata.TableBucket;
+import org.apache.fluss.metadata.TableChange;
+import org.apache.fluss.metadata.TableDescriptor;
+import org.apache.fluss.metadata.TablePath;
+import org.apache.fluss.row.InternalRow;
+import org.apache.fluss.server.testutils.FlussClusterExtension;
+import org.apache.fluss.types.DataTypes;
+
+import org.apache.flink.core.execution.JobClient;
+import org.junit.jupiter.api.BeforeAll;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.extension.RegisterExtension;
+
+import java.time.Duration;
+import java.util.Collections;
+import java.util.List;
+
+import static org.apache.fluss.testutils.DataTestUtils.row;
+import static org.apache.fluss.testutils.InternalRowAssert.assertThatRow;
+import static org.apache.fluss.testutils.common.CommonTestUtils.retry;
+import static org.assertj.core.api.Assertions.assertThat;
+
+/** End-to-end IT case for looking up expired Fluss partitions from Paimon. */
+class HistoricalPartitionLookupITCase extends FlinkPaimonTieringTestBase {
+
+    private static final String EXPIRED_PARTITION_NAME = "20240101";
+    private static final int INITIAL_PARTITION_RETENTION = 100000;
+    private static final int EXPIRED_PARTITION_RETENTION = 1;
+
+    @RegisterExtension
+    public static final FlussClusterExtension FLUSS_CLUSTER_EXTENSION =
+            FlussClusterExtension.builder()
+                    .setClusterConf(initConfig())
+                    .setNumOfTabletServers(3)
+                    .build();
+
+    @BeforeAll
+    protected static void beforeAll() {
+        FlinkPaimonTieringTestBase.beforeAll(FLUSS_CLUSTER_EXTENSION.getClientConfig());
+    }
+
+    @Test
+    void testLookupExpiredPartitionFromPaimon() throws Exception {
+        TablePath tablePath = TablePath.of(DEFAULT_DB, "historical_lookup_pk");
+        Schema oldSchema = partitionedPkSchema();
+        long tableId = createTable(tablePath, partitionedPkDescriptor(oldSchema));
+
+        // Keep the initial retention wide enough so this old partition can be created and written
+        // through the normal Fluss path before it is treated as historical.
+        PartitionSpec expiredPartitionSpec = partitionSpec(EXPIRED_PARTITION_NAME);
+        admin.createPartition(tablePath, expiredPartitionSpec, false).get();
+        long partitionId = getPartitionId(tablePath, EXPIRED_PARTITION_NAME);
+        FLUSS_CLUSTER_EXTENSION.waitUntilTablePartitionReady(tableId, partitionId);
+
+        InternalRow expectedOldRow = row(1, EXPIRED_PARTITION_NAME, "Alice");
+        writeRows(tablePath, Collections.singletonList(expectedOldRow), false);
+
+        TableBucket tableBucket = new TableBucket(tableId, partitionId, 0);
+        FLUSS_CLUSTER_EXTENSION.triggerAndWaitSnapshots(Collections.singleton(tableBucket));
+
+        JobClient jobClient = buildTieringJob(execEnv);
+        try {
+            assertReplicaStatus(tableBucket, 1);
+        } finally {
+            jobClient.cancel().get();
+        }
+
+        admin.alterTable(
+                        tablePath,
+                        Collections.singletonList(
+                                TableChange.addColumn(
+                                        "extra",
+                                        DataTypes.STRING(),
+                                        "extra column",
+                                        TableChange.ColumnPosition.last())),
+                        false)
+                .get();
+        Schema evolvedSchema = evolvedPartitionedPkSchema();
+
+        InternalRow expectedNewRow = row(2, EXPIRED_PARTITION_NAME, "Bob", "new-value");
+        writeRows(tablePath, Collections.singletonList(expectedNewRow), false);
+        FLUSS_CLUSTER_EXTENSION.triggerAndWaitSnapshots(Collections.singleton(tableBucket));
+
+        jobClient = buildTieringJob(execEnv);
+        try {
+            assertReplicaStatus(tableBucket, 2);
+        } finally {
+            jobClient.cancel().get();
+        }
+
+        // Create the lookuper with the initial retention, cache the still-existing normal
+        // partition route, then keep using it after the retention changes and the partition is
+        // deleted. It should use the latest retention to reroute to historical lookup.
+        try (Connection lookupConn = ConnectionFactory.createConnection(clientConf);
+                Table table = lookupConn.getTable(tablePath)) {
+            Lookuper lookuper = table.newLookup().createLookuper();
+            InternalRow lookupRow =
+                    lookuper.lookup(row(2, EXPIRED_PARTITION_NAME)).get().getSingletonRow();
+            assertThatRow(lookupRow)
+                    .withSchema(evolvedSchema.getRowType())
+                    .isEqualTo(expectedNewRow);
+
+            admin.alterTable(
+                            tablePath,
+                            Collections.singletonList(
+                                    TableChange.set(
+                                            ConfigOptions.TABLE_AUTO_PARTITION_NUM_RETENTION.key(),
+                                            String.valueOf(EXPIRED_PARTITION_RETENTION))),
+                            false)
+                    .get();
+
+            admin.dropPartition(tablePath, expiredPartitionSpec, true).get();
+            waitUntilPartitionDropped(tablePath, EXPIRED_PARTITION_NAME);
+
+            lookupRow = lookuper.lookup(row(1, EXPIRED_PARTITION_NAME)).get().getSingletonRow();
+            assertThatRow(lookupRow)
+                    .withSchema(evolvedSchema.getRowType())
+                    .isEqualTo(row(1, EXPIRED_PARTITION_NAME, "Alice", null));
+
+            lookupRow = lookuper.lookup(row(2, EXPIRED_PARTITION_NAME)).get().getSingletonRow();
+            assertThatRow(lookupRow)
+                    .withSchema(evolvedSchema.getRowType())
+                    .isEqualTo(expectedNewRow);
+        }
+    }
+
+    @Override
+    protected FlussClusterExtension getFlussClusterExtension() {
+        return FLUSS_CLUSTER_EXTENSION;
+    }
+
+    private static Schema partitionedPkSchema() {
+        return Schema.newBuilder()
+                .column("id", DataTypes.INT())
+                .column("dt", DataTypes.STRING())
+                .column("name", DataTypes.STRING())
+                .primaryKey("id", "dt")

Review Comment:
   Could we add a test with a non-default bucket key (bucket key != primary key)? The current ITCase uses a default bucket key, so it only covers the Paimon-encoder path — a non-default bucket key would exercise the CompactedKeyEncoder path, where the historical-lookup key encoding may not match what Paimon expects (this affects point lookups too, since the encoder is table-level).



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [lookup] Support historical partition lookup for lake tables [fluss]

Posted by "luoyuxia (via GitHub)" <gi...@apache.org>.
luoyuxia commented on code in PR #3630:
URL: https://github.com/apache/fluss/pull/3630#discussion_r3601994554


##########
fluss-server/src/main/java/org/apache/fluss/server/replica/ReplicaManager.java:
##########
@@ -774,6 +871,49 @@ public void lookups(
         lookups(false, null, null, entriesPerBucket, apiVersion, responseCallback);
     }
 
+    /** Lookup historical lake data for requests that carry original partition names. */
+    public void historicalLookups(

Review Comment:
   This PR only introduces the lake lookup path. The follow-up PR that adds write support will also implement the historical RocksDB lookup and wire it in here, so the final lookup order will be buffer → historical RocksDB → lake, as described in the FIP.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [lookup] Support historical partition lookup for lake tables [fluss]

Posted by "platinumhamburg (via GitHub)" <gi...@apache.org>.
platinumhamburg commented on code in PR #3630:
URL: https://github.com/apache/fluss/pull/3630#discussion_r3600514655


##########
fluss-client/src/main/java/org/apache/fluss/client/lookup/PrimaryKeyLookuper.java:
##########
@@ -115,19 +125,89 @@ public CompletableFuture<LookupResult> lookup(InternalRow lookupKey) {
                                 tableInfo.getTablePath(),
                                 metadataUpdater);
             } catch (PartitionNotExistException e) {
-                return CompletableFuture.completedFuture(new LookupResult(Collections.emptyList()));
+                return mayFallbackToHistoricalLookup(
+                        bucketingFunction.bucketing(bkBytes, numBuckets), pkBytes, lookupKey);
             }
         }
 
         int bucketId = bucketingFunction.bucketing(bkBytes, numBuckets);
         TableBucket tableBucket = new TableBucket(tableInfo.getTableId(), partitionId, bucketId);
+        return lookupBucket(tableBucket, pkBytes, insertIfNotExists, null, lookupKey);
+    }
+
+    /** Falls back to historical lookup if the lookup key belongs to a historical partition. */
+    private CompletableFuture<LookupResult> mayFallbackToHistoricalLookup(
+            int bucketId, byte[] keyBytes, InternalRow lookupKey) {
+        String originalPartitionName = partitionGetter.getPartition(lookupKey);
+        if (!isHistoricalLookupCandidatePartition(
+                tableInfo, originalPartitionName, Instant.now())) {

Review Comment:
   The Coordinator removes expired partitions using the CoordinatorServer clock, but this code calculates the retention boundary again using the client clock.
   
   For an hourly partitioned table with retention set to 1, the Coordinator may be just past 10:00 and remove partition `2024010108`, while a client is still just before 10:00. The client refreshes metadata and sees that the partition is gone, but this check still treats it as retained and returns an empty result. A few seconds later, the same lookup starts using the historical path without any data or metadata change.
   
   Can this decision use the Coordinator's time instead of `Instant.now()` on each client? A test with the Coordinator and client on opposite sides of an hour boundary would cover this case.



##########
fluss-client/src/main/java/org/apache/fluss/client/lookup/PrimaryKeyLookuper.java:
##########
@@ -115,19 +125,89 @@ public CompletableFuture<LookupResult> lookup(InternalRow lookupKey) {
                                 tableInfo.getTablePath(),
                                 metadataUpdater);
             } catch (PartitionNotExistException e) {
-                return CompletableFuture.completedFuture(new LookupResult(Collections.emptyList()));
+                return mayFallbackToHistoricalLookup(
+                        bucketingFunction.bucketing(bkBytes, numBuckets), pkBytes, lookupKey);
             }
         }
 
         int bucketId = bucketingFunction.bucketing(bkBytes, numBuckets);
         TableBucket tableBucket = new TableBucket(tableInfo.getTableId(), partitionId, bucketId);
+        return lookupBucket(tableBucket, pkBytes, insertIfNotExists, null, lookupKey);
+    }
+
+    /** Falls back to historical lookup if the lookup key belongs to a historical partition. */
+    private CompletableFuture<LookupResult> mayFallbackToHistoricalLookup(
+            int bucketId, byte[] keyBytes, InternalRow lookupKey) {
+        String originalPartitionName = partitionGetter.getPartition(lookupKey);
+        if (!isHistoricalLookupCandidatePartition(

Review Comment:
   `tableInfo` is fixed when the `Lookuper` is created, but `table.auto-partition.num-retention` can be changed while the lookuper is running.
   
   Suppose a client creates a lookuper with retention set to 7. On January 10, retention is changed to 1, so the Coordinator removes partition `20240105`. The existing lookuper still checks `20240105` with retention 7 and returns an empty result, while a client using the updated table info sends the same lookup through the historical path.
   
   Can we use the current table configuration here, or let the Coordinator make this decision? The current IT changes retention before creating the lookuper, so it does not cover this case.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [lookup] Support historical partition lookup for lake tables [fluss]

Posted by "luoyuxia (via GitHub)" <gi...@apache.org>.
luoyuxia commented on PR #3630:
URL: https://github.com/apache/fluss/pull/3630#issuecomment-5140114897

   @jark
   Thanks for reviewing again. Comments has been addressed.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [lookup] Support historical partition lookup for lake tables [fluss]

Posted by "luoyuxia (via GitHub)" <gi...@apache.org>.
luoyuxia commented on PR #3630:
URL: https://github.com/apache/fluss/pull/3630#issuecomment-5099010029

   @wuchong Comments has been addressed. PTAL


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [lookup] Support historical partition lookup for lake tables [fluss]

Posted by "luoyuxia (via GitHub)" <gi...@apache.org>.
luoyuxia commented on PR #3630:
URL: https://github.com/apache/fluss/pull/3630#issuecomment-5003157002

   @fresh-borzoni @platinumhamburg Thanks for your review. I already address your comments


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [lookup] Support historical partition lookup for lake tables [fluss]

Posted by "beryllw (via GitHub)" <gi...@apache.org>.
beryllw commented on code in PR #3630:
URL: https://github.com/apache/fluss/pull/3630#discussion_r3609672188


##########
fluss-lake/fluss-lake-paimon/src/test/java/org/apache/fluss/lake/paimon/lookup/HistoricalPartitionLookupITCase.java:
##########
@@ -0,0 +1,218 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.fluss.lake.paimon.lookup;
+
+import org.apache.fluss.client.Connection;
+import org.apache.fluss.client.ConnectionFactory;
+import org.apache.fluss.client.lookup.Lookuper;
+import org.apache.fluss.client.table.Table;
+import org.apache.fluss.config.AutoPartitionTimeUnit;
+import org.apache.fluss.config.ConfigOptions;
+import org.apache.fluss.lake.paimon.testutils.FlinkPaimonTieringTestBase;
+import org.apache.fluss.metadata.PartitionInfo;
+import org.apache.fluss.metadata.PartitionSpec;
+import org.apache.fluss.metadata.Schema;
+import org.apache.fluss.metadata.TableBucket;
+import org.apache.fluss.metadata.TableChange;
+import org.apache.fluss.metadata.TableDescriptor;
+import org.apache.fluss.metadata.TablePath;
+import org.apache.fluss.row.InternalRow;
+import org.apache.fluss.server.testutils.FlussClusterExtension;
+import org.apache.fluss.types.DataTypes;
+
+import org.apache.flink.core.execution.JobClient;
+import org.junit.jupiter.api.BeforeAll;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.extension.RegisterExtension;
+
+import java.time.Duration;
+import java.util.Collections;
+import java.util.List;
+
+import static org.apache.fluss.testutils.DataTestUtils.row;
+import static org.apache.fluss.testutils.InternalRowAssert.assertThatRow;
+import static org.apache.fluss.testutils.common.CommonTestUtils.retry;
+import static org.assertj.core.api.Assertions.assertThat;
+
+/** End-to-end IT case for looking up expired Fluss partitions from Paimon. */
+class HistoricalPartitionLookupITCase extends FlinkPaimonTieringTestBase {
+
+    private static final String EXPIRED_PARTITION_NAME = "20240101";
+    private static final int INITIAL_PARTITION_RETENTION = 100000;
+    private static final int EXPIRED_PARTITION_RETENTION = 1;
+
+    @RegisterExtension
+    public static final FlussClusterExtension FLUSS_CLUSTER_EXTENSION =
+            FlussClusterExtension.builder()
+                    .setClusterConf(initConfig())
+                    .setNumOfTabletServers(3)
+                    .build();
+
+    @BeforeAll
+    protected static void beforeAll() {
+        FlinkPaimonTieringTestBase.beforeAll(FLUSS_CLUSTER_EXTENSION.getClientConfig());
+    }
+
+    @Test
+    void testLookupExpiredPartitionFromPaimon() throws Exception {
+        TablePath tablePath = TablePath.of(DEFAULT_DB, "historical_lookup_pk");
+        Schema oldSchema = partitionedPkSchema();
+        long tableId = createTable(tablePath, partitionedPkDescriptor(oldSchema));
+
+        // Keep the initial retention wide enough so this old partition can be created and written
+        // through the normal Fluss path before it is treated as historical.
+        PartitionSpec expiredPartitionSpec = partitionSpec(EXPIRED_PARTITION_NAME);
+        admin.createPartition(tablePath, expiredPartitionSpec, false).get();
+        long partitionId = getPartitionId(tablePath, EXPIRED_PARTITION_NAME);
+        FLUSS_CLUSTER_EXTENSION.waitUntilTablePartitionReady(tableId, partitionId);
+
+        InternalRow expectedOldRow = row(1, EXPIRED_PARTITION_NAME, "Alice");
+        writeRows(tablePath, Collections.singletonList(expectedOldRow), false);
+
+        TableBucket tableBucket = new TableBucket(tableId, partitionId, 0);
+        FLUSS_CLUSTER_EXTENSION.triggerAndWaitSnapshots(Collections.singleton(tableBucket));
+
+        JobClient jobClient = buildTieringJob(execEnv);
+        try {
+            assertReplicaStatus(tableBucket, 1);
+        } finally {
+            jobClient.cancel().get();
+        }
+
+        admin.alterTable(
+                        tablePath,
+                        Collections.singletonList(
+                                TableChange.addColumn(
+                                        "extra",
+                                        DataTypes.STRING(),
+                                        "extra column",
+                                        TableChange.ColumnPosition.last())),
+                        false)
+                .get();
+        Schema evolvedSchema = evolvedPartitionedPkSchema();
+
+        InternalRow expectedNewRow = row(2, EXPIRED_PARTITION_NAME, "Bob", "new-value");
+        writeRows(tablePath, Collections.singletonList(expectedNewRow), false);
+        FLUSS_CLUSTER_EXTENSION.triggerAndWaitSnapshots(Collections.singleton(tableBucket));
+
+        jobClient = buildTieringJob(execEnv);
+        try {
+            assertReplicaStatus(tableBucket, 2);
+        } finally {
+            jobClient.cancel().get();
+        }
+
+        // Create the lookuper with the initial retention, cache the still-existing normal
+        // partition route, then keep using it after the retention changes and the partition is
+        // deleted. It should use the latest retention to reroute to historical lookup.
+        try (Connection lookupConn = ConnectionFactory.createConnection(clientConf);
+                Table table = lookupConn.getTable(tablePath)) {
+            Lookuper lookuper = table.newLookup().createLookuper();
+            InternalRow lookupRow =
+                    lookuper.lookup(row(2, EXPIRED_PARTITION_NAME)).get().getSingletonRow();
+            assertThatRow(lookupRow)
+                    .withSchema(evolvedSchema.getRowType())
+                    .isEqualTo(expectedNewRow);
+
+            admin.alterTable(
+                            tablePath,
+                            Collections.singletonList(
+                                    TableChange.set(
+                                            ConfigOptions.TABLE_AUTO_PARTITION_NUM_RETENTION.key(),
+                                            String.valueOf(EXPIRED_PARTITION_RETENTION))),
+                            false)
+                    .get();
+
+            admin.dropPartition(tablePath, expiredPartitionSpec, true).get();
+            waitUntilPartitionDropped(tablePath, EXPIRED_PARTITION_NAME);
+
+            lookupRow = lookuper.lookup(row(1, EXPIRED_PARTITION_NAME)).get().getSingletonRow();
+            assertThatRow(lookupRow)
+                    .withSchema(evolvedSchema.getRowType())
+                    .isEqualTo(row(1, EXPIRED_PARTITION_NAME, "Alice", null));
+
+            lookupRow = lookuper.lookup(row(2, EXPIRED_PARTITION_NAME)).get().getSingletonRow();
+            assertThatRow(lookupRow)
+                    .withSchema(evolvedSchema.getRowType())
+                    .isEqualTo(expectedNewRow);
+        }
+    }
+
+    @Override
+    protected FlussClusterExtension getFlussClusterExtension() {
+        return FLUSS_CLUSTER_EXTENSION;
+    }
+
+    private static Schema partitionedPkSchema() {
+        return Schema.newBuilder()
+                .column("id", DataTypes.INT())
+                .column("dt", DataTypes.STRING())
+                .column("name", DataTypes.STRING())
+                .primaryKey("id", "dt")

Review Comment:
   It might be worth adding a test case with a non-default bucket key (e.g. bucket key ⊊ primary key, which also enables prefix lookup) — the current ITCase uses a default bucket key, so it doesn't exercise the CompactedKeyEncoder path where the encodings could diverge.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [lookup] Support historical partition lookup for lake tables [fluss]

Posted by "luoyuxia (via GitHub)" <gi...@apache.org>.
luoyuxia commented on code in PR #3630:
URL: https://github.com/apache/fluss/pull/3630#discussion_r3602145810


##########
fluss-client/src/main/java/org/apache/fluss/client/lookup/HistoricalPartitionResolver.java:
##########
@@ -0,0 +1,186 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.fluss.client.lookup;
+
+import org.apache.fluss.annotation.Internal;
+import org.apache.fluss.client.admin.Admin;
+import org.apache.fluss.client.metadata.MetadataUpdater;
+import org.apache.fluss.exception.PartitionNotExistException;
+import org.apache.fluss.metadata.PhysicalTablePath;
+import org.apache.fluss.metadata.ResolvedPartitionSpec;
+import org.apache.fluss.metadata.TableInfo;
+import org.apache.fluss.metadata.TablePath;
+
+import javax.annotation.Nullable;
+
+import java.util.Objects;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+
+import static org.apache.fluss.utils.PartitionUtils.toHistoricalPartitionSpec;
+import static org.apache.fluss.utils.Preconditions.checkNotNull;
+
+/**
+ * Resolves original partitions to their historical system partition ids.
+ *
+ * <p>Historical lookup requests still carry the original partition name, but the lookup RPC must be
+ * sent to the generated historical system partition. This resolver bridges that gap on the client
+ * side.
+ */
+@Internal
+class HistoricalPartitionResolver {
+
+    private final MetadataUpdater metadataUpdater;
+    private final Admin admin;
+    private final ConcurrentHashMap<HistoricalPartitionKey, CompletableFuture<Long>>
+            inflightResolves;
+
+    HistoricalPartitionResolver(MetadataUpdater metadataUpdater, Admin admin) {
+        this.metadataUpdater = checkNotNull(metadataUpdater, "metadataUpdater must not be null.");
+        this.admin = checkNotNull(admin, "admin must not be null.");
+        this.inflightResolves = new ConcurrentHashMap<>();
+    }
+
+    CompletableFuture<Long> resolveHistoricalPartitionId(
+            TableInfo tableInfo, String originalPartitionName) {
+        HistoricalPartitionKey key =
+                new HistoricalPartitionKey(
+                        tableInfo.getTableId(), tableInfo.getTablePath(), originalPartitionName);
+        // Multiple lookups for the same original partition can be issued concurrently. Coalesce
+        // them so only one metadata refresh/create path runs for a given partition.
+        CompletableFuture<Long> result = new CompletableFuture<>();
+        CompletableFuture<Long> previous = inflightResolves.putIfAbsent(key, result);
+        if (previous != null) {
+            return previous;
+        }
+
+        resolveHistoricalPartitionIdInternal(tableInfo, originalPartitionName)
+                .whenComplete(
+                        (partitionId, error) -> {
+                            if (error != null) {
+                                result.completeExceptionally(error);
+                            } else {
+                                result.complete(partitionId);
+                            }
+                            inflightResolves.remove(key, result);
+                        });
+        return result;
+    }
+
+    private CompletableFuture<Long> resolveHistoricalPartitionIdInternal(
+            TableInfo tableInfo, String originalPartitionName) {
+        CompletableFuture<Long> result = new CompletableFuture<>();
+        ResolvedPartitionSpec historicalPartitionSpec;
+        PhysicalTablePath historicalPartitionPath;
+        try {
+            // The server-side historical lookup path is keyed by the historical system partition,
+            // not by the original partition that existed before retention cleanup.
+            historicalPartitionSpec = toHistoricalPartitionSpec(tableInfo, originalPartitionName);
+            historicalPartitionPath =
+                    PhysicalTablePath.of(
+                            tableInfo.getTablePath(), historicalPartitionSpec.getPartitionName());
+
+            // Prefer cached metadata, then refresh once before creating the system partition.
+            Long partitionId = getCachedPartitionId(historicalPartitionPath);
+            if (partitionId != null) {
+                result.complete(partitionId);
+                return result;
+            }
+
+            tryRefreshHistoricalPartition(historicalPartitionPath);
+            partitionId = getCachedPartitionId(historicalPartitionPath);
+            if (partitionId != null) {
+                result.complete(partitionId);
+                return result;
+            }
+        } catch (Throwable t) {
+            result.completeExceptionally(t);
+            return result;
+        }
+
+        // If the historical system partition still does not exist, create it idempotently and
+        // refresh metadata again so the caller can route the lookup request by partition id.
+        admin.createPartition(

Review Comment:
   I agree that creating real replicas(no rocksdb for read now) from a read path is not ideal. However, the historical system partition is currently also the routing and ownership anchor: the client needs its partition ID to resolve a bucket leader, and the server uses the corresponding replica to validate the request and obtain table/schema metadata.
   
   Routing historical lookups without creating the partition would require a separate virtual-routing path: selecting a TabletServer without partition leadership, resolving table and schema metadata without a replica, handling retries, and gating rolling upgrades. That is broader than removing the createPartition call.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [lookup] Support historical partition lookup for lake tables [fluss]

Posted by "luoyuxia (via GitHub)" <gi...@apache.org>.
luoyuxia commented on code in PR #3630:
URL: https://github.com/apache/fluss/pull/3630#discussion_r3601898629


##########
fluss-rpc/src/main/proto/FlussApi.proto:
##########
@@ -946,6 +946,8 @@ message PbLookupReqForBucket {
   optional int64 partition_id = 1;
   required int32 bucket_id = 2;
   repeated bytes keys = 3;
+  // The original partition name for historical lookup. It is unset for normal lookup.
+  optional string original_partition_name = 4;

Review Comment:
   I agree that this can still return a silent miss while the cluster is running mixed TabletServer versions. However, before this change, lookup against an expired or otherwise non-existent partition already returned null. 
   Before this pr, it will check partition exists or not, if not, it'll return null then



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [lookup] Support historical partition lookup for lake tables [fluss]

Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #3630:
URL: https://github.com/apache/fluss/pull/3630#discussion_r3556242744


##########
fluss-server/src/main/java/org/apache/fluss/server/coordinator/CoordinatorService.java:
##########
@@ -723,16 +725,34 @@ public CompletableFuture<CreatePartitionResponse> createPartition(
 
         // first, validate the partition spec, and get resolved partition spec.
         PartitionSpec partitionSpec = getPartitionSpec(request.getPartitionSpec());
-        validatePartitionSpec(tablePath, table.partitionKeys, partitionSpec, true);
+        ResolvedPartitionSpec partitionToCreate;
+        if (isHistoricalPartitionCreate(table, partitionSpec)) {
+            // Historical system partitions are lookup metadata, so creating one requires the same
+            // permission as reading the table instead of writing table data.
+            authorizeTable(OperationType.READ, tablePath);
+            if (!request.isIgnoreIfNotExists()) {
+                throw new InvalidPartitionException(
+                        "Creating historical system partition requires ignoreIfExists=true.");
+            }

Review Comment:
   Error message references `ignoreIfExists=true`, but the request flag checked here is `ignoreIfNotExists`. This can mislead users when they hit this validation path for historical system partitions.



##########
fluss-client/src/main/java/org/apache/fluss/client/lookup/PrimaryKeyLookuper.java:
##########
@@ -115,15 +130,57 @@ public CompletableFuture<LookupResult> lookup(InternalRow lookupKey) {
                                 tableInfo.getTablePath(),
                                 metadataUpdater);
             } catch (PartitionNotExistException e) {
+                if (isHistoricalLookupCandidatePartition(
+                        tableInfo, originalPartitionName, Instant.now())) {
+                    if (insertIfNotExists) {
+                        return completedExceptionally(
+                                new UnsupportedOperationException(
+                                        "Lookup with insertIfNotExists is not supported for historical partition lookup."));
+                    }

Review Comment:
   This branch returns a failed future with `UnsupportedOperationException`, which is not a Fluss `ApiException` and can lead to inconsistent error handling for callers (compared to the server-side `InvalidTableException` used for the same constraint). Prefer returning an `InvalidTableException` so the error is classified as a user-facing, non-retriable request validation failure.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org