You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@fluss.apache.org by "zuston (via GitHub)" <gi...@apache.org> on 2026/08/26 09:03:53 UTC
[PR] [server] Add support of sharable file cache capacity for historical partition lookup [fluss]
zuston opened a new pull request, #4113:
URL: https://github.com/apache/fluss/pull/4113
### Purpose
Linked issue: close #3955
This PR depends on the https://github.com/apache/fluss/pull/4050 .
### Brief change log
As discussed in [#3955](https://github.com/apache/fluss/issues/3955), sharing the file cache across multiple table lookups requires maintaining a Fluss-side fork of Paimon’s LocalTableQuery, adapted to use the shared cache.
### Tests
<!-- List UT and IT cases to verify this change -->
### API and Format
<!-- Does this change affect API or storage format -->
### Documentation
<!-- Does this change introduce a new feature -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [server] Add support of sharable file cache capacity for historical partition lookup [fluss]
Posted by "fresh-borzoni (via GitHub)" <gi...@apache.org>.
fresh-borzoni commented on code in PR #4113:
URL: https://github.com/apache/fluss/pull/4113#discussion_r3894498500
##########
fluss-server/src/main/java/org/apache/fluss/server/replica/historical/HistoricalLakeLookupManager.java:
##########
@@ -180,6 +185,8 @@ class HistoricalLakeLookupManager implements AutoCloseable {
.executor(Runnable::run)
.removalListener(this::onLookuperRemoved)
.build();
+ this.lookuperIdSequence = new AtomicLong();
+ this.lookupRuntime = createLookupRuntime(conf);
Review Comment:
This builds the IOManager against the cache root, then startup() deletes and recreates that
directory. Only works because Paimon creates the spill dir lazily on first createChannel.
Mb move it into startup()?
##########
fluss-lake/fluss-lake-paimon/src/test/java/org/apache/fluss/lake/paimon/lookup/SharedLookupFileCacheTest.java:
##########
@@ -0,0 +1,93 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.fluss.lake.paimon.lookup;
+
+import org.apache.paimon.lookup.LookupStoreReader;
+import org.apache.paimon.mergetree.LookupFile;
+import org.apache.paimon.options.MemorySize;
+import org.apache.paimon.shade.caffeine2.com.github.benmanes.caffeine.cache.Cache;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.io.TempDir;
+
+import java.io.File;
+import java.io.IOException;
+import java.io.RandomAccessFile;
+import java.time.Duration;
+import java.util.Arrays;
+
+import static org.assertj.core.api.Assertions.assertThat;
+
+/** Tests for {@link SharedLookupFileCache}. */
+class SharedLookupFileCacheTest {
+
+ @TempDir private File tempDir;
+
+ @Test
+ void testNamespaceIsolationAndGlobalLimit() throws Exception {
+ File firstFile = lookupFile("first.lookup");
+ File secondFile = lookupFile("second.lookup");
+ File thirdFile = lookupFile("third.lookup");
+
+ try (SharedLookupFileCache sharedCache =
+ new SharedLookupFileCache(Duration.ofHours(1), MemorySize.ofKibiBytes(2))) {
+ Cache<String, LookupFile> firstNamespace = sharedCache.namespaced("first");
+ Cache<String, LookupFile> secondNamespace = sharedCache.namespaced("second");
+ LookupFile firstLookupFile = lookupFile(firstFile);
+ LookupFile secondLookupFile = lookupFile(secondFile);
+
+ firstNamespace.put("same-file-name", firstLookupFile);
+ secondNamespace.put("same-file-name", secondLookupFile);
+ assertThat(firstNamespace.getIfPresent("same-file-name")).isSameAs(firstLookupFile);
+ assertThat(secondNamespace.getIfPresent("same-file-name")).isSameAs(secondLookupFile);
+
+ firstNamespace.invalidateAll();
+ assertThat(firstFile).doesNotExist();
+ assertThat(secondFile).exists();
+
+ sharedCache.updateMaxDiskSize(MemorySize.ofKibiBytes(1));
+ secondNamespace.put("third-file", lookupFile(thirdFile));
+ assertThat(Arrays.asList(secondFile, thirdFile).stream().filter(File::exists).count())
+ .isLessThanOrEqualTo(1L);
Review Comment:
The eviction here only ever happens inside secondNamespace, so it doesn't show one namespace
pushing another's files out which is the point of the PR. Worth adding, and asserting the count
rather than <= 1?
##########
fluss-lake/fluss-lake-paimon/src/main/java/org/apache/fluss/lake/paimon/PaimonLakeStorage.java:
##########
@@ -55,13 +65,50 @@ public LakeSource<PaimonSplit> createLakeSource(TablePath tablePath) {
}
@Override
- public LakeTableLookuper createLakeTableLookuper(TablePath tablePath, LookuperContext context) {
- return new PaimonLakeTableLookuper(
- paimonConfig,
- tablePath,
- context.ioTmpDir(),
- context.tableConfig(),
- context.lookupCacheMaxDiskBytes(),
- context.diskWriteGuard());
+ public LakeTableLookupRuntime createLakeTableLookupRuntime(
+ String ioTmpDir, long lookupCacheMaxDiskBytes) {
+ return new PaimonLakeTableLookupRuntime(ioTmpDir, lookupCacheMaxDiskBytes);
+ }
+
+ /** Paimon lookup runtime sharing one I/O manager across table lookupers. */
+ private static final class PaimonLakeTableLookupRuntime implements LakeTableLookupRuntime {
+ private final IOManager ioManager;
+ private final SharedLookupFileCache lookupFileCache;
+
+ private PaimonLakeTableLookupRuntime(String ioTmpDir, long lookupCacheMaxDiskBytes) {
+ checkArgument(
+ lookupCacheMaxDiskBytes > 0, "lookupCacheMaxDiskBytes must be greater than 0.");
+ this.ioManager = IOManager.create(checkNotNull(ioTmpDir, "ioTmpDir must not be null."));
+ // ponytail: one runtime-wide retention; add a server option if this needs tuning.
+ this.lookupFileCache =
+ new SharedLookupFileCache(
+ CoreOptions.LOOKUP_CACHE_FILE_RETENTION.defaultValue(),
Review Comment:
Before, LocalTableQuery read lookup.cache-file-retention off the table's own options. Now it's
always the 1h default, this is the only reader of it left in the PR. Is it intentional?
##########
fluss-lake/fluss-lake-paimon/src/main/java/org/apache/fluss/lake/paimon/lookup/SharedLookupFileCache.java:
##########
@@ -0,0 +1,112 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.fluss.lake.paimon.lookup;
+
+import org.apache.fluss.annotation.Internal;
+
+import org.apache.paimon.mergetree.LookupFile;
+import org.apache.paimon.options.MemorySize;
+import org.apache.paimon.shade.caffeine2.com.github.benmanes.caffeine.cache.Cache;
+import org.apache.paimon.shade.caffeine2.com.github.benmanes.caffeine.cache.Caffeine;
+import org.apache.paimon.shade.caffeine2.com.github.benmanes.caffeine.cache.RemovalCause;
+
+import javax.annotation.Nullable;
+
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.time.Duration;
+import java.util.Objects;
+
+import static org.apache.fluss.utils.Preconditions.checkNotNull;
+import static org.apache.paimon.mergetree.LookupUtils.fileKibiBytes;
+
+/** A weighted lookup-file cache shared by multiple Paimon table lookupers. */
+@Internal
+public final class SharedLookupFileCache implements AutoCloseable {
+
+ private final Cache<Key, LookupFile> cache;
+
+ /** Creates a shared lookup-file cache. */
+ public SharedLookupFileCache(Duration fileRetention, MemorySize maxDiskSize) {
+ checkNotNull(fileRetention, "fileRetention must not be null.");
+ checkNotNull(maxDiskSize, "maxDiskSize must not be null.");
+ this.cache =
+ Caffeine.newBuilder()
Review Comment:
#3955 asks for eviction metrics. capacityEvictions counts lookuper evictions, not file evictions, and
without recordStats() we can't count those. Shall we add it?
##########
fluss-lake/fluss-lake-paimon/src/main/java/org/apache/fluss/lake/paimon/lookup/NamespacedLookupFileCache.java:
##########
@@ -0,0 +1,205 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.fluss.lake.paimon.lookup;
+
+import org.apache.paimon.mergetree.LookupFile;
+import org.apache.paimon.shade.caffeine2.com.github.benmanes.caffeine.cache.Cache;
+import org.apache.paimon.shade.caffeine2.com.github.benmanes.caffeine.cache.Policy;
+import org.apache.paimon.shade.caffeine2.com.github.benmanes.caffeine.cache.stats.CacheStats;
+
+import javax.annotation.Nullable;
+
+import java.util.AbstractMap;
+import java.util.HashSet;
+import java.util.LinkedHashMap;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.ConcurrentMap;
+import java.util.function.Function;
+
+import static org.apache.fluss.utils.Preconditions.checkNotNull;
+
+/** Namespace-scoped cache view delegating all storage to a shared lookup-file cache. */
+final class NamespacedLookupFileCache implements Cache<String, LookupFile> {
+
+ private final Cache<SharedLookupFileCache.Key, LookupFile> sharedCache;
+ private final String namespace;
+
+ NamespacedLookupFileCache(
+ Cache<SharedLookupFileCache.Key, LookupFile> sharedCache, String namespace) {
+ this.sharedCache = checkNotNull(sharedCache, "sharedCache must not be null.");
+ this.namespace = checkNotNull(namespace, "namespace must not be null.");
+ }
+
+ @Override
+ public @Nullable LookupFile getIfPresent(Object fileName) {
+ return fileName instanceof String ? sharedCache.getIfPresent(key((String) fileName)) : null;
+ }
+
+ @Override
+ public LookupFile get(
+ String fileName, Function<? super String, ? extends LookupFile> mappingFunction) {
+ return sharedCache.get(key(fileName), ignored -> mappingFunction.apply(fileName));
+ }
+
+ @Override
+ public Map<String, LookupFile> getAllPresent(Iterable<?> fileNames) {
+ Map<String, LookupFile> result = new LinkedHashMap<>();
+ for (Object fileName : fileNames) {
+ LookupFile lookupFile = getIfPresent(fileName);
+ if (lookupFile != null) {
+ result.put((String) fileName, lookupFile);
+ }
+ }
+ return result;
+ }
+
+ @Override
+ public void put(String fileName, LookupFile lookupFile) {
+ sharedCache.put(key(fileName), lookupFile);
+ }
+
+ @Override
+ public void putAll(Map<? extends String, ? extends LookupFile> entries) {
+ entries.forEach(this::put);
+ }
+
+ @Override
+ public void invalidate(Object fileName) {
+ if (fileName instanceof String) {
+ sharedCache.invalidate(key((String) fileName));
+ }
+ }
+
+ @Override
+ public void invalidateAll(Iterable<?> fileNames) {
+ for (Object fileName : fileNames) {
+ invalidate(fileName);
+ }
+ }
+
+ @Override
+ public void invalidateAll() {
+ // ponytail: O(n) namespace scan; add a namespace index if cache cardinality makes close
Review Comment:
what is this ponytail, is it some kind of TODO?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [server] Add support of sharable file cache capacity for historical partition lookup [fluss]
Posted by "luoyuxia (via GitHub)" <gi...@apache.org>.
luoyuxia commented on code in PR #4113:
URL: https://github.com/apache/fluss/pull/4113#discussion_r4102328942
##########
fluss-common/src/main/java/org/apache/fluss/lake/lakestorage/LakeTableLookupRuntime.java:
##########
@@ -0,0 +1,90 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.fluss.lake.lakestorage;
+
+import org.apache.fluss.annotation.PublicEvolving;
+import org.apache.fluss.config.Configuration;
+import org.apache.fluss.config.TableConfig;
+import org.apache.fluss.metadata.TablePath;
+
+import static org.apache.fluss.utils.Preconditions.checkNotNull;
+
+/** TabletServer-scoped runtime for creating lake table lookupers. */
+@PublicEvolving
+public interface LakeTableLookupRuntime extends AutoCloseable {
Review Comment:
How about renaming this to `LakeLookuper`? It manages shared lookup resources across tables within a TabletServer and creates `LakeTableLookuper` instances. The two names would make the scopes clearer: `LakeLookuper` for the lake-storage-level entry point, and `LakeTableLookuper` for lookups on an individual table. The factory method and implementation could then be named `LakeStorage.createLakeLookuper()` and `PaimonLakeLookuper`.
Also, since this is a new `@PublicEvolving` API, could we add `@since 1.0` to its class Javadoc?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [server] Add support of sharable file cache capacity for historical partition lookup [fluss]
Posted by "luoyuxia (via GitHub)" <gi...@apache.org>.
luoyuxia commented on code in PR #4113:
URL: https://github.com/apache/fluss/pull/4113#discussion_r4102328942
##########
fluss-common/src/main/java/org/apache/fluss/lake/lakestorage/LakeTableLookupRuntime.java:
##########
@@ -0,0 +1,90 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.fluss.lake.lakestorage;
+
+import org.apache.fluss.annotation.PublicEvolving;
+import org.apache.fluss.config.Configuration;
+import org.apache.fluss.config.TableConfig;
+import org.apache.fluss.metadata.TablePath;
+
+import static org.apache.fluss.utils.Preconditions.checkNotNull;
+
+/** TabletServer-scoped runtime for creating lake table lookupers. */
+@PublicEvolving
+public interface LakeTableLookupRuntime extends AutoCloseable {
Review Comment:
How about renaming this to `LakeLookuper`? It manages shared lookup resources across tables within a TabletServer and creates `LakeTableLookuper` instances. The two names would make the scopes clearer: `LakeLookuper` for the lake-storage-level entry point, and `LakeTableLookuper` for lookups on an individual table. The factory method and implementation could then be named `LakeStorage.createLakeLookuper()` and `PaimonLakeLookuper`.
Also, since this is a new `@PublicEvolving` API, could we add `@since 1.0` to its class Javadoc?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [server] Add support of sharable file cache capacity for historical partition lookup [fluss]
Posted by "luoyuxia (via GitHub)" <gi...@apache.org>.
luoyuxia commented on code in PR #4113:
URL: https://github.com/apache/fluss/pull/4113#discussion_r4103737947
##########
fluss-lake/fluss-lake-paimon/src/main/java/org/apache/fluss/lake/paimon/lookup/PaimonLocalTableQuery.java:
##########
@@ -0,0 +1,316 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.fluss.lake.paimon.lookup;
+
+import org.apache.paimon.CoreOptions;
+import org.apache.paimon.FileStore;
+import org.apache.paimon.KeyValue;
+import org.apache.paimon.KeyValueFileStore;
+import org.apache.paimon.data.BinaryRow;
+import org.apache.paimon.data.InternalRow;
+import org.apache.paimon.data.serializer.InternalRowSerializer;
+import org.apache.paimon.data.serializer.InternalSerializers;
+import org.apache.paimon.data.serializer.RowCompactedSerializer;
+import org.apache.paimon.deletionvectors.DeletionVector;
+import org.apache.paimon.disk.IOManager;
+import org.apache.paimon.fs.FileIO;
+import org.apache.paimon.io.DataFileMeta;
+import org.apache.paimon.io.KeyValueFileReaderFactory;
+import org.apache.paimon.io.cache.CacheManager;
+import org.apache.paimon.lookup.LookupStoreFactory;
+import org.apache.paimon.mergetree.Levels;
+import org.apache.paimon.mergetree.LookupFile;
+import org.apache.paimon.mergetree.LookupLevels;
+import org.apache.paimon.mergetree.lookup.LookupSerializerFactory;
+import org.apache.paimon.mergetree.lookup.PersistValueProcessor;
+import org.apache.paimon.mergetree.lookup.RemoteLookupFileManager;
+import org.apache.paimon.operation.metrics.PartialLookupMetrics;
+import org.apache.paimon.options.Options;
+import org.apache.paimon.reader.RecordReader;
+import org.apache.paimon.shade.caffeine2.com.github.benmanes.caffeine.cache.Cache;
+import org.apache.paimon.table.FileStoreTable;
+import org.apache.paimon.table.query.TableQuery;
+import org.apache.paimon.types.RowType;
+import org.apache.paimon.utils.Filter;
+import org.apache.paimon.utils.KeyComparatorSupplier;
+import org.apache.paimon.utils.Preconditions;
+
+import javax.annotation.Nullable;
+
+import java.io.IOException;
+import java.util.Comparator;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.locks.ReadWriteLock;
+import java.util.concurrent.locks.ReentrantReadWriteLock;
+import java.util.function.Supplier;
+
+import static org.apache.paimon.lookup.LookupStoreFactory.bloomFilterBuilderFactory;
+import static org.apache.paimon.mergetree.LookupFile.localFilePrefix;
+
+/**
+ * Fluss-owned fork of Paimon 2.0.0's {@code LocalTableQuery}, using a shared namespaced lookup
+ * cache.
+ */
+final class PaimonLocalTableQuery implements TableQuery {
+
+ private final Map<BinaryRow, Map<Integer, BucketLookupState>> tableView;
+
+ private final CoreOptions options;
+
+ private final Supplier<Comparator<InternalRow>> keyComparatorSupplier;
+
+ private final KeyValueFileReaderFactory.Builder readerFactoryBuilder;
+
+ private final LookupStoreFactory lookupStoreFactory;
+
+ private final int startLevel;
+
+ private IOManager ioManager;
+
+ private final Cache<String, LookupFile> lookupFileCache;
+
+ private final RowType rowType;
+ private final RowType partitionType;
+ private final FileIO fileIO;
+
+ @Nullable private Filter<InternalRow> cacheRowFilter;
+
+ @Nullable private PartialLookupMetrics partialLookupMetrics;
+
+ PaimonLocalTableQuery(
+ FileStoreTable table,
+ SharedLookupFileCache sharedLookupFileCache,
+ String cacheNamespace) {
+ this.options = table.coreOptions();
+ this.lookupFileCache = sharedLookupFileCache.namespaced(cacheNamespace);
+ this.tableView = new ConcurrentHashMap<>();
+ FileStore<?> tableStore = table.store();
+ if (!(tableStore instanceof KeyValueFileStore)) {
+ throw new UnsupportedOperationException(
+ "Table Query only supports table with primary key.");
+ }
+ KeyValueFileStore store = (KeyValueFileStore) tableStore;
+
+ this.readerFactoryBuilder = store.newReaderFactoryBuilder();
+ this.rowType = table.schema().logicalRowType();
+ this.partitionType = table.schema().logicalPartitionType();
+ this.fileIO = table.fileIO();
+ RowType keyType = readerFactoryBuilder.keyType();
+ this.keyComparatorSupplier = new KeyComparatorSupplier(readerFactoryBuilder.keyType());
+ this.lookupStoreFactory =
+ LookupStoreFactory.create(
+ options,
+ new CacheManager(
+ options.lookupCacheMaxMemory(),
+ options.lookupCacheHighPrioPoolRatio()),
+ new RowCompactedSerializer(keyType).createSliceComparator());
+ startLevel = options.needLookup() ? 1 : 0;
+ }
+
+ public void refreshFiles(
+ BinaryRow partition,
+ int bucket,
+ List<DataFileMeta> beforeFiles,
+ List<DataFileMeta> dataFiles) {
+ // Both tableView and its nested bucket maps are ConcurrentHashMaps; this nested
+ // computeIfAbsent pattern relies on each map providing atomic insertion.
+ BucketLookupState state =
+ tableView
+ .computeIfAbsent(partition, k -> new ConcurrentHashMap<>())
+ .computeIfAbsent(bucket, k -> new BucketLookupState());
+ state.lock.writeLock().lock();
+ try {
+ if (state.lookupLevels == null) {
+ // Initial phase: ignore beforeFiles as they represent deletions from previous state
+ state.lookupLevels = createLookupLevels(partition, bucket, dataFiles);
+ } else {
+ state.lookupLevels.getLevels().update(beforeFiles, dataFiles);
+ }
+ } finally {
+ state.lock.writeLock().unlock();
+ }
+ }
+
+ private LookupLevels<KeyValue> createLookupLevels(
+ BinaryRow partition, int bucket, List<DataFileMeta> dataFiles) {
+ Levels levels = new Levels(keyComparatorSupplier.get(), dataFiles, options.numLevels());
+ // TODO pass DeletionVector factory
+ KeyValueFileReaderFactory factory =
+ readerFactoryBuilder.build(partition, bucket, DeletionVector.emptyFactory());
+ Options options = this.options.toConfiguration();
+
+ RowType readValueType = readerFactoryBuilder.readValueType();
+ LookupLevels<KeyValue> lookupLevels =
+ new LookupLevels<>(
+ schemaId -> readValueType,
+ 0L,
+ levels,
+ keyComparatorSupplier.get(),
+ readerFactoryBuilder.keyType(),
+ PersistValueProcessor.factory(readValueType),
+ LookupSerializerFactory.INSTANCE.get(),
+ file -> {
+ RecordReader<KeyValue> reader = factory.createRecordReader(file);
+ if (cacheRowFilter != null) {
+ reader =
+ reader.filter(
+ keyValue -> cacheRowFilter.test(keyValue.value()));
+ }
+ return reader;
+ },
+ file ->
+ Preconditions.checkNotNull(ioManager, "IOManager is required.")
+ .createChannel(
+ localFilePrefix(
+ partitionType, partition, bucket, file))
+ .getPathFile(),
+ lookupStoreFactory,
+ bloomFilterBuilderFactory(options),
+ lookupFileCache);
+
+ // Optimization - download lookup files if already persisted to object store
+ // We download these files if three conditions are met
+ // 1) lookup.remote-file.enabled is true - files are persisted in the first place
+ // 2) deletion-vectors.enabled is false - SSTables only contain row positions, not values,
+ // when DVs are enabled
+ // 3) The client is accessing the full data row, as opposed to a projection
+ // - The persisted remote SSTable files are created during compaction and hold the entire
+ // data row value
+ // - We could deserialize and project in memory, but we'll have to read much more data,
+ // not as clear of a win
+ boolean fullValueRead = readerFactoryBuilder.readValueType().equals(rowType);
+ if (this.options.lookupRemoteFileEnabled()
+ && !this.options.deletionVectorsEnabled()
+ && fullValueRead) {
+ // Calling the constructor tells `lookupLevels` to load remote files
+ new RemoteLookupFileManager<>(
+ fileIO,
+ factory.pathFactory(),
+ lookupLevels,
+ this.options.lookupRemoteLevelThreshold());
+ }
+
+ return lookupLevels;
+ }
+
+ @Nullable
+ @Override
+ public InternalRow lookup(BinaryRow partition, int bucket, InternalRow key) throws IOException {
+ PartialLookupMetrics currentMetrics = partialLookupMetrics;
+ LookupLevels.LookupContext context =
+ currentMetrics == null ? null : new LookupLevels.LookupContext();
+ try {
+ return lookup(partition, bucket, key, context);
+ } finally {
+ if (currentMetrics != null) {
+ currentMetrics.reportLookup(context != null && context.remoteAccessed());
+ }
+ }
+ }
+
+ @Nullable
+ private InternalRow lookup(
+ BinaryRow partition,
+ int bucket,
+ InternalRow key,
+ @Nullable LookupLevels.LookupContext context)
+ throws IOException {
+ Map<Integer, BucketLookupState> buckets = tableView.get(partition);
+ if (buckets == null || buckets.isEmpty()) {
+ return null;
+ }
+ BucketLookupState state = buckets.get(bucket);
+ if (state == null) {
+ return null;
+ }
+
+ state.lock.readLock().lock();
+ try {
+ LookupLevels<KeyValue> lookupLevels = state.lookupLevels;
+ if (lookupLevels == null) {
+ return null;
+ }
+
+ KeyValue kv = lookupLevels.lookup(key, startLevel, context);
+ if (kv == null || kv.valueKind().isRetract()) {
+ return null;
+ } else {
+ return kv.value();
+ }
+ } finally {
+ state.lock.readLock().unlock();
+ }
+ }
+
+ @Override
+ public PaimonLocalTableQuery withValueProjection(int[] projection) {
+ this.readerFactoryBuilder.withReadValueType(rowType.project(projection));
+ return this;
+ }
+
+ public PaimonLocalTableQuery withIOManager(IOManager ioManager) {
+ this.ioManager = ioManager;
+ return this;
+ }
+
+ public PaimonLocalTableQuery withCacheRowFilter(Filter<InternalRow> cacheRowFilter) {
+ this.cacheRowFilter = cacheRowFilter;
+ return this;
+ }
+
+ public PaimonLocalTableQuery withMetrics(@Nullable PartialLookupMetrics metrics) {
+ this.partialLookupMetrics = metrics;
+ return this;
+ }
+
+ @Override
+ public InternalRowSerializer createValueSerializer() {
+ return InternalSerializers.create(readerFactoryBuilder.readValueType());
+ }
+
+ @Override
+ public void close() throws IOException {
+ // ConcurrentHashMap iteration is weakly consistent. close is expected not to race with
+ // refreshFiles for the same query instance; callers may rebuild this query after close.
+ for (Map.Entry<BinaryRow, Map<Integer, BucketLookupState>> buckets : tableView.entrySet()) {
+ for (Map.Entry<Integer, BucketLookupState> bucket : buckets.getValue().entrySet()) {
+ BucketLookupState state = bucket.getValue();
+ state.lock.writeLock().lock();
+ try {
+ if (state.lookupLevels != null) {
+ state.lookupLevels.close();
+ }
+ } finally {
+ state.lock.writeLock().unlock();
+ }
+ }
+ }
+ lookupFileCache.invalidateAll();
Review Comment:
#3955 asks for cached-file ownership and eviction to be independent of the in-memory lookuper cache. Closing a lookuper here still removes all files in its namespace; `LookupLevels.close()` above also invalidates its owned entries.
For example, querying 11 small tables can trigger the 10-lookuper capacity limit and delete an entire table's local cache even when all 11 tables' files would fit within the shared disk budget. Querying the evicted table again then requires rebuilding those files. This leaves the lifecycle-decoupling part of #3955 unfinished.
Could we separate ordinary lookuper eviction from explicit table/schema/config invalidation, so the shared cache can retain reusable files across lookuper recreation? This also needs a reusable cache identity: `cacheNamespace(...)` currently includes a fresh `lookuperId` for each instance. Removing only this `invalidateAll()` would therefore be insufficient. A regression test with more than 10 tables whose files fit within the global budget would help verify the intended reuse.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [server] Add support of sharable file cache capacity for historical partition lookup [fluss]
Posted by "luoyuxia (via GitHub)" <gi...@apache.org>.
luoyuxia commented on code in PR #4113:
URL: https://github.com/apache/fluss/pull/4113#discussion_r4119733805
##########
fluss-server/src/main/java/org/apache/fluss/server/replica/historical/HistoricalLakeLookupManager.java:
##########
@@ -263,9 +274,33 @@ List<byte[]> lookup(
@Override
public void close() {
+ lifecycleLock.writeLock().lock();
+ try {
+ synchronized (this) {
+ if (closed) {
+ return;
+ }
+ closed = true;
+ }
+ } finally {
+ lifecycleLock.writeLock().unlock();
+ }
lakeTableLookupers.invalidateAll();
lakeTableLookupers.cleanUp();
requiredLakeSnapshotIds.clear();
+ closeSharedResourcesIfUnused();
+ }
+
+ private void closeSharedResourcesIfUnused() {
+ LakeTableLookuperManager manager;
+ synchronized (this) {
+ if (!closed || openLookupers.get() != 0) {
Review Comment:
Could we simplify lifecycle management by using `lifecycleLock` consistently, instead of combining it with `synchronized (this)` and deferred shared-resource cleanup?
Concretely:
- Keep the read lock in `acquireLookuper()`, and use the write lock in `startup()` and the state-changing part of `reconfigure()`.
- In `close()`, take the write lock only to check/set `closed` and detach `lookuperManager`. Then release the lock, invalidate the cached lookupers, and close the captured shared manager in a `finally` block.
- Remove `openLookupers` and `closeSharedResourcesIfUnused()`, so `closeLookuper()` only closes its own lookuper. Retain the per-lookuper `activeLookups` and synchronization for eviction and replacement during normal operation.
This manager is closed during TabletServer shutdown, after the historical task executor's bounded shutdown wait. At that point, allowing remaining lookups to fail and accepting possible temporary-cache leftovers seems reasonable. This would give us one lifecycle locking scheme and a more direct shutdown path.
##########
fluss-lake/fluss-lake-paimon/src/main/java/org/apache/fluss/lake/paimon/lookup/PaimonLocalTableQuery.java:
##########
@@ -0,0 +1,321 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.fluss.lake.paimon.lookup;
+
+import org.apache.paimon.CoreOptions;
+import org.apache.paimon.FileStore;
+import org.apache.paimon.KeyValue;
+import org.apache.paimon.KeyValueFileStore;
+import org.apache.paimon.data.BinaryRow;
+import org.apache.paimon.data.InternalRow;
+import org.apache.paimon.data.serializer.InternalRowSerializer;
+import org.apache.paimon.data.serializer.InternalSerializers;
+import org.apache.paimon.data.serializer.RowCompactedSerializer;
+import org.apache.paimon.deletionvectors.DeletionVector;
+import org.apache.paimon.disk.IOManager;
+import org.apache.paimon.fs.FileIO;
+import org.apache.paimon.io.DataFileMeta;
+import org.apache.paimon.io.KeyValueFileReaderFactory;
+import org.apache.paimon.io.cache.CacheManager;
+import org.apache.paimon.lookup.LookupStoreFactory;
+import org.apache.paimon.mergetree.Levels;
+import org.apache.paimon.mergetree.LookupFile;
+import org.apache.paimon.mergetree.LookupLevels;
+import org.apache.paimon.mergetree.lookup.LookupSerializerFactory;
+import org.apache.paimon.mergetree.lookup.PersistValueProcessor;
+import org.apache.paimon.mergetree.lookup.RemoteLookupFileManager;
+import org.apache.paimon.operation.metrics.PartialLookupMetrics;
+import org.apache.paimon.options.Options;
+import org.apache.paimon.reader.RecordReader;
+import org.apache.paimon.shade.caffeine2.com.github.benmanes.caffeine.cache.Cache;
+import org.apache.paimon.table.FileStoreTable;
+import org.apache.paimon.table.query.TableQuery;
+import org.apache.paimon.types.RowType;
+import org.apache.paimon.utils.Filter;
+import org.apache.paimon.utils.KeyComparatorSupplier;
+import org.apache.paimon.utils.Preconditions;
+
+import javax.annotation.Nullable;
+
+import java.io.IOException;
+import java.util.Comparator;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.locks.ReadWriteLock;
+import java.util.concurrent.locks.ReentrantReadWriteLock;
+import java.util.function.Supplier;
+
+import static org.apache.paimon.lookup.LookupStoreFactory.bloomFilterBuilderFactory;
+import static org.apache.paimon.mergetree.LookupFile.localFilePrefix;
+
+/**
+ * Adapted from Paimon 2.0.0's <a
+ * href="https://github.com/apache/paimon/blob/604e6d5e131c74a8d127333a2a3ad6d0319732bf/paimon-core/src/main/java/org/apache/paimon/table/query/LocalTableQuery.java">LocalTableQuery</a>.
+ *
+ * <p>Accepts a shared lookup-file cache and an instance namespace, passing that namespace's view to
+ * {@link LookupLevels}. This replaces the lazy, query-owned cache built from table retention and
+ * disk-size options. Closing a query only invalidates its namespace; the shared cache and I/O
+ * manager belong to the lake table lookuper manager.
+ */
+final class PaimonLocalTableQuery implements TableQuery {
Review Comment:
Could we add a TODO here to consider introducing an API in Paimon that allows `LocalTableQuery` to accept an externally managed shared lookup-file cache? This would let Fluss reuse Paimon's implementation directly and eventually remove this copied class.
##########
fluss-common/src/main/java/org/apache/fluss/lake/lakestorage/LakeTableLookuperManager.java:
##########
@@ -0,0 +1,154 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.fluss.lake.lakestorage;
+
+import org.apache.fluss.annotation.PublicEvolving;
+import org.apache.fluss.config.Configuration;
+import org.apache.fluss.config.TableConfig;
+import org.apache.fluss.metadata.TablePath;
+
+import java.time.Duration;
+
+import static org.apache.fluss.utils.Preconditions.checkArgument;
+import static org.apache.fluss.utils.Preconditions.checkNotNull;
+
+/**
+ * Creates lake table lookupers and manages their shared resources within one TabletServer.
+ *
+ * <p>The caller must close all created lookupers after their requests have finished before closing
+ * this manager. Closing a lookuper must not release resources shared with other lookupers.
+ *
+ * @since 1.1
+ */
+@PublicEvolving
+public interface LakeTableLookuperManager extends AutoCloseable {
+
+ /**
+ * Creates a table-level point lookuper for the specified lake table.
+ *
+ * @param tablePath the logical path identifying the table in the lakehouse storage
+ * @param context runtime context for creating the lookuper
+ * @return a table-level point lookuper
+ */
+ LakeTableLookuper createLakeTableLookuper(TablePath tablePath, Context context);
+
+ /**
+ * Applies a new snapshot of the runtime resource settings to existing and future lookupers
+ * without replacing them.
+ *
+ * <p>This method may be called concurrently with lookuper creation and lookups. Implementations
+ * must apply the settings in a thread-safe manner. Lake-format and table-specific configuration
+ * is outside the scope of this method.
+ *
+ * @param options the new runtime resource settings
+ */
+ void reconfigure(LookupRuntimeOptions options);
+
+ /** Returns the cumulative number of lookup files evicted by the shared disk-space budget. */
+ default long fileCacheCapacityEvictions() {
+ return 0L;
+ }
+
+ /**
+ * Immutable snapshot of the format-independent resource settings for a lookup runtime.
+ *
+ * <p>These settings apply to resources shared by all table lookupers in one runtime.
+ * Lake-format and table-specific configuration is supplied separately when creating a lookuper.
+ */
+ final class LookupRuntimeOptions {
Review Comment:
Could we rename `LookupRuntimeOptions` to `LookupCacheOptions`? It currently contains only `localCacheMaxBytes` and `expireAfterAccess`, both of which configure the shared lookup cache. The more specific name would make its scope clearer. Please update the references and Javadoc accordingly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [server] Add support of sharable file cache capacity for historical partition lookup [fluss]
Posted by "zuston (via GitHub)" <gi...@apache.org>.
zuston commented on PR #4113:
URL: https://github.com/apache/fluss/pull/4113#issuecomment-5489912453
> #3955 also asks that the lookuper cache stop deciding when files get evicted. Paimon ties file lifetime to the query, so I don't think that's doable without the upstream withLookupFileCache change you mentioned.
Yes. We may also need to extend this PR to address the issue raised in #4158. I’ll work on it this week.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [server] Add support of sharable file cache capacity for historical partition lookup [fluss]
Posted by "luoyuxia (via GitHub)" <gi...@apache.org>.
luoyuxia merged PR #4113:
URL: https://github.com/apache/fluss/pull/4113
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [server] Add support of sharable file cache capacity for historical partition lookup [fluss]
Posted by "luoyuxia (via GitHub)" <gi...@apache.org>.
luoyuxia commented on code in PR #4113:
URL: https://github.com/apache/fluss/pull/4113#discussion_r4103756409
##########
fluss-common/src/main/java/org/apache/fluss/lake/lakestorage/LakeTableLookupRuntime.java:
##########
@@ -0,0 +1,90 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.fluss.lake.lakestorage;
+
+import org.apache.fluss.annotation.PublicEvolving;
+import org.apache.fluss.config.Configuration;
+import org.apache.fluss.config.TableConfig;
+import org.apache.fluss.metadata.TablePath;
+
+import static org.apache.fluss.utils.Preconditions.checkNotNull;
+
+/** TabletServer-scoped runtime for creating lake table lookupers. */
+@PublicEvolving
+public interface LakeTableLookupRuntime extends AutoCloseable {
Review Comment:
How about renaming this to `LakeTableLookuperManager`? It creates `LakeTableLookuper` instances and manages their shared lookup resources and disk-cache budget within a TabletServer. This name makes its relationship with `LakeTableLookuper` explicit and reflects that the interface manages lookup resources rather than performing table lookups itself. The factory method and implementation could then be named `LakeStorage.createLakeTableLookuperManager()` and `PaimonLakeTableLookuperManager`.
Also, since this is a new `@PublicEvolving` API, could we add `@since 1.0` to its class Javadoc?
##########
fluss-server/src/main/java/org/apache/fluss/server/replica/historical/HistoricalLakeLookupManager.java:
##########
@@ -413,11 +458,16 @@ private static boolean hasLakeConfigChanged(Configuration currentConf, Configura
extractLakeProperties(currentConf), extractLakeProperties(newConf));
}
- private long cacheBytesPerTable(double ratio) {
+ private long cacheBytes(double ratio) {
checkArgument(ratio > 0.0 && ratio <= 1.0, "ratio must be within (0.0, 1.0].");
long totalCacheBytes =
Math.min(dataDirVolumeBytes, (long) Math.ceil(dataDirVolumeBytes * ratio));
- return Math.max(1L, totalCacheBytes / MAX_CACHED_TABLES);
+ return Math.max(1L, totalCacheBytes);
+ }
+
+ private static String cacheNamespace(
+ long tableId, int schemaId, long lakeConfigVersion, long lookuperId) {
Review Comment:
Could we add a comment explaining that this namespace scopes shared-cache entries to one lookuper instance? In particular, `lookuperId` is allocated by `lookuperIdSequence.getAndIncrement()` each time a lookuper is created. Recreating a lookuper for the same table therefore gets a new namespace even when its schema and lake configuration are unchanged.
Please also explain why this is needed: an old lookuper may still be serving in-flight requests when its replacement is created. Separate namespaces prevent closing the old lookuper from invalidating the replacement's cached files.
##########
fluss-server/src/main/java/org/apache/fluss/server/replica/historical/HistoricalLakeLookupManager.java:
##########
@@ -413,11 +458,16 @@ private static boolean hasLakeConfigChanged(Configuration currentConf, Configura
extractLakeProperties(currentConf), extractLakeProperties(newConf));
}
- private long cacheBytesPerTable(double ratio) {
+ private long cacheBytes(double ratio) {
checkArgument(ratio > 0.0 && ratio <= 1.0, "ratio must be within (0.0, 1.0].");
long totalCacheBytes =
Math.min(dataDirVolumeBytes, (long) Math.ceil(dataDirVolumeBytes * ratio));
- return Math.max(1L, totalCacheBytes / MAX_CACHED_TABLES);
+ return Math.max(1L, totalCacheBytes);
+ }
+
+ private static String cacheNamespace(
+ long tableId, int schemaId, long lakeConfigVersion, long lookuperId) {
+ return tableId + "-" + schemaId + "-" + lakeConfigVersion + "-" + lookuperId;
Review Comment:
Nit (optional): could we use `String.format("%d-%d-%d-%d", tableId, schemaId, lakeConfigVersion, lookuperId)` to make the namespace format explicit?
##########
fluss-lake/fluss-lake-paimon/src/main/java/org/apache/fluss/lake/paimon/lookup/PaimonLocalTableQuery.java:
##########
@@ -0,0 +1,316 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.fluss.lake.paimon.lookup;
+
+import org.apache.paimon.CoreOptions;
+import org.apache.paimon.FileStore;
+import org.apache.paimon.KeyValue;
+import org.apache.paimon.KeyValueFileStore;
+import org.apache.paimon.data.BinaryRow;
+import org.apache.paimon.data.InternalRow;
+import org.apache.paimon.data.serializer.InternalRowSerializer;
+import org.apache.paimon.data.serializer.InternalSerializers;
+import org.apache.paimon.data.serializer.RowCompactedSerializer;
+import org.apache.paimon.deletionvectors.DeletionVector;
+import org.apache.paimon.disk.IOManager;
+import org.apache.paimon.fs.FileIO;
+import org.apache.paimon.io.DataFileMeta;
+import org.apache.paimon.io.KeyValueFileReaderFactory;
+import org.apache.paimon.io.cache.CacheManager;
+import org.apache.paimon.lookup.LookupStoreFactory;
+import org.apache.paimon.mergetree.Levels;
+import org.apache.paimon.mergetree.LookupFile;
+import org.apache.paimon.mergetree.LookupLevels;
+import org.apache.paimon.mergetree.lookup.LookupSerializerFactory;
+import org.apache.paimon.mergetree.lookup.PersistValueProcessor;
+import org.apache.paimon.mergetree.lookup.RemoteLookupFileManager;
+import org.apache.paimon.operation.metrics.PartialLookupMetrics;
+import org.apache.paimon.options.Options;
+import org.apache.paimon.reader.RecordReader;
+import org.apache.paimon.shade.caffeine2.com.github.benmanes.caffeine.cache.Cache;
+import org.apache.paimon.table.FileStoreTable;
+import org.apache.paimon.table.query.TableQuery;
+import org.apache.paimon.types.RowType;
+import org.apache.paimon.utils.Filter;
+import org.apache.paimon.utils.KeyComparatorSupplier;
+import org.apache.paimon.utils.Preconditions;
+
+import javax.annotation.Nullable;
+
+import java.io.IOException;
+import java.util.Comparator;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.locks.ReadWriteLock;
+import java.util.concurrent.locks.ReentrantReadWriteLock;
+import java.util.function.Supplier;
+
+import static org.apache.paimon.lookup.LookupStoreFactory.bloomFilterBuilderFactory;
+import static org.apache.paimon.mergetree.LookupFile.localFilePrefix;
+
+/**
+ * Fluss-owned fork of Paimon 2.0.0's {@code LocalTableQuery}, using a shared namespaced lookup
+ * cache.
+ */
+final class PaimonLocalTableQuery implements TableQuery {
+
+ private final Map<BinaryRow, Map<Integer, BucketLookupState>> tableView;
+
+ private final CoreOptions options;
+
+ private final Supplier<Comparator<InternalRow>> keyComparatorSupplier;
+
+ private final KeyValueFileReaderFactory.Builder readerFactoryBuilder;
+
+ private final LookupStoreFactory lookupStoreFactory;
+
+ private final int startLevel;
+
+ private IOManager ioManager;
+
+ private final Cache<String, LookupFile> lookupFileCache;
+
+ private final RowType rowType;
+ private final RowType partitionType;
+ private final FileIO fileIO;
+
+ @Nullable private Filter<InternalRow> cacheRowFilter;
+
+ @Nullable private PartialLookupMetrics partialLookupMetrics;
+
+ PaimonLocalTableQuery(
+ FileStoreTable table,
+ SharedLookupFileCache sharedLookupFileCache,
+ String cacheNamespace) {
+ this.options = table.coreOptions();
+ this.lookupFileCache = sharedLookupFileCache.namespaced(cacheNamespace);
+ this.tableView = new ConcurrentHashMap<>();
+ FileStore<?> tableStore = table.store();
+ if (!(tableStore instanceof KeyValueFileStore)) {
+ throw new UnsupportedOperationException(
+ "Table Query only supports table with primary key.");
+ }
+ KeyValueFileStore store = (KeyValueFileStore) tableStore;
+
+ this.readerFactoryBuilder = store.newReaderFactoryBuilder();
+ this.rowType = table.schema().logicalRowType();
+ this.partitionType = table.schema().logicalPartitionType();
+ this.fileIO = table.fileIO();
+ RowType keyType = readerFactoryBuilder.keyType();
+ this.keyComparatorSupplier = new KeyComparatorSupplier(readerFactoryBuilder.keyType());
+ this.lookupStoreFactory =
+ LookupStoreFactory.create(
+ options,
+ new CacheManager(
+ options.lookupCacheMaxMemory(),
+ options.lookupCacheHighPrioPoolRatio()),
+ new RowCompactedSerializer(keyType).createSliceComparator());
+ startLevel = options.needLookup() ? 1 : 0;
+ }
+
+ public void refreshFiles(
+ BinaryRow partition,
+ int bucket,
+ List<DataFileMeta> beforeFiles,
+ List<DataFileMeta> dataFiles) {
+ // Both tableView and its nested bucket maps are ConcurrentHashMaps; this nested
+ // computeIfAbsent pattern relies on each map providing atomic insertion.
+ BucketLookupState state =
+ tableView
+ .computeIfAbsent(partition, k -> new ConcurrentHashMap<>())
+ .computeIfAbsent(bucket, k -> new BucketLookupState());
+ state.lock.writeLock().lock();
+ try {
+ if (state.lookupLevels == null) {
+ // Initial phase: ignore beforeFiles as they represent deletions from previous state
+ state.lookupLevels = createLookupLevels(partition, bucket, dataFiles);
+ } else {
+ state.lookupLevels.getLevels().update(beforeFiles, dataFiles);
+ }
+ } finally {
+ state.lock.writeLock().unlock();
+ }
+ }
+
+ private LookupLevels<KeyValue> createLookupLevels(
+ BinaryRow partition, int bucket, List<DataFileMeta> dataFiles) {
+ Levels levels = new Levels(keyComparatorSupplier.get(), dataFiles, options.numLevels());
+ // TODO pass DeletionVector factory
+ KeyValueFileReaderFactory factory =
+ readerFactoryBuilder.build(partition, bucket, DeletionVector.emptyFactory());
+ Options options = this.options.toConfiguration();
+
+ RowType readValueType = readerFactoryBuilder.readValueType();
+ LookupLevels<KeyValue> lookupLevels =
+ new LookupLevels<>(
+ schemaId -> readValueType,
+ 0L,
+ levels,
+ keyComparatorSupplier.get(),
+ readerFactoryBuilder.keyType(),
+ PersistValueProcessor.factory(readValueType),
+ LookupSerializerFactory.INSTANCE.get(),
+ file -> {
+ RecordReader<KeyValue> reader = factory.createRecordReader(file);
+ if (cacheRowFilter != null) {
+ reader =
+ reader.filter(
+ keyValue -> cacheRowFilter.test(keyValue.value()));
+ }
+ return reader;
+ },
+ file ->
+ Preconditions.checkNotNull(ioManager, "IOManager is required.")
+ .createChannel(
+ localFilePrefix(
+ partitionType, partition, bucket, file))
+ .getPathFile(),
+ lookupStoreFactory,
+ bloomFilterBuilderFactory(options),
+ lookupFileCache);
+
+ // Optimization - download lookup files if already persisted to object store
+ // We download these files if three conditions are met
+ // 1) lookup.remote-file.enabled is true - files are persisted in the first place
+ // 2) deletion-vectors.enabled is false - SSTables only contain row positions, not values,
+ // when DVs are enabled
+ // 3) The client is accessing the full data row, as opposed to a projection
+ // - The persisted remote SSTable files are created during compaction and hold the entire
+ // data row value
+ // - We could deserialize and project in memory, but we'll have to read much more data,
+ // not as clear of a win
+ boolean fullValueRead = readerFactoryBuilder.readValueType().equals(rowType);
+ if (this.options.lookupRemoteFileEnabled()
+ && !this.options.deletionVectorsEnabled()
+ && fullValueRead) {
+ // Calling the constructor tells `lookupLevels` to load remote files
+ new RemoteLookupFileManager<>(
+ fileIO,
+ factory.pathFactory(),
+ lookupLevels,
+ this.options.lookupRemoteLevelThreshold());
+ }
+
+ return lookupLevels;
+ }
+
+ @Nullable
+ @Override
+ public InternalRow lookup(BinaryRow partition, int bucket, InternalRow key) throws IOException {
+ PartialLookupMetrics currentMetrics = partialLookupMetrics;
+ LookupLevels.LookupContext context =
+ currentMetrics == null ? null : new LookupLevels.LookupContext();
+ try {
+ return lookup(partition, bucket, key, context);
+ } finally {
+ if (currentMetrics != null) {
+ currentMetrics.reportLookup(context != null && context.remoteAccessed());
+ }
+ }
+ }
+
+ @Nullable
+ private InternalRow lookup(
+ BinaryRow partition,
+ int bucket,
+ InternalRow key,
+ @Nullable LookupLevels.LookupContext context)
+ throws IOException {
+ Map<Integer, BucketLookupState> buckets = tableView.get(partition);
+ if (buckets == null || buckets.isEmpty()) {
+ return null;
+ }
+ BucketLookupState state = buckets.get(bucket);
+ if (state == null) {
+ return null;
+ }
+
+ state.lock.readLock().lock();
+ try {
+ LookupLevels<KeyValue> lookupLevels = state.lookupLevels;
+ if (lookupLevels == null) {
+ return null;
+ }
+
+ KeyValue kv = lookupLevels.lookup(key, startLevel, context);
+ if (kv == null || kv.valueKind().isRetract()) {
+ return null;
+ } else {
+ return kv.value();
+ }
+ } finally {
+ state.lock.readLock().unlock();
+ }
+ }
+
+ @Override
+ public PaimonLocalTableQuery withValueProjection(int[] projection) {
+ this.readerFactoryBuilder.withReadValueType(rowType.project(projection));
+ return this;
+ }
+
+ public PaimonLocalTableQuery withIOManager(IOManager ioManager) {
+ this.ioManager = ioManager;
+ return this;
+ }
+
+ public PaimonLocalTableQuery withCacheRowFilter(Filter<InternalRow> cacheRowFilter) {
+ this.cacheRowFilter = cacheRowFilter;
+ return this;
+ }
+
+ public PaimonLocalTableQuery withMetrics(@Nullable PartialLookupMetrics metrics) {
+ this.partialLookupMetrics = metrics;
+ return this;
+ }
+
+ @Override
+ public InternalRowSerializer createValueSerializer() {
+ return InternalSerializers.create(readerFactoryBuilder.readValueType());
+ }
+
+ @Override
+ public void close() throws IOException {
+ // ConcurrentHashMap iteration is weakly consistent. close is expected not to race with
+ // refreshFiles for the same query instance; callers may rebuild this query after close.
+ for (Map.Entry<BinaryRow, Map<Integer, BucketLookupState>> buckets : tableView.entrySet()) {
+ for (Map.Entry<Integer, BucketLookupState> bucket : buckets.getValue().entrySet()) {
+ BucketLookupState state = bucket.getValue();
+ state.lock.writeLock().lock();
+ try {
+ if (state.lookupLevels != null) {
+ state.lookupLevels.close();
+ }
+ } finally {
+ state.lock.writeLock().unlock();
+ }
+ }
+ }
+ lookupFileCache.invalidateAll();
Review Comment:
Could we remove the fixed `MAX_CACHED_TABLES = 10` limit from `HistoricalLakeLookupManager` now that lookup files share a global disk budget?
Querying 11 small tables can still evict a table lookuper and delete all its cached files here, even when all of those files would fit within the shared disk budget. Keeping lookuper idle expiration while letting the shared file cache enforce the disk budget would avoid this premature eviction.
##########
fluss-lake/fluss-lake-paimon/src/main/java/org/apache/fluss/lake/paimon/lookup/PaimonLocalTableQuery.java:
##########
@@ -0,0 +1,316 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.fluss.lake.paimon.lookup;
+
+import org.apache.paimon.CoreOptions;
+import org.apache.paimon.FileStore;
+import org.apache.paimon.KeyValue;
+import org.apache.paimon.KeyValueFileStore;
+import org.apache.paimon.data.BinaryRow;
+import org.apache.paimon.data.InternalRow;
+import org.apache.paimon.data.serializer.InternalRowSerializer;
+import org.apache.paimon.data.serializer.InternalSerializers;
+import org.apache.paimon.data.serializer.RowCompactedSerializer;
+import org.apache.paimon.deletionvectors.DeletionVector;
+import org.apache.paimon.disk.IOManager;
+import org.apache.paimon.fs.FileIO;
+import org.apache.paimon.io.DataFileMeta;
+import org.apache.paimon.io.KeyValueFileReaderFactory;
+import org.apache.paimon.io.cache.CacheManager;
+import org.apache.paimon.lookup.LookupStoreFactory;
+import org.apache.paimon.mergetree.Levels;
+import org.apache.paimon.mergetree.LookupFile;
+import org.apache.paimon.mergetree.LookupLevels;
+import org.apache.paimon.mergetree.lookup.LookupSerializerFactory;
+import org.apache.paimon.mergetree.lookup.PersistValueProcessor;
+import org.apache.paimon.mergetree.lookup.RemoteLookupFileManager;
+import org.apache.paimon.operation.metrics.PartialLookupMetrics;
+import org.apache.paimon.options.Options;
+import org.apache.paimon.reader.RecordReader;
+import org.apache.paimon.shade.caffeine2.com.github.benmanes.caffeine.cache.Cache;
+import org.apache.paimon.table.FileStoreTable;
+import org.apache.paimon.table.query.TableQuery;
+import org.apache.paimon.types.RowType;
+import org.apache.paimon.utils.Filter;
+import org.apache.paimon.utils.KeyComparatorSupplier;
+import org.apache.paimon.utils.Preconditions;
+
+import javax.annotation.Nullable;
+
+import java.io.IOException;
+import java.util.Comparator;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.locks.ReadWriteLock;
+import java.util.concurrent.locks.ReentrantReadWriteLock;
+import java.util.function.Supplier;
+
+import static org.apache.paimon.lookup.LookupStoreFactory.bloomFilterBuilderFactory;
+import static org.apache.paimon.mergetree.LookupFile.localFilePrefix;
+
+/**
+ * Fluss-owned fork of Paimon 2.0.0's {@code LocalTableQuery}, using a shared namespaced lookup
Review Comment:
Could we expand the class Javadoc with a link to the exact upstream source and a short summary of the changes? Here is the [Paimon 2.0.0 `LocalTableQuery` source](https://github.com/apache/paimon/blob/604e6d5e131c74a8d127333a2a3ad6d0319732bf/paimon-core/src/main/java/org/apache/paimon/table/query/LocalTableQuery.java).
The main adaptations to document are:
- Accepting an externally supplied `SharedLookupFileCache` and `cacheNamespace`, and passing the namespaced view to `LookupLevels`.
- Replacing the lazy, query-owned cache created from the table's retention/disk-size options with that injected cache view.
- Keeping query-close cleanup scoped to its namespace; the shared cache itself is owned by the lookup runtime.
Please also add this copied/modified source file to the existing Apache Paimon section in the root [`LICENSE`](https://github.com/apache/fluss/blob/63dc46c623e222bc8160dc5d5958d6393a0fb632/LICENSE#L363), following the existing source-attribution convention. The ASF license header is already present in this file.
Finally, could we open an issue in the Paimon community, or link an existing one, to allow `LocalTableQuery` to accept a caller-provided shared lookup-file cache with explicit ownership and close semantics? This would let multiple table queries share a global cache budget and allow Fluss to eventually use the upstream implementation directly and remove this fork.
##########
fluss-lake/fluss-lake-paimon/src/main/java/org/apache/fluss/lake/paimon/lookup/NamespacedLookupFileCache.java:
##########
@@ -0,0 +1,205 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.fluss.lake.paimon.lookup;
+
+import org.apache.paimon.mergetree.LookupFile;
+import org.apache.paimon.shade.caffeine2.com.github.benmanes.caffeine.cache.Cache;
+import org.apache.paimon.shade.caffeine2.com.github.benmanes.caffeine.cache.Policy;
+import org.apache.paimon.shade.caffeine2.com.github.benmanes.caffeine.cache.stats.CacheStats;
+
+import javax.annotation.Nullable;
+
+import java.util.AbstractMap;
+import java.util.HashSet;
+import java.util.LinkedHashMap;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.ConcurrentMap;
+import java.util.function.Function;
+
+import static org.apache.fluss.utils.Preconditions.checkNotNull;
+
+/** Namespace-scoped cache view delegating all storage to a shared lookup-file cache. */
+final class NamespacedLookupFileCache implements Cache<String, LookupFile> {
+
+ private final Cache<SharedLookupFileCache.Key, LookupFile> sharedCache;
+ private final String namespace;
+
+ NamespacedLookupFileCache(
+ Cache<SharedLookupFileCache.Key, LookupFile> sharedCache, String namespace) {
+ this.sharedCache = checkNotNull(sharedCache, "sharedCache must not be null.");
+ this.namespace = checkNotNull(namespace, "namespace must not be null.");
+ }
+
+ @Override
+ public @Nullable LookupFile getIfPresent(Object fileName) {
+ return fileName instanceof String ? sharedCache.getIfPresent(key((String) fileName)) : null;
+ }
+
+ @Override
+ public LookupFile get(
+ String fileName, Function<? super String, ? extends LookupFile> mappingFunction) {
+ return sharedCache.get(key(fileName), ignored -> mappingFunction.apply(fileName));
+ }
+
+ @Override
+ public Map<String, LookupFile> getAllPresent(Iterable<?> fileNames) {
Review Comment:
Nit: could we align the nullability annotations on these overrides with the Caffeine `Cache` interface? For example, `getAllPresent` has a non-null parameter and return value, and `stats()` / `asMap()` have non-null return values. The missing annotations lead to IDE warnings such as "Not annotated parameter overrides @NonNull parameter" and "Not annotated method overrides method annotated with @NonNull".
Please preserve the nullable return contracts as well: both `getIfPresent()` and `get()` can return null.
##########
fluss-lake/fluss-lake-paimon/src/main/java/org/apache/fluss/lake/paimon/lookup/SharedLookupFileCache.java:
##########
@@ -0,0 +1,112 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.fluss.lake.paimon.lookup;
+
+import org.apache.fluss.annotation.Internal;
+
+import org.apache.paimon.mergetree.LookupFile;
+import org.apache.paimon.options.MemorySize;
+import org.apache.paimon.shade.caffeine2.com.github.benmanes.caffeine.cache.Cache;
+import org.apache.paimon.shade.caffeine2.com.github.benmanes.caffeine.cache.Caffeine;
+import org.apache.paimon.shade.caffeine2.com.github.benmanes.caffeine.cache.RemovalCause;
+
+import javax.annotation.Nullable;
+
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.time.Duration;
+import java.util.Objects;
+
+import static org.apache.fluss.utils.Preconditions.checkNotNull;
+import static org.apache.paimon.mergetree.LookupUtils.fileKibiBytes;
+
+/** A weighted lookup-file cache shared by multiple Paimon table lookupers. */
+@Internal
+public final class SharedLookupFileCache implements AutoCloseable {
+
+ private final Cache<Key, LookupFile> cache;
+
+ /** Creates a shared lookup-file cache. */
+ public SharedLookupFileCache(Duration fileRetention, MemorySize maxDiskSize) {
+ checkNotNull(fileRetention, "fileRetention must not be null.");
+ checkNotNull(maxDiskSize, "maxDiskSize must not be null.");
+ this.cache =
+ Caffeine.newBuilder()
Review Comment:
Agreed, we could consider exposing a separate TabletServer-level metric for file evictions from this shared cache. That would show eviction pressure from the global disk budget even when no table lookuper is evicted. For a capacity-eviction counter, explicit invalidation during lookuper cleanup should be excluded.
##########
fluss-lake/fluss-lake-paimon/src/main/java/org/apache/fluss/lake/paimon/PaimonLakeStorage.java:
##########
@@ -55,13 +65,50 @@ public LakeSource<PaimonSplit> createLakeSource(TablePath tablePath) {
}
@Override
- public LakeTableLookuper createLakeTableLookuper(TablePath tablePath, LookuperContext context) {
- return new PaimonLakeTableLookuper(
- paimonConfig,
- tablePath,
- context.ioTmpDir(),
- context.tableConfig(),
- context.lookupCacheMaxDiskBytes(),
- context.diskWriteGuard());
+ public LakeTableLookupRuntime createLakeTableLookupRuntime(
+ String ioTmpDir, long lookupCacheMaxDiskBytes) {
+ return new PaimonLakeTableLookupRuntime(ioTmpDir, lookupCacheMaxDiskBytes);
+ }
+
+ /** Paimon lookup runtime sharing one I/O manager across table lookupers. */
+ private static final class PaimonLakeTableLookupRuntime implements LakeTableLookupRuntime {
+ private final IOManager ioManager;
+ private final SharedLookupFileCache lookupFileCache;
+
+ private PaimonLakeTableLookupRuntime(String ioTmpDir, long lookupCacheMaxDiskBytes) {
+ checkArgument(
+ lookupCacheMaxDiskBytes > 0, "lookupCacheMaxDiskBytes must be greater than 0.");
+ this.ioManager = IOManager.create(checkNotNull(ioTmpDir, "ioTmpDir must not be null."));
+ // ponytail: one runtime-wide retention; add a server option if this needs tuning.
+ this.lookupFileCache =
+ new SharedLookupFileCache(
+ CoreOptions.LOOKUP_CACHE_FILE_RETENTION.defaultValue(),
Review Comment:
+1. Could we reuse the configured `server.historical-partition.lookuper-cache.expire-after-access` value for the shared file cache's expire-after-access duration, instead of hard-coding the Paimon default? This would give both caches a configurable timeout (3 hours by default) without adding per-table expiration handling.
The two caches would still track access independently: using a table lookuper refreshes its timeout, while accessing a file refreshes that file's timeout. Please also propagate dynamic changes to this setting to the shared file cache, and document the setting's expanded scope and that it replaces the Paimon table-level `lookup.cache-file-retention` for historical lookups.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [server] Add support of sharable file cache capacity for historical partition lookup [fluss]
Posted by "luoyuxia (via GitHub)" <gi...@apache.org>.
luoyuxia commented on code in PR #4113:
URL: https://github.com/apache/fluss/pull/4113#discussion_r4102328942
##########
fluss-common/src/main/java/org/apache/fluss/lake/lakestorage/LakeTableLookupRuntime.java:
##########
@@ -0,0 +1,90 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.fluss.lake.lakestorage;
+
+import org.apache.fluss.annotation.PublicEvolving;
+import org.apache.fluss.config.Configuration;
+import org.apache.fluss.config.TableConfig;
+import org.apache.fluss.metadata.TablePath;
+
+import static org.apache.fluss.utils.Preconditions.checkNotNull;
+
+/** TabletServer-scoped runtime for creating lake table lookupers. */
+@PublicEvolving
+public interface LakeTableLookupRuntime extends AutoCloseable {
Review Comment:
How about renaming this to `LakeLookuper`? It manages shared lookup resources across tables within a TabletServer and creates `LakeTableLookuper` instances. The two names would make the scopes clearer: `LakeLookuper` for the lake-storage-level entry point, and `LakeTableLookuper` for lookups on an individual table. The factory method and implementation could then be named `LakeStorage.createLakeLookuper()` and `PaimonLakeLookuper`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [server] Add support of sharable file cache capacity for historical partition lookup [fluss]
Posted by "luoyuxia (via GitHub)" <gi...@apache.org>.
luoyuxia commented on code in PR #4113:
URL: https://github.com/apache/fluss/pull/4113#discussion_r4120273359
##########
fluss-server/src/test/java/org/apache/fluss/server/replica/historical/HistoricalLakeLookupManagerTest.java:
##########
@@ -40,21 +42,31 @@
import com.github.benmanes.caffeine.cache.Ticker;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.io.TempDir;
+import org.mockito.ArgumentCaptor;
import java.io.File;
import java.io.RandomAccessFile;
import java.time.Duration;
import java.util.ArrayList;
import java.util.Collections;
import java.util.List;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
import java.util.concurrent.FutureTask;
import java.util.concurrent.ScheduledFuture;
+import java.util.concurrent.TimeUnit;
import java.util.concurrent.atomic.AtomicLong;
import java.util.concurrent.atomic.AtomicReference;
import static org.apache.fluss.record.TestData.PARTITION_TABLE_ID;
import static org.apache.fluss.record.TestData.PARTITION_TABLE_INFO;
import static org.assertj.core.api.Assertions.assertThat;
+import static org.assertj.core.api.Assertions.assertThatThrownBy;
+import static org.mockito.Mockito.mock;
Review Comment:
Sorry, I missed this in my earlier review. We generally avoid Mockito because mock-based tests tend to be difficult to maintain. Please see [Flink's guideline on avoiding Mockito and using reusable test implementations](https://flink.apache.org/zh/how-to-contribute/code-style-and-quality-common/#avoid-mockito---use-reusable-test-implementations).
Could we use reusable test implementations here instead? If removing Mockito would be too difficult, I would also be fine with removing the affected Mockito-dependent tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [server] Add support of sharable file cache capacity for historical partition lookup [fluss]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #4113:
URL: https://github.com/apache/fluss/pull/4113#discussion_r4093228618
##########
fluss-lake/fluss-lake-paimon/src/main/java/org/apache/fluss/lake/paimon/lookup/PaimonLocalTableQuery.java:
##########
@@ -0,0 +1,316 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.fluss.lake.paimon.lookup;
+
+import org.apache.paimon.CoreOptions;
+import org.apache.paimon.FileStore;
+import org.apache.paimon.KeyValue;
+import org.apache.paimon.KeyValueFileStore;
+import org.apache.paimon.data.BinaryRow;
+import org.apache.paimon.data.InternalRow;
+import org.apache.paimon.data.serializer.InternalRowSerializer;
+import org.apache.paimon.data.serializer.InternalSerializers;
+import org.apache.paimon.data.serializer.RowCompactedSerializer;
+import org.apache.paimon.deletionvectors.DeletionVector;
+import org.apache.paimon.disk.IOManager;
+import org.apache.paimon.fs.FileIO;
+import org.apache.paimon.io.DataFileMeta;
+import org.apache.paimon.io.KeyValueFileReaderFactory;
+import org.apache.paimon.io.cache.CacheManager;
+import org.apache.paimon.lookup.LookupStoreFactory;
+import org.apache.paimon.mergetree.Levels;
+import org.apache.paimon.mergetree.LookupFile;
+import org.apache.paimon.mergetree.LookupLevels;
+import org.apache.paimon.mergetree.lookup.LookupSerializerFactory;
+import org.apache.paimon.mergetree.lookup.PersistValueProcessor;
+import org.apache.paimon.mergetree.lookup.RemoteLookupFileManager;
+import org.apache.paimon.operation.metrics.PartialLookupMetrics;
+import org.apache.paimon.options.Options;
+import org.apache.paimon.reader.RecordReader;
+import org.apache.paimon.shade.caffeine2.com.github.benmanes.caffeine.cache.Cache;
+import org.apache.paimon.table.FileStoreTable;
+import org.apache.paimon.table.query.TableQuery;
+import org.apache.paimon.types.RowType;
+import org.apache.paimon.utils.Filter;
+import org.apache.paimon.utils.KeyComparatorSupplier;
+import org.apache.paimon.utils.Preconditions;
+
+import javax.annotation.Nullable;
+
+import java.io.IOException;
+import java.util.Comparator;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.locks.ReadWriteLock;
+import java.util.concurrent.locks.ReentrantReadWriteLock;
+import java.util.function.Supplier;
+
+import static org.apache.paimon.lookup.LookupStoreFactory.bloomFilterBuilderFactory;
+import static org.apache.paimon.mergetree.LookupFile.localFilePrefix;
+
+/**
+ * Fluss-owned fork of Paimon 2.0.0's {@code LocalTableQuery}, using a shared namespaced lookup
+ * cache.
+ */
+final class PaimonLocalTableQuery implements TableQuery {
+
+ private final Map<BinaryRow, Map<Integer, BucketLookupState>> tableView;
+
+ private final CoreOptions options;
+
+ private final Supplier<Comparator<InternalRow>> keyComparatorSupplier;
+
+ private final KeyValueFileReaderFactory.Builder readerFactoryBuilder;
+
+ private final LookupStoreFactory lookupStoreFactory;
+
+ private final int startLevel;
+
+ private IOManager ioManager;
+
+ private final Cache<String, LookupFile> lookupFileCache;
+
+ private final RowType rowType;
+ private final RowType partitionType;
+ private final FileIO fileIO;
+
+ @Nullable private Filter<InternalRow> cacheRowFilter;
+
+ @Nullable private PartialLookupMetrics partialLookupMetrics;
+
+ PaimonLocalTableQuery(
+ FileStoreTable table,
+ SharedLookupFileCache sharedLookupFileCache,
+ String cacheNamespace) {
+ this.options = table.coreOptions();
+ this.lookupFileCache = sharedLookupFileCache.namespaced(cacheNamespace);
Review Comment:
The forked query no longer preserves Paimon's query-auth guard. When `query-auth.enabled` is set, the local lookup path must be rejected because it reads raw values outside row filters and column masks; proceeding here can expose unmasked data through historical lookups. Add the same `options.queryAuthEnabled()` check before initializing the query.
##########
fluss-server/src/main/java/org/apache/fluss/server/replica/historical/HistoricalLakeLookupManager.java:
##########
@@ -180,6 +185,8 @@ class HistoricalLakeLookupManager implements AutoCloseable {
.executor(Runnable::run)
.removalListener(this::onLookuperRemoved)
.build();
+ this.lookuperIdSequence = new AtomicLong();
+ this.lookupRuntime = createLookupRuntime(conf);
Review Comment:
This constructs the shared runtime before `startup()` removes and recreates `historicalLookupCacheRootDir`. The Paimon IO manager owns temporary directories below the path it is given, so startup can delete directories already held by this runtime and make later channel creation fail. Delay runtime construction until after startup cleanup (or close and recreate it around the cleanup).
##########
fluss-lake/fluss-lake-paimon/src/main/java/org/apache/fluss/lake/paimon/lookup/SharedLookupFileCache.java:
##########
@@ -0,0 +1,112 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.fluss.lake.paimon.lookup;
+
+import org.apache.fluss.annotation.Internal;
+
+import org.apache.paimon.mergetree.LookupFile;
+import org.apache.paimon.options.MemorySize;
+import org.apache.paimon.shade.caffeine2.com.github.benmanes.caffeine.cache.Cache;
+import org.apache.paimon.shade.caffeine2.com.github.benmanes.caffeine.cache.Caffeine;
+import org.apache.paimon.shade.caffeine2.com.github.benmanes.caffeine.cache.RemovalCause;
+
+import javax.annotation.Nullable;
+
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.time.Duration;
+import java.util.Objects;
+
+import static org.apache.fluss.utils.Preconditions.checkNotNull;
+import static org.apache.paimon.mergetree.LookupUtils.fileKibiBytes;
+
+/** A weighted lookup-file cache shared by multiple Paimon table lookupers. */
+@Internal
+public final class SharedLookupFileCache implements AutoCloseable {
+
+ private final Cache<Key, LookupFile> cache;
+
+ /** Creates a shared lookup-file cache. */
+ public SharedLookupFileCache(Duration fileRetention, MemorySize maxDiskSize) {
+ checkNotNull(fileRetention, "fileRetention must not be null.");
+ checkNotNull(maxDiskSize, "maxDiskSize must not be null.");
+ this.cache =
+ Caffeine.newBuilder()
+ .expireAfterAccess(fileRetention)
+ .maximumWeight(Math.max(1L, maxDiskSize.getKibiBytes()))
+ .weigher(
+ (Key key, LookupFile lookupFile) ->
+ Math.max(1, fileKibiBytes(lookupFile.localFile())))
+ .removalListener(SharedLookupFileCache::removeLookupFile)
Review Comment:
The removal listener closes a `LookupFile` immediately on global weight/TTL eviction, but active lookups only hold the per-bucket read lock and do not pin their cache entries. A lookup in one table can therefore be reading a file when another table's insertion evicts and deletes it, causing a mid-lookup I/O failure. Evicted files need reference-counted/deferred close coordination with active lookups.
This issue also appears on line 55 of the same file.
##########
fluss-lake/fluss-lake-paimon/src/main/java/org/apache/fluss/lake/paimon/PaimonLakeStorage.java:
##########
@@ -55,13 +65,50 @@ public LakeSource<PaimonSplit> createLakeSource(TablePath tablePath) {
}
@Override
- public LakeTableLookuper createLakeTableLookuper(TablePath tablePath, LookuperContext context) {
- return new PaimonLakeTableLookuper(
- paimonConfig,
- tablePath,
- context.ioTmpDir(),
- context.tableConfig(),
- context.lookupCacheMaxDiskBytes(),
- context.diskWriteGuard());
+ public LakeTableLookupRuntime createLakeTableLookupRuntime(
+ String ioTmpDir, long lookupCacheMaxDiskBytes) {
+ return new PaimonLakeTableLookupRuntime(ioTmpDir, lookupCacheMaxDiskBytes);
+ }
+
+ /** Paimon lookup runtime sharing one I/O manager across table lookupers. */
+ private static final class PaimonLakeTableLookupRuntime implements LakeTableLookupRuntime {
+ private final IOManager ioManager;
+ private final SharedLookupFileCache lookupFileCache;
+
+ private PaimonLakeTableLookupRuntime(String ioTmpDir, long lookupCacheMaxDiskBytes) {
+ checkArgument(
+ lookupCacheMaxDiskBytes > 0, "lookupCacheMaxDiskBytes must be greater than 0.");
+ this.ioManager = IOManager.create(checkNotNull(ioTmpDir, "ioTmpDir must not be null."));
+ // ponytail: one runtime-wide retention; add a server option if this needs tuning.
+ this.lookupFileCache =
+ new SharedLookupFileCache(
+ CoreOptions.LOOKUP_CACHE_FILE_RETENTION.defaultValue(),
Review Comment:
This hard-codes the Paimon default retention and no longer honors a configured `lookup.cache-file-retention`. Before this change, the copied `FileStoreTable` carried the configured Paimon lookup options into `LocalTableQuery`; now all TabletServers use the default TTL regardless of lake configuration. Derive the shared cache retention from the configured Paimon options or expose an equivalent server setting.
##########
fluss-server/src/main/java/org/apache/fluss/server/replica/historical/HistoricalLakeLookupManager.java:
##########
@@ -264,6 +271,12 @@ public void close() {
lakeTableLookupers.invalidateAll();
lakeTableLookupers.cleanUp();
requiredLakeSnapshotIds.clear();
+ LakeTableLookupRuntime runtime;
+ synchronized (this) {
+ runtime = lookupRuntime;
+ lookupRuntime = null;
+ }
+ IOUtils.closeQuietly(runtime, "historical lake lookup runtime");
Review Comment:
Invalidating table lookupers defers their close while `activeLookups` is nonzero, but the shared runtime is closed immediately afterward. That closes the shared cache and IO manager while an active lookup can still be using them, defeating the manager's drain/deferred-close logic and making shutdown race with in-flight lookups. Close the runtime only after all cached lookupers have actually closed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [server] Add support of sharable file cache capacity for historical partition lookup [fluss]
Posted by "luoyuxia (via GitHub)" <gi...@apache.org>.
luoyuxia commented on code in PR #4113:
URL: https://github.com/apache/fluss/pull/4113#discussion_r4103737947
##########
fluss-lake/fluss-lake-paimon/src/main/java/org/apache/fluss/lake/paimon/lookup/PaimonLocalTableQuery.java:
##########
@@ -0,0 +1,316 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.fluss.lake.paimon.lookup;
+
+import org.apache.paimon.CoreOptions;
+import org.apache.paimon.FileStore;
+import org.apache.paimon.KeyValue;
+import org.apache.paimon.KeyValueFileStore;
+import org.apache.paimon.data.BinaryRow;
+import org.apache.paimon.data.InternalRow;
+import org.apache.paimon.data.serializer.InternalRowSerializer;
+import org.apache.paimon.data.serializer.InternalSerializers;
+import org.apache.paimon.data.serializer.RowCompactedSerializer;
+import org.apache.paimon.deletionvectors.DeletionVector;
+import org.apache.paimon.disk.IOManager;
+import org.apache.paimon.fs.FileIO;
+import org.apache.paimon.io.DataFileMeta;
+import org.apache.paimon.io.KeyValueFileReaderFactory;
+import org.apache.paimon.io.cache.CacheManager;
+import org.apache.paimon.lookup.LookupStoreFactory;
+import org.apache.paimon.mergetree.Levels;
+import org.apache.paimon.mergetree.LookupFile;
+import org.apache.paimon.mergetree.LookupLevels;
+import org.apache.paimon.mergetree.lookup.LookupSerializerFactory;
+import org.apache.paimon.mergetree.lookup.PersistValueProcessor;
+import org.apache.paimon.mergetree.lookup.RemoteLookupFileManager;
+import org.apache.paimon.operation.metrics.PartialLookupMetrics;
+import org.apache.paimon.options.Options;
+import org.apache.paimon.reader.RecordReader;
+import org.apache.paimon.shade.caffeine2.com.github.benmanes.caffeine.cache.Cache;
+import org.apache.paimon.table.FileStoreTable;
+import org.apache.paimon.table.query.TableQuery;
+import org.apache.paimon.types.RowType;
+import org.apache.paimon.utils.Filter;
+import org.apache.paimon.utils.KeyComparatorSupplier;
+import org.apache.paimon.utils.Preconditions;
+
+import javax.annotation.Nullable;
+
+import java.io.IOException;
+import java.util.Comparator;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.locks.ReadWriteLock;
+import java.util.concurrent.locks.ReentrantReadWriteLock;
+import java.util.function.Supplier;
+
+import static org.apache.paimon.lookup.LookupStoreFactory.bloomFilterBuilderFactory;
+import static org.apache.paimon.mergetree.LookupFile.localFilePrefix;
+
+/**
+ * Fluss-owned fork of Paimon 2.0.0's {@code LocalTableQuery}, using a shared namespaced lookup
+ * cache.
+ */
+final class PaimonLocalTableQuery implements TableQuery {
+
+ private final Map<BinaryRow, Map<Integer, BucketLookupState>> tableView;
+
+ private final CoreOptions options;
+
+ private final Supplier<Comparator<InternalRow>> keyComparatorSupplier;
+
+ private final KeyValueFileReaderFactory.Builder readerFactoryBuilder;
+
+ private final LookupStoreFactory lookupStoreFactory;
+
+ private final int startLevel;
+
+ private IOManager ioManager;
+
+ private final Cache<String, LookupFile> lookupFileCache;
+
+ private final RowType rowType;
+ private final RowType partitionType;
+ private final FileIO fileIO;
+
+ @Nullable private Filter<InternalRow> cacheRowFilter;
+
+ @Nullable private PartialLookupMetrics partialLookupMetrics;
+
+ PaimonLocalTableQuery(
+ FileStoreTable table,
+ SharedLookupFileCache sharedLookupFileCache,
+ String cacheNamespace) {
+ this.options = table.coreOptions();
+ this.lookupFileCache = sharedLookupFileCache.namespaced(cacheNamespace);
+ this.tableView = new ConcurrentHashMap<>();
+ FileStore<?> tableStore = table.store();
+ if (!(tableStore instanceof KeyValueFileStore)) {
+ throw new UnsupportedOperationException(
+ "Table Query only supports table with primary key.");
+ }
+ KeyValueFileStore store = (KeyValueFileStore) tableStore;
+
+ this.readerFactoryBuilder = store.newReaderFactoryBuilder();
+ this.rowType = table.schema().logicalRowType();
+ this.partitionType = table.schema().logicalPartitionType();
+ this.fileIO = table.fileIO();
+ RowType keyType = readerFactoryBuilder.keyType();
+ this.keyComparatorSupplier = new KeyComparatorSupplier(readerFactoryBuilder.keyType());
+ this.lookupStoreFactory =
+ LookupStoreFactory.create(
+ options,
+ new CacheManager(
+ options.lookupCacheMaxMemory(),
+ options.lookupCacheHighPrioPoolRatio()),
+ new RowCompactedSerializer(keyType).createSliceComparator());
+ startLevel = options.needLookup() ? 1 : 0;
+ }
+
+ public void refreshFiles(
+ BinaryRow partition,
+ int bucket,
+ List<DataFileMeta> beforeFiles,
+ List<DataFileMeta> dataFiles) {
+ // Both tableView and its nested bucket maps are ConcurrentHashMaps; this nested
+ // computeIfAbsent pattern relies on each map providing atomic insertion.
+ BucketLookupState state =
+ tableView
+ .computeIfAbsent(partition, k -> new ConcurrentHashMap<>())
+ .computeIfAbsent(bucket, k -> new BucketLookupState());
+ state.lock.writeLock().lock();
+ try {
+ if (state.lookupLevels == null) {
+ // Initial phase: ignore beforeFiles as they represent deletions from previous state
+ state.lookupLevels = createLookupLevels(partition, bucket, dataFiles);
+ } else {
+ state.lookupLevels.getLevels().update(beforeFiles, dataFiles);
+ }
+ } finally {
+ state.lock.writeLock().unlock();
+ }
+ }
+
+ private LookupLevels<KeyValue> createLookupLevels(
+ BinaryRow partition, int bucket, List<DataFileMeta> dataFiles) {
+ Levels levels = new Levels(keyComparatorSupplier.get(), dataFiles, options.numLevels());
+ // TODO pass DeletionVector factory
+ KeyValueFileReaderFactory factory =
+ readerFactoryBuilder.build(partition, bucket, DeletionVector.emptyFactory());
+ Options options = this.options.toConfiguration();
+
+ RowType readValueType = readerFactoryBuilder.readValueType();
+ LookupLevels<KeyValue> lookupLevels =
+ new LookupLevels<>(
+ schemaId -> readValueType,
+ 0L,
+ levels,
+ keyComparatorSupplier.get(),
+ readerFactoryBuilder.keyType(),
+ PersistValueProcessor.factory(readValueType),
+ LookupSerializerFactory.INSTANCE.get(),
+ file -> {
+ RecordReader<KeyValue> reader = factory.createRecordReader(file);
+ if (cacheRowFilter != null) {
+ reader =
+ reader.filter(
+ keyValue -> cacheRowFilter.test(keyValue.value()));
+ }
+ return reader;
+ },
+ file ->
+ Preconditions.checkNotNull(ioManager, "IOManager is required.")
+ .createChannel(
+ localFilePrefix(
+ partitionType, partition, bucket, file))
+ .getPathFile(),
+ lookupStoreFactory,
+ bloomFilterBuilderFactory(options),
+ lookupFileCache);
+
+ // Optimization - download lookup files if already persisted to object store
+ // We download these files if three conditions are met
+ // 1) lookup.remote-file.enabled is true - files are persisted in the first place
+ // 2) deletion-vectors.enabled is false - SSTables only contain row positions, not values,
+ // when DVs are enabled
+ // 3) The client is accessing the full data row, as opposed to a projection
+ // - The persisted remote SSTable files are created during compaction and hold the entire
+ // data row value
+ // - We could deserialize and project in memory, but we'll have to read much more data,
+ // not as clear of a win
+ boolean fullValueRead = readerFactoryBuilder.readValueType().equals(rowType);
+ if (this.options.lookupRemoteFileEnabled()
+ && !this.options.deletionVectorsEnabled()
+ && fullValueRead) {
+ // Calling the constructor tells `lookupLevels` to load remote files
+ new RemoteLookupFileManager<>(
+ fileIO,
+ factory.pathFactory(),
+ lookupLevels,
+ this.options.lookupRemoteLevelThreshold());
+ }
+
+ return lookupLevels;
+ }
+
+ @Nullable
+ @Override
+ public InternalRow lookup(BinaryRow partition, int bucket, InternalRow key) throws IOException {
+ PartialLookupMetrics currentMetrics = partialLookupMetrics;
+ LookupLevels.LookupContext context =
+ currentMetrics == null ? null : new LookupLevels.LookupContext();
+ try {
+ return lookup(partition, bucket, key, context);
+ } finally {
+ if (currentMetrics != null) {
+ currentMetrics.reportLookup(context != null && context.remoteAccessed());
+ }
+ }
+ }
+
+ @Nullable
+ private InternalRow lookup(
+ BinaryRow partition,
+ int bucket,
+ InternalRow key,
+ @Nullable LookupLevels.LookupContext context)
+ throws IOException {
+ Map<Integer, BucketLookupState> buckets = tableView.get(partition);
+ if (buckets == null || buckets.isEmpty()) {
+ return null;
+ }
+ BucketLookupState state = buckets.get(bucket);
+ if (state == null) {
+ return null;
+ }
+
+ state.lock.readLock().lock();
+ try {
+ LookupLevels<KeyValue> lookupLevels = state.lookupLevels;
+ if (lookupLevels == null) {
+ return null;
+ }
+
+ KeyValue kv = lookupLevels.lookup(key, startLevel, context);
+ if (kv == null || kv.valueKind().isRetract()) {
+ return null;
+ } else {
+ return kv.value();
+ }
+ } finally {
+ state.lock.readLock().unlock();
+ }
+ }
+
+ @Override
+ public PaimonLocalTableQuery withValueProjection(int[] projection) {
+ this.readerFactoryBuilder.withReadValueType(rowType.project(projection));
+ return this;
+ }
+
+ public PaimonLocalTableQuery withIOManager(IOManager ioManager) {
+ this.ioManager = ioManager;
+ return this;
+ }
+
+ public PaimonLocalTableQuery withCacheRowFilter(Filter<InternalRow> cacheRowFilter) {
+ this.cacheRowFilter = cacheRowFilter;
+ return this;
+ }
+
+ public PaimonLocalTableQuery withMetrics(@Nullable PartialLookupMetrics metrics) {
+ this.partialLookupMetrics = metrics;
+ return this;
+ }
+
+ @Override
+ public InternalRowSerializer createValueSerializer() {
+ return InternalSerializers.create(readerFactoryBuilder.readValueType());
+ }
+
+ @Override
+ public void close() throws IOException {
+ // ConcurrentHashMap iteration is weakly consistent. close is expected not to race with
+ // refreshFiles for the same query instance; callers may rebuild this query after close.
+ for (Map.Entry<BinaryRow, Map<Integer, BucketLookupState>> buckets : tableView.entrySet()) {
+ for (Map.Entry<Integer, BucketLookupState> bucket : buckets.getValue().entrySet()) {
+ BucketLookupState state = bucket.getValue();
+ state.lock.writeLock().lock();
+ try {
+ if (state.lookupLevels != null) {
+ state.lookupLevels.close();
+ }
+ } finally {
+ state.lock.writeLock().unlock();
+ }
+ }
+ }
+ lookupFileCache.invalidateAll();
Review Comment:
#3955 asks for cached-file ownership and eviction to be independent of the in-memory lookuper cache. Closing a lookuper here still removes all files in its namespace; `LookupLevels.close()` above also invalidates its owned entries.
For example, querying 11 small tables can trigger the 10-lookuper capacity limit and delete an entire table's local cache even when all 11 tables' files would fit within the shared disk budget. Querying the evicted table again then requires rebuilding those files. This leaves the lifecycle-decoupling part of #3955 unfinished.
Could we separate ordinary lookuper eviction from explicit table/schema/config invalidation, so the shared cache can retain reusable files across lookuper recreation? This also needs a reusable cache identity: `cacheNamespace(...)` currently includes a fresh `lookuperId` for each instance. Removing only this `invalidateAll()` would therefore be insufficient. A regression test with more than 10 tables whose files fit within the global budget would help verify the intended reuse.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org