You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@fluss.apache.org by "luoyuxia (via GitHub)" <gi...@apache.org> on 2026/08/31 01:38:19 UTC
[PR] [lake/paimon] Prevent concurrent local lookup false misses [fluss]
luoyuxia opened a new pull request, #4158:
URL: https://github.com/apache/fluss/pull/4158
<!--
Generated-by: Codex (GPT-5) following [the guidelines](https://github.com/apache/fluss/blob/main/AGENTS.md)
-->
### Purpose
Linked issue: [apache/paimon#9483](https://github.com/apache/paimon/issues/9483)
Paimon 2.0 allows lookups for different partition-buckets to run concurrently, but their local lookup-store readers share a comparator with mutable reader state. Concurrent lookups against different files can therefore return `null` for existing keys.
This PR adds a Fluss-side correctness workaround while the problem is addressed upstream. Calls to one `LocalTableQuery` are serialized, including the retry after refreshing stale files. Consequently, a cache hit waits while another lookup on the same table is downloading a local lookup file.
### Brief change log
- Serialize calls to `LocalTableQuery.lookup` with a dedicated per-lookuper lock.
- Add a regression test that submits cold-cache lookups for two different partitions concurrently.
- Update the compaction and snapshot-expiration test so it no longer assumes that a cache hit bypasses an in-progress lookup-file download.
### Tests
- `./mvnw -pl fluss-lake/fluss-lake-paimon -Dtest=PaimonLakeTableLookuperTest test`
- 11 tests passed.
- Checkstyle reported 0 violations.
- Spotless check passed.
- `./mvnw install -DskipTests -pl fluss-lake/fluss-lake-paimon -am`
### API and Format
No API or storage format changes.
### Documentation
No documentation changes. This is a correctness workaround for an upstream concurrency issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [lake/paimon] Prevent concurrent local lookup false misses [fluss]
Posted by "luoyuxia (via GitHub)" <gi...@apache.org>.
luoyuxia commented on code in PR #4158:
URL: https://github.com/apache/fluss/pull/4158#discussion_r3891320234
##########
fluss-lake/fluss-lake-paimon/src/main/java/org/apache/fluss/lake/paimon/lookup/PaimonLakeTableLookuper.java:
##########
@@ -339,18 +340,28 @@ private org.apache.paimon.data.BinaryRow getKey(byte[] key, LookupContext contex
throws IOException {
List<DataFileMeta> filesBeforeLookup = initializeFiles(partition, bucket);
try {
- return localTableQuery.lookup(partition, bucket, key);
+ return lookupLocalTable(partition, bucket, key);
} catch (IOException firstError) {
refreshFilesIfUnchanged(partition, bucket, filesBeforeLookup);
try {
- return localTableQuery.lookup(partition, bucket, key);
+ return lookupLocalTable(partition, bucket, key);
} catch (IOException retryError) {
retryError.addSuppressed(firstError);
throw retryError;
}
}
}
+ private @Nullable org.apache.paimon.data.InternalRow lookupLocalTable(
Review Comment:
Saidly yes. It'll force to seralize the lookup operations.
I think we should introduce the dedicated localTableQuery in fluss side to bypass this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [lake/paimon] Prevent concurrent local lookup false misses [fluss]
Posted by "luoyuxia (via GitHub)" <gi...@apache.org>.
luoyuxia commented on code in PR #4158:
URL: https://github.com/apache/fluss/pull/4158#discussion_r3894421486
##########
fluss-lake/fluss-lake-paimon/src/main/java/org/apache/fluss/lake/paimon/lookup/PaimonLakeTableLookuper.java:
##########
@@ -339,18 +340,28 @@ private org.apache.paimon.data.BinaryRow getKey(byte[] key, LookupContext contex
throws IOException {
List<DataFileMeta> filesBeforeLookup = initializeFiles(partition, bucket);
try {
- return localTableQuery.lookup(partition, bucket, key);
+ return lookupLocalTable(partition, bucket, key);
} catch (IOException firstError) {
refreshFilesIfUnchanged(partition, bucket, filesBeforeLookup);
try {
- return localTableQuery.lookup(partition, bucket, key);
+ return lookupLocalTable(partition, bucket, key);
} catch (IOException retryError) {
retryError.addSuppressed(firstError);
throw retryError;
}
}
}
+ private @Nullable org.apache.paimon.data.InternalRow lookupLocalTable(
Review Comment:
Added an explicit TODO in 9e93d116f. It tracks removing the lock once apache/paimon#9483 is fixed in the Paimon version used by Fluss, with a Fluss-side `LocalTableQuery` implementation following #4113 as the fallback if concurrent lookup is needed sooner.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [lake/paimon] Prevent concurrent local lookup false misses [fluss]
Posted by "luoyuxia (via GitHub)" <gi...@apache.org>.
luoyuxia merged PR #4158:
URL: https://github.com/apache/fluss/pull/4158
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [lake/paimon] Prevent concurrent local lookup false misses [fluss]
Posted by "zuston (via GitHub)" <gi...@apache.org>.
zuston commented on code in PR #4158:
URL: https://github.com/apache/fluss/pull/4158#discussion_r3891302928
##########
fluss-lake/fluss-lake-paimon/src/main/java/org/apache/fluss/lake/paimon/lookup/PaimonLakeTableLookuper.java:
##########
@@ -339,18 +340,28 @@ private org.apache.paimon.data.BinaryRow getKey(byte[] key, LookupContext contex
throws IOException {
List<DataFileMeta> filesBeforeLookup = initializeFiles(partition, bucket);
try {
- return localTableQuery.lookup(partition, bucket, key);
+ return lookupLocalTable(partition, bucket, key);
} catch (IOException firstError) {
refreshFilesIfUnchanged(partition, bucket, filesBeforeLookup);
try {
- return localTableQuery.lookup(partition, bucket, key);
+ return lookupLocalTable(partition, bucket, key);
} catch (IOException retryError) {
retryError.addSuppressed(firstError);
throw retryError;
}
}
}
+ private @Nullable org.apache.paimon.data.InternalRow lookupLocalTable(
Review Comment:
Does this mean we have to seralize the lookup operations to ensure thread safe?
Is this possible to introduce the dedicated localTableQuery in fluss side to bypass this ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [lake/paimon] Prevent concurrent local lookup false misses [fluss]
Posted by "zuston (via GitHub)" <gi...@apache.org>.
zuston commented on code in PR #4158:
URL: https://github.com/apache/fluss/pull/4158#discussion_r3891332601
##########
fluss-lake/fluss-lake-paimon/src/main/java/org/apache/fluss/lake/paimon/lookup/PaimonLakeTableLookuper.java:
##########
@@ -339,18 +340,28 @@ private org.apache.paimon.data.BinaryRow getKey(byte[] key, LookupContext contex
throws IOException {
List<DataFileMeta> filesBeforeLookup = initializeFiles(partition, bucket);
try {
- return localTableQuery.lookup(partition, bucket, key);
+ return lookupLocalTable(partition, bucket, key);
} catch (IOException firstError) {
refreshFilesIfUnchanged(partition, bucket, filesBeforeLookup);
try {
- return localTableQuery.lookup(partition, bucket, key);
+ return lookupLocalTable(partition, bucket, key);
} catch (IOException retryError) {
retryError.addSuppressed(firstError);
throw retryError;
}
}
}
+ private @Nullable org.apache.paimon.data.InternalRow lookupLocalTable(
Review Comment:
+1 . this is necessary, I have tried in https://github.com/apache/fluss/pull/4113
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org