You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "yuzelin (via GitHub)" <gi...@apache.org> on 2025/01/03 07:24:22 UTC
[PR] [refactor] Refactor LookupJoin: add logs for troubleshoot and polish DynamicPartitionLoader [paimon]
yuzelin opened a new pull request, #4828:
URL: https://github.com/apache/paimon/pull/4828
<!-- Please specify the module before the PR name: [core] ... or [flink] ... -->
### Purpose
<!-- Linking this pull request to the issue -->
Linked issue: close #xxx
<!-- What is the purpose of the change -->
### Tests
<!-- List UT and IT cases to verify this change -->
### API and Format
<!-- Does this change affect API or storage format -->
### Documentation
<!-- Does this change introduce a new feature -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [refactor] Refactor LookupJoin: add logs for troubleshoot and polish DynamicPartitionLoader [paimon]
Posted by "LinMingQiang (via GitHub)" <gi...@apache.org>.
LinMingQiang commented on PR #4828:
URL: https://github.com/apache/paimon/pull/4828#issuecomment-2572226189
+1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [refactor] Refactor LookupJoin: add logs for troubleshoot and polish DynamicPartitionLoader [paimon]
Posted by "LsomeYeah (via GitHub)" <gi...@apache.org>.
LsomeYeah commented on code in PR #4828:
URL: https://github.com/apache/paimon/pull/4828#discussion_r1903919798
##########
paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/lookup/FileStoreLookupFunction.java:
##########
@@ -168,12 +163,20 @@ private void open() throws Exception {
int[] projection = projectFields.stream().mapToInt(fieldNames::indexOf).toArray();
FileStoreTable storeTable = (FileStoreTable) table;
+ LOG.info("Creating lookup table for {}.", table.name());
if (options.get(LOOKUP_CACHE_MODE) == LookupCacheMode.AUTO
&& new HashSet<>(table.primaryKeys()).equals(new HashSet<>(joinKeys))) {
+ String message =
+ String.format(
+ "{}={} and primary keys equal to join keys, trying to create PrimaryKeyPartialLookupTable.",
+ LOOKUP_CACHE_MODE.key(),
+ LookupCacheMode.AUTO);
Review Comment:
It seems the 'message' variable is not used anywhere else.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [refactor] Refactor LookupJoin: add logs for troubleshoot and polish DynamicPartitionLoader [paimon]
Posted by "wwj6591812 (via GitHub)" <gi...@apache.org>.
wwj6591812 commented on PR #4828:
URL: https://github.com/apache/paimon/pull/4828#issuecomment-2568879380
+1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [refactor] Refactor LookupJoin: add logs for troubleshoot and polish DynamicPartitionLoader [paimon]
Posted by "LinMingQiang (via GitHub)" <gi...@apache.org>.
LinMingQiang commented on code in PR #4828:
URL: https://github.com/apache/paimon/pull/4828#discussion_r1901554476
##########
paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/lookup/FileStoreLookupFunction.java:
##########
@@ -206,7 +200,7 @@ private void open() throws Exception {
partitionLoader.checkRefresh();
List<BinaryRow> partitions = partitionLoader.partitions();
if (!partitions.isEmpty()) {
- lookupTable.specificPartitionFilter(createSpecificPartFilter(partitions));
+ lookupTable.specificPartitionFilter(partitionLoader.createSpecificPartFilter());
}
}
Review Comment:
Suggestion > Add a log where LookupTable is created: Which type of LookupTable will be created and why..
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [refactor] Refactor LookupJoin: add logs for troubleshoot and polish DynamicPartitionLoader [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on PR #4828:
URL: https://github.com/apache/paimon/pull/4828#issuecomment-2575492888
+1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [refactor] Refactor LookupJoin: add logs for troubleshoot and polish DynamicPartitionLoader [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi merged PR #4828:
URL: https://github.com/apache/paimon/pull/4828
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [refactor] Refactor LookupJoin: add logs for troubleshoot and polish DynamicPartitionLoader [paimon]
Posted by "LinMingQiang (via GitHub)" <gi...@apache.org>.
LinMingQiang commented on code in PR #4828:
URL: https://github.com/apache/paimon/pull/4828#discussion_r1903595333
##########
paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/lookup/FileStoreLookupFunction.java:
##########
@@ -183,6 +180,9 @@ private void open() throws Exception {
path,
joinKeys,
getRequireCachedBucketIds());
+ LOG.info(
+ "Created lookup table for {}. Type is PrimaryKeyPartialLookupTable with LocalQueryExecutor.",
+ table.name());
} catch (UnsupportedOperationException ignore2) {
}
}
Review Comment:
Logging the reason why it is not supported. (Cus table bucket mode is not HASH_FIXED, so will be use FullCacheLookupTable).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [refactor] Refactor LookupJoin: add logs for troubleshoot and polish DynamicPartitionLoader [paimon]
Posted by "LinMingQiang (via GitHub)" <gi...@apache.org>.
LinMingQiang commented on code in PR #4828:
URL: https://github.com/apache/paimon/pull/4828#discussion_r1903595333
##########
paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/lookup/FileStoreLookupFunction.java:
##########
@@ -183,6 +180,9 @@ private void open() throws Exception {
path,
joinKeys,
getRequireCachedBucketIds());
+ LOG.info(
+ "Created lookup table for {}. Type is PrimaryKeyPartialLookupTable with LocalQueryExecutor.",
+ table.name());
} catch (UnsupportedOperationException ignore2) {
}
}
Review Comment:
Logging the reason why it is not supported. (Cus table bucket mode is not HASH_FIXED).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org