You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "xuyangzhong (via GitHub)" <gi...@apache.org> on 2024/11/25 05:57:00 UTC
[PR] [FLINK-36688][table-planner] Sort metadata keys when reusing source [flink]
xuyangzhong opened a new pull request, #25681:
URL: https://github.com/apache/flink/pull/25681
## What is the purpose of the change
Sort the metadata keys when reusing source.
## Brief change log
- *Sort metadata keys in `ScanReuser`*
- *Add tests to verity this pr*
## Verifying this change
New tests are added to verify this fix.
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): no
- The public API, i.e., is any changed class annotated with `@Public(Evolving)`: no
- The serializers: no
- The runtime per-record code paths (performance sensitive): no
- Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no
- The S3 file system connector: no
## Documentation
- Does this pull request introduce a new feature? no
- If yes, how is the feature documented?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-36688][table-planner] Sort metadata keys when reusing source [flink]
Posted by "flinkbot (via GitHub)" <gi...@apache.org>.
flinkbot commented on PR #25681:
URL: https://github.com/apache/flink/pull/25681#issuecomment-2496935245
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "212cbc6067ce7fe29e0e08cdfdb1125531f5e8c4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "212cbc6067ce7fe29e0e08cdfdb1125531f5e8c4",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 212cbc6067ce7fe29e0e08cdfdb1125531f5e8c4 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-36688][table-planner] Sort metadata keys when reusing source [flink]
Posted by "snuyanzin (via GitHub)" <gi...@apache.org>.
snuyanzin commented on PR #25681:
URL: https://github.com/apache/flink/pull/25681#issuecomment-2513069651
@davidradl thanks for the review, do you have anything in your mind what else could be improved?
I'm asking since I'm going to merge it in coming days
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-36688][table-planner] Sort metadata keys when reusing source [flink]
Posted by "davidradl (via GitHub)" <gi...@apache.org>.
davidradl commented on PR #25681:
URL: https://github.com/apache/flink/pull/25681#issuecomment-2507637171
Reviewed by Chi on 28/11/24. Asked submitter questions
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-36688][table-planner] Sort metadata keys when reusing source [flink]
Posted by "davidradl (via GitHub)" <gi...@apache.org>.
davidradl commented on PR #25681:
URL: https://github.com/apache/flink/pull/25681#issuecomment-2517512507
> @davidradl thanks for the review, do you have anything in your mind what else could be improved? I'm asking since I'm going to merge it in coming days
@snuyanzin looks good to me
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-36688][table-planner] Sort metadata keys when reusing source [flink]
Posted by "davidradl (via GitHub)" <gi...@apache.org>.
davidradl commented on code in PR #25681:
URL: https://github.com/apache/flink/pull/25681#discussion_r1862411403
##########
flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/plan/reuse/ScanReuser.java:
##########
@@ -165,7 +166,7 @@ public List<RelNode> reuseDuplicatedScan(List<RelNode> relNodes) {
}
int[][] allProjectFields = allProjectFieldSet.toArray(new int[0][]);
- List<String> allMetaKeys = new ArrayList<>(allMetaKeySet);
+ List<String> allMetaKeys = sortMetadataKeys(allMetaKeySet, pickTable.tableSource());
Review Comment:
NIT: method name could be `enforceMetadataKeyOrder` or `copyMetadataKeyOrder`; sort implies there is a sort order like ascending.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-36688][table-planner] Sort metadata keys when reusing source [flink]
Posted by "snuyanzin (via GitHub)" <gi...@apache.org>.
snuyanzin commented on code in PR #25681:
URL: https://github.com/apache/flink/pull/25681#discussion_r1856602333
##########
flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/plan/reuse/ScanReuserUtils.java:
##########
@@ -270,6 +273,16 @@ public static List<String> metadataKeys(TableSourceTable source) {
: meta.getMetadataKeys();
}
+ public static List<String> sortMetadataKeys(
+ Set<String> allUsedMetadataKeys, DynamicTableSource source) {
+ List<String> allOrderedMetadataKeysFromTable =
+ new ArrayList<>(extractMetadataMap(source).keySet());
+
Review Comment:
nit: can we reuse `extractMetadataMap(source).keySet()` directly rather than creating a new object
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-36688][table-planner] Sort metadata keys when reusing source [flink]
Posted by "snuyanzin (via GitHub)" <gi...@apache.org>.
snuyanzin merged PR #25681:
URL: https://github.com/apache/flink/pull/25681
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org