You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "AHeise (via GitHub)" <gi...@apache.org> on 2026/10/07 07:44:13 UTC

[PR] [FLINK-38569][table-planner] Don't record a ReadingMetadata ability without metadata keys [flink]

AHeise opened a new pull request, #29415:
URL: https://github.com/apache/flink/pull/29415

   ## What is the purpose of the change
   
   For a source that implements `SupportsReadingMetadata`, projection push down records a `ReadingMetadata` ability even if the query reads no metadata column:
   
   ```sql
   CREATE TABLE src (a INT, b BIGINT) WITH ('connector' = 'values');
   SELECT a FROM src WHERE b > 1;
   ```
   
   ```
   TableSourceScan(table=[[default_catalog, default_database, src, filter=[...], project=[a], metadata=[]]], fields=[a])
   ```
   
   and the compiled plan carries a no-op `{"type" : "ReadingMetadata", "metadataKeys" : [ ], ...}`.
   
   ## Brief change log
   
   - `PushProjectIntoTableSourceScanRule` records the `ReadingMetadataSpec` only if a metadata key is projected, or if the scan already has one that must be narrowed (the table declares metadata columns, see `DynamicSourceUtils#validateAndApplyMetadata`).
   - `ScanReuser` skips the spec if the reused scans use no metadata key. It rebuilds the source without the earlier projection and metadata specs, so there is nothing to narrow.
   
   | Table / query | Before | After |
   |---|---|---|
   | physical columns only | `metadata=[]` | no `ReadingMetadata` |
   | metadata column declared, not projected | `metadata=[]` | `metadata=[]` (narrows the earlier spec) |
   | metadata column projected | `metadata=[m]` | `metadata=[m]` |
   
   ## Verifying this change
   
   This change is already covered by existing tests:
   
   - Expected plans of 53 plan test classes and the `*JsonPlanTest`s are regenerated; the diff only drops `metadata=[]` from scan digests and empty `ReadingMetadata` entries from JSON plans.
   - Plans where an earlier spec is narrowed keep `metadata=[]`, e.g. `AggregateTest#testCountStartWithNestedRow`, `PushLocalAggIntoTableSourceScanRuleTest#testCanPushDownLocalAggWithAuxGrouping`.
   - All restore tests pass unchanged: the stored compiled plans that contain an empty `ReadingMetadata` entry still restore.
   - `ExplainTest` golden files and the `EXPLAIN` examples in the docs are updated the same way.
   - Resolves the TODO in `DeltaJoinUtil`; `ReadingMetadataSpec` stays a supported spec there, since narrowing still produces one with empty keys.
   
   ## Does this pull request potentially affect one of the following parts:
   
     - Dependencies (does it add or upgrade a dependency): no
     - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: no
     - The serializers: no
     - The runtime per-record code paths (performance sensitive): no
     - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no
     - The S3 file system connector: no
   
   - A source that supports metadata no longer receives an `applyReadableMetadata` call with an empty key list when the table declares no metadata columns.
   - Plan tests of external connectors (e.g. Kafka) lose `metadata=[]` in the same way.
   - `ScanReuserUtils#containsRexNodeSpecAfterProjection` counts a `ReadingMetadataSpec` as a projection, so a metadata source without projection push down can now be reused in cases where it was not before. Without metadata the field indices do not shift, so this is safe.
   
   ## Documentation
   
     - Does this pull request introduce a new feature? no
     - If yes, how is the feature documented? not applicable
   
   ---
   
   ##### Was generative AI tooling used to co-author this PR?
   
   - [X] Yes (please specify the tool below)
   
   Generated-by: Claude Code (Claude Opus 5.5)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-38569][table-planner] Don't record a ReadingMetadata ability without metadata keys [flink]

Posted by "AHeise (via GitHub)" <gi...@apache.org>.
AHeise commented on code in PR #29415:
URL: https://github.com/apache/flink/pull/29415#discussion_r4206493440


##########
flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/plan/rules/logical/PushProjectIntoTableSourceScanRule.java:
##########
@@ -200,6 +199,11 @@ private boolean supportsProjectionPushDown(DynamicTableSource tableSource) {
         return tableSource instanceof SupportsProjectionPushDown;
     }
 
+    private static boolean hasReadingMetadataSpec(TableSourceTable source) {
+        return Arrays.stream(source.abilitySpecs())
+                .anyMatch(spec -> spec instanceof ReadingMetadataSpec);
+    }
+
     private boolean supportsMetadata(DynamicTableSource tableSource) {

Review Comment:
   Done, also `supportsNestedProjection`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-38569][table-planner] Don't record a ReadingMetadata ability without metadata keys [flink]

Posted by "snuyanzin (via GitHub)" <gi...@apache.org>.
snuyanzin commented on code in PR #29415:
URL: https://github.com/apache/flink/pull/29415#discussion_r4206337502


##########
flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/plan/rules/logical/PushProjectIntoTableSourceScanRule.java:
##########
@@ -200,6 +199,11 @@ private boolean supportsProjectionPushDown(DynamicTableSource tableSource) {
         return tableSource instanceof SupportsProjectionPushDown;
     }
 
+    private static boolean hasReadingMetadataSpec(TableSourceTable source) {
+        return Arrays.stream(source.abilitySpecs())
+                .anyMatch(spec -> spec instanceof ReadingMetadataSpec);
+    }
+
     private boolean supportsMetadata(DynamicTableSource tableSource) {

Review Comment:
   nit: since you are here probably `supportsMetadata` and `supportsProjectionPushDown` above could be also static



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-38569][table-planner] Don't record a ReadingMetadata ability without metadata keys [flink]

Posted by "AHeise (via GitHub)" <gi...@apache.org>.
AHeise commented on PR #29415:
URL: https://github.com/apache/flink/pull/29415#issuecomment-6033600881

   cc @fresh-borzoni: sorry I saw too late you already worked on this in #27378. Compared to your approach, the difference is in the FLINK-23911 case you pointed out: the empty spec is still recorded when it narrows an earlier one, because the source is rebuilt from the specs at translation and on restore. Reviews welcome.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-38569][table-planner] Don't record a ReadingMetadata ability without metadata keys [flink]

Posted by "AHeise (via GitHub)" <gi...@apache.org>.
AHeise merged PR #29415:
URL: https://github.com/apache/flink/pull/29415


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-38569][table-planner] Don't record a ReadingMetadata ability without metadata keys [flink]

Posted by "AHeise (via GitHub)" <gi...@apache.org>.
AHeise commented on code in PR #29415:
URL: https://github.com/apache/flink/pull/29415#discussion_r4206635519


##########
flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/plan/rules/logical/PushProjectIntoTableSourceScanRule.java:
##########
@@ -348,7 +352,10 @@ private RowType performPushDown(
                             .map(col -> col.getMetadataKey().orElse(col.getName()))
                             .collect(Collectors.toList());
 
-            abilitySpecs.add(new ReadingMetadataSpec(projectedMetadataKeys, newProducedType));
+            // An earlier ReadingMetadataSpec must be narrowed even if no metadata is projected.
+            if (!projectedMetadataKeys.isEmpty() || hasReadingMetadataSpec(source)) {
+                abilitySpecs.add(new ReadingMetadataSpec(projectedMetadataKeys, newProducedType));
+            }

Review Comment:
   Checked all `SupportsReadingMetadata` table sources in the apache connector repos (kafka, aws/kinesis, pulsar, cdc ×7, rocketmq ×2, http) plus filesystem and `ExternalDynamicSource` in core. All but one start with the physical type and no metadata keys, so a missing call equals an empty one.
   
   The exception is `HttpLookupTableSource`: it sets `producedDataType` only in `applyReadableMetadata`, and the lookup function drops the row instead of returning null enrichment columns for a failed call with continue-on-error if it is unset. It already does that today when no projection is pushed, so I filed FLINK-40940 to fix the connector rather than keep the planner calling it.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-38569][table-planner] Don't record a ReadingMetadata ability without metadata keys [flink]

Posted by "flinkbot (via GitHub)" <gi...@apache.org>.
flinkbot commented on PR #29415:
URL: https://github.com/apache/flink/pull/29415#issuecomment-6033686563

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "7da0b1eb3f34adc4c7f19325b26dbf5a48206fac",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "7da0b1eb3f34adc4c7f19325b26dbf5a48206fac",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 7da0b1eb3f34adc4c7f19325b26dbf5a48206fac UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-38569][table-planner] Don't record a ReadingMetadata ability without metadata keys [flink]

Posted by "snuyanzin (via GitHub)" <gi...@apache.org>.
snuyanzin commented on code in PR #29415:
URL: https://github.com/apache/flink/pull/29415#discussion_r4206318964


##########
flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/plan/rules/logical/PushProjectIntoTableSourceScanRule.java:
##########
@@ -348,7 +352,10 @@ private RowType performPushDown(
                             .map(col -> col.getMetadataKey().orElse(col.getName()))
                             .collect(Collectors.toList());
 
-            abilitySpecs.add(new ReadingMetadataSpec(projectedMetadataKeys, newProducedType));
+            // An earlier ReadingMetadataSpec must be narrowed even if no metadata is projected.
+            if (!projectedMetadataKeys.isEmpty() || hasReadingMetadataSpec(source)) {
+                abilitySpecs.add(new ReadingMetadataSpec(projectedMetadataKeys, newProducedType));
+            }

Review Comment:
   I hope connectors do not assume it is always present 🙈 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org