You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "dawidwys (via GitHub)" <gi...@apache.org> on 2023/09/13 14:18:44 UTC
[GitHub] [flink] dawidwys opened a new pull request, #23412: [FLINK-33083] Properly apply ReadingMetadataSpec for a TableSourceScan
dawidwys opened a new pull request, #23412:
URL: https://github.com/apache/flink/pull/23412
## What is the purpose of the change
The PR fixes creating a TableSourceScan to also include a proper ReadingMetadataSpec which has been applied on the source.
## Brief change log
* revert the solution added in #22894 because it works around the outcome of the bug rather than fixes the bug
* create a ReadingMetadataSpec when applying it on the `TableSource` in a `TableSourceScan`
## Verifying this change
* tests added in #22894 should still pass
* added a dedicated test `TableSourceJsonPlanITCase#testReadingMetadataWithProjectionPushDownDisabled`
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): (yes / **no**)
- The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes / **no**)
- The serializers: (yes / **no** / don't know)
- The runtime per-record code paths (performance sensitive): (yes / **no** / don't know)
- Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (yes / **no** / don't know)
- The S3 file system connector: (yes / **no** / don't know)
## Documentation
- Does this pull request introduce a new feature? (yes / **no**)
- If yes, how is the feature documented? (**not applicable** / docs / JavaDocs / not documented)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] dawidwys commented on a diff in pull request #23412: [FLINK-33083] Properly apply ReadingMetadataSpec for a TableSourceScan
Posted by "dawidwys (via GitHub)" <gi...@apache.org>.
dawidwys commented on code in PR #23412:
URL: https://github.com/apache/flink/pull/23412#discussion_r1325658492
##########
flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/plan/nodes/exec/spec/DynamicTableSourceSpec.java:
##########
@@ -95,29 +92,19 @@ private DynamicTableSource getTableSource(FlinkContext context, FlinkTypeFactory
FactoryUtil.createDynamicTableSource(
factory,
contextResolvedTable.getIdentifier(),
- resolvedCatalogTable,
+ contextResolvedTable.getResolvedTable(),
loadOptionsFromCatalogTable(contextResolvedTable, context),
context.getTableConfig(),
context.getClassLoader(),
contextResolvedTable.isTemporary());
- // validate DynamicSource and apply Metadata
- DynamicSourceUtils.prepareDynamicSource(
- contextResolvedTable.getIdentifier().toString(),
- resolvedCatalogTable,
- tableSource,
- false,
- context.getTableConfig().getConfiguration());
if (sourceAbilities != null) {
- // Note: use DynamicSourceUtils.createProducedType to produce the type info so that
- // keep consistent with sql2Rel phase which also called the method producing
- // deterministic format (PHYSICAL COLUMNS + METADATA COLUMNS) when converts a given
- // DynamicTableSource to a RelNode.
- // TODO should do a refactor(e.g., add serialized input type info into each
- // SourceAbilitySpec so as to avoid this implicit logic dependency)
RowType newProducedType =
- DynamicSourceUtils.createProducedType(
- contextResolvedTable.getResolvedSchema(), tableSource);
+ (RowType)
+ contextResolvedTable
+ .getResolvedSchema()
+ .toSourceRowDataType()
+ .getLogicalType();
Review Comment:
Not sure, if I understand your question.
Before applying any specs the table has a schema as described by
```
contextResolvedTable.getResolvedSchema()
.toSourceRowDataType()
.getLogicalType()
```
Only once the `ReadingMetadataSpec` is applied it will change the type to `spec.getProducedType().get()`. In this particular case of reading metadata in a `TableSourceScan` it is equal to `DynamicSourceUtils.createProducedType(contextResolvedTable.getResolvedSchema(), tableSource);`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] twalthr commented on a diff in pull request #23412: [FLINK-33083] Properly apply ReadingMetadataSpec for a TableSourceScan
Posted by "twalthr (via GitHub)" <gi...@apache.org>.
twalthr commented on code in PR #23412:
URL: https://github.com/apache/flink/pull/23412#discussion_r1326853075
##########
flink-table/flink-table-planner/src/test/resources/org/apache/flink/table/planner/plan/batch/sql/TableSourceTest.xml:
##########
@@ -109,13 +109,13 @@ FROM T
<![CDATA[
LogicalProject(id=[$0], nested1=[$1.nested1], results=[+(+($1.nested1.value, $1.nested2.num), $2)])
+- LogicalProject(id=[$0], deepNested=[$1], metadata_1=[$2], metadata_2=[$3])
- +- LogicalTableScan(table=[[default_catalog, default_database, T]])
+ +- LogicalTableScan(table=[[default_catalog, default_database, T, metadata=[metadata_1, metadata_2]]])
]]>
</Resource>
<Resource name="optimized exec plan">
<![CDATA[
Calc(select=[id, deepNested_nested1 AS nested1, ((deepNested_nested1.value + deepNested_nested2_num) + metadata_1) AS results])
-+- TableSourceScan(table=[[default_catalog, default_database, T, project=[id, deepNested_nested1, deepNested_nested2_num], metadata=[metadata_1]]], fields=[id, deepNested_nested1, deepNested_nested2_num, metadata_1])
++- TableSourceScan(table=[[default_catalog, default_database, T, metadata=[metadata_1, metadata_2], project=[id, deepNested_nested1, deepNested_nested2_num], metadata=[metadata_1]]], fields=[id, deepNested_nested1, deepNested_nested2_num, metadata_1])
Review Comment:
This looks wrong. The source should use only `metadata_1`. Looking at path revisions of this class, this was the case last year.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] dawidwys commented on a diff in pull request #23412: [FLINK-33083] Properly apply ReadingMetadataSpec for a TableSourceScan
Posted by "dawidwys (via GitHub)" <gi...@apache.org>.
dawidwys commented on code in PR #23412:
URL: https://github.com/apache/flink/pull/23412#discussion_r1330056518
##########
flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/plan/nodes/exec/spec/DynamicTableSourceSpec.java:
##########
@@ -95,29 +92,19 @@ private DynamicTableSource getTableSource(FlinkContext context, FlinkTypeFactory
FactoryUtil.createDynamicTableSource(
factory,
contextResolvedTable.getIdentifier(),
- resolvedCatalogTable,
+ contextResolvedTable.getResolvedTable(),
loadOptionsFromCatalogTable(contextResolvedTable, context),
context.getTableConfig(),
context.getClassLoader(),
contextResolvedTable.isTemporary());
- // validate DynamicSource and apply Metadata
- DynamicSourceUtils.prepareDynamicSource(
- contextResolvedTable.getIdentifier().toString(),
- resolvedCatalogTable,
- tableSource,
- false,
- context.getTableConfig().getConfiguration());
if (sourceAbilities != null) {
- // Note: use DynamicSourceUtils.createProducedType to produce the type info so that
- // keep consistent with sql2Rel phase which also called the method producing
- // deterministic format (PHYSICAL COLUMNS + METADATA COLUMNS) when converts a given
- // DynamicTableSource to a RelNode.
- // TODO should do a refactor(e.g., add serialized input type info into each
- // SourceAbilitySpec so as to avoid this implicit logic dependency)
RowType newProducedType =
- DynamicSourceUtils.createProducedType(
- contextResolvedTable.getResolvedSchema(), tableSource);
+ (RowType)
+ contextResolvedTable
+ .getResolvedSchema()
+ .toSourceRowDataType()
+ .getLogicalType();
Review Comment:
I updated the PR to use `toPhysicalRowDataType` as the initial type.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] twalthr commented on a diff in pull request #23412: [FLINK-33083] Properly apply ReadingMetadataSpec for a TableSourceScan
Posted by "twalthr (via GitHub)" <gi...@apache.org>.
twalthr commented on code in PR #23412:
URL: https://github.com/apache/flink/pull/23412#discussion_r1326845652
##########
flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/plan/nodes/exec/spec/DynamicTableSourceSpec.java:
##########
@@ -95,29 +92,19 @@ private DynamicTableSource getTableSource(FlinkContext context, FlinkTypeFactory
FactoryUtil.createDynamicTableSource(
factory,
contextResolvedTable.getIdentifier(),
- resolvedCatalogTable,
+ contextResolvedTable.getResolvedTable(),
loadOptionsFromCatalogTable(contextResolvedTable, context),
context.getTableConfig(),
context.getClassLoader(),
contextResolvedTable.isTemporary());
- // validate DynamicSource and apply Metadata
- DynamicSourceUtils.prepareDynamicSource(
- contextResolvedTable.getIdentifier().toString(),
- resolvedCatalogTable,
- tableSource,
- false,
- context.getTableConfig().getConfiguration());
if (sourceAbilities != null) {
- // Note: use DynamicSourceUtils.createProducedType to produce the type info so that
- // keep consistent with sql2Rel phase which also called the method producing
- // deterministic format (PHYSICAL COLUMNS + METADATA COLUMNS) when converts a given
- // DynamicTableSource to a RelNode.
- // TODO should do a refactor(e.g., add serialized input type info into each
- // SourceAbilitySpec so as to avoid this implicit logic dependency)
RowType newProducedType =
- DynamicSourceUtils.createProducedType(
- contextResolvedTable.getResolvedSchema(), tableSource);
+ (RowType)
+ contextResolvedTable
+ .getResolvedSchema()
+ .toSourceRowDataType()
+ .getLogicalType();
Review Comment:
By just looking at the code, I would say `newProducedType` should be initialized with only physical columns. Not sure what the tests say to this. But `toSourceRowDataType` includes also computed column which makes no sense for the DynamicTableSource as the source doesn't know this concept. After the spec is applied in the loop, the type changes to `PHYSICAL COLUMNS + METADATA COLUMNS`. In any case, the producedType should always contain what comes out of the source.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] dawidwys commented on a diff in pull request #23412: [FLINK-33083] Properly apply ReadingMetadataSpec for a TableSourceScan
Posted by "dawidwys (via GitHub)" <gi...@apache.org>.
dawidwys commented on code in PR #23412:
URL: https://github.com/apache/flink/pull/23412#discussion_r1330056041
##########
flink-table/flink-table-planner/src/test/resources/org/apache/flink/table/planner/plan/batch/sql/TableSourceTest.xml:
##########
@@ -109,13 +109,13 @@ FROM T
<![CDATA[
LogicalProject(id=[$0], nested1=[$1.nested1], results=[+(+($1.nested1.value, $1.nested2.num), $2)])
+- LogicalProject(id=[$0], deepNested=[$1], metadata_1=[$2], metadata_2=[$3])
- +- LogicalTableScan(table=[[default_catalog, default_database, T]])
+ +- LogicalTableScan(table=[[default_catalog, default_database, T, metadata=[metadata_1, metadata_2]]])
]]>
</Resource>
<Resource name="optimized exec plan">
<![CDATA[
Calc(select=[id, deepNested_nested1 AS nested1, ((deepNested_nested1.value + deepNested_nested2_num) + metadata_1) AS results])
-+- TableSourceScan(table=[[default_catalog, default_database, T, project=[id, deepNested_nested1, deepNested_nested2_num], metadata=[metadata_1]]], fields=[id, deepNested_nested1, deepNested_nested2_num, metadata_1])
++- TableSourceScan(table=[[default_catalog, default_database, T, metadata=[metadata_1, metadata_2], project=[id, deepNested_nested1, deepNested_nested2_num], metadata=[metadata_1]]], fields=[id, deepNested_nested1, deepNested_nested2_num, metadata_1])
Review Comment:
I updated the PR so that we leave only a single instance of each spec
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] swuferhong commented on a diff in pull request #23412: [FLINK-33083] Properly apply ReadingMetadataSpec for a TableSourceScan
Posted by "swuferhong (via GitHub)" <gi...@apache.org>.
swuferhong commented on code in PR #23412:
URL: https://github.com/apache/flink/pull/23412#discussion_r1325238554
##########
flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/plan/nodes/exec/spec/DynamicTableSourceSpec.java:
##########
@@ -95,29 +92,19 @@ private DynamicTableSource getTableSource(FlinkContext context, FlinkTypeFactory
FactoryUtil.createDynamicTableSource(
factory,
contextResolvedTable.getIdentifier(),
- resolvedCatalogTable,
+ contextResolvedTable.getResolvedTable(),
loadOptionsFromCatalogTable(contextResolvedTable, context),
context.getTableConfig(),
context.getClassLoader(),
contextResolvedTable.isTemporary());
- // validate DynamicSource and apply Metadata
- DynamicSourceUtils.prepareDynamicSource(
- contextResolvedTable.getIdentifier().toString(),
- resolvedCatalogTable,
- tableSource,
- false,
- context.getTableConfig().getConfiguration());
if (sourceAbilities != null) {
- // Note: use DynamicSourceUtils.createProducedType to produce the type info so that
- // keep consistent with sql2Rel phase which also called the method producing
- // deterministic format (PHYSICAL COLUMNS + METADATA COLUMNS) when converts a given
- // DynamicTableSource to a RelNode.
- // TODO should do a refactor(e.g., add serialized input type info into each
- // SourceAbilitySpec so as to avoid this implicit logic dependency)
RowType newProducedType =
- DynamicSourceUtils.createProducedType(
- contextResolvedTable.getResolvedSchema(), tableSource);
+ (RowType)
+ contextResolvedTable
+ .getResolvedSchema()
+ .toSourceRowDataType()
+ .getLogicalType();
Review Comment:
Hi, @dawidwys, thanks for your PR. I have a small question that why we change the spec column index from `PHYSICAL COLUMNS + METADATA COLUMNS` to the original user-defined schema order here ? If we want to change this order, more code that uses source specs need to be modified, such as `ScanReuser.reuseDuplicatedScan`, cc @twalthr .
##########
flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/plan/nodes/exec/spec/DynamicTableSourceSpec.java:
##########
@@ -95,29 +92,19 @@ private DynamicTableSource getTableSource(FlinkContext context, FlinkTypeFactory
FactoryUtil.createDynamicTableSource(
factory,
contextResolvedTable.getIdentifier(),
- resolvedCatalogTable,
+ contextResolvedTable.getResolvedTable(),
loadOptionsFromCatalogTable(contextResolvedTable, context),
context.getTableConfig(),
context.getClassLoader(),
contextResolvedTable.isTemporary());
- // validate DynamicSource and apply Metadata
- DynamicSourceUtils.prepareDynamicSource(
- contextResolvedTable.getIdentifier().toString(),
- resolvedCatalogTable,
- tableSource,
- false,
- context.getTableConfig().getConfiguration());
if (sourceAbilities != null) {
- // Note: use DynamicSourceUtils.createProducedType to produce the type info so that
- // keep consistent with sql2Rel phase which also called the method producing
- // deterministic format (PHYSICAL COLUMNS + METADATA COLUMNS) when converts a given
- // DynamicTableSource to a RelNode.
- // TODO should do a refactor(e.g., add serialized input type info into each
- // SourceAbilitySpec so as to avoid this implicit logic dependency)
RowType newProducedType =
- DynamicSourceUtils.createProducedType(
- contextResolvedTable.getResolvedSchema(), tableSource);
+ (RowType)
+ contextResolvedTable
+ .getResolvedSchema()
+ .toSourceRowDataType()
+ .getLogicalType();
Review Comment:
cc @twalthr
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] swuferhong commented on a diff in pull request #23412: [FLINK-33083] Properly apply ReadingMetadataSpec for a TableSourceScan
Posted by "swuferhong (via GitHub)" <gi...@apache.org>.
swuferhong commented on code in PR #23412:
URL: https://github.com/apache/flink/pull/23412#discussion_r1325700289
##########
flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/plan/nodes/exec/spec/DynamicTableSourceSpec.java:
##########
@@ -95,29 +92,19 @@ private DynamicTableSource getTableSource(FlinkContext context, FlinkTypeFactory
FactoryUtil.createDynamicTableSource(
factory,
contextResolvedTable.getIdentifier(),
- resolvedCatalogTable,
+ contextResolvedTable.getResolvedTable(),
loadOptionsFromCatalogTable(contextResolvedTable, context),
context.getTableConfig(),
context.getClassLoader(),
contextResolvedTable.isTemporary());
- // validate DynamicSource and apply Metadata
- DynamicSourceUtils.prepareDynamicSource(
- contextResolvedTable.getIdentifier().toString(),
- resolvedCatalogTable,
- tableSource,
- false,
- context.getTableConfig().getConfiguration());
if (sourceAbilities != null) {
- // Note: use DynamicSourceUtils.createProducedType to produce the type info so that
- // keep consistent with sql2Rel phase which also called the method producing
- // deterministic format (PHYSICAL COLUMNS + METADATA COLUMNS) when converts a given
- // DynamicTableSource to a RelNode.
- // TODO should do a refactor(e.g., add serialized input type info into each
- // SourceAbilitySpec so as to avoid this implicit logic dependency)
RowType newProducedType =
- DynamicSourceUtils.createProducedType(
- contextResolvedTable.getResolvedSchema(), tableSource);
+ (RowType)
+ contextResolvedTable
+ .getResolvedSchema()
+ .toSourceRowDataType()
+ .getLogicalType();
Review Comment:
Hi, @dawidwys . For this case:
```
String ddl = "CREATE TABLE MyTable1 (\n"
+ " metadata_0 int METADATA VIRTUAL,\n"
+ " a0 int,\n"
+ " a1 int,\n"
+ " a2 int,\n"
+ " ts STRING,\n "
+ " rowtime as TO_TIMESTAMP(`ts`),\n"
+ " WATERMARK FOR rowtime AS rowtime - INTERVAL '1' SECOND\n"
+ ") WITH ( .....
+ ")";
```
which `METADATA` column defined as first column.
```
DynamicSourceUtils.createProducedType(contextResolvedTable.getResolvedSchema(), tableSource)
```
will get different schema order with
```
contextResolvedTable.getResolvedSchema()
.toSourceRowDataType()
.getLogicalType()
```
IMO, I think it's better to use the origin order than convert the order to `PHYSICAL COLUMNS + METADATA COLUMNS`, but this need @twalthr to confirm.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] dawidwys commented on pull request #23412: [FLINK-33083] Properly apply ReadingMetadataSpec for a TableSourceScan
Posted by "dawidwys (via GitHub)" <gi...@apache.org>.
dawidwys commented on PR #23412:
URL: https://github.com/apache/flink/pull/23412#issuecomment-1717730004
fyi @lincoln-lil @swuferhong I think we should be a bit more careful when fixing those kind of bugs so that we don't introduce hacks and workarounds
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] swuferhong commented on a diff in pull request #23412: [FLINK-33083] Properly apply ReadingMetadataSpec for a TableSourceScan
Posted by "swuferhong (via GitHub)" <gi...@apache.org>.
swuferhong commented on code in PR #23412:
URL: https://github.com/apache/flink/pull/23412#discussion_r1325700289
##########
flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/plan/nodes/exec/spec/DynamicTableSourceSpec.java:
##########
@@ -95,29 +92,19 @@ private DynamicTableSource getTableSource(FlinkContext context, FlinkTypeFactory
FactoryUtil.createDynamicTableSource(
factory,
contextResolvedTable.getIdentifier(),
- resolvedCatalogTable,
+ contextResolvedTable.getResolvedTable(),
loadOptionsFromCatalogTable(contextResolvedTable, context),
context.getTableConfig(),
context.getClassLoader(),
contextResolvedTable.isTemporary());
- // validate DynamicSource and apply Metadata
- DynamicSourceUtils.prepareDynamicSource(
- contextResolvedTable.getIdentifier().toString(),
- resolvedCatalogTable,
- tableSource,
- false,
- context.getTableConfig().getConfiguration());
if (sourceAbilities != null) {
- // Note: use DynamicSourceUtils.createProducedType to produce the type info so that
- // keep consistent with sql2Rel phase which also called the method producing
- // deterministic format (PHYSICAL COLUMNS + METADATA COLUMNS) when converts a given
- // DynamicTableSource to a RelNode.
- // TODO should do a refactor(e.g., add serialized input type info into each
- // SourceAbilitySpec so as to avoid this implicit logic dependency)
RowType newProducedType =
- DynamicSourceUtils.createProducedType(
- contextResolvedTable.getResolvedSchema(), tableSource);
+ (RowType)
+ contextResolvedTable
+ .getResolvedSchema()
+ .toSourceRowDataType()
+ .getLogicalType();
Review Comment:
Hi, @dawidwys . For this case:
```
String ddl = "CREATE TABLE MyTable1 (\n"
+ " metadata_0 int METADATA VIRTUAL,\n"
+ " a0 int,\n"
+ " a1 int,\n"
+ " a2 int,\n"
+ " ts STRING,\n "
+ " rowtime as TO_TIMESTAMP(`ts`),\n"
+ " WATERMARK FOR rowtime AS rowtime - INTERVAL '1' SECOND\n"
+ ") WITH ( .....
+ ")";
```
which `METADATA` column defined as first column.
```
DynamicSourceUtils.createProducedType(contextResolvedTable.getResolvedSchema(), tableSource)
```
will get different schema order with
```
contextResolvedTable.getResolvedSchema()
.toSourceRowDataType()
.getLogicalType()
```
IMO, I think it's better to use the origin order than convert the order to `PHYSICAL COLUMNS + METADATA COLUMNS`, but this need @twalthr to confirm it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] swuferhong commented on a diff in pull request #23412: [FLINK-33083] Properly apply ReadingMetadataSpec for a TableSourceScan
Posted by "swuferhong (via GitHub)" <gi...@apache.org>.
swuferhong commented on code in PR #23412:
URL: https://github.com/apache/flink/pull/23412#discussion_r1325320081
##########
flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/plan/nodes/exec/spec/DynamicTableSourceSpec.java:
##########
@@ -95,29 +92,19 @@ private DynamicTableSource getTableSource(FlinkContext context, FlinkTypeFactory
FactoryUtil.createDynamicTableSource(
factory,
contextResolvedTable.getIdentifier(),
- resolvedCatalogTable,
+ contextResolvedTable.getResolvedTable(),
loadOptionsFromCatalogTable(contextResolvedTable, context),
context.getTableConfig(),
context.getClassLoader(),
contextResolvedTable.isTemporary());
- // validate DynamicSource and apply Metadata
- DynamicSourceUtils.prepareDynamicSource(
- contextResolvedTable.getIdentifier().toString(),
- resolvedCatalogTable,
- tableSource,
- false,
- context.getTableConfig().getConfiguration());
if (sourceAbilities != null) {
- // Note: use DynamicSourceUtils.createProducedType to produce the type info so that
- // keep consistent with sql2Rel phase which also called the method producing
- // deterministic format (PHYSICAL COLUMNS + METADATA COLUMNS) when converts a given
- // DynamicTableSource to a RelNode.
- // TODO should do a refactor(e.g., add serialized input type info into each
- // SourceAbilitySpec so as to avoid this implicit logic dependency)
RowType newProducedType =
- DynamicSourceUtils.createProducedType(
- contextResolvedTable.getResolvedSchema(), tableSource);
+ (RowType)
+ contextResolvedTable
+ .getResolvedSchema()
+ .toSourceRowDataType()
+ .getLogicalType();
Review Comment:
cc @twalthr
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] swuferhong commented on a diff in pull request #23412: [FLINK-33083] Properly apply ReadingMetadataSpec for a TableSourceScan
Posted by "swuferhong (via GitHub)" <gi...@apache.org>.
swuferhong commented on code in PR #23412:
URL: https://github.com/apache/flink/pull/23412#discussion_r1325238554
##########
flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/plan/nodes/exec/spec/DynamicTableSourceSpec.java:
##########
@@ -95,29 +92,19 @@ private DynamicTableSource getTableSource(FlinkContext context, FlinkTypeFactory
FactoryUtil.createDynamicTableSource(
factory,
contextResolvedTable.getIdentifier(),
- resolvedCatalogTable,
+ contextResolvedTable.getResolvedTable(),
loadOptionsFromCatalogTable(contextResolvedTable, context),
context.getTableConfig(),
context.getClassLoader(),
contextResolvedTable.isTemporary());
- // validate DynamicSource and apply Metadata
- DynamicSourceUtils.prepareDynamicSource(
- contextResolvedTable.getIdentifier().toString(),
- resolvedCatalogTable,
- tableSource,
- false,
- context.getTableConfig().getConfiguration());
if (sourceAbilities != null) {
- // Note: use DynamicSourceUtils.createProducedType to produce the type info so that
- // keep consistent with sql2Rel phase which also called the method producing
- // deterministic format (PHYSICAL COLUMNS + METADATA COLUMNS) when converts a given
- // DynamicTableSource to a RelNode.
- // TODO should do a refactor(e.g., add serialized input type info into each
- // SourceAbilitySpec so as to avoid this implicit logic dependency)
RowType newProducedType =
- DynamicSourceUtils.createProducedType(
- contextResolvedTable.getResolvedSchema(), tableSource);
+ (RowType)
+ contextResolvedTable
+ .getResolvedSchema()
+ .toSourceRowDataType()
+ .getLogicalType();
Review Comment:
Hi, @dawidwys, thanks for your PR. I have a small question that why we change the spec column index from `PHYSICAL COLUMNS + METADATA COLUMNS` to the original user-defined schema order here ? If we want to change this order, more code that uses source specs need to be modified, such as `ScanReuser.reuseDuplicatedScan`, cc @tillrohrmann .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] twalthr commented on a diff in pull request #23412: [FLINK-33083] Properly apply ReadingMetadataSpec for a TableSourceScan
Posted by "twalthr (via GitHub)" <gi...@apache.org>.
twalthr commented on code in PR #23412:
URL: https://github.com/apache/flink/pull/23412#discussion_r1326845652
##########
flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/plan/nodes/exec/spec/DynamicTableSourceSpec.java:
##########
@@ -95,29 +92,19 @@ private DynamicTableSource getTableSource(FlinkContext context, FlinkTypeFactory
FactoryUtil.createDynamicTableSource(
factory,
contextResolvedTable.getIdentifier(),
- resolvedCatalogTable,
+ contextResolvedTable.getResolvedTable(),
loadOptionsFromCatalogTable(contextResolvedTable, context),
context.getTableConfig(),
context.getClassLoader(),
contextResolvedTable.isTemporary());
- // validate DynamicSource and apply Metadata
- DynamicSourceUtils.prepareDynamicSource(
- contextResolvedTable.getIdentifier().toString(),
- resolvedCatalogTable,
- tableSource,
- false,
- context.getTableConfig().getConfiguration());
if (sourceAbilities != null) {
- // Note: use DynamicSourceUtils.createProducedType to produce the type info so that
- // keep consistent with sql2Rel phase which also called the method producing
- // deterministic format (PHYSICAL COLUMNS + METADATA COLUMNS) when converts a given
- // DynamicTableSource to a RelNode.
- // TODO should do a refactor(e.g., add serialized input type info into each
- // SourceAbilitySpec so as to avoid this implicit logic dependency)
RowType newProducedType =
- DynamicSourceUtils.createProducedType(
- contextResolvedTable.getResolvedSchema(), tableSource);
+ (RowType)
+ contextResolvedTable
+ .getResolvedSchema()
+ .toSourceRowDataType()
+ .getLogicalType();
Review Comment:
By just looking at the code, I would say `newProducedType` should be initialized with only physical columns. Not sure what the tests say to this. But `toSourceRowDataType` includes also computed column which makes no sense for the DynamicTableSource as the source doesn't know this concept. After the spec is applied in the loop, the type changes to `PHYSICAL COLUMNS + METADATA COLUMNS`. In any case, the producedType should always contain what comes out of the source (not scan).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] dawidwys merged pull request #23412: [FLINK-33083] Properly apply ReadingMetadataSpec for a TableSourceScan
Posted by "dawidwys (via GitHub)" <gi...@apache.org>.
dawidwys merged PR #23412:
URL: https://github.com/apache/flink/pull/23412
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #23412: [FLINK-33083] Properly apply ReadingMetadataSpec for a TableSourceScan
Posted by "flinkbot (via GitHub)" <gi...@apache.org>.
flinkbot commented on PR #23412:
URL: https://github.com/apache/flink/pull/23412#issuecomment-1717739419
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "75b651538a1fa083c211ab8c4020822590b89043",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "75b651538a1fa083c211ab8c4020822590b89043",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 75b651538a1fa083c211ab8c4020822590b89043 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] twalthr commented on a diff in pull request #23412: [FLINK-33083] Properly apply ReadingMetadataSpec for a TableSourceScan
Posted by "twalthr (via GitHub)" <gi...@apache.org>.
twalthr commented on code in PR #23412:
URL: https://github.com/apache/flink/pull/23412#discussion_r1326845652
##########
flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/plan/nodes/exec/spec/DynamicTableSourceSpec.java:
##########
@@ -95,29 +92,19 @@ private DynamicTableSource getTableSource(FlinkContext context, FlinkTypeFactory
FactoryUtil.createDynamicTableSource(
factory,
contextResolvedTable.getIdentifier(),
- resolvedCatalogTable,
+ contextResolvedTable.getResolvedTable(),
loadOptionsFromCatalogTable(contextResolvedTable, context),
context.getTableConfig(),
context.getClassLoader(),
contextResolvedTable.isTemporary());
- // validate DynamicSource and apply Metadata
- DynamicSourceUtils.prepareDynamicSource(
- contextResolvedTable.getIdentifier().toString(),
- resolvedCatalogTable,
- tableSource,
- false,
- context.getTableConfig().getConfiguration());
if (sourceAbilities != null) {
- // Note: use DynamicSourceUtils.createProducedType to produce the type info so that
- // keep consistent with sql2Rel phase which also called the method producing
- // deterministic format (PHYSICAL COLUMNS + METADATA COLUMNS) when converts a given
- // DynamicTableSource to a RelNode.
- // TODO should do a refactor(e.g., add serialized input type info into each
- // SourceAbilitySpec so as to avoid this implicit logic dependency)
RowType newProducedType =
- DynamicSourceUtils.createProducedType(
- contextResolvedTable.getResolvedSchema(), tableSource);
+ (RowType)
+ contextResolvedTable
+ .getResolvedSchema()
+ .toSourceRowDataType()
+ .getLogicalType();
Review Comment:
By just looking at the code, I would say `newProducedType` should be initialized with only physical columns. Not sure what the tests say to this. But `toSourceRowDataType` includes also computed column which makes no sense for the DynamicTableSource as the source doesn't know this concept. After the spec is applied in the loop, the type changes for `PHYSICAL COLUMNS + METADATA COLUMNS`. In any case, the producedType should always contain what comes out of the source.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] twalthr commented on a diff in pull request #23412: [FLINK-33083] Properly apply ReadingMetadataSpec for a TableSourceScan
Posted by "twalthr (via GitHub)" <gi...@apache.org>.
twalthr commented on code in PR #23412:
URL: https://github.com/apache/flink/pull/23412#discussion_r1326861376
##########
flink-table/flink-table-planner/src/test/resources/org/apache/flink/table/planner/plan/batch/sql/TableSourceTest.xml:
##########
@@ -109,13 +109,13 @@ FROM T
<![CDATA[
LogicalProject(id=[$0], nested1=[$1.nested1], results=[+(+($1.nested1.value, $1.nested2.num), $2)])
+- LogicalProject(id=[$0], deepNested=[$1], metadata_1=[$2], metadata_2=[$3])
- +- LogicalTableScan(table=[[default_catalog, default_database, T]])
+ +- LogicalTableScan(table=[[default_catalog, default_database, T, metadata=[metadata_1, metadata_2]]])
]]>
</Resource>
<Resource name="optimized exec plan">
<![CDATA[
Calc(select=[id, deepNested_nested1 AS nested1, ((deepNested_nested1.value + deepNested_nested2_num) + metadata_1) AS results])
-+- TableSourceScan(table=[[default_catalog, default_database, T, project=[id, deepNested_nested1, deepNested_nested2_num], metadata=[metadata_1]]], fields=[id, deepNested_nested1, deepNested_nested2_num, metadata_1])
++- TableSourceScan(table=[[default_catalog, default_database, T, metadata=[metadata_1, metadata_2], project=[id, deepNested_nested1, deepNested_nested2_num], metadata=[metadata_1]]], fields=[id, deepNested_nested1, deepNested_nested2_num, metadata_1])
Review Comment:
I just noticed that the parenthesis are also not correct. What is going on here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] dawidwys commented on a diff in pull request #23412: [FLINK-33083] Properly apply ReadingMetadataSpec for a TableSourceScan
Posted by "dawidwys (via GitHub)" <gi...@apache.org>.
dawidwys commented on code in PR #23412:
URL: https://github.com/apache/flink/pull/23412#discussion_r1326868368
##########
flink-table/flink-table-planner/src/test/resources/org/apache/flink/table/planner/plan/batch/sql/TableSourceTest.xml:
##########
@@ -109,13 +109,13 @@ FROM T
<![CDATA[
LogicalProject(id=[$0], nested1=[$1.nested1], results=[+(+($1.nested1.value, $1.nested2.num), $2)])
+- LogicalProject(id=[$0], deepNested=[$1], metadata_1=[$2], metadata_2=[$3])
- +- LogicalTableScan(table=[[default_catalog, default_database, T]])
+ +- LogicalTableScan(table=[[default_catalog, default_database, T, metadata=[metadata_1, metadata_2]]])
]]>
</Resource>
<Resource name="optimized exec plan">
<![CDATA[
Calc(select=[id, deepNested_nested1 AS nested1, ((deepNested_nested1.value + deepNested_nested2_num) + metadata_1) AS results])
-+- TableSourceScan(table=[[default_catalog, default_database, T, project=[id, deepNested_nested1, deepNested_nested2_num], metadata=[metadata_1]]], fields=[id, deepNested_nested1, deepNested_nested2_num, metadata_1])
++- TableSourceScan(table=[[default_catalog, default_database, T, metadata=[metadata_1, metadata_2], project=[id, deepNested_nested1, deepNested_nested2_num], metadata=[metadata_1]]], fields=[id, deepNested_nested1, deepNested_nested2_num, metadata_1])
Review Comment:
> I just noticed that the parenthesis are also not correct. What is going on here?
Where are they wrong?
> This looks wrong. The source should use only metadata_1. Looking at path revisions of this class, this was the case last year.
Bear in mind there are two `ReadingMetadataSpec`. One applied during a scan and one during projection pushdown. That's also the case when creating the `ExecNode` tree that `applyReadableMetadata` is called twice.
The spec from projection pushdown uses only `metadata_1`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org