You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by GitBox <gi...@apache.org> on 2020/06/15 06:57:37 UTC
[GitHub] [flink] dawidwys opened a new pull request #12649: [FLINK-18286] Implement type inference for GET/FLATTEN
dawidwys opened a new pull request #12649:
URL: https://github.com/apache/flink/pull/12649
## What is the purpose of the change
This PR implements type inference for GET/FLATTEN expressions. Additionally it aligns the behaviour between Table API & SQL. To achieve that it fixes handling nullability of nested columns in operators such as DOT, ITEM and nested fields coming directly from a Table.
## Verifying this change
Added tests in
* `InputTypeStrategiesTest` for `COMPOSITE` strategy
* `TypeStrategiesTest` for `GET` strategy
* `CompositeTypeAccessExpressionITCase` IT tests for accessing nested fields in both SQL & Table API
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): (yes / **no**)
- The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes / **no**)
- The serializers: (yes / **no** / don't know)
- The runtime per-record code paths (performance sensitive): (yes / **no** / don't know)
- Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: (yes / **no** / don't know)
- The S3 file system connector: (yes / **no** / don't know)
## Documentation
- Does this pull request introduce a new feature? (yes / **no**)
- If yes, how is the feature documented? (**not applicable** / docs / JavaDocs / not documented)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12649: [FLINK-18286] Implement type inference for GET/FLATTEN
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #12649:
URL: https://github.com/apache/flink/pull/12649#issuecomment-643957887
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ba5bc62416b82d8e6f1be2c848242de2d3c546e5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3489",
"triggerID" : "ba5bc62416b82d8e6f1be2c848242de2d3c546e5",
"triggerType" : "PUSH"
}, {
"hash" : "c38f0da0bd042779c6c9a096e305cb40e8b1e367",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3531",
"triggerID" : "c38f0da0bd042779c6c9a096e305cb40e8b1e367",
"triggerType" : "PUSH"
}, {
"hash" : "b3dd92d0c41cee2472bb521eec89741c42bc05cd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3580",
"triggerID" : "b3dd92d0c41cee2472bb521eec89741c42bc05cd",
"triggerType" : "PUSH"
}, {
"hash" : "326222e8eb85270297876eba22ffda57ebe592c3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3639",
"triggerID" : "326222e8eb85270297876eba22ffda57ebe592c3",
"triggerType" : "PUSH"
}, {
"hash" : "67905cf92819be26e49afbcf38794e340cca8e9c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3677",
"triggerID" : "67905cf92819be26e49afbcf38794e340cca8e9c",
"triggerType" : "PUSH"
}, {
"hash" : "4588093f97836f4f04c7c2c9992ea5049bf71c4c",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3694",
"triggerID" : "4588093f97836f4f04c7c2c9992ea5049bf71c4c",
"triggerType" : "PUSH"
}, {
"hash" : "43596996d023eb47c9589f95d8baf8a3cee83016",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3774",
"triggerID" : "43596996d023eb47c9589f95d8baf8a3cee83016",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4588093f97836f4f04c7c2c9992ea5049bf71c4c Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3694)
* 43596996d023eb47c9589f95d8baf8a3cee83016 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3774)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12649: [FLINK-18286] Implement type inference for GET/FLATTEN
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #12649:
URL: https://github.com/apache/flink/pull/12649#issuecomment-643957887
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ba5bc62416b82d8e6f1be2c848242de2d3c546e5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3489",
"triggerID" : "ba5bc62416b82d8e6f1be2c848242de2d3c546e5",
"triggerType" : "PUSH"
}, {
"hash" : "c38f0da0bd042779c6c9a096e305cb40e8b1e367",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3531",
"triggerID" : "c38f0da0bd042779c6c9a096e305cb40e8b1e367",
"triggerType" : "PUSH"
}, {
"hash" : "b3dd92d0c41cee2472bb521eec89741c42bc05cd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3580",
"triggerID" : "b3dd92d0c41cee2472bb521eec89741c42bc05cd",
"triggerType" : "PUSH"
}, {
"hash" : "326222e8eb85270297876eba22ffda57ebe592c3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3639",
"triggerID" : "326222e8eb85270297876eba22ffda57ebe592c3",
"triggerType" : "PUSH"
}, {
"hash" : "67905cf92819be26e49afbcf38794e340cca8e9c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3677",
"triggerID" : "67905cf92819be26e49afbcf38794e340cca8e9c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 67905cf92819be26e49afbcf38794e340cca8e9c Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3677)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] dawidwys edited a comment on pull request #12649: [FLINK-18286] Implement type inference for GET/FLATTEN
Posted by GitBox <gi...@apache.org>.
dawidwys edited a comment on pull request #12649:
URL: https://github.com/apache/flink/pull/12649#issuecomment-644565539
Let me explain what the changes are about. They are not trying to fix the nullability of the record type in all operators, but in my opinion it actually improves the behaviour described in the thread you posted (thank you for it, it is actually a good read).
First of all we do very much want to respect the original nullability, that's the main purpose of those changes and also why we have a special handling in `FlinkTypeFactory:348`. We have that because Calcite actually modifies nested columns of a `ROW` type, if the type is nullable.
In calcite a type `ROW<i INT NOT NULL>` becomes `ROW<i INT>` See `org.apache.calcite.rel.type.RelDataTypeFactoryImpl#createTypeWithNullability:326-338`. This behaviour is especially problematic for STRUCTURED types, which correspond to java's pojos. Let's take a look at an example:
```
class Address {
int id;
String city;
}
```
such a pojo we would like to map to a `Address(id INT NOT NULL, city STRING)`, where the `Address` type itself (outer row) is nullable, but the nested `id` field is `NOT NULL`. This is not possible in Calcite by default. Thats why we have the changes in `org.apache.calcite.rel.type.RelDataTypeFactoryImpl#createTypeWithNullability:326-338`.
On the other hand if we access a nested field of a nullable composite type we do want to adjust the nullability, cause accessing a field of a null row should obviously produce null values. If we do
```
CREATE TABLE test (
address Address
);
// the type should be nullable INT, if the outer row is null we can not produce a NOT NULL id
SELECT t.address.id FROM test t;
```
BTW this behaviour is already present in `SqlDotOperator#inferReturnType`, but as far as I can tell that method is not used.
To summarize it again. I do very much want to respect the nullability of the record attributes. Thats the main goal of the changes. Calcite as of now does not respect the nullability of nested fields, which we had to fix in FLINK-16344. The changes here affect the return type of field accessors, not the record itself.
I hope it gives a better overview what are the changes. If you have any questions let me know, I will try to elaborate more.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #12649: [FLINK-18286] Implement type inference for GET/FLATTEN
Posted by GitBox <gi...@apache.org>.
flinkbot commented on pull request #12649:
URL: https://github.com/apache/flink/pull/12649#issuecomment-643957887
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ba5bc62416b82d8e6f1be2c848242de2d3c546e5",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ba5bc62416b82d8e6f1be2c848242de2d3c546e5",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ba5bc62416b82d8e6f1be2c848242de2d3c546e5 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12649: [FLINK-18286] Implement type inference for GET/FLATTEN
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #12649:
URL: https://github.com/apache/flink/pull/12649#issuecomment-643957887
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ba5bc62416b82d8e6f1be2c848242de2d3c546e5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3489",
"triggerID" : "ba5bc62416b82d8e6f1be2c848242de2d3c546e5",
"triggerType" : "PUSH"
}, {
"hash" : "c38f0da0bd042779c6c9a096e305cb40e8b1e367",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3531",
"triggerID" : "c38f0da0bd042779c6c9a096e305cb40e8b1e367",
"triggerType" : "PUSH"
}, {
"hash" : "b3dd92d0c41cee2472bb521eec89741c42bc05cd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3580",
"triggerID" : "b3dd92d0c41cee2472bb521eec89741c42bc05cd",
"triggerType" : "PUSH"
}, {
"hash" : "326222e8eb85270297876eba22ffda57ebe592c3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3639",
"triggerID" : "326222e8eb85270297876eba22ffda57ebe592c3",
"triggerType" : "PUSH"
}, {
"hash" : "67905cf92819be26e49afbcf38794e340cca8e9c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3677",
"triggerID" : "67905cf92819be26e49afbcf38794e340cca8e9c",
"triggerType" : "PUSH"
}, {
"hash" : "4588093f97836f4f04c7c2c9992ea5049bf71c4c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4588093f97836f4f04c7c2c9992ea5049bf71c4c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 67905cf92819be26e49afbcf38794e340cca8e9c Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3677)
* 4588093f97836f4f04c7c2c9992ea5049bf71c4c UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] dawidwys commented on pull request #12649: [FLINK-18286] Implement type inference for GET/FLATTEN
Posted by GitBox <gi...@apache.org>.
dawidwys commented on pull request #12649:
URL: https://github.com/apache/flink/pull/12649#issuecomment-644565539
Let me explain what the changes are about. They are not trying to fix the nullability of the record type in all operators, but in my opinion it actually improves the behaviour described in the thread you posted (thank you for it, it is actually a good read).
First of all we do very much want to respect the original nullability, that's the main purpose of those changes and also why we have a special handling in `FlinkTypeFactory:348`. We have that because Calcite actually modifies nested columns of a `ROW` type, if the type is nullable.
In calcite a type `ROW<i INT NOT NULL>` becomes `ROW<i INT>` See `org.apache.calcite.rel.type.RelDataTypeFactoryImpl#createTypeWithNullability:326-338`. This behaviour is especially problematic for STRUCTURED types, which correspond to java's pojos. Let's take a look at an example:
```
class Address {
int id;
String city;
}
```
such a pojo we would like to map to a `Address(id INT NOT NULL, city STRING)`, where the `Address` type itself (outer row) is nullable, but the nested `id` field is `NOT NULL`. This is not possible in Calcite by default. Thats why we have the changes in `org.apache.calcite.rel.type.RelDataTypeFactoryImpl#createTypeWithNullability:326-338`.
On the other hand if we access a nested field of a nullable composite type we do want to adjust the nullability, cause accessing a field of a null row should obviously produce null values. If we do
```
CREATE TABLE test (
address Address
);
// the type should be nullable INT, if the outer row is null we can not produce a not null id
SELECT t.address.id FROM test t;
```
BTW this behaviour is already present in `SqlDotOperator#inferReturnType`, but as far as I can tell that method is not used.
To summarize it again. I do very much want to respect the nullability of the record attributes. Thats the main goal of the changes. Calcite as of now does not respect the nullability of nested fields, which we had to fix in FLINK-16344. The changes here affect the return type of field accessors, not the record itself.
I hope it gives a better overview what are the changes. If you have any questions let me know, I will try to elaborate more.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12649: [FLINK-18286] Implement type inference for GET/FLATTEN
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #12649:
URL: https://github.com/apache/flink/pull/12649#issuecomment-643957887
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ba5bc62416b82d8e6f1be2c848242de2d3c546e5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3489",
"triggerID" : "ba5bc62416b82d8e6f1be2c848242de2d3c546e5",
"triggerType" : "PUSH"
}, {
"hash" : "c38f0da0bd042779c6c9a096e305cb40e8b1e367",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3531",
"triggerID" : "c38f0da0bd042779c6c9a096e305cb40e8b1e367",
"triggerType" : "PUSH"
}, {
"hash" : "b3dd92d0c41cee2472bb521eec89741c42bc05cd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3580",
"triggerID" : "b3dd92d0c41cee2472bb521eec89741c42bc05cd",
"triggerType" : "PUSH"
}, {
"hash" : "326222e8eb85270297876eba22ffda57ebe592c3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3639",
"triggerID" : "326222e8eb85270297876eba22ffda57ebe592c3",
"triggerType" : "PUSH"
}, {
"hash" : "67905cf92819be26e49afbcf38794e340cca8e9c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3677",
"triggerID" : "67905cf92819be26e49afbcf38794e340cca8e9c",
"triggerType" : "PUSH"
}, {
"hash" : "4588093f97836f4f04c7c2c9992ea5049bf71c4c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3694",
"triggerID" : "4588093f97836f4f04c7c2c9992ea5049bf71c4c",
"triggerType" : "PUSH"
}, {
"hash" : "43596996d023eb47c9589f95d8baf8a3cee83016",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3774",
"triggerID" : "43596996d023eb47c9589f95d8baf8a3cee83016",
"triggerType" : "PUSH"
}, {
"hash" : "93f58e49ad79331331d4d85bbe7720ecfe39f968",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4559",
"triggerID" : "93f58e49ad79331331d4d85bbe7720ecfe39f968",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 93f58e49ad79331331d4d85bbe7720ecfe39f968 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4559)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] dawidwys commented on pull request #12649: [FLINK-18286] Implement type inference for GET/FLATTEN
Posted by GitBox <gi...@apache.org>.
dawidwys commented on pull request #12649:
URL: https://github.com/apache/flink/pull/12649#issuecomment-659371624
Some of the changes were contributed to Calcite (changes that made us copy some of Calcite classes). See: https://github.com/apache/calcite/commit/e2942feef4d52c577a46cc3f2b92fa462035bb20
Therefore I will merge it once I get a green build. Unless there are objections.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] danny0405 commented on pull request #12649: [FLINK-18286] Implement type inference for GET/FLATTEN
Posted by GitBox <gi...@apache.org>.
danny0405 commented on pull request #12649:
URL: https://github.com/apache/flink/pull/12649#issuecomment-644630624
Got you ~ The conclusion in the JIRA issue is synced with your needs: the nullability of structure type should not affect its members. I would try to fire a fix in Calcite, there may needs more discussion there ~
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12649: [FLINK-18286] Implement type inference for GET/FLATTEN
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #12649:
URL: https://github.com/apache/flink/pull/12649#issuecomment-643957887
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ba5bc62416b82d8e6f1be2c848242de2d3c546e5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3489",
"triggerID" : "ba5bc62416b82d8e6f1be2c848242de2d3c546e5",
"triggerType" : "PUSH"
}, {
"hash" : "c38f0da0bd042779c6c9a096e305cb40e8b1e367",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3531",
"triggerID" : "c38f0da0bd042779c6c9a096e305cb40e8b1e367",
"triggerType" : "PUSH"
}, {
"hash" : "b3dd92d0c41cee2472bb521eec89741c42bc05cd",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3580",
"triggerID" : "b3dd92d0c41cee2472bb521eec89741c42bc05cd",
"triggerType" : "PUSH"
}, {
"hash" : "326222e8eb85270297876eba22ffda57ebe592c3",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3639",
"triggerID" : "326222e8eb85270297876eba22ffda57ebe592c3",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b3dd92d0c41cee2472bb521eec89741c42bc05cd Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3580)
* 326222e8eb85270297876eba22ffda57ebe592c3 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3639)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12649: [FLINK-18286] Implement type inference for GET/FLATTEN
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #12649:
URL: https://github.com/apache/flink/pull/12649#issuecomment-643957887
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ba5bc62416b82d8e6f1be2c848242de2d3c546e5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3489",
"triggerID" : "ba5bc62416b82d8e6f1be2c848242de2d3c546e5",
"triggerType" : "PUSH"
}, {
"hash" : "c38f0da0bd042779c6c9a096e305cb40e8b1e367",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3531",
"triggerID" : "c38f0da0bd042779c6c9a096e305cb40e8b1e367",
"triggerType" : "PUSH"
}, {
"hash" : "b3dd92d0c41cee2472bb521eec89741c42bc05cd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3580",
"triggerID" : "b3dd92d0c41cee2472bb521eec89741c42bc05cd",
"triggerType" : "PUSH"
}, {
"hash" : "326222e8eb85270297876eba22ffda57ebe592c3",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3639",
"triggerID" : "326222e8eb85270297876eba22ffda57ebe592c3",
"triggerType" : "PUSH"
}, {
"hash" : "67905cf92819be26e49afbcf38794e340cca8e9c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "67905cf92819be26e49afbcf38794e340cca8e9c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 326222e8eb85270297876eba22ffda57ebe592c3 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3639)
* 67905cf92819be26e49afbcf38794e340cca8e9c UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] dawidwys edited a comment on pull request #12649: [FLINK-18286] Implement type inference for GET/FLATTEN
Posted by GitBox <gi...@apache.org>.
dawidwys edited a comment on pull request #12649:
URL: https://github.com/apache/flink/pull/12649#issuecomment-644618279
Yes, but does it always. We want that behaviour only when accessing the field.
Imagine a user function that accepts `ADDRESS(id INT NOT NULL, city STRING)`.
```
class UDF extends ScalarFunction {
public Object eval(Address address) {
if (address != null) {
// based on the type I know address.id is NOT NULL here
int id = address.id;
}
}
}
```
but this is not possible in Calcite. There is no way to construct a `ADDRESS(id INT NOT NULL, city STRING)`. It will be changed to `ADDRESS(id INT, city STRING)`. And based on that there is no way to have the behaviour as above. The id is always nullable. **Without this behaviour we can not have primitive fields in POJOs.**
On the other hand I want the type adjusted if I pass the field only, as this might be null without additional checks:
```
class UDF2 extends ScalarFunction {
public Object eval(int id {
}
}
// This is illegal. We can not ensure that t.address.id is NOT NULL
SELECT UDF2(t.address.id) FROM test t;
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] dawidwys commented on pull request #12649: [FLINK-18286] Implement type inference for GET/FLATTEN
Posted by GitBox <gi...@apache.org>.
dawidwys commented on pull request #12649:
URL: https://github.com/apache/flink/pull/12649#issuecomment-643942158
@danny0405 Could you check changes in 11f71d1b13e2b6456722b5ce5f1081a6f9d1cef9 in classes:
* `SqlDotOperator`
* `SqlItemOperator`
* `SqlValidatorScope`
and say if they make sense and if we can also get those changes into Calcite? (I think those are fixes for valid bugs in Calcite).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12649: [FLINK-18286] Implement type inference for GET/FLATTEN
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #12649:
URL: https://github.com/apache/flink/pull/12649#issuecomment-643957887
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ba5bc62416b82d8e6f1be2c848242de2d3c546e5",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3489",
"triggerID" : "ba5bc62416b82d8e6f1be2c848242de2d3c546e5",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ba5bc62416b82d8e6f1be2c848242de2d3c546e5 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3489)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12649: [FLINK-18286] Implement type inference for GET/FLATTEN
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #12649:
URL: https://github.com/apache/flink/pull/12649#issuecomment-643957887
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ba5bc62416b82d8e6f1be2c848242de2d3c546e5",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3489",
"triggerID" : "ba5bc62416b82d8e6f1be2c848242de2d3c546e5",
"triggerType" : "PUSH"
}, {
"hash" : "c38f0da0bd042779c6c9a096e305cb40e8b1e367",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3531",
"triggerID" : "c38f0da0bd042779c6c9a096e305cb40e8b1e367",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ba5bc62416b82d8e6f1be2c848242de2d3c546e5 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3489)
* c38f0da0bd042779c6c9a096e305cb40e8b1e367 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3531)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12649: [FLINK-18286] Implement type inference for GET/FLATTEN
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #12649:
URL: https://github.com/apache/flink/pull/12649#issuecomment-643957887
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ba5bc62416b82d8e6f1be2c848242de2d3c546e5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3489",
"triggerID" : "ba5bc62416b82d8e6f1be2c848242de2d3c546e5",
"triggerType" : "PUSH"
}, {
"hash" : "c38f0da0bd042779c6c9a096e305cb40e8b1e367",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3531",
"triggerID" : "c38f0da0bd042779c6c9a096e305cb40e8b1e367",
"triggerType" : "PUSH"
}, {
"hash" : "b3dd92d0c41cee2472bb521eec89741c42bc05cd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3580",
"triggerID" : "b3dd92d0c41cee2472bb521eec89741c42bc05cd",
"triggerType" : "PUSH"
}, {
"hash" : "326222e8eb85270297876eba22ffda57ebe592c3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3639",
"triggerID" : "326222e8eb85270297876eba22ffda57ebe592c3",
"triggerType" : "PUSH"
}, {
"hash" : "67905cf92819be26e49afbcf38794e340cca8e9c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3677",
"triggerID" : "67905cf92819be26e49afbcf38794e340cca8e9c",
"triggerType" : "PUSH"
}, {
"hash" : "4588093f97836f4f04c7c2c9992ea5049bf71c4c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3694",
"triggerID" : "4588093f97836f4f04c7c2c9992ea5049bf71c4c",
"triggerType" : "PUSH"
}, {
"hash" : "43596996d023eb47c9589f95d8baf8a3cee83016",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3774",
"triggerID" : "43596996d023eb47c9589f95d8baf8a3cee83016",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 43596996d023eb47c9589f95d8baf8a3cee83016 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3774)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] dawidwys merged pull request #12649: [FLINK-18286] Implement type inference for GET/FLATTEN
Posted by GitBox <gi...@apache.org>.
dawidwys merged pull request #12649:
URL: https://github.com/apache/flink/pull/12649
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12649: [FLINK-18286] Implement type inference for GET/FLATTEN
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #12649:
URL: https://github.com/apache/flink/pull/12649#issuecomment-643957887
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ba5bc62416b82d8e6f1be2c848242de2d3c546e5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3489",
"triggerID" : "ba5bc62416b82d8e6f1be2c848242de2d3c546e5",
"triggerType" : "PUSH"
}, {
"hash" : "c38f0da0bd042779c6c9a096e305cb40e8b1e367",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3531",
"triggerID" : "c38f0da0bd042779c6c9a096e305cb40e8b1e367",
"triggerType" : "PUSH"
}, {
"hash" : "b3dd92d0c41cee2472bb521eec89741c42bc05cd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3580",
"triggerID" : "b3dd92d0c41cee2472bb521eec89741c42bc05cd",
"triggerType" : "PUSH"
}, {
"hash" : "326222e8eb85270297876eba22ffda57ebe592c3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3639",
"triggerID" : "326222e8eb85270297876eba22ffda57ebe592c3",
"triggerType" : "PUSH"
}, {
"hash" : "67905cf92819be26e49afbcf38794e340cca8e9c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3677",
"triggerID" : "67905cf92819be26e49afbcf38794e340cca8e9c",
"triggerType" : "PUSH"
}, {
"hash" : "4588093f97836f4f04c7c2c9992ea5049bf71c4c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3694",
"triggerID" : "4588093f97836f4f04c7c2c9992ea5049bf71c4c",
"triggerType" : "PUSH"
}, {
"hash" : "43596996d023eb47c9589f95d8baf8a3cee83016",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3774",
"triggerID" : "43596996d023eb47c9589f95d8baf8a3cee83016",
"triggerType" : "PUSH"
}, {
"hash" : "93f58e49ad79331331d4d85bbe7720ecfe39f968",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "93f58e49ad79331331d4d85bbe7720ecfe39f968",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 43596996d023eb47c9589f95d8baf8a3cee83016 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3774)
* 93f58e49ad79331331d4d85bbe7720ecfe39f968 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12649: [FLINK-18286] Implement type inference for GET/FLATTEN
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #12649:
URL: https://github.com/apache/flink/pull/12649#issuecomment-643957887
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ba5bc62416b82d8e6f1be2c848242de2d3c546e5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3489",
"triggerID" : "ba5bc62416b82d8e6f1be2c848242de2d3c546e5",
"triggerType" : "PUSH"
}, {
"hash" : "c38f0da0bd042779c6c9a096e305cb40e8b1e367",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3531",
"triggerID" : "c38f0da0bd042779c6c9a096e305cb40e8b1e367",
"triggerType" : "PUSH"
}, {
"hash" : "b3dd92d0c41cee2472bb521eec89741c42bc05cd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3580",
"triggerID" : "b3dd92d0c41cee2472bb521eec89741c42bc05cd",
"triggerType" : "PUSH"
}, {
"hash" : "326222e8eb85270297876eba22ffda57ebe592c3",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3639",
"triggerID" : "326222e8eb85270297876eba22ffda57ebe592c3",
"triggerType" : "PUSH"
}, {
"hash" : "67905cf92819be26e49afbcf38794e340cca8e9c",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3677",
"triggerID" : "67905cf92819be26e49afbcf38794e340cca8e9c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 326222e8eb85270297876eba22ffda57ebe592c3 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3639)
* 67905cf92819be26e49afbcf38794e340cca8e9c Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3677)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12649: [FLINK-18286] Implement type inference for GET/FLATTEN
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #12649:
URL: https://github.com/apache/flink/pull/12649#issuecomment-643957887
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ba5bc62416b82d8e6f1be2c848242de2d3c546e5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3489",
"triggerID" : "ba5bc62416b82d8e6f1be2c848242de2d3c546e5",
"triggerType" : "PUSH"
}, {
"hash" : "c38f0da0bd042779c6c9a096e305cb40e8b1e367",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3531",
"triggerID" : "c38f0da0bd042779c6c9a096e305cb40e8b1e367",
"triggerType" : "PUSH"
}, {
"hash" : "b3dd92d0c41cee2472bb521eec89741c42bc05cd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3580",
"triggerID" : "b3dd92d0c41cee2472bb521eec89741c42bc05cd",
"triggerType" : "PUSH"
}, {
"hash" : "326222e8eb85270297876eba22ffda57ebe592c3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3639",
"triggerID" : "326222e8eb85270297876eba22ffda57ebe592c3",
"triggerType" : "PUSH"
}, {
"hash" : "67905cf92819be26e49afbcf38794e340cca8e9c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3677",
"triggerID" : "67905cf92819be26e49afbcf38794e340cca8e9c",
"triggerType" : "PUSH"
}, {
"hash" : "4588093f97836f4f04c7c2c9992ea5049bf71c4c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3694",
"triggerID" : "4588093f97836f4f04c7c2c9992ea5049bf71c4c",
"triggerType" : "PUSH"
}, {
"hash" : "43596996d023eb47c9589f95d8baf8a3cee83016",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3774",
"triggerID" : "43596996d023eb47c9589f95d8baf8a3cee83016",
"triggerType" : "PUSH"
}, {
"hash" : "93f58e49ad79331331d4d85bbe7720ecfe39f968",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4559",
"triggerID" : "93f58e49ad79331331d4d85bbe7720ecfe39f968",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 43596996d023eb47c9589f95d8baf8a3cee83016 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3774)
* 93f58e49ad79331331d4d85bbe7720ecfe39f968 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4559)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12649: [FLINK-18286] Implement type inference for GET/FLATTEN
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #12649:
URL: https://github.com/apache/flink/pull/12649#issuecomment-643957887
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ba5bc62416b82d8e6f1be2c848242de2d3c546e5",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3489",
"triggerID" : "ba5bc62416b82d8e6f1be2c848242de2d3c546e5",
"triggerType" : "PUSH"
}, {
"hash" : "c38f0da0bd042779c6c9a096e305cb40e8b1e367",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c38f0da0bd042779c6c9a096e305cb40e8b1e367",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ba5bc62416b82d8e6f1be2c848242de2d3c546e5 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3489)
* c38f0da0bd042779c6c9a096e305cb40e8b1e367 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] danny0405 commented on a change in pull request #12649: [FLINK-18286] Implement type inference for GET/FLATTEN
Posted by GitBox <gi...@apache.org>.
danny0405 commented on a change in pull request #12649:
URL: https://github.com/apache/flink/pull/12649#discussion_r443528280
##########
File path: flink-table/flink-table-planner-blink/src/main/java/org/apache/calcite/sql/fun/SqlDotOperator.java
##########
@@ -0,0 +1,194 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.calcite.sql.fun;
+
+import org.apache.calcite.rel.type.RelDataType;
+import org.apache.calcite.rel.type.RelDataTypeFactory;
+import org.apache.calcite.rel.type.RelDataTypeField;
+import org.apache.calcite.sql.SqlCall;
+import org.apache.calcite.sql.SqlCallBinding;
+import org.apache.calcite.sql.SqlKind;
+import org.apache.calcite.sql.SqlNode;
+import org.apache.calcite.sql.SqlOperandCountRange;
+import org.apache.calcite.sql.SqlOperatorBinding;
+import org.apache.calcite.sql.SqlSpecialOperator;
+import org.apache.calcite.sql.SqlUtil;
+import org.apache.calcite.sql.SqlWriter;
+import org.apache.calcite.sql.parser.SqlParserPos;
+import org.apache.calcite.sql.type.OperandTypes;
+import org.apache.calcite.sql.type.SqlOperandCountRanges;
+import org.apache.calcite.sql.type.SqlSingleOperandTypeChecker;
+import org.apache.calcite.sql.type.SqlTypeFamily;
+import org.apache.calcite.sql.type.SqlTypeName;
+import org.apache.calcite.sql.util.SqlBasicVisitor;
+import org.apache.calcite.sql.util.SqlVisitor;
+import org.apache.calcite.sql.validate.SqlValidator;
+import org.apache.calcite.sql.validate.SqlValidatorScope;
+import org.apache.calcite.sql.validate.SqlValidatorUtil;
+import org.apache.calcite.util.Litmus;
+import org.apache.calcite.util.Static;
+
+import java.util.Arrays;
+
+/**
+ * The dot operator {@code .}, used to access a field of a
+ * record. For example, {@code a.b}.
+ *
+ * <p>This class was copied over from Calcite to fix the derived type.
+ * If the ROW is nullable force the accessed field to be nullable as well.
+ */
+public class SqlDotOperator extends SqlSpecialOperator {
+ SqlDotOperator() {
+ super("DOT", SqlKind.DOT, 100, true, null, null, null);
+ }
+
+ @Override public ReduceResult reduceExpr(int ordinal, TokenSequence list) {
+ SqlNode left = list.node(ordinal - 1);
+ SqlNode right = list.node(ordinal + 1);
+ return new ReduceResult(ordinal - 1,
+ ordinal + 2,
+ createCall(
+ SqlParserPos.sum(
+ Arrays.asList(left.getParserPosition(),
+ right.getParserPosition(),
+ list.pos(ordinal))),
+ left,
+ right));
+ }
+
+ @Override public void unparse(SqlWriter writer, SqlCall call, int leftPrec,
+ int rightPrec) {
+ final SqlWriter.Frame frame =
+ writer.startList(SqlWriter.FrameTypeEnum.IDENTIFIER);
+ call.operand(0).unparse(writer, leftPrec, 0);
+ writer.sep(".");
+ call.operand(1).unparse(writer, 0, 0);
+ writer.endList(frame);
+ }
+
+ @Override public SqlOperandCountRange getOperandCountRange() {
+ return SqlOperandCountRanges.of(2);
+ }
+
+ @Override public <R> void acceptCall(SqlVisitor<R> visitor, SqlCall call,
+ boolean onlyExpressions, SqlBasicVisitor.ArgHandler<R> argHandler) {
+ if (onlyExpressions) {
+ // Do not visit operands[1] -- it is not an expression.
+ argHandler.visitChild(visitor, call, 0, call.operand(0));
+ } else {
+ super.acceptCall(visitor, call, onlyExpressions, argHandler);
+ }
+ }
+
+ @Override public RelDataType deriveType(SqlValidator validator,
+ SqlValidatorScope scope, SqlCall call) {
+ final SqlNode operand = call.getOperandList().get(0);
+ final RelDataType nodeType =
+ validator.deriveType(scope, operand);
+ assert nodeType != null;
+ if (!nodeType.isStruct()) {
+ throw SqlUtil.newContextException(operand.getParserPosition(),
+ Static.RESOURCE.incompatibleTypes());
+ }
+
+ final SqlNode fieldId = call.operand(1);
+ final String fieldName = fieldId.toString();
+ final RelDataTypeField field =
+ nodeType.getField(fieldName, false, false);
+ if (field == null) {
+ throw SqlUtil.newContextException(fieldId.getParserPosition(),
+ Static.RESOURCE.unknownField(fieldName));
+ }
+ RelDataType type = field.getType();
+ if (nodeType.isNullable() && !type.isNullable()) {
+ type = validator.getTypeFactory().createTypeWithNullability(type, true);
+ }
Review comment:
In Calcite, when the record type is nullable, the attributes are always nullable, so, this fix is not that necessary from the Calcite side.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12649: [FLINK-18286] Implement type inference for GET/FLATTEN
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #12649:
URL: https://github.com/apache/flink/pull/12649#issuecomment-643957887
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ba5bc62416b82d8e6f1be2c848242de2d3c546e5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3489",
"triggerID" : "ba5bc62416b82d8e6f1be2c848242de2d3c546e5",
"triggerType" : "PUSH"
}, {
"hash" : "c38f0da0bd042779c6c9a096e305cb40e8b1e367",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3531",
"triggerID" : "c38f0da0bd042779c6c9a096e305cb40e8b1e367",
"triggerType" : "PUSH"
}, {
"hash" : "b3dd92d0c41cee2472bb521eec89741c42bc05cd",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3580",
"triggerID" : "b3dd92d0c41cee2472bb521eec89741c42bc05cd",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c38f0da0bd042779c6c9a096e305cb40e8b1e367 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3531)
* b3dd92d0c41cee2472bb521eec89741c42bc05cd Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3580)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] dawidwys edited a comment on pull request #12649: [FLINK-18286] Implement type inference for GET/FLATTEN
Posted by GitBox <gi...@apache.org>.
dawidwys edited a comment on pull request #12649:
URL: https://github.com/apache/flink/pull/12649#issuecomment-644618279
Yes, but it does always. We want that behaviour only when accessing the field.
Imagine a user function that accepts `ADDRESS(id INT NOT NULL, city STRING)`.
```
class UDF extends ScalarFunction {
public Object eval(Address address) {
if (address != null) {
// based on the type I know address.id is NOT NULL here
int id = address.id;
}
}
}
```
but this is not possible in Calcite. There is no way to construct a `ADDRESS(id INT NOT NULL, city STRING)`. It will be changed to `ADDRESS(id INT, city STRING)`. And based on that there is no way to have the behaviour as above. The id is always nullable. **Without this behaviour we can not have primitive fields in POJOs.**
On the other hand I want the type adjusted if I pass the field only, as this might be null without additional checks:
```
class UDF2 extends ScalarFunction {
public Object eval(int id {
}
}
// This is illegal. We can not ensure that t.address.id is NOT NULL
SELECT UDF2(t.address.id) FROM test t;
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] dawidwys edited a comment on pull request #12649: [FLINK-18286] Implement type inference for GET/FLATTEN
Posted by GitBox <gi...@apache.org>.
dawidwys edited a comment on pull request #12649:
URL: https://github.com/apache/flink/pull/12649#issuecomment-644618279
Yes, but does it always. We want that behaviour only when accessing the field.
Imagine a user function that accepts `ADDRESS(id INT NOT NULL, city STRING)`.
```
class UDF extends ScalarFunction {
public Object eval(Address address) {
if (address != null) {
// based on the type I know address.id is NOT NULL here
int id = address.id;
}
}
}
```
but this is not possible in Calcite. There is no way to construct a `ADDRESS(id INT NOT NULL, city STRING)`. It will be changed to `ADDRESS(id INT, city STRING)`. And based on that there is no way to have the behaviour as above. The id is always nullable. Without this behaviour we can not have primitive fields in POJOs.
On the other hand I want the type adjusted if I pass the field only, as this might be null without additional checks:
```
class UDF2 extends ScalarFunction {
public Object eval(int id {
}
}
// This is illegal. We can not ensure that t.address.id is NOT NULL
SELECT UDF2(t.address.id) FROM test t;
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] dawidwys commented on a change in pull request #12649: [FLINK-18286] Implement type inference for GET/FLATTEN
Posted by GitBox <gi...@apache.org>.
dawidwys commented on a change in pull request #12649:
URL: https://github.com/apache/flink/pull/12649#discussion_r440794777
##########
File path: flink-table/flink-table-planner-blink/src/test/java/org/apache/flink/table/planner/expressions/CompositeTypeAccessExpressionITCase.java
##########
@@ -0,0 +1,264 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.planner.expressions;
+
+import org.apache.flink.table.annotation.DataTypeHint;
+import org.apache.flink.table.annotation.FunctionHint;
+import org.apache.flink.table.api.DataTypes;
+import org.apache.flink.table.api.EnvironmentSettings;
+import org.apache.flink.table.api.TableEnvironment;
+import org.apache.flink.table.api.TableResult;
+import org.apache.flink.table.api.TableSchema;
+import org.apache.flink.table.api.ValidationException;
+import org.apache.flink.table.functions.BuiltInFunctionDefinitions;
+import org.apache.flink.table.functions.ScalarFunction;
+import org.apache.flink.table.functions.TableFunction;
+import org.apache.flink.table.types.logical.LogicalTypeRoot;
+import org.apache.flink.types.Row;
+import org.apache.flink.util.CloseableIterator;
+
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.ExpectedException;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+import org.junit.runners.Suite;
+
+import java.util.Arrays;
+import java.util.List;
+
+import static java.util.Collections.singletonMap;
+import static org.apache.flink.table.api.DataTypes.ARRAY;
+import static org.apache.flink.table.api.DataTypes.BIGINT;
+import static org.apache.flink.table.api.DataTypes.FIELD;
+import static org.apache.flink.table.api.DataTypes.MAP;
+import static org.apache.flink.table.api.DataTypes.ROW;
+import static org.apache.flink.table.api.DataTypes.STRING;
+import static org.apache.flink.table.api.Expressions.$;
+import static org.apache.flink.table.api.Expressions.call;
+import static org.hamcrest.CoreMatchers.equalTo;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertThat;
+
+/**
+ * Tests for functions that access nested fields/elements of composite/collection types.
+ */
+@RunWith(Suite.class)
+@Suite.SuiteClasses(
+ {
+ CompositeTypeAccessExpressionITCase.TableFieldAccess.class,
+ CompositeTypeAccessExpressionITCase.CallFieldAccess.class
+ }
+)
+public class CompositeTypeAccessExpressionITCase {
+
+ /**
+ * Regular tests. See also {@link CallFieldAccess} for tests that access a nested field of an expression or
+ * for {@link BuiltInFunctionDefinitions#FLATTEN} which produces multiple columns from a single one.
+ */
+ public static class TableFieldAccess extends BuiltInFunctionTestBase {
+ @Parameterized.Parameters(name = "{index}: {0}")
+ public static List<TestSpec> testData() {
+ return Arrays.asList(
+
+ // Actually in case of SQL it does not use the GET method, but
+ // a custom logic for accessing nested fields of a Table.
+ TestSpec.forFunction(BuiltInFunctionDefinitions.GET)
+ .onFieldsWithData(null, Row.of(1))
+ .andDataTypes(
+ ROW(FIELD("nested", BIGINT().notNull())).nullable(),
+ ROW(FIELD("nested", BIGINT().notNull())).notNull()
+ )
+ .testTableApiResult($("f0").get("nested"), null, BIGINT().nullable())
+ .testTableApiResult($("f1").get("nested"), 1L, BIGINT().notNull())
+ .testSqlResult("f0.nested", null, BIGINT().nullable())
+ .testSqlResult("f1.nested", 1L, BIGINT().notNull()),
+
+ // In Calcite it maps to FlinkSqlOperatorTable.ITEM
+ TestSpec.forFunction(BuiltInFunctionDefinitions.AT)
+ .onFieldsWithData(null, new int[] {1}, null, singletonMap("nested", 1), null, Row.of(1))
+ .andDataTypes(
+ ARRAY(BIGINT().notNull()).nullable(),
+ ARRAY(BIGINT().notNull()).notNull(),
+ MAP(STRING(), BIGINT().notNull()).nullable(),
+ MAP(STRING(), BIGINT().notNull()).notNull(),
+ ROW(FIELD("nested", BIGINT().notNull())).nullable(),
+ ROW(FIELD("nested", BIGINT().notNull())).notNull()
+ )
+ // accessing elements of MAP or ARRAY is a runtime operations,
+ // we do not know about the size or contents during the inference
+ // therefore the results are always nullable
+ .testSqlResult("f0[1]", null, BIGINT().nullable())
Review comment:
I have it in a different branch. I found this PR already quite big.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] danny0405 commented on pull request #12649: [FLINK-18286] Implement type inference for GET/FLATTEN
Posted by GitBox <gi...@apache.org>.
danny0405 commented on pull request #12649:
URL: https://github.com/apache/flink/pull/12649#issuecomment-644612644
> This is not possible in Calcite by default. Thats why we have the changes in `org.apache.calcite.rel.type.RelDataTypeFactoryImpl#createTypeWithNullability:326-338`.
The calcite code says:
> If nullable -> Do a deep copy, setting all fields of the record type
> to be nullable regardless of initial nullability.
This seems just what you need, what do you mean it is not possible ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12649: [FLINK-18286] Implement type inference for GET/FLATTEN
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #12649:
URL: https://github.com/apache/flink/pull/12649#issuecomment-643957887
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ba5bc62416b82d8e6f1be2c848242de2d3c546e5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3489",
"triggerID" : "ba5bc62416b82d8e6f1be2c848242de2d3c546e5",
"triggerType" : "PUSH"
}, {
"hash" : "c38f0da0bd042779c6c9a096e305cb40e8b1e367",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3531",
"triggerID" : "c38f0da0bd042779c6c9a096e305cb40e8b1e367",
"triggerType" : "PUSH"
}, {
"hash" : "b3dd92d0c41cee2472bb521eec89741c42bc05cd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b3dd92d0c41cee2472bb521eec89741c42bc05cd",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c38f0da0bd042779c6c9a096e305cb40e8b1e367 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3531)
* b3dd92d0c41cee2472bb521eec89741c42bc05cd UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] dawidwys commented on pull request #12649: [FLINK-18286] Implement type inference for GET/FLATTEN
Posted by GitBox <gi...@apache.org>.
dawidwys commented on pull request #12649:
URL: https://github.com/apache/flink/pull/12649#issuecomment-644618279
Yes, but does it always. We want that behaviour only when accessing the field.
Imagine a user function that accepts `ADDRESS(id INT NOT NULL, city STRING)`.
```
class UDF extends ScalarFunction {
public Object eval(Address address) {
if (address != null) {
// based on the type I know address.id is NOT NULL here
int id = address.id;
}
}
}
```
but this is not possible in Calcite. There is no way to construct a `ADDRESS(id INT NOT NULL, city STRING)`. It will be changed to `ADDRESS(id INT, city STRING)`. And based on that there is no way to have the behaviour as above. The id is always nullable.
On the other hand I want the type adjusted if I pass the field only, as this might be null without additional checks:
```
class UDF2 extends ScalarFunction {
public Object eval(int id {
}
}
// This is illegal. We can not ensure that t.address.id is NOT NULL
SELECT UDF2(t.address.id) FROM test t;
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12649: [FLINK-18286] Implement type inference for GET/FLATTEN
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #12649:
URL: https://github.com/apache/flink/pull/12649#issuecomment-643957887
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ba5bc62416b82d8e6f1be2c848242de2d3c546e5",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3489",
"triggerID" : "ba5bc62416b82d8e6f1be2c848242de2d3c546e5",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ba5bc62416b82d8e6f1be2c848242de2d3c546e5 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3489)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12649: [FLINK-18286] Implement type inference for GET/FLATTEN
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #12649:
URL: https://github.com/apache/flink/pull/12649#issuecomment-643957887
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ba5bc62416b82d8e6f1be2c848242de2d3c546e5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3489",
"triggerID" : "ba5bc62416b82d8e6f1be2c848242de2d3c546e5",
"triggerType" : "PUSH"
}, {
"hash" : "c38f0da0bd042779c6c9a096e305cb40e8b1e367",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3531",
"triggerID" : "c38f0da0bd042779c6c9a096e305cb40e8b1e367",
"triggerType" : "PUSH"
}, {
"hash" : "b3dd92d0c41cee2472bb521eec89741c42bc05cd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3580",
"triggerID" : "b3dd92d0c41cee2472bb521eec89741c42bc05cd",
"triggerType" : "PUSH"
}, {
"hash" : "326222e8eb85270297876eba22ffda57ebe592c3",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3639",
"triggerID" : "326222e8eb85270297876eba22ffda57ebe592c3",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 326222e8eb85270297876eba22ffda57ebe592c3 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3639)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12649: [FLINK-18286] Implement type inference for GET/FLATTEN
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #12649:
URL: https://github.com/apache/flink/pull/12649#issuecomment-643957887
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ba5bc62416b82d8e6f1be2c848242de2d3c546e5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3489",
"triggerID" : "ba5bc62416b82d8e6f1be2c848242de2d3c546e5",
"triggerType" : "PUSH"
}, {
"hash" : "c38f0da0bd042779c6c9a096e305cb40e8b1e367",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3531",
"triggerID" : "c38f0da0bd042779c6c9a096e305cb40e8b1e367",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c38f0da0bd042779c6c9a096e305cb40e8b1e367 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3531)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12649: [FLINK-18286] Implement type inference for GET/FLATTEN
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #12649:
URL: https://github.com/apache/flink/pull/12649#issuecomment-643957887
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ba5bc62416b82d8e6f1be2c848242de2d3c546e5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3489",
"triggerID" : "ba5bc62416b82d8e6f1be2c848242de2d3c546e5",
"triggerType" : "PUSH"
}, {
"hash" : "c38f0da0bd042779c6c9a096e305cb40e8b1e367",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3531",
"triggerID" : "c38f0da0bd042779c6c9a096e305cb40e8b1e367",
"triggerType" : "PUSH"
}, {
"hash" : "b3dd92d0c41cee2472bb521eec89741c42bc05cd",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3580",
"triggerID" : "b3dd92d0c41cee2472bb521eec89741c42bc05cd",
"triggerType" : "PUSH"
}, {
"hash" : "326222e8eb85270297876eba22ffda57ebe592c3",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "326222e8eb85270297876eba22ffda57ebe592c3",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b3dd92d0c41cee2472bb521eec89741c42bc05cd Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3580)
* 326222e8eb85270297876eba22ffda57ebe592c3 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12649: [FLINK-18286] Implement type inference for GET/FLATTEN
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #12649:
URL: https://github.com/apache/flink/pull/12649#issuecomment-643957887
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ba5bc62416b82d8e6f1be2c848242de2d3c546e5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3489",
"triggerID" : "ba5bc62416b82d8e6f1be2c848242de2d3c546e5",
"triggerType" : "PUSH"
}, {
"hash" : "c38f0da0bd042779c6c9a096e305cb40e8b1e367",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3531",
"triggerID" : "c38f0da0bd042779c6c9a096e305cb40e8b1e367",
"triggerType" : "PUSH"
}, {
"hash" : "b3dd92d0c41cee2472bb521eec89741c42bc05cd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3580",
"triggerID" : "b3dd92d0c41cee2472bb521eec89741c42bc05cd",
"triggerType" : "PUSH"
}, {
"hash" : "326222e8eb85270297876eba22ffda57ebe592c3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3639",
"triggerID" : "326222e8eb85270297876eba22ffda57ebe592c3",
"triggerType" : "PUSH"
}, {
"hash" : "67905cf92819be26e49afbcf38794e340cca8e9c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3677",
"triggerID" : "67905cf92819be26e49afbcf38794e340cca8e9c",
"triggerType" : "PUSH"
}, {
"hash" : "4588093f97836f4f04c7c2c9992ea5049bf71c4c",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3694",
"triggerID" : "4588093f97836f4f04c7c2c9992ea5049bf71c4c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4588093f97836f4f04c7c2c9992ea5049bf71c4c Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3694)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] dawidwys commented on a change in pull request #12649: [FLINK-18286] Implement type inference for GET/FLATTEN
Posted by GitBox <gi...@apache.org>.
dawidwys commented on a change in pull request #12649:
URL: https://github.com/apache/flink/pull/12649#discussion_r440894139
##########
File path: flink-table/flink-table-common/src/main/java/org/apache/flink/table/types/inference/TypeStrategies.java
##########
@@ -315,6 +317,36 @@ public static TypeStrategy nullable(TypeStrategy initialStrategy) {
return Optional.of(fromLogicalToDataType(inferredType));
};
+ /**
+ * Type strategy that returns a type of a field nested inside a composite type that is described by the second argument.
+ * The second argument must be a literal that describes either the nested field name or index.
+ */
+ public static final TypeStrategy GET = callContext -> {
+ List<DataType> argumentDataTypes = callContext.getArgumentDataTypes();
+ DataType rowDataType = argumentDataTypes.get(0);
+ TableSchema nestedSchema = DataTypeUtils.expandCompositeTypeToSchema(rowDataType);
Review comment:
I think it does already support the `DISTINCT` type. Moreover `AtomicDataType` can not hold a composite type. `AtomicDataType` can not hold the bridging classes of nested fields. IMO `Distinct` type can be any of `FieldsDataType`, `AtomicDataType`, 'KeyValueDataType` etc.
The way we support `DISTINCT` type is that we do not check the `LogicalTypeRoot` in `expandCompositeTypeToSchema`, but we check only for the `FieldsDataType`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12649: [FLINK-18286] Implement type inference for GET/FLATTEN
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #12649:
URL: https://github.com/apache/flink/pull/12649#issuecomment-643957887
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ba5bc62416b82d8e6f1be2c848242de2d3c546e5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3489",
"triggerID" : "ba5bc62416b82d8e6f1be2c848242de2d3c546e5",
"triggerType" : "PUSH"
}, {
"hash" : "c38f0da0bd042779c6c9a096e305cb40e8b1e367",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3531",
"triggerID" : "c38f0da0bd042779c6c9a096e305cb40e8b1e367",
"triggerType" : "PUSH"
}, {
"hash" : "b3dd92d0c41cee2472bb521eec89741c42bc05cd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3580",
"triggerID" : "b3dd92d0c41cee2472bb521eec89741c42bc05cd",
"triggerType" : "PUSH"
}, {
"hash" : "326222e8eb85270297876eba22ffda57ebe592c3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3639",
"triggerID" : "326222e8eb85270297876eba22ffda57ebe592c3",
"triggerType" : "PUSH"
}, {
"hash" : "67905cf92819be26e49afbcf38794e340cca8e9c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3677",
"triggerID" : "67905cf92819be26e49afbcf38794e340cca8e9c",
"triggerType" : "PUSH"
}, {
"hash" : "4588093f97836f4f04c7c2c9992ea5049bf71c4c",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3694",
"triggerID" : "4588093f97836f4f04c7c2c9992ea5049bf71c4c",
"triggerType" : "PUSH"
}, {
"hash" : "43596996d023eb47c9589f95d8baf8a3cee83016",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "43596996d023eb47c9589f95d8baf8a3cee83016",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4588093f97836f4f04c7c2c9992ea5049bf71c4c Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3694)
* 43596996d023eb47c9589f95d8baf8a3cee83016 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] dawidwys commented on pull request #12649: [FLINK-18286] Implement type inference for GET/FLATTEN
Posted by GitBox <gi...@apache.org>.
dawidwys commented on pull request #12649:
URL: https://github.com/apache/flink/pull/12649#issuecomment-643942301
@twalthr Could you review the PR?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] twalthr commented on a change in pull request #12649: [FLINK-18286] Implement type inference for GET/FLATTEN
Posted by GitBox <gi...@apache.org>.
twalthr commented on a change in pull request #12649:
URL: https://github.com/apache/flink/pull/12649#discussion_r440252362
##########
File path: flink-table/flink-table-common/src/main/java/org/apache/flink/table/types/inference/InputTypeStrategies.java
##########
@@ -198,6 +201,36 @@ public static InputTypeStrategy comparable(
*/
public static final LiteralArgumentTypeStrategy LITERAL_OR_NULL = new LiteralArgumentTypeStrategy(true);
+ /**
+ * Strategy that checks that the argument has a composite type.
+ */
+ public static final ArgumentTypeStrategy COMPOSITE = new ArgumentTypeStrategy() {
Review comment:
nit: move to separate class like all other argument strategies
##########
File path: flink-table/flink-table-planner-blink/src/test/java/org/apache/flink/table/planner/expressions/CompositeTypeAccessExpressionITCase.java
##########
@@ -0,0 +1,264 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.planner.expressions;
+
+import org.apache.flink.table.annotation.DataTypeHint;
+import org.apache.flink.table.annotation.FunctionHint;
+import org.apache.flink.table.api.DataTypes;
+import org.apache.flink.table.api.EnvironmentSettings;
+import org.apache.flink.table.api.TableEnvironment;
+import org.apache.flink.table.api.TableResult;
+import org.apache.flink.table.api.TableSchema;
+import org.apache.flink.table.api.ValidationException;
+import org.apache.flink.table.functions.BuiltInFunctionDefinitions;
+import org.apache.flink.table.functions.ScalarFunction;
+import org.apache.flink.table.functions.TableFunction;
+import org.apache.flink.table.types.logical.LogicalTypeRoot;
+import org.apache.flink.types.Row;
+import org.apache.flink.util.CloseableIterator;
+
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.ExpectedException;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+import org.junit.runners.Suite;
+
+import java.util.Arrays;
+import java.util.List;
+
+import static java.util.Collections.singletonMap;
+import static org.apache.flink.table.api.DataTypes.ARRAY;
+import static org.apache.flink.table.api.DataTypes.BIGINT;
+import static org.apache.flink.table.api.DataTypes.FIELD;
+import static org.apache.flink.table.api.DataTypes.MAP;
+import static org.apache.flink.table.api.DataTypes.ROW;
+import static org.apache.flink.table.api.DataTypes.STRING;
+import static org.apache.flink.table.api.Expressions.$;
+import static org.apache.flink.table.api.Expressions.call;
+import static org.hamcrest.CoreMatchers.equalTo;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertThat;
+
+/**
+ * Tests for functions that access nested fields/elements of composite/collection types.
+ */
+@RunWith(Suite.class)
+@Suite.SuiteClasses(
+ {
+ CompositeTypeAccessExpressionITCase.TableFieldAccess.class,
+ CompositeTypeAccessExpressionITCase.CallFieldAccess.class
+ }
+)
+public class CompositeTypeAccessExpressionITCase {
+
+ /**
+ * Regular tests. See also {@link CallFieldAccess} for tests that access a nested field of an expression or
+ * for {@link BuiltInFunctionDefinitions#FLATTEN} which produces multiple columns from a single one.
+ */
+ public static class TableFieldAccess extends BuiltInFunctionTestBase {
+ @Parameterized.Parameters(name = "{index}: {0}")
+ public static List<TestSpec> testData() {
+ return Arrays.asList(
+
+ // Actually in case of SQL it does not use the GET method, but
+ // a custom logic for accessing nested fields of a Table.
+ TestSpec.forFunction(BuiltInFunctionDefinitions.GET)
+ .onFieldsWithData(null, Row.of(1))
+ .andDataTypes(
+ ROW(FIELD("nested", BIGINT().notNull())).nullable(),
+ ROW(FIELD("nested", BIGINT().notNull())).notNull()
+ )
+ .testTableApiResult($("f0").get("nested"), null, BIGINT().nullable())
+ .testTableApiResult($("f1").get("nested"), 1L, BIGINT().notNull())
+ .testSqlResult("f0.nested", null, BIGINT().nullable())
+ .testSqlResult("f1.nested", 1L, BIGINT().notNull()),
+
+ // In Calcite it maps to FlinkSqlOperatorTable.ITEM
+ TestSpec.forFunction(BuiltInFunctionDefinitions.AT)
+ .onFieldsWithData(null, new int[] {1}, null, singletonMap("nested", 1), null, Row.of(1))
+ .andDataTypes(
+ ARRAY(BIGINT().notNull()).nullable(),
+ ARRAY(BIGINT().notNull()).notNull(),
+ MAP(STRING(), BIGINT().notNull()).nullable(),
+ MAP(STRING(), BIGINT().notNull()).notNull(),
+ ROW(FIELD("nested", BIGINT().notNull())).nullable(),
+ ROW(FIELD("nested", BIGINT().notNull())).notNull()
+ )
+ // accessing elements of MAP or ARRAY is a runtime operations,
+ // we do not know about the size or contents during the inference
+ // therefore the results are always nullable
+ .testSqlResult("f0[1]", null, BIGINT().nullable())
Review comment:
should we port `AT` as well or in a separate PR?
##########
File path: flink-table/flink-table-planner-blink/src/test/java/org/apache/flink/table/planner/expressions/CompositeTypeAccessExpressionITCase.java
##########
@@ -0,0 +1,264 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.planner.expressions;
+
+import org.apache.flink.table.annotation.DataTypeHint;
+import org.apache.flink.table.annotation.FunctionHint;
+import org.apache.flink.table.api.DataTypes;
+import org.apache.flink.table.api.EnvironmentSettings;
+import org.apache.flink.table.api.TableEnvironment;
+import org.apache.flink.table.api.TableResult;
+import org.apache.flink.table.api.TableSchema;
+import org.apache.flink.table.api.ValidationException;
+import org.apache.flink.table.functions.BuiltInFunctionDefinitions;
+import org.apache.flink.table.functions.ScalarFunction;
+import org.apache.flink.table.functions.TableFunction;
+import org.apache.flink.table.types.logical.LogicalTypeRoot;
+import org.apache.flink.types.Row;
+import org.apache.flink.util.CloseableIterator;
+
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.ExpectedException;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+import org.junit.runners.Suite;
+
+import java.util.Arrays;
+import java.util.List;
+
+import static java.util.Collections.singletonMap;
+import static org.apache.flink.table.api.DataTypes.ARRAY;
+import static org.apache.flink.table.api.DataTypes.BIGINT;
+import static org.apache.flink.table.api.DataTypes.FIELD;
+import static org.apache.flink.table.api.DataTypes.MAP;
+import static org.apache.flink.table.api.DataTypes.ROW;
+import static org.apache.flink.table.api.DataTypes.STRING;
+import static org.apache.flink.table.api.Expressions.$;
+import static org.apache.flink.table.api.Expressions.call;
+import static org.hamcrest.CoreMatchers.equalTo;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertThat;
+
+/**
+ * Tests for functions that access nested fields/elements of composite/collection types.
+ */
+@RunWith(Suite.class)
+@Suite.SuiteClasses(
+ {
+ CompositeTypeAccessExpressionITCase.TableFieldAccess.class,
+ CompositeTypeAccessExpressionITCase.CallFieldAccess.class
+ }
+)
+public class CompositeTypeAccessExpressionITCase {
+
+ /**
+ * Regular tests. See also {@link CallFieldAccess} for tests that access a nested field of an expression or
+ * for {@link BuiltInFunctionDefinitions#FLATTEN} which produces multiple columns from a single one.
+ */
+ public static class TableFieldAccess extends BuiltInFunctionTestBase {
+ @Parameterized.Parameters(name = "{index}: {0}")
+ public static List<TestSpec> testData() {
+ return Arrays.asList(
+
+ // Actually in case of SQL it does not use the GET method, but
+ // a custom logic for accessing nested fields of a Table.
+ TestSpec.forFunction(BuiltInFunctionDefinitions.GET)
+ .onFieldsWithData(null, Row.of(1))
+ .andDataTypes(
+ ROW(FIELD("nested", BIGINT().notNull())).nullable(),
+ ROW(FIELD("nested", BIGINT().notNull())).notNull()
+ )
+ .testTableApiResult($("f0").get("nested"), null, BIGINT().nullable())
+ .testTableApiResult($("f1").get("nested"), 1L, BIGINT().notNull())
+ .testSqlResult("f0.nested", null, BIGINT().nullable())
+ .testSqlResult("f1.nested", 1L, BIGINT().notNull()),
+
+ // In Calcite it maps to FlinkSqlOperatorTable.ITEM
+ TestSpec.forFunction(BuiltInFunctionDefinitions.AT)
+ .onFieldsWithData(null, new int[] {1}, null, singletonMap("nested", 1), null, Row.of(1))
+ .andDataTypes(
+ ARRAY(BIGINT().notNull()).nullable(),
+ ARRAY(BIGINT().notNull()).notNull(),
+ MAP(STRING(), BIGINT().notNull()).nullable(),
+ MAP(STRING(), BIGINT().notNull()).notNull(),
+ ROW(FIELD("nested", BIGINT().notNull())).nullable(),
+ ROW(FIELD("nested", BIGINT().notNull())).notNull()
+ )
+ // accessing elements of MAP or ARRAY is a runtime operations,
+ // we do not know about the size or contents during the inference
+ // therefore the results are always nullable
+ .testSqlResult("f0[1]", null, BIGINT().nullable())
+ .testSqlResult("f1[1]", 1L, BIGINT().nullable())
+ .testSqlResult("f2['nested']", null, BIGINT().nullable())
+ .testSqlResult("f3['nested']", 1L, BIGINT().nullable())
+
+ // we know all the fields of a type up front, therefore we can
+ // derive more accurate types during the inference
+ .testSqlResult("f4['nested']", null, BIGINT().nullable())
+ .testSqlResult("f5['nested']", 1L, BIGINT().notNull())
+ );
+ }
+ }
+
+ /**
+ * A class for customized tests.
+ */
+ public static class CallFieldAccess {
Review comment:
nit: very good tests in general but can we rename the two suits to `FieldAccessAfterCall` and `FieldAccessFromTable` or similar to highlight the difference?
##########
File path: flink-table/flink-table-common/src/main/java/org/apache/flink/table/functions/BuiltInFunctionDefinitions.java
##########
@@ -972,13 +972,28 @@
new BuiltInFunctionDefinition.Builder()
.name("flatten")
.kind(OTHER)
- .outputTypeStrategy(TypeStrategies.MISSING)
+ .inputTypeStrategy(sequence(InputTypeStrategies.COMPOSITE))
+ .outputTypeStrategy(callContext -> {
+ throw new UnsupportedOperationException("FLATTEN should be resolved to GET expressions");
+ })
.build();
public static final BuiltInFunctionDefinition GET =
new BuiltInFunctionDefinition.Builder()
.name("get")
.kind(OTHER)
- .outputTypeStrategy(TypeStrategies.MISSING)
+ .inputTypeStrategy(
+ sequence(
+ InputTypeStrategies.COMPOSITE,
+ and(
+ InputTypeStrategies.LITERAL,
+ or(
+ logical(LogicalTypeRoot.INTEGER),
+ logical(LogicalTypeFamily.CHARACTER_STRING, false)
Review comment:
remove `false`? the nullability is already excluded by `LITERAL`
##########
File path: flink-table/flink-table-common/src/main/java/org/apache/flink/table/types/inference/InputTypeStrategies.java
##########
@@ -198,6 +201,36 @@ public static InputTypeStrategy comparable(
*/
public static final LiteralArgumentTypeStrategy LITERAL_OR_NULL = new LiteralArgumentTypeStrategy(true);
+ /**
+ * Strategy that checks that the argument has a composite type.
+ */
+ public static final ArgumentTypeStrategy COMPOSITE = new ArgumentTypeStrategy() {
+ @Override
+ public Optional<DataType> inferArgumentType(
+ CallContext callContext,
+ int argumentPos,
+ boolean throwOnFailure) {
+ DataType dataType = callContext.getArgumentDataTypes().get(argumentPos);
Review comment:
nit: wrong indention
##########
File path: flink-table/flink-table-planner-blink/src/test/java/org/apache/flink/table/planner/expressions/CompositeTypeAccessExpressionITCase.java
##########
@@ -0,0 +1,264 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.planner.expressions;
+
+import org.apache.flink.table.annotation.DataTypeHint;
+import org.apache.flink.table.annotation.FunctionHint;
+import org.apache.flink.table.api.DataTypes;
+import org.apache.flink.table.api.EnvironmentSettings;
+import org.apache.flink.table.api.TableEnvironment;
+import org.apache.flink.table.api.TableResult;
+import org.apache.flink.table.api.TableSchema;
+import org.apache.flink.table.api.ValidationException;
+import org.apache.flink.table.functions.BuiltInFunctionDefinitions;
+import org.apache.flink.table.functions.ScalarFunction;
+import org.apache.flink.table.functions.TableFunction;
+import org.apache.flink.table.types.logical.LogicalTypeRoot;
+import org.apache.flink.types.Row;
+import org.apache.flink.util.CloseableIterator;
+
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.ExpectedException;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+import org.junit.runners.Suite;
+
+import java.util.Arrays;
+import java.util.List;
+
+import static java.util.Collections.singletonMap;
+import static org.apache.flink.table.api.DataTypes.ARRAY;
+import static org.apache.flink.table.api.DataTypes.BIGINT;
+import static org.apache.flink.table.api.DataTypes.FIELD;
+import static org.apache.flink.table.api.DataTypes.MAP;
+import static org.apache.flink.table.api.DataTypes.ROW;
+import static org.apache.flink.table.api.DataTypes.STRING;
+import static org.apache.flink.table.api.Expressions.$;
+import static org.apache.flink.table.api.Expressions.call;
+import static org.hamcrest.CoreMatchers.equalTo;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertThat;
+
+/**
+ * Tests for functions that access nested fields/elements of composite/collection types.
+ */
+@RunWith(Suite.class)
+@Suite.SuiteClasses(
+ {
+ CompositeTypeAccessExpressionITCase.TableFieldAccess.class,
+ CompositeTypeAccessExpressionITCase.CallFieldAccess.class
+ }
+)
+public class CompositeTypeAccessExpressionITCase {
+
+ /**
+ * Regular tests. See also {@link CallFieldAccess} for tests that access a nested field of an expression or
+ * for {@link BuiltInFunctionDefinitions#FLATTEN} which produces multiple columns from a single one.
+ */
+ public static class TableFieldAccess extends BuiltInFunctionTestBase {
+ @Parameterized.Parameters(name = "{index}: {0}")
+ public static List<TestSpec> testData() {
+ return Arrays.asList(
+
+ // Actually in case of SQL it does not use the GET method, but
+ // a custom logic for accessing nested fields of a Table.
+ TestSpec.forFunction(BuiltInFunctionDefinitions.GET)
+ .onFieldsWithData(null, Row.of(1))
+ .andDataTypes(
+ ROW(FIELD("nested", BIGINT().notNull())).nullable(),
+ ROW(FIELD("nested", BIGINT().notNull())).notNull()
+ )
+ .testTableApiResult($("f0").get("nested"), null, BIGINT().nullable())
Review comment:
use the newly introduced `testResult` such that a result must be defined only once
##########
File path: flink-table/flink-table-planner-blink/src/main/scala/org/apache/flink/table/planner/codegen/ExprCodeGenerator.scala
##########
@@ -724,6 +724,9 @@ class ExprCodeGenerator(ctx: CodeGeneratorContext, nullableInput: Boolean)
val key = operands(1)
generateMapGet(ctx, operands.head, key)
+ case t: LogicalType if TypeCheckUtils.isRow(t) =>
Review comment:
use `LogicalTypeChecks` instead
##########
File path: flink-table/flink-table-common/src/main/java/org/apache/flink/table/types/inference/TypeStrategies.java
##########
@@ -315,6 +317,36 @@ public static TypeStrategy nullable(TypeStrategy initialStrategy) {
return Optional.of(fromLogicalToDataType(inferredType));
};
+ /**
+ * Type strategy that returns a type of a field nested inside a composite type that is described by the second argument.
+ * The second argument must be a literal that describes either the nested field name or index.
+ */
+ public static final TypeStrategy GET = callContext -> {
+ List<DataType> argumentDataTypes = callContext.getArgumentDataTypes();
+ DataType rowDataType = argumentDataTypes.get(0);
+ TableSchema nestedSchema = DataTypeUtils.expandCompositeTypeToSchema(rowDataType);
Review comment:
I don't like that we use `TableSchema` here. This class doesn't belong here. Furthermore, it does not support distinct type yet. Maybe we can just introduce a `DataTypeUtils.getFieldType(int) and DataTypeUtils.getFieldType(name)`. Can we make it support distinct type (an `AtomicDataType`)?
##########
File path: flink-table/flink-table-common/src/test/java/org/apache/flink/table/types/inference/InputTypeStrategiesTest.java
##########
@@ -582,7 +587,43 @@
"My constraint says %s must be nullable.",
args -> args.get(0).getLogicalType().isNullable()))))
.calledWithArgumentTypes(DataTypes.BOOLEAN().notNull())
- .expectErrorMessage("My constraint says BOOLEAN NOT NULL must be nullable.")
+ .expectErrorMessage("My constraint says BOOLEAN NOT NULL must be nullable."),
+
+ TestSpec
+ .forStrategy(
+ "Composite type strategy with ROW",
+ sequence(InputTypeStrategies.COMPOSITE)
+ )
+ .calledWithArgumentTypes(DataTypes.ROW(DataTypes.FIELD("f0", DataTypes.BIGINT())))
+ .expectSignature("f(<COMPOSITE>)")
+ .expectArgumentTypes(DataTypes.ROW(DataTypes.FIELD("f0", DataTypes.BIGINT()))),
+
+ TestSpec
+ .forStrategy(
+ "Composite type strategy with STRUCTURED type",
+ sequence(InputTypeStrategies.COMPOSITE)
+ )
+ .calledWithArgumentTypes(new FieldsDataType(
Review comment:
shall we upgrade the test base to accept `AbstractDataType` similar to `DataStructureConverterTest`? it would make the code more readable.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #12649: [FLINK-18286] Implement type inference for GET/FLATTEN
Posted by GitBox <gi...@apache.org>.
flinkbot commented on pull request #12649:
URL: https://github.com/apache/flink/pull/12649#issuecomment-643942361
Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community
to review your pull request. We will use this comment to track the progress of the review.
## Automated Checks
Last check on commit ba5bc62416b82d8e6f1be2c848242de2d3c546e5 (Mon Jun 15 07:00:18 UTC 2020)
**Warnings:**
* No documentation files were touched! Remember to keep the Flink docs up to date!
<sub>Mention the bot in a comment to re-run the automated checks.</sub>
## Review Progress
* ❓ 1. The [description] looks good.
* ❓ 2. There is [consensus] that the contribution should go into to Flink.
* ❓ 3. Needs [attention] from.
* ❓ 4. The change fits into the overall [architecture].
* ❓ 5. Overall code [quality] is good.
Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process.<details>
The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required <summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`)
- `@flinkbot approve all` to approve all aspects
- `@flinkbot approve-until architecture` to approve everything until `architecture`
- `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention
- `@flinkbot disapprove architecture` to remove an approval you gave earlier
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12649: [FLINK-18286] Implement type inference for GET/FLATTEN
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #12649:
URL: https://github.com/apache/flink/pull/12649#issuecomment-643957887
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ba5bc62416b82d8e6f1be2c848242de2d3c546e5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3489",
"triggerID" : "ba5bc62416b82d8e6f1be2c848242de2d3c546e5",
"triggerType" : "PUSH"
}, {
"hash" : "c38f0da0bd042779c6c9a096e305cb40e8b1e367",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3531",
"triggerID" : "c38f0da0bd042779c6c9a096e305cb40e8b1e367",
"triggerType" : "PUSH"
}, {
"hash" : "b3dd92d0c41cee2472bb521eec89741c42bc05cd",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3580",
"triggerID" : "b3dd92d0c41cee2472bb521eec89741c42bc05cd",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b3dd92d0c41cee2472bb521eec89741c42bc05cd Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3580)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] twalthr commented on a change in pull request #12649: [FLINK-18286] Implement type inference for GET/FLATTEN
Posted by GitBox <gi...@apache.org>.
twalthr commented on a change in pull request #12649:
URL: https://github.com/apache/flink/pull/12649#discussion_r443372860
##########
File path: flink-table/flink-table-planner-blink/src/main/scala/org/apache/flink/table/planner/codegen/ExprCodeGenerator.scala
##########
@@ -719,18 +721,18 @@ class ExprCodeGenerator(ctx: CodeGeneratorContext, nullableInput: Boolean)
generateMap(ctx, resultType, operands)
case ITEM =>
- operands.head.resultType match {
- case t: LogicalType if TypeCheckUtils.isArray(t) =>
+ operands.head.resultType.getTypeRoot match {
+ case LogicalTypeRoot.ARRAY =>
val array = operands.head
val index = operands(1)
requireInteger(index)
generateArrayElementAt(ctx, array, index)
- case t: LogicalType if TypeCheckUtils.isMap(t) =>
+ case LogicalTypeRoot.MAP =>
val key = operands(1)
generateMapGet(ctx, operands.head, key)
- case t: LogicalType if TypeCheckUtils.isRow(t) =>
+ case LogicalTypeRoot.ROW =>
Review comment:
LogicalTypeRoot.ROW | LogicalTypeRoot.STRUCTURED_TYPE
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12649: [FLINK-18286] Implement type inference for GET/FLATTEN
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #12649:
URL: https://github.com/apache/flink/pull/12649#issuecomment-643957887
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ba5bc62416b82d8e6f1be2c848242de2d3c546e5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3489",
"triggerID" : "ba5bc62416b82d8e6f1be2c848242de2d3c546e5",
"triggerType" : "PUSH"
}, {
"hash" : "c38f0da0bd042779c6c9a096e305cb40e8b1e367",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3531",
"triggerID" : "c38f0da0bd042779c6c9a096e305cb40e8b1e367",
"triggerType" : "PUSH"
}, {
"hash" : "b3dd92d0c41cee2472bb521eec89741c42bc05cd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3580",
"triggerID" : "b3dd92d0c41cee2472bb521eec89741c42bc05cd",
"triggerType" : "PUSH"
}, {
"hash" : "326222e8eb85270297876eba22ffda57ebe592c3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3639",
"triggerID" : "326222e8eb85270297876eba22ffda57ebe592c3",
"triggerType" : "PUSH"
}, {
"hash" : "67905cf92819be26e49afbcf38794e340cca8e9c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3677",
"triggerID" : "67905cf92819be26e49afbcf38794e340cca8e9c",
"triggerType" : "PUSH"
}, {
"hash" : "4588093f97836f4f04c7c2c9992ea5049bf71c4c",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3694",
"triggerID" : "4588093f97836f4f04c7c2c9992ea5049bf71c4c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 67905cf92819be26e49afbcf38794e340cca8e9c Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3677)
* 4588093f97836f4f04c7c2c9992ea5049bf71c4c Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3694)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org