You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "bersprockets (via GitHub)" <gi...@apache.org> on 2023/05/23 00:44:08 UTC
[GitHub] [spark] bersprockets opened a new pull request, #41267: [SPARK-43718][SQL] Set nullable correctly for keys in USING joins
bersprockets opened a new pull request, #41267:
URL: https://github.com/apache/spark/pull/41267
### What changes were proposed in this pull request?
In `Anaylzer#commonNaturalJoinProcessing`, set nullable correctly when adding the join keys to the Project's hidden_output tag.
### Why are the changes needed?
The value of the hidden_output tag will be used for resolution of attributes in parent operators, so incorrect nullabilty can cause problems.
For example, assume this data:
```
create or replace temp view t1 as values (1), (2), (3) as (c1);
create or replace temp view t2 as values (2), (3), (4) as (c1);
```
The following query produces incorrect results:
```
spark-sql (default)> select explode(array(t1.c1, t2.c1)) as x1
from t1
full outer join t2
using (c1);
1
-1 <== should be null
2
2
3
3
-1 <== should be null
4
Time taken: 0.663 seconds, Fetched 8 row(s)
spark-sql (default)>
```
Similar issues occur with right outer join and left outer join.
`t1.c1` and `t2.c1` have the wrong nullability at the time the array is resolved, so the array's `containsNull` value is incorrect.
`UpdateNullability` will update the nullability of `t1.c1` and `t2.c1` in the `CreateArray` arguments, but will not update `containsNull` in the function's data type.
Queries that don't use arrays also can get wrong results. Assume this data:
```
create or replace temp view t1 as values (0), (1), (2) as (c1);
create or replace temp view t2 as values (1), (2), (3) as (c1);
create or replace temp view t3 as values (1, 2), (3, 4), (4, 5) as (a, b);
```
The following query produces incorrect results:
```
select t1.c1 as t1_c1, t2.c1 as t2_c1, b
from t1
full outer join t2
using (c1),
lateral (
select b
from t3
where a = coalesce(t2.c1, 1)
) lt3;
1 1 2
NULL 3 4
Time taken: 2.395 seconds, Fetched 2 row(s)
spark-sql (default)>
```
The result should be the following:
```
0 NULL 2
1 1 2
NULL 3 4
```
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
New tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #41267: [SPARK-43718][SQL] Set nullable correctly for keys in USING joins
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on PR #41267:
URL: https://github.com/apache/spark/pull/41267#issuecomment-1558520783
late LGTM
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #41267: [SPARK-43718][SQL] Set nullable correctly for keys in USING joins
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on PR #41267:
URL: https://github.com/apache/spark/pull/41267#issuecomment-1558496978
cc @rednaxelafx @cloud-fan FYI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun closed pull request #41267: [SPARK-43718][SQL] Set nullable correctly for keys in USING joins
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun closed pull request #41267: [SPARK-43718][SQL] Set nullable correctly for keys in USING joins
URL: https://github.com/apache/spark/pull/41267
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org