You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2022/01/07 20:15:37 UTC
[GitHub] [spark] cdegroc opened a new pull request #35140: [SPARK-37829][SQL] DataFrame.joinWith should return null rows for missing values
cdegroc opened a new pull request #35140:
URL: https://github.com/apache/spark/pull/35140
### What changes were proposed in this pull request?
Add a unit test demonstrating the regression on `DataFrame.joinWith`.
Revert [commit cd92f25be5a221e0d4618925f7bc9dfd3bb8cb59](https://github.com/apache/spark/commit/cd92f25be5a221e0d4618925f7bc9dfd3bb8cb59) making the test pass.
### Why are the changes needed?
Doing an outer-join using joinWith on DataFrames used to return missing values as null in Spark 2.4.8, but returns them as Rows with null values in Spark 3.0.0+.
The regression has been introduced in [commit cd92f25be5a221e0d4618925f7bc9dfd3bb8cb59](https://github.com/apache/spark/commit/cd92f25be5a221e0d4618925f7bc9dfd3bb8cb59).
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
A unit test was added.
Ran unit tests for the `sql-core` and `sql-catalyst` submodules with `./build/mvn clean package -pl sql/core,cql/catalyst`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #35140: [SPARK-37829][SQL] DataFrame.joinWith should return null rows for missing values
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #35140:
URL: https://github.com/apache/spark/pull/35140#issuecomment-1008094749
Can one of the admins verify this patch?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #35140: [SPARK-37829][SQL] DataFrame.joinWith should return null rows for missing values
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #35140:
URL: https://github.com/apache/spark/pull/35140#issuecomment-1008208298
cc @cloud-fan @viirya FYI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cdegroc commented on pull request #35140: [SPARK-37829][SQL] DataFrame.joinWith should return null rows for missing values
Posted by GitBox <gi...@apache.org>.
cdegroc commented on pull request #35140:
URL: https://github.com/apache/spark/pull/35140#issuecomment-1008674828
@gengliangwang both PRs solve SPARK-37829 in the same way. Only one PR should be picked (up to the maintainers).
The difference is that this one solves it by completely reverting the commit that introduced the bug, while the other keeps the current version of the code but fixes it to resolve the bug.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gengliangwang commented on pull request #35140: [SPARK-37829][SQL] DataFrame.joinWith should return null rows for missing values
Posted by GitBox <gi...@apache.org>.
gengliangwang commented on pull request #35140:
URL: https://github.com/apache/spark/pull/35140#issuecomment-1008667431
@cdegroc What is the relationship between this PR and https://github.com/apache/spark/pull/35139? The titles are the same
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org