You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Hyukjin Kwon (JIRA)" <ji...@apache.org> on 2019/07/08 01:06:00 UTC
[jira] [Updated] (SPARK-27921) Convert applicable *.sql tests into
UDF integrated test base
[ https://issues.apache.org/jira/browse/SPARK-27921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hyukjin Kwon updated SPARK-27921:
---------------------------------
Description:
This JIRA targets to improve Python test coverage in particular about {{ExtractPythonUDFs}}.
This rule has caused many regressions or issues such as SPARK-27803, SPARK-26147, SPARK-26864, SPARK-26293, SPARK-25314 and SPARK-24721.
We should convert *.sql test cases that can be affected by this rule {{ExtractPythonUDFs}} like [https://github.com/apache/spark/blob/f5317f10b25bd193cf5026a8f4fd1cd1ded8f5b4/sql/core/src/test/resources/sql-tests/inputs/udf/udf-inner-join.sql]
Namely most of plan related test cases might have to be converted.
Here is the rough contribution guide to follow:
1. Copy and paste {{xxx.sql}} file into {{udf/udf-xxx.sql}}
2. Keep the comments and state that this file was copied from {{xxx.sql}}, for now.
3. Run it below:
{code:java}
SPARK_GENERATE_GOLDEN_FILES=1 build/sbt "sql/test-only *SQLQueryTestSuite -- -z udf/udf-xxx.sql"
git add .
{code}
4. Insert {{udf(...)}} into each statement. It is not required to add more combinations.
And it is not strict about where to insert.
5. Run it below again:
{code:java}
SPARK_GENERATE_GOLDEN_FILES=1 build/sbt "sql/test-only *SQLQueryTestSuite -- -z udf/udf-xxx.sql"
git diff
{code}
6. Compare results with original file, {{xxx.sql}}. If there are no notable diff, open a PR.
7. If there are diff, file or find the JIRA, skip the tests with comments.
was:
This JIRA targets to improve Python test coverage in particular about {{ExtractPythonUDFs}}.
This rule has caused many regressions or issues such as SPARK-27803, SPARK-26147, SPARK-26864, SPARK-26293, SPARK-25314 and SPARK-24721.
We should convert *.sql test cases that can be affected by this rule {{ExtractPythonUDFs}} like https://github.com/apache/spark/blob/f5317f10b25bd193cf5026a8f4fd1cd1ded8f5b4/sql/core/src/test/resources/sql-tests/inputs/udf/udf-inner-join.sql
Namely most of plan related test cases might have to be converted.
> Convert applicable *.sql tests into UDF integrated test base
> ------------------------------------------------------------
>
> Key: SPARK-27921
> URL: https://issues.apache.org/jira/browse/SPARK-27921
> Project: Spark
> Issue Type: Umbrella
> Components: PySpark, SQL
> Affects Versions: 3.0.0
> Reporter: Hyukjin Kwon
> Priority: Major
>
> This JIRA targets to improve Python test coverage in particular about {{ExtractPythonUDFs}}.
> This rule has caused many regressions or issues such as SPARK-27803, SPARK-26147, SPARK-26864, SPARK-26293, SPARK-25314 and SPARK-24721.
> We should convert *.sql test cases that can be affected by this rule {{ExtractPythonUDFs}} like [https://github.com/apache/spark/blob/f5317f10b25bd193cf5026a8f4fd1cd1ded8f5b4/sql/core/src/test/resources/sql-tests/inputs/udf/udf-inner-join.sql]
> Namely most of plan related test cases might have to be converted.
> Here is the rough contribution guide to follow:
> 1. Copy and paste {{xxx.sql}} file into {{udf/udf-xxx.sql}}
> 2. Keep the comments and state that this file was copied from {{xxx.sql}}, for now.
> 3. Run it below:
> {code:java}
> SPARK_GENERATE_GOLDEN_FILES=1 build/sbt "sql/test-only *SQLQueryTestSuite -- -z udf/udf-xxx.sql"
> git add .
> {code}
> 4. Insert {{udf(...)}} into each statement. It is not required to add more combinations.
> And it is not strict about where to insert.
> 5. Run it below again:
> {code:java}
> SPARK_GENERATE_GOLDEN_FILES=1 build/sbt "sql/test-only *SQLQueryTestSuite -- -z udf/udf-xxx.sql"
> git diff
> {code}
> 6. Compare results with original file, {{xxx.sql}}. If there are no notable diff, open a PR.
> 7. If there are diff, file or find the JIRA, skip the tests with comments.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org