You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2022/06/09 09:02:14 UTC
[GitHub] [spark] wangyum opened a new pull request, #36821: [SPARK-39226][SQL][FOLLOWUP] Update the migration guide after fixing the precision of the return type of round-like functions
wangyum opened a new pull request, #36821:
URL: https://github.com/apache/spark/pull/36821
### What changes were proposed in this pull request?
Update the migration guide after fixing the precision of the return type of round-like functions.
How to reproduce this issue:
```sql
-- Spark 3.2
CREATE TABLE t1(CURNCY_AMT DECIMAL(18,6)) using parquet;
CREATE VIEW v1 AS SELECT BROUND(CURNCY_AMT, 6) AS CURNCY_AMT FROM t1;
```
```sql
-- Spark 3.3
SELECT * FROM v1;
org.apache.spark.sql.AnalysisException: [CANNOT_UP_CAST_DATATYPE] Cannot up cast CURNCY_AMT from "DECIMAL(19,6)" to "DECIMAL(18,6)".
```
### Why are the changes needed?
Update the migration guide.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
N/A
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk commented on pull request #36821: [SPARK-39226][DOCS][FOLLOWUP] Update the migration guide after fixing the precision of the return type of round-like functions
Posted by GitBox <gi...@apache.org>.
MaxGekk commented on PR #36821:
URL: https://github.com/apache/spark/pull/36821#issuecomment-1150986767
Failed with:
```
[info] Main Java API documentation to /__w/spark/spark/target/javaunidoc...
[error] /__w/spark/spark/connector/avro/target/java/org/apache/spark/sql/avro/AvroDataToCatalyst.java:3:1: error: illegal combination of modifiers: abstract and static
[error] static public abstract R apply (T1 v1, T2 v2, T3 v3) ;
[error] ^
```
Is this related?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk closed pull request #36821: [SPARK-39226][DOCS][FOLLOWUP] Update the migration guide after fixing the precision of the return type of round-like functions
Posted by GitBox <gi...@apache.org>.
MaxGekk closed pull request #36821: [SPARK-39226][DOCS][FOLLOWUP] Update the migration guide after fixing the precision of the return type of round-like functions
URL: https://github.com/apache/spark/pull/36821
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] wangyum commented on pull request #36821: [SPARK-39226][SQL][FOLLOWUP] Update the migration guide after fixing the precision of the return type of round-like functions
Posted by GitBox <gi...@apache.org>.
wangyum commented on PR #36821:
URL: https://github.com/apache/spark/pull/36821#issuecomment-1150862860
cc @cloud-fan @MaxGekk
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] wangyum commented on pull request #36821: [SPARK-39226][DOCS][FOLLOWUP] Update the migration guide after fixing the precision of the return type of round-like functions
Posted by GitBox <gi...@apache.org>.
wangyum commented on PR #36821:
URL: https://github.com/apache/spark/pull/36821#issuecomment-1151029112
> Failed with:
>
> ```
> [info] Main Java API documentation to /__w/spark/spark/target/javaunidoc...
> [error] /__w/spark/spark/connector/avro/target/java/org/apache/spark/sql/avro/AvroDataToCatalyst.java:3:1: error: illegal combination of modifiers: abstract and static
> [error] static public abstract R apply (T1 v1, T2 v2, T3 v3) ;
> [error] ^
> ```
>
> Is this related?
I do not think it is related.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk commented on pull request #36821: [SPARK-39226][DOCS][FOLLOWUP] Update the migration guide after fixing the precision of the return type of round-like functions
Posted by GitBox <gi...@apache.org>.
MaxGekk commented on PR #36821:
URL: https://github.com/apache/spark/pull/36821#issuecomment-1151137268
+1, LGTM. Merging to master/3.3.
Thank you, @wangyum and @yaooqinn for review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk commented on a diff in pull request #36821: [SPARK-39226][DOCS][FOLLOWUP] Update the migration guide after fixing the precision of the return type of round-like functions
Posted by GitBox <gi...@apache.org>.
MaxGekk commented on code in PR #36821:
URL: https://github.com/apache/spark/pull/36821#discussion_r893296714
##########
docs/sql-migration-guide.md:
##########
@@ -65,6 +65,8 @@ license: |
- Since Spark 3.3, when reading values from a JSON attribute defined as `FloatType` or `DoubleType`, the strings `"+Infinity"`, `"+INF"`, and `"-INF"` are now parsed to the appropriate values, in addition to the already supported `"Infinity"` and `"-Infinity"` variations. This change was made to improve consistency with Jackson's parsing of the unquoted versions of these values. Also, the `allowNonNumericNumbers` option is now respected so these strings will now be considered invalid if this option is disabled.
- Since Spark 3.3, Spark will try to use built-in data source writer instead of Hive serde in `INSERT OVERWRITE DIRECTORY`. This behavior is effective only if `spark.sql.hive.convertMetastoreParquet` or `spark.sql.hive.convertMetastoreOrc` is enabled respectively for Parquet and ORC formats. To restore the behavior before Spark 3.3, you can set `spark.sql.hive.convertMetastoreInsertDir` to `false`.
+
+ - Since Spark 3.3, the precision of the return type of round-like functions has been fixed. This may cause Spark throw a `CANNOT_UP_CAST_DATATYPE` exception when using views created by prior versions. In such cases, you need to recreate the views using ALTER VIEW AS or CREATE OR REPLACE VIEW AS with newer Spark versions.
Review Comment:
nit: more precisely: ... Spark throw AnalysisException of the `CANNOT_UP_CAST_DATATYPE` error class
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] wangyum commented on a diff in pull request #36821: [SPARK-39226][DOCS][FOLLOWUP] Update the migration guide after fixing the precision of the return type of round-like functions
Posted by GitBox <gi...@apache.org>.
wangyum commented on code in PR #36821:
URL: https://github.com/apache/spark/pull/36821#discussion_r893411974
##########
docs/sql-migration-guide.md:
##########
@@ -65,6 +65,8 @@ license: |
- Since Spark 3.3, when reading values from a JSON attribute defined as `FloatType` or `DoubleType`, the strings `"+Infinity"`, `"+INF"`, and `"-INF"` are now parsed to the appropriate values, in addition to the already supported `"Infinity"` and `"-Infinity"` variations. This change was made to improve consistency with Jackson's parsing of the unquoted versions of these values. Also, the `allowNonNumericNumbers` option is now respected so these strings will now be considered invalid if this option is disabled.
- Since Spark 3.3, Spark will try to use built-in data source writer instead of Hive serde in `INSERT OVERWRITE DIRECTORY`. This behavior is effective only if `spark.sql.hive.convertMetastoreParquet` or `spark.sql.hive.convertMetastoreOrc` is enabled respectively for Parquet and ORC formats. To restore the behavior before Spark 3.3, you can set `spark.sql.hive.convertMetastoreInsertDir` to `false`.
+
+ - Since Spark 3.3, the precision of the return type of round-like functions has been fixed. This may cause Spark throw a `CANNOT_UP_CAST_DATATYPE` exception when using views created by prior versions. In such cases, you need to recreate the views using ALTER VIEW AS or CREATE OR REPLACE VIEW AS with newer Spark versions.
Review Comment:
fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org