You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2019/07/12 14:47:55 UTC
[GitHub] [spark] wangyum opened a new pull request #25137:
[SPARK-28348][SQL] Decimal precision promotion for binary arithmetic with
casted decimal type
wangyum opened a new pull request #25137: [SPARK-28348][SQL] Decimal precision promotion for binary arithmetic with casted decimal type
URL: https://github.com/apache/spark/pull/25137
## What changes were proposed in this pull request?
How to reproduce this issue:
```sql
create table spark_28348 using parquet as select * from values
(cast(-34338492.215397047 as decimal(38, 18)))
as v(c1);
```
```shell
scala> spark.sql("select cast(c1 * cast(-34338492.215397047 as decimal(38, 18)) as decimal(38, 18)) as c1 from spark_28348").show(false)
+-----------------------------------+
|c1 |
+-----------------------------------+
|1179132047626883.596862000000000000|
+-----------------------------------+
scala> spark.sql("select cast(c1 * cast(-34338492.215397047 as decimal(38, 18)) as decimal(38, 18)) as c1 from spark_28348").explain(false);
== Physical Plan ==
*(1) Project [cast(CheckOverflow((promote_precision(c1#1) * -34338492.215397047000000000), DecimalType(38,6), true) as decimal(38,18)) AS c1#13]
+- *(1) ColumnarToRow
+- FileScan parquet default.spark_28348[c1#1] Batched: true, DataFilters: [], Format: Parquet, Location: InMemoryFileIndex[file:/user/hive/warehouse/spark_28348], PartitionFilters: [], PushedFilters: [], ReadSchema: struct<c1:decimal(38,18)>
```
It first `CheckOverflow` with `DecimalType(38,6)` and then cast to `decimal(38,18)`.
We can use `decimal(38,18)` to `CheckOverflow`. This has two benefits:
1. Reduce one cast.
2. Higher data precision.
After this PR:
```shell
scala> spark.sql("select cast(c1 * cast(-34338492.215397047 as decimal(38, 18)) as decimal(38, 18)) as c1 from spark_28348").show(false)
+-----------------------------------+
|c1 |
+-----------------------------------+
|1179132047626883.596862135856320209|
+-----------------------------------+
scala> spark.sql("select cast(c1 * cast(-34338492.215397047 as decimal(38, 18)) as decimal(38, 18)) as c1 from spark_28348").explain(false);
== Physical Plan ==
*(1) Project [CheckOverflow((promote_precision(c1#1) * -34338492.215397047000000000), DecimalType(38,18), true) AS c1#17]
+- *(1) ColumnarToRow
+- FileScan parquet default.spark_28348[c1#1] Batched: true, DataFilters: [], Format: Parquet, Location: InMemoryFileIndex[file:/user/hive/warehouse/spark_28348], PartitionFilters: [], PushedFilters: [], ReadSchema: struct<c1:decimal(38,18)>
```
## How was this patch tested?
unit tests
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org