[jira] [Commented] (SPARK-40032) Support Decimal128 type

["jiaan.geng (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-40032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17577942#comment-17577942 ] 

jiaan.geng commented on SPARK-40032:
------------------------------------

We are editing the design doc. 

> Support Decimal128 type
> -----------------------
>
>                 Key: SPARK-40032
>                 URL: https://issues.apache.org/jira/browse/SPARK-40032
>             Project: Spark
>          Issue Type: New Feature
>          Components: SQL
>    Affects Versions: 3.4.0
>            Reporter: jiaan.geng
>            Priority: Major
>         Attachments: Performance comparison between decimal128 and spark decimal benchmark.pdf
>
>
> Spark SQL today supports the DECIMAL data type. The implementation of Decimal that can hold a BigDecimal or Long.  Decimal provides some operators like +, -, *, / and so on.
> Take the + as example, the implementation show below.
> {code:java}
>   def + (that: Decimal): Decimal = {
>     if (decimalVal.eq(null) && that.decimalVal.eq(null) && scale == that.scale) {
>       Decimal(longVal + that.longVal, Math.max(precision, that.precision) + 1, scale)
>     } else {
>       Decimal(toBigDecimal.bigDecimal.add(that.toBigDecimal.bigDecimal))
>     }
>   }
> {code}
> We can see there exists two addition and call Decimal.apply. The add operator of BigDecimal will construct a new BigDecimal instance.
> The implementation of Decimal.apply will call new to construct a new Decimal instance with the new BigDecimal instance.
> As we know, Decimal instance will hold the new BigDecimal instance.
> If a large table has a Decimal field called 'colA, the execution of SUM('colA) will involve the creation of a large number of Decimal instances and BigDecimal instances. These Decimal instances and BigDecimal instances will lead to garbage collection to occur frequently.
> Decimal128 is a high-performance decimal about 8X more efficient than Java BigDecimal for typical operations. It uses a finite (128 bit) precision and can handle up to decimal(38, X). It is also "mutable" so you can change the contents of an existing object. This helps reduce the cost of new() and garbage collection.
> We have generate a benchmark report for compare Spark Decimal, Java BigDecimal and Decimal128. Please see the attachment.
> In this new feature, we will introduce DECIMAL128 to accelerate decimal calculation.
> h3. Milestone 1 – Spark Decimal equivalency ( The new Decimal type Decimal128 meets or exceeds all function of the existing SQL Decimal):
> * Add a new DataType implementation for Decimal128.
> * Support Decimal128 in Dataset/UDF.
> * Decimal128 literals
> * Decimal128 arithmetic(e.g. Decimal128 + Decimal128, Decimal128 - Decimal)
> * Decimal or Math functions/operators: POWER, LOG, Round, etc
> * Cast to and from Decimal128, cast String/Decimal to Decimal128, cast Decimal128 to string (pretty printing)/Decimal, with the * * SQL syntax to specify the types
> * Support sorting Decimal128.
> h3. Milestone 2 – Persistence:
>  * Ability to create tables of type Decimal128
>  * Ability to write to common file formats such as Parquet and JSON.
>  * INSERT, SELECT, UPDATE, MERGE
>  * Discovery
> h3. Milestone 3 – Client support
>  * JDBC support
>  * Hive Thrift server
> h3. Milestone 4 – PySpark and Spark R integration
>  * Python UDF can take and return Decimal128
>  * DataFrame support



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org