[jira] [Created] (SPARK-50293) Support high precision DECIMAL256 in SparkSQL and Dataframe

["Eric Sun (Jira)" <ji...@apache.org>]

Eric Sun created SPARK-50293:
--------------------------------

             Summary: Support high precision DECIMAL256 in SparkSQL and Dataframe
                 Key: SPARK-50293
                 URL: https://issues.apache.org/jira/browse/SPARK-50293
             Project: Spark
          Issue Type: Wish
          Components: Spark Core, SQL
    Affects Versions: 3.5.3, 4.0.0
            Reporter: Eric Sun


Financial services can't use DOUBLE to handle money values, and DECIMAL(38,x) is not sufficient for Crypto because we need to handle DECIMAL(54,24) or DECIMAL(60,24) {color:#ff0000}*without*{color} precision loss.

Originally, long time ago, SparkSQL supported Java BigDecimal for some time, but later it reverted the support to be *compatible* with the HIVE standard (only up to Decimal(38,x)).
{quote}[https://github.com/apache/spark/blob/master/sql/api/src/main/scala/org/apache/spark/sql/types/Decimal.scala] fully uses BigInteger and BigDecimal already.
{quote}
----
Postgres can handle financial data with very high precision. If Databricks can support this feature, it will be a great differentiator (compared with other Big Data analytical systems) for the entire Financial Service sector.

 
The request is to support Java BigDecimal and/or Decimal256 (Apache Arrow supports it already) in Spark Tungsten Dataframe, SQL and Unity Catalog.

 
 * [/java/format/src/main/java/org/apache/arrow/flatbuf/Decimal.java|https://github.com/apache/arrow/blob/main/java/format/src/main/java/org/apache/arrow/flatbuf/Decimal.java]
 ** [/java/vector/src/main/java/org/apache/arrow/vector/Decimal256Vector.java|https://github.com/apache/arrow/blob/main/java/vector/src/main/java/org/apache/arrow/vector/Decimal256Vector.java]
 ** [/java/vector/src/main/java/org/apache/arrow/vector/util/DecimalUtility.java#L46|https://github.com/apache/arrow/blob/main/java/vector/src/main/java/org/apache/arrow/vector/util/DecimalUtility.java#L46]
 ** [/java/vector/src/test/java/org/apache/arrow/vector/TestDecimal256Vector.java|https://github.com/apache/arrow/blob/main/java/vector/src/test/java/org/apache/arrow/vector/TestDecimal256Vector.java]



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org