[GitHub] rednaxelafx opened a new pull request #23587: [SPARK-26664][SQL] Make DecimalType's minimum adjusted scale configurable

[GitBox <gi...@apache.org>]

rednaxelafx opened a new pull request #23587: [SPARK-26664][SQL] Make DecimalType's minimum adjusted scale configurable
URL: https://github.com/apache/spark/pull/23587
 
 
   ## What changes were proposed in this pull request?
   
   Introduce a new conf flag that allows the user to set the value of `DecimalType.MINIMAL_ADJUSTED_SCALE`, currently a constant of 6, to match their workloads' needs.
   
   The new flag is `spark.sql.decimalOperations.minimumAdjustedScale`.
   
   https://github.com/apache/spark/pull/20023 introduced a new conf flag `spark.sql.decimalOperations.allowPrecisionLoss` to match SQL Server's and new Hive's behavior of allowing precision loss when performing multiplication/division on big and small decimal numbers.
   Along with this feature, a constant `MINIMAL_ADJUSTED_SCALE` was set to 6 for when precision loss is allowed.
   
   Some customer workload may needed a larger adjusted scale to match their business needs, and in exchange they may be willing to tolerate some more calculations overflowing the max precision, leading to nulls. So they would like the minimum adjusted scale to be configurable. Thus the need for a new conf.
   
   The default behavior after introducing this conf flag is not changed.
   
   ## How was this patch tested?
   
   Added a new section in SQL tests to test the behavior of setting minimumAdjustedScale=12.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org