[GitHub] mgaido91 opened a new pull request #23308: [SPARK-26308][SQL] Infer abstract decimal type for java/scala BigDecimal

[GitBox <gi...@apache.org>]

mgaido91 opened a new pull request #23308: [SPARK-26308][SQL] Infer abstract decimal type for java/scala BigDecimal
URL: https://github.com/apache/spark/pull/23308
 
 
   ## What changes were proposed in this pull request?
   
   Currently, when we infer the schema for scala/java decimals, we return as data type the `SYSTEM_DEFAULT` implementation, ie. the decimal type with precision 38 and scale 18. But this is not right, as we know nothing about the right precision and scale and these values can be not enough to store the data. This problem arises in particular with UDF, where we cast all the input of type `DecimalType` to a `DecimalType(38, 18)`: in case this is not enough, null is returned as input for the UDF.
   
   The PR changes the resolution of `BigDecimal`/`Decimal` to the abstract `DecimalType`. Please notice that the same problem is still present for Decimals in arrays, structs and maps, because the related types don't accept an `AbstractDataType`. In a followup work, we can introduce abstract types for them accepting abstract types in order to fix the issue for them too.
   
   ## How was this patch tested?
   
   added UT
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org