You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Jonathan Ross (Jira)" <ji...@apache.org> on 2023/05/30 18:50:00 UTC
[jira] [Updated] (SPARK-43887) Reduce function doesn't infer Decimal type precision correctly
[ https://issues.apache.org/jira/browse/SPARK-43887?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ross updated SPARK-43887:
----------------------------------
Priority: Major (was: Critical)
> Reduce function doesn't infer Decimal type precision correctly
> --------------------------------------------------------------
>
> Key: SPARK-43887
> URL: https://issues.apache.org/jira/browse/SPARK-43887
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 3.4.0
> Reporter: Jonathan Ross
> Priority: Major
>
> A working reduce function example
>
> {code:java}
> select reduce(arrays, array(0, 0), (acc,x) -> array(get(acc,0)+get(x,0), get(acc,1)+get(x,1))) from (select array(array(1, 2), array(1000, 2000)) as arrays){code}
>
> produces
> {code:java}
> [1001,2002]{code}
>
> But, if you introduce decimals, it breaks:
> {code:java}
> select reduce(arrays, array(0, 0), (acc,x) -> array(get(acc,0)+get(x,0), get(acc,1)+get(x,1))) from (select array(array(1, 2), array(1000.00, 2000)) as arrays); {code}
> produces:
> {code:java}
> [DATATYPE_MISMATCH.UNEXPECTED_INPUT_TYPE] Cannot resolve "reduce(arrays, array(0, 0), lambdafunction(array((get(namedlambdavariable(), 0) + get(namedlambdavariable(), 0)), (get(namedlambdavariable(), 1) + get(namedlambdavariable(), 1))), namedlambdavariable(), namedlambdavariable()), lambdafunction(namedlambdavariable(), namedlambdavariable()))" due to data type mismatch: Parameter 3 requires the "ARRAY<INT>" type, however "lambdafunction(array((get(namedlambdavariable(), 0) + get(namedlambdavariable(), 0)), (get(namedlambdavariable(), 1) + get(namedlambdavariable(), 1))), namedlambdavariable(), namedlambdavariable())" has the type "ARRAY<DECIMAL(13,2)>".; line 1 pos 0; Project [reduce(arrays#6, array(0, 0), lambdafunction(array((cast(get(lambda acc#7, 0) as decimal(10,0)) + get(lambda x#8, 0)), (cast(get(lambda acc#7, 1) as decimal(10,0)) + get(lambda x#8, 1))), lambda acc#7, lambda x#8, false), lambdafunction(lambda id#9, lambda id#9, false)) AS reduce(arrays, array(0, 0), lambdafunction(array((get(namedlambdavariable(), 0) + get(namedlambdavariable(), 0)), (get(namedlambdavariable(), 1) + get(namedlambdavariable(), 1))), namedlambdavariable(), namedlambdavariable()), lambdafunction(namedlambdavariable(), namedlambdavariable()))#10] +- SubqueryAlias __auto_generated_subquery_name +- Project [array(cast(array(1, 2) as array<decimal(12,2)>), array(cast(1000.00 as decimal(12,2)), cast(2000 as decimal(12,2)))) AS arrays#6] +- OneRowRelation
> {code}
> We've reproduced the issue even when trying to specify the decimal cast of all inputs in the array and the types in the accumulator.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org