You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Apache Spark (Jira)" <ji...@apache.org> on 2022/05/03 20:40:00 UTC
[jira] [Assigned] (SPARK-39093) Dividing interval by integral can result in codegen compilation error
[ https://issues.apache.org/jira/browse/SPARK-39093?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Apache Spark reassigned SPARK-39093:
------------------------------------
Assignee: (was: Apache Spark)
> Dividing interval by integral can result in codegen compilation error
> ---------------------------------------------------------------------
>
> Key: SPARK-39093
> URL: https://issues.apache.org/jira/browse/SPARK-39093
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 3.2.1, 3.3.0, 3.4.0
> Reporter: Bruce Robbins
> Priority: Minor
>
> Assume this data:
> {noformat}
> create or replace temp view v1 as
> select * FROM VALUES
> (interval '10' months, interval '10' day, 2)
> as v1(period, duration, num);
> cache table v1;
> {noformat}
> These two queries work:
> {noformat}
> spark-sql> select period/num from v1;
> 0-5
> Time taken: 0.143 seconds, Fetched 1 row(s)
> {noformat}
> {noformat}
> spark-sql> select duration/num from v1;
> 5 00:00:00.000000000
> Time taken: 0.094 seconds, Fetched 1 row(s)
> {noformat}
> However, these two queries get a codegen compilation error:
> {noformat}
> spark-sql> select period/(num + 3) from v1;
> 22/05/03 08:56:37 ERROR CodeGenerator: failed to compile: org.codehaus.commons.compiler.CompileException: File 'generated.java', Line 40, Column 44: Expression "project_value_2" is not an rvalue
> ...
> 22/05/03 08:56:37 WARN UnsafeProjection: Expr codegen error and falling back to interpreter mode
> ...
> 0-2
> Time taken: 0.149 seconds, Fetched 1 row(s)
> {noformat}
> {noformat}
> spark-sql> select duration/(num + 3) from v1;
> 22/05/03 08:57:29 ERROR CodeGenerator: failed to compile: org.codehaus.commons.compiler.CompileException: File 'generated.java', Line 40, Column 54: Expression "project_value_2" is not an rvalue
> ...
> 22/05/03 08:57:29 WARN UnsafeProjection: Expr codegen error and falling back to interpreter mode
> ...
> 2 00:00:00.000000000
> Time taken: 0.089 seconds, Fetched 1 row(s)
> {noformat}
> Even the first two queries will get a compilation error if you turn off whole-stage codegen:
> {noformat}
> spark-sql> set spark.sql.codegen.wholeStage=false;
> spark.sql.codegen.wholeStage false
> Time taken: 0.055 seconds, Fetched 1 row(s)
> spark-sql> select period/num from v1;
> 22/05/03 09:16:42 ERROR CodeGenerator: failed to compile: org.codehaus.commons.compiler.CompileException: File 'generated.java', Line 37, Column 5: Expression "value_1" is not an rvalue
> ....
> 0-5
> Time taken: 0.175 seconds, Fetched 1 row(s)
> spark-sql> select duration/num from v1;
> 22/05/03 09:17:41 ERROR CodeGenerator: failed to compile: org.codehaus.commons.compiler.CompileException: File 'generated.java', Line 37, Column 5: Expression "value_1" is not an rvalue
> ...
> 5 00:00:00.000000000
> Time taken: 0.104 seconds, Fetched 1 row(s)
> {noformat}
> Note that in the error cases, the queries still return a result because Spark falls back on interpreting the divide expression (so I marked this as "minor").
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org