You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Sylvain Zimmer (JIRA)" <ji...@apache.org> on 2016/07/29 22:56:20 UTC
[jira] [Updated] (SPARK-16807) Optimize some ABS() statements
[ https://issues.apache.org/jira/browse/SPARK-16807?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sylvain Zimmer updated SPARK-16807:
-----------------------------------
Description:
I'm not a Catalyst expert, but I think some use cases for the ABS() function could generate simpler code.
This is the code generated when doing something like {{ABS(x - y) > 0}} or {{ABS(x - y) = 0}} in Spark SQL:
{code}
/* 267 */ float filter_value6 = -1.0f;
/* 268 */ filter_value6 = agg_value27 - agg_value32;
/* 269 */ float filter_value5 = -1.0f;
/* 270 */ filter_value5 = (float)(java.lang.Math.abs(filter_value6));
/* 271 */
/* 272 */ boolean filter_value4 = false;
/* 273 */ filter_value4 = org.apache.spark.util.Utils.nanSafeCompareFloats(filter_value5, 0.0f) > 0;
/* 274 */ if (!filter_value4) continue;
{code}
Maybe it could all be simplified to something like this?
{code}
filter_value4 = (agg_value27 != agg_value32)
{code}
(Of course you could write {{x != y}} directly in the SQL query, but the {{0}} in my example could be a configurable threshold, not something you can hardcode)
was:
I'm not a Catalyst expert, but I think some use cases for the ABS() function could generate simpler code.
This is the code generated when doing something like {{ABS(x - y) > 0}} or {{ABS(x - y) = 0}} in Spark SQL:
{code}
/* 267 */ float filter_value6 = -1.0f;
/* 268 */ filter_value6 = agg_value27 - agg_value32;
/* 269 */ float filter_value5 = -1.0f;
/* 270 */ filter_value5 = (float)(java.lang.Math.abs(filter_value6));
/* 271 */
/* 272 */ boolean filter_value4 = false;
/* 273 */ filter_value4 = org.apache.spark.util.Utils.nanSafeCompareFloats(filter_value5, 0.0f) > 0;
/* 274 */ if (!filter_value4) continue;
{code}
Maybe it could all be simplified to something like this?
{code}
filter_value4 = (agg_value27 != agg_value32)
{code}
> Optimize some ABS() statements
> ------------------------------
>
> Key: SPARK-16807
> URL: https://issues.apache.org/jira/browse/SPARK-16807
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Reporter: Sylvain Zimmer
>
> I'm not a Catalyst expert, but I think some use cases for the ABS() function could generate simpler code.
> This is the code generated when doing something like {{ABS(x - y) > 0}} or {{ABS(x - y) = 0}} in Spark SQL:
> {code}
> /* 267 */ float filter_value6 = -1.0f;
> /* 268 */ filter_value6 = agg_value27 - agg_value32;
> /* 269 */ float filter_value5 = -1.0f;
> /* 270 */ filter_value5 = (float)(java.lang.Math.abs(filter_value6));
> /* 271 */
> /* 272 */ boolean filter_value4 = false;
> /* 273 */ filter_value4 = org.apache.spark.util.Utils.nanSafeCompareFloats(filter_value5, 0.0f) > 0;
> /* 274 */ if (!filter_value4) continue;
> {code}
> Maybe it could all be simplified to something like this?
> {code}
> filter_value4 = (agg_value27 != agg_value32)
> {code}
> (Of course you could write {{x != y}} directly in the SQL query, but the {{0}} in my example could be a configurable threshold, not something you can hardcode)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org