You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Hiroshi Inoue (JIRA)" <ji...@apache.org> on 2017/06/23 06:46:00 UTC
[jira] [Closed] (SPARK-15795) Enable more optimizations in whole
stage codegen when isNull is a compile-time constant
[ https://issues.apache.org/jira/browse/SPARK-15795?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hiroshi Inoue closed SPARK-15795.
---------------------------------
Resolution: Invalid
Since this implementation is hacky and we need other solutions.
> Enable more optimizations in whole stage codegen when isNull is a compile-time constant
> ---------------------------------------------------------------------------------------
>
> Key: SPARK-15795
> URL: https://issues.apache.org/jira/browse/SPARK-15795
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Reporter: Hiroshi Inoue
>
> Whole stage codegen often creates {{isNull}} variable initialized with constant _false_, like
> {{boolean mapelements_isNull = false || false;}}
> If there is no further assignment for this {{isNull}} variable, whole stage codegen can do more optimizations by assuming {{isNull}} as a compile-time constant.
> In the example below, which is generated for a dataset map operation, {{mapelements_isNull}} defined at line 115 can be assumed by a compile-time constant (false).
> By assuming this as a constant, the whole stage codegen eliminates {{zeroOutNullBytes}} at line 119 and an if-statement at line 121.
> In addition to the benefits of improved readability of generated code, eliminating {{zeroOutNullBytes}} will give performance advantage since it is difficult to remove for Java JIT compiler.
> {code}
> /* 107 */ // CONSUME: Project [id#0L AS l#3L]
> /* 108 */ // CONSUME: DeserializeToObject l#3: bigint, obj#16: bigint
> /* 109 */ // CONSUME: MapElements <function1>, obj#17: bigint
> /* 110 */ // CONSUME: SerializeFromObject [input[0, bigint, true] AS value#18L]
> /* 111 */ // <function1>.apply
> /* 112 */ Object mapelements_obj = ((Expression) references[1]).eval(null);
> /* 113 */ scala.Function1 mapelements_value1 = (scala.Function1) mapelements_obj;
> /* 114 */
> /* 115 */ boolean mapelements_isNull = false || false;
> /* 116 */ final long mapelements_value = mapelements_isNull ? -1L : (Long) mapelements_value1.apply(range_value);
> /* 117 */
> /* 118 */ // CONSUME: WholeStageCodegen
> /* 119 */ serializefromobject_rowWriter.zeroOutNullBytes();
> /* 120 */
> /* 121 */ if (mapelements_isNull) {
> /* 122 */ serializefromobject_rowWriter.setNullAt(0);
> /* 123 */ } else {
> /* 124 */ serializefromobject_rowWriter.write(0, mapelements_value);
> /* 125 */ }
> /* 126 */ append(serializefromobject_result);
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org