You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by PierreB <pi...@realimpactanalytics.com> on 2015/12/21 21:45:43 UTC
Re: Tungsten gives unexpected results when selecting null elements
in array
For info, this is the generated code:
<pre>
GeneratedExpressionCode(
cursor8 = 16;
convertedStruct6.pointTo(buffer7, Platform.BYTE_ARRAY_OFFSET, 1,
cursor8);
/* input[0, ArrayType(StringType,true)][0] */
/* input[0, ArrayType(StringType,true)] */
boolean isNull2 = i.isNullAt(0);
ArrayData primitive3 = isNull2 ? null : (i.getArray(0));
boolean isNull0 = isNull2;
UTF8String primitive1 = null;
if (!isNull0) {
/* 0 */
if (!false) {
final int index = (int) 0;
if (index >= primitive3.numElements() || index < 0) {
isNull0 = true;
} else {
primitive1 = primitive3.getUTF8String(index);
}
} else {
isNull0 = true;
}
}
int numBytes10 = cursor8 + (isNull0 ? 0 :
org.apache.spark.sql.catalyst.expressions.UnsafeRowWriters$UTF8StringWriter.getSize(primitive1));
if (buffer7.length < numBytes10) {
// This will not happen frequently, because the buffer is
re-used.
byte[] tmpBuffer9 = new byte[numBytes10 * 2];
Platform.copyMemory(buffer7, Platform.BYTE_ARRAY_OFFSET,
tmpBuffer9, Platform.BYTE_ARRAY_OFFSET, buffer7.length);
buffer7 = tmpBuffer9;
}
convertedStruct6.pointTo(buffer7, Platform.BYTE_ARRAY_OFFSET, 1,
numBytes10);
if (isNull0) {
convertedStruct6.setNullAt(0);
} else {
cursor8 +=
org.apache.spark.sql.catalyst.expressions.UnsafeRowWriters$UTF8StringWriter.write(convertedStruct6,
0, cursor8, primitive1);
}
,false,convertedStruct6)
</pre>
The culprit line is the following:
<pre>
int numBytes10 = cursor8 + (isNull0 ? 0 :
org.apache.spark.sql.catalyst.expressions.UnsafeRowWriters$UTF8StringWriter.getSize(primitive1));
</pre>
--
View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/Tungsten-gives-unexpected-results-when-selecting-null-elements-in-array-tp15717p15718.html
Sent from the Apache Spark Developers List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org