You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by cloud-fan <gi...@git.apache.org> on 2016/08/09 10:19:29 UTC
[GitHub] spark pull request #14562: [SPARK-16973][SQL] remove the buffer offsets in I...
GitHub user cloud-fan opened a pull request:
https://github.com/apache/spark/pull/14562
[SPARK-16973][SQL] remove the buffer offsets in ImperativeAggregate
## What changes were proposed in this pull request?
the `mutableAggBufferOffset` and `inputAggBufferOffset` in `ImperativeAggregate` are really hard to understand and tightly coupled with aggregation implementation. What's worse, all `ImperativeAggregate` implementations need to understand this concept and deal with it carefully.
This PR isolate this buffet offsets concept into the base class `ImperativeAggregate`, by introducing a sliced row. Then all `ImperativeAggregate` implementations don't need to care about it anymore.
## How was this patch tested?
existing tests.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/cloud-fan/spark agg-minor
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/14562.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #14562
----
commit 9658056f420fc4280cc5ac656e7d45fc28a262de
Author: Wenchen Fan <we...@databricks.com>
Date: 2016-08-08T06:34:22Z
remove the buffer offsets in ImperativeAggregate
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14562: [SPARK-16973][SQL] remove the buffer offsets in I...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/14562#discussion_r74182507
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/slicedRows.scala ---
@@ -0,0 +1,108 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.expressions
+
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.util.{ArrayData, MapData}
+import org.apache.spark.sql.types.{DataType, Decimal}
+import org.apache.spark.unsafe.types.{CalendarInterval, UTF8String}
+
+/**
+ * A sliced row provides a view of the underlying row, and forwards all read/write requests to the
+ * underlying row, after shifting the index by offset.
+ *
+ * Note that, the underlying row field in this class is mutable, so that we can reuse the same
+ * instance of this class while processing many rows.
+ */
+trait BaseSlicedInternalRow extends InternalRow {
+ def baseRow: InternalRow
+ def offset: Int
+
+ protected def toBaseIndex(index: Int): Int = {
+ assert(index >= 0, "index (" + index + ") should >= 0")
+ assert(index < numFields, "index (" + index + ") should < " + numFields)
+ index + offset
+ }
+
+ override def isNullAt(i: Int): Boolean = baseRow.isNullAt(toBaseIndex(i))
+ override def getBoolean(i: Int): Boolean = baseRow.getBoolean(toBaseIndex(i))
+ override def getByte(i: Int): Byte = baseRow.getByte(toBaseIndex(i))
+ override def getShort(i: Int): Short = baseRow.getShort(toBaseIndex(i))
+ override def getInt(i: Int): Int = baseRow.getInt(toBaseIndex(i))
+ override def getLong(i: Int): Long = baseRow.getLong(toBaseIndex(i))
+ override def getFloat(i: Int): Float = baseRow.getFloat(toBaseIndex(i))
+ override def getDouble(i: Int): Double = baseRow.getDouble(toBaseIndex((i)))
+ override def getDecimal(i: Int, precision: Int, scale: Int): Decimal =
+ baseRow.getDecimal(toBaseIndex(i), precision, scale)
+ override def getUTF8String(i: Int): UTF8String = baseRow.getUTF8String(toBaseIndex(i))
+ override def getBinary(i: Int): Array[Byte] = baseRow.getBinary(toBaseIndex(i))
+ override def getInterval(i: Int): CalendarInterval = baseRow.getInterval(toBaseIndex(i))
+ override def getStruct(i: Int, numFields: Int): InternalRow =
+ baseRow.getStruct(toBaseIndex(i), numFields)
+ override def getArray(i: Int): ArrayData = baseRow.getArray(toBaseIndex(i))
+ override def getMap(i: Int): MapData = baseRow.getMap(toBaseIndex(i))
+ override def get(i: Int, dataType: DataType): AnyRef = baseRow.get(toBaseIndex(i), dataType)
+
+ override def anyNull: Boolean = {
+ val len = offset + numFields
+ var i = offset
+ while (i < len) {
+ if (isNullAt(i)) { return true }
+ i += 1
+ }
+ false
+ }
+
+ override def copy(): InternalRow = {
+ throw new UnsupportedOperationException("Cannot copy a SlicedMutableRow")
+ }
+}
+
+case class SlicedInternalRow(offset: Int, numFields: Int) extends BaseSlicedInternalRow {
+ private var _baseRow: InternalRow = _
+ def target(row: InternalRow): SlicedInternalRow = {
+ _baseRow = row
+ this
+ }
+
+ def baseRow: InternalRow = _baseRow
+}
+
+case class SlicedMutableRow(offset: Int, numFields: Int)
--- End diff --
does this need to be a case class?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14562: [SPARK-16973][SQL] remove the buffer offsets in Imperati...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14562
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/63908/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14562: [SPARK-16973][SQL] remove the buffer offsets in Imperati...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14562
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/63552/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14562: [SPARK-16973][SQL] remove the buffer offsets in I...
Posted by hvanhovell <gi...@git.apache.org>.
Github user hvanhovell commented on a diff in the pull request:
https://github.com/apache/spark/pull/14562#discussion_r74052366
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/interfaces.scala ---
@@ -219,111 +219,118 @@ sealed abstract class AggregateFunction extends Expression with ImplicitCastInpu
}
/**
- * API for aggregation functions that are expressed in terms of imperative initialize(), update(),
- * and merge() functions which operate on Row-based aggregation buffers.
- *
- * Within these functions, code should access fields of the mutable aggregation buffer by adding the
- * bufferSchema-relative field number to `mutableAggBufferOffset` then using this new field number
- * to access the buffer Row. This is necessary because this aggregation function's buffer is
- * embedded inside of a larger shared aggregation buffer when an aggregation operator evaluates
- * multiple aggregate functions at the same time.
- *
- * We need to perform similar field number arithmetic when merging multiple intermediate
- * aggregate buffers together in `merge()` (in this case, use `inputAggBufferOffset` when accessing
- * the input buffer).
- *
- * Correct ImperativeAggregate evaluation depends on the correctness of `mutableAggBufferOffset` and
- * `inputAggBufferOffset`, but not on the correctness of the attribute ids in `aggBufferAttributes`
- * and `inputAggBufferAttributes`.
+ * API for aggregation functions that are expressed in terms of imperative doInitialize(),
+ * doUpdate(), doMerge() and doComplete() functions which operate on Row-based aggregation buffers.
*/
abstract class ImperativeAggregate extends AggregateFunction with CodegenFallback {
/**
- * The offset of this function's first buffer value in the underlying shared mutable aggregation
- * buffer.
+ * The aggregation operator keeps a large shared mutable buffer row for all aggregate functions,
+ * each aggregate function should only access a slice of this shared buffer.
+ */
+ private var mutableBufferRow: SlicedMutableRow = _
+
+ /**
+ * During partial aggregation, the input buffer row to be merged is shared among all aggregate
+ * functions, each aggregate function should only access a slice of this input buffer.
+ */
+ private var inputBufferRow: SlicedInternalRow = _
+
+ /**
+ * Set the offset of this function's start buffer value in the underlying shared mutable
+ * aggregation buffer.
*
* For example, we have two aggregate functions `avg(x)` and `avg(y)`, which share the same
- * aggregation buffer. In this shared buffer, the position of the first buffer value of `avg(x)`
- * will be 0 and the position of the first buffer value of `avg(y)` will be 2:
+ * aggregation buffer. In this shared buffer, the position of the start buffer value of `avg(x)`
+ * will be 0 and the position of the start buffer value of `avg(y)` will be 2:
* {{{
- * avg(x) mutableAggBufferOffset = 0
+ * avg(x) mutable buffer offset is 0
* |
* v
* +--------+--------+--------+--------+
* | sum1 | count1 | sum2 | count2 |
* +--------+--------+--------+--------+
* ^
* |
- * avg(y) mutableAggBufferOffset = 2
+ * avg(y) mutable buffer offset is 2
* }}}
*/
- protected val mutableAggBufferOffset: Int
-
- /**
- * Returns a copy of this ImperativeAggregate with an updated mutableAggBufferOffset.
- * This new copy's attributes may have different ids than the original.
- */
- def withNewMutableAggBufferOffset(newMutableAggBufferOffset: Int): ImperativeAggregate
+ final def setMutableBufferOffset(offset: Int): Unit = {
+ assert(mutableBufferRow == null)
+ mutableBufferRow = SlicedMutableRow(offset, aggBufferAttributes.length)
+ }
/**
- * The offset of this function's start buffer value in the underlying shared input aggregation
+ * Set the offset of this function's start buffer value in the underlying shared input aggregation
* buffer. An input aggregation buffer is used when we merge two aggregation buffers together in
- * the `update()` function and is immutable (we merge an input aggregation buffer and a mutable
+ * the `merge()` function and is immutable (we merge an input aggregation buffer and a mutable
* aggregation buffer and then store the new buffer values to the mutable aggregation buffer).
*
* An input aggregation buffer may contain extra fields, such as grouping keys, at its start, so
- * mutableAggBufferOffset and inputAggBufferOffset are often different.
+ * mutable buffer offset and input buffer offset are often different.
*
* For example, say we have a grouping expression, `key`, and two aggregate functions,
- * `avg(x)` and `avg(y)`. In the shared input aggregation buffer, the position of the first
- * buffer value of `avg(x)` will be 1 and the position of the first buffer value of `avg(y)`
+ * `avg(x)` and `avg(y)`. In the shared input aggregation buffer, the position of the start
+ * buffer value of `avg(x)` will be 1 and the position of the start buffer value of `avg(y)`
* will be 3 (position 0 is used for the value of `key`):
* {{{
- * avg(x) inputAggBufferOffset = 1
+ * avg(x) input buffer offset is 1
* |
* v
* +--------+--------+--------+--------+--------+
* | key | sum1 | count1 | sum2 | count2 |
* +--------+--------+--------+--------+--------+
* ^
* |
- * avg(y) inputAggBufferOffset = 3
+ * avg(y) input buffer offset is 3
* }}}
*/
- protected val inputAggBufferOffset: Int
+ final def setInputBufferOffset(offset: Int): Unit = {
+ assert(inputBufferRow == null)
--- End diff --
This seems to be creating causing some trouble.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14562: [SPARK-16973][SQL] remove the buffer offsets in I...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/14562#discussion_r75076293
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/HyperLogLogPlusPlus.scala ---
@@ -142,23 +129,16 @@ case class HyperLogLogPlusPlus(
override def inputTypes: Seq[AbstractDataType] = Seq(AnyDataType)
- override def aggBufferSchema: StructType = StructType.fromAttributes(aggBufferAttributes)
-
/** Allocate enough words to store all registers. */
- override val aggBufferAttributes: Seq[AttributeReference] = Seq.tabulate(numWords) { i =>
- AttributeReference(s"MS[$i]", LongType)()
- }
-
- // Note: although this simply copies aggBufferAttributes, this common code can not be placed
- // in the superclass because that will lead to initialization ordering issues.
--- End diff --
finally get rid of it!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14562: [SPARK-16973][SQL] remove the buffer offsets in I...
Posted by clockfly <gi...@git.apache.org>.
Github user clockfly commented on a diff in the pull request:
https://github.com/apache/spark/pull/14562#discussion_r75039535
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/slicedRows.scala ---
@@ -0,0 +1,110 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.expressions
+
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.util.{ArrayData, MapData}
+import org.apache.spark.sql.types.{DataType, Decimal}
+import org.apache.spark.unsafe.types.{CalendarInterval, UTF8String}
+
+/**
+ * A sliced row provides a view of the underlying row, and forwards all read/write requests to the
+ * underlying row, after shifting the index by offset.
+ *
+ * Note that, the underlying row field in this class is mutable, so that we can reuse the same
+ * instance of this class while processing many rows.
+ */
+trait BaseSlicedInternalRow extends InternalRow {
+ protected def baseRow: InternalRow
+ protected def offset: Int
+
+ protected def toBaseIndex(index: Int): Int = {
--- End diff --
Rename to `baseIndexOf`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14562: [SPARK-16973][SQL] remove the buffer offsets in Imperati...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14562
**[Test build #63908 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/63908/consoleFull)** for PR 14562 at commit [`5c8f324`](https://github.com/apache/spark/commit/5c8f324a9fdc58ef2bb145c842bdff74febd99bb).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `abstract class Collect extends ImperativeAggregateImpl `
* `sealed abstract class AggregateFunction extends Expression `
* `case class ImperativeAggregate(`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14562: [SPARK-16973][SQL] remove the buffer offsets in Imperati...
Posted by clockfly <gi...@git.apache.org>.
Github user clockfly commented on the issue:
https://github.com/apache/spark/pull/14562
Maybe there is another alternative, for example, we can define an InternalRowReader, which wraps the offset.
Sub-class of ImperativeAggregate need to use the InternalRowReader to read the fields from InternalRow.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14562: [SPARK-16973][SQL] remove the buffer offsets in I...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/14562#discussion_r74055718
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/interfaces.scala ---
@@ -219,111 +219,118 @@ sealed abstract class AggregateFunction extends Expression with ImplicitCastInpu
}
/**
- * API for aggregation functions that are expressed in terms of imperative initialize(), update(),
- * and merge() functions which operate on Row-based aggregation buffers.
- *
- * Within these functions, code should access fields of the mutable aggregation buffer by adding the
- * bufferSchema-relative field number to `mutableAggBufferOffset` then using this new field number
- * to access the buffer Row. This is necessary because this aggregation function's buffer is
- * embedded inside of a larger shared aggregation buffer when an aggregation operator evaluates
- * multiple aggregate functions at the same time.
- *
- * We need to perform similar field number arithmetic when merging multiple intermediate
- * aggregate buffers together in `merge()` (in this case, use `inputAggBufferOffset` when accessing
- * the input buffer).
- *
- * Correct ImperativeAggregate evaluation depends on the correctness of `mutableAggBufferOffset` and
- * `inputAggBufferOffset`, but not on the correctness of the attribute ids in `aggBufferAttributes`
- * and `inputAggBufferAttributes`.
+ * API for aggregation functions that are expressed in terms of imperative doInitialize(),
+ * doUpdate(), doMerge() and doComplete() functions which operate on Row-based aggregation buffers.
*/
abstract class ImperativeAggregate extends AggregateFunction with CodegenFallback {
/**
- * The offset of this function's first buffer value in the underlying shared mutable aggregation
- * buffer.
+ * The aggregation operator keeps a large shared mutable buffer row for all aggregate functions,
+ * each aggregate function should only access a slice of this shared buffer.
+ */
+ private var mutableBufferRow: SlicedMutableRow = _
+
+ /**
+ * During partial aggregation, the input buffer row to be merged is shared among all aggregate
+ * functions, each aggregate function should only access a slice of this input buffer.
+ */
+ private var inputBufferRow: SlicedInternalRow = _
+
+ /**
+ * Set the offset of this function's start buffer value in the underlying shared mutable
+ * aggregation buffer.
*
* For example, we have two aggregate functions `avg(x)` and `avg(y)`, which share the same
- * aggregation buffer. In this shared buffer, the position of the first buffer value of `avg(x)`
- * will be 0 and the position of the first buffer value of `avg(y)` will be 2:
+ * aggregation buffer. In this shared buffer, the position of the start buffer value of `avg(x)`
+ * will be 0 and the position of the start buffer value of `avg(y)` will be 2:
* {{{
- * avg(x) mutableAggBufferOffset = 0
+ * avg(x) mutable buffer offset is 0
* |
* v
* +--------+--------+--------+--------+
* | sum1 | count1 | sum2 | count2 |
* +--------+--------+--------+--------+
* ^
* |
- * avg(y) mutableAggBufferOffset = 2
+ * avg(y) mutable buffer offset is 2
* }}}
*/
- protected val mutableAggBufferOffset: Int
-
- /**
- * Returns a copy of this ImperativeAggregate with an updated mutableAggBufferOffset.
- * This new copy's attributes may have different ids than the original.
- */
- def withNewMutableAggBufferOffset(newMutableAggBufferOffset: Int): ImperativeAggregate
+ final def setMutableBufferOffset(offset: Int): Unit = {
+ assert(mutableBufferRow == null)
+ mutableBufferRow = SlicedMutableRow(offset, aggBufferAttributes.length)
+ }
/**
- * The offset of this function's start buffer value in the underlying shared input aggregation
+ * Set the offset of this function's start buffer value in the underlying shared input aggregation
* buffer. An input aggregation buffer is used when we merge two aggregation buffers together in
- * the `update()` function and is immutable (we merge an input aggregation buffer and a mutable
+ * the `merge()` function and is immutable (we merge an input aggregation buffer and a mutable
* aggregation buffer and then store the new buffer values to the mutable aggregation buffer).
*
* An input aggregation buffer may contain extra fields, such as grouping keys, at its start, so
- * mutableAggBufferOffset and inputAggBufferOffset are often different.
+ * mutable buffer offset and input buffer offset are often different.
*
* For example, say we have a grouping expression, `key`, and two aggregate functions,
- * `avg(x)` and `avg(y)`. In the shared input aggregation buffer, the position of the first
- * buffer value of `avg(x)` will be 1 and the position of the first buffer value of `avg(y)`
+ * `avg(x)` and `avg(y)`. In the shared input aggregation buffer, the position of the start
+ * buffer value of `avg(x)` will be 1 and the position of the start buffer value of `avg(y)`
* will be 3 (position 0 is used for the value of `key`):
* {{{
- * avg(x) inputAggBufferOffset = 1
+ * avg(x) input buffer offset is 1
* |
* v
* +--------+--------+--------+--------+--------+
* | key | sum1 | count1 | sum2 | count2 |
* +--------+--------+--------+--------+--------+
* ^
* |
- * avg(y) inputAggBufferOffset = 3
+ * avg(y) input buffer offset is 3
* }}}
*/
- protected val inputAggBufferOffset: Int
+ final def setInputBufferOffset(offset: Int): Unit = {
+ assert(inputBufferRow == null)
--- End diff --
This is based on my understanding. I'll figure it out tomorrow.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14562: [SPARK-16973][SQL] remove the buffer offsets in Imperati...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/14562
@hvanhovell I'm not sure about the performance, will benchmark it later, hopefully they can be inlined by JVM successfully.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14562: [SPARK-16973][SQL] remove the buffer offsets in I...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/14562#discussion_r74187449
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/slicedRows.scala ---
@@ -0,0 +1,108 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.expressions
+
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.util.{ArrayData, MapData}
+import org.apache.spark.sql.types.{DataType, Decimal}
+import org.apache.spark.unsafe.types.{CalendarInterval, UTF8String}
+
+/**
+ * A sliced row provides a view of the underlying row, and forwards all read/write requests to the
+ * underlying row, after shifting the index by offset.
+ *
+ * Note that, the underlying row field in this class is mutable, so that we can reuse the same
+ * instance of this class while processing many rows.
+ */
+trait BaseSlicedInternalRow extends InternalRow {
+ def baseRow: InternalRow
+ def offset: Int
+
+ protected def toBaseIndex(index: Int): Int = {
+ assert(index >= 0, "index (" + index + ") should >= 0")
+ assert(index < numFields, "index (" + index + ") should < " + numFields)
+ index + offset
+ }
+
+ override def isNullAt(i: Int): Boolean = baseRow.isNullAt(toBaseIndex(i))
+ override def getBoolean(i: Int): Boolean = baseRow.getBoolean(toBaseIndex(i))
+ override def getByte(i: Int): Byte = baseRow.getByte(toBaseIndex(i))
+ override def getShort(i: Int): Short = baseRow.getShort(toBaseIndex(i))
+ override def getInt(i: Int): Int = baseRow.getInt(toBaseIndex(i))
+ override def getLong(i: Int): Long = baseRow.getLong(toBaseIndex(i))
+ override def getFloat(i: Int): Float = baseRow.getFloat(toBaseIndex(i))
+ override def getDouble(i: Int): Double = baseRow.getDouble(toBaseIndex((i)))
+ override def getDecimal(i: Int, precision: Int, scale: Int): Decimal =
+ baseRow.getDecimal(toBaseIndex(i), precision, scale)
+ override def getUTF8String(i: Int): UTF8String = baseRow.getUTF8String(toBaseIndex(i))
+ override def getBinary(i: Int): Array[Byte] = baseRow.getBinary(toBaseIndex(i))
+ override def getInterval(i: Int): CalendarInterval = baseRow.getInterval(toBaseIndex(i))
+ override def getStruct(i: Int, numFields: Int): InternalRow =
+ baseRow.getStruct(toBaseIndex(i), numFields)
+ override def getArray(i: Int): ArrayData = baseRow.getArray(toBaseIndex(i))
+ override def getMap(i: Int): MapData = baseRow.getMap(toBaseIndex(i))
+ override def get(i: Int, dataType: DataType): AnyRef = baseRow.get(toBaseIndex(i), dataType)
+
+ override def anyNull: Boolean = {
+ val len = offset + numFields
+ var i = offset
+ while (i < len) {
+ if (isNullAt(i)) { return true }
+ i += 1
+ }
+ false
+ }
+
+ override def copy(): InternalRow = {
+ throw new UnsupportedOperationException("Cannot copy a SlicedMutableRow")
+ }
+}
+
+case class SlicedInternalRow(offset: Int, numFields: Int) extends BaseSlicedInternalRow {
--- End diff --
It generates a lot of crap in bytecode, so would be good to not generate them unless they are useful.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14562: [SPARK-16973][SQL] remove the buffer offsets in I...
Posted by clockfly <gi...@git.apache.org>.
Github user clockfly commented on a diff in the pull request:
https://github.com/apache/spark/pull/14562#discussion_r75038877
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/interfaces.scala ---
@@ -219,111 +219,125 @@ sealed abstract class AggregateFunction extends Expression with ImplicitCastInpu
}
/**
- * API for aggregation functions that are expressed in terms of imperative initialize(), update(),
- * and merge() functions which operate on Row-based aggregation buffers.
- *
- * Within these functions, code should access fields of the mutable aggregation buffer by adding the
- * bufferSchema-relative field number to `mutableAggBufferOffset` then using this new field number
- * to access the buffer Row. This is necessary because this aggregation function's buffer is
- * embedded inside of a larger shared aggregation buffer when an aggregation operator evaluates
- * multiple aggregate functions at the same time.
- *
- * We need to perform similar field number arithmetic when merging multiple intermediate
- * aggregate buffers together in `merge()` (in this case, use `inputAggBufferOffset` when accessing
- * the input buffer).
- *
- * Correct ImperativeAggregate evaluation depends on the correctness of `mutableAggBufferOffset` and
- * `inputAggBufferOffset`, but not on the correctness of the attribute ids in `aggBufferAttributes`
- * and `inputAggBufferAttributes`.
+ * API for aggregation functions that are expressed in terms of imperative doInitialize(),
+ * doUpdate(), doMerge() and doComplete() functions which operate on Row-based aggregation buffers.
*/
abstract class ImperativeAggregate extends AggregateFunction with CodegenFallback {
+ // Although `mutableBufferRow` and `inputBufferRow` are 2 mutable fields in `ImperativeAggregate`,
+ // they can only be set once, thus make `ImperativeAggregate` kind of immutable and stateless.
+
+ /**
+ * The aggregation operator keeps a large shared mutable buffer row for all aggregate functions,
+ * each aggregate function should only access a slice of this shared buffer.
+ */
+ private var mutableBufferRow: SlicedMutableRow = _
+
+ /**
+ * During partial aggregation, the input buffer row to be merged is shared among all aggregate
+ * functions, each aggregate function should only access a slice of this input buffer.
+ */
+ private var inputBufferRow: SlicedInternalRow = _
+
/**
- * The offset of this function's first buffer value in the underlying shared mutable aggregation
- * buffer.
+ * Set the offset of this function's start buffer value in the underlying shared mutable
+ * aggregation buffer.
*
* For example, we have two aggregate functions `avg(x)` and `avg(y)`, which share the same
- * aggregation buffer. In this shared buffer, the position of the first buffer value of `avg(x)`
- * will be 0 and the position of the first buffer value of `avg(y)` will be 2:
+ * aggregation buffer. In this shared buffer, the position of the start buffer value of `avg(x)`
+ * will be 0 and the position of the start buffer value of `avg(y)` will be 2:
* {{{
- * avg(x) mutableAggBufferOffset = 0
+ * avg(x) mutable buffer offset is 0
* |
* v
* +--------+--------+--------+--------+
* | sum1 | count1 | sum2 | count2 |
* +--------+--------+--------+--------+
* ^
* |
- * avg(y) mutableAggBufferOffset = 2
+ * avg(y) mutable buffer offset is 2
* }}}
*/
- protected val mutableAggBufferOffset: Int
-
- /**
- * Returns a copy of this ImperativeAggregate with an updated mutableAggBufferOffset.
- * This new copy's attributes may have different ids than the original.
- */
- def withNewMutableAggBufferOffset(newMutableAggBufferOffset: Int): ImperativeAggregate
+ final def setMutableBufferOffset(offset: Int): Unit = {
+ assert(mutableBufferRow == null)
+ mutableBufferRow = new SlicedMutableRow(offset, aggBufferAttributes.length)
+ }
/**
- * The offset of this function's start buffer value in the underlying shared input aggregation
+ * Set the offset of this function's start buffer value in the underlying shared input aggregation
* buffer. An input aggregation buffer is used when we merge two aggregation buffers together in
- * the `update()` function and is immutable (we merge an input aggregation buffer and a mutable
+ * the `merge()` function and is immutable (we merge an input aggregation buffer and a mutable
* aggregation buffer and then store the new buffer values to the mutable aggregation buffer).
*
* An input aggregation buffer may contain extra fields, such as grouping keys, at its start, so
- * mutableAggBufferOffset and inputAggBufferOffset are often different.
+ * mutable buffer offset and input buffer offset are often different.
*
* For example, say we have a grouping expression, `key`, and two aggregate functions,
- * `avg(x)` and `avg(y)`. In the shared input aggregation buffer, the position of the first
- * buffer value of `avg(x)` will be 1 and the position of the first buffer value of `avg(y)`
+ * `avg(x)` and `avg(y)`. In the shared input aggregation buffer, the position of the start
+ * buffer value of `avg(x)` will be 1 and the position of the start buffer value of `avg(y)`
* will be 3 (position 0 is used for the value of `key`):
* {{{
- * avg(x) inputAggBufferOffset = 1
+ * avg(x) input buffer offset is 1
* |
* v
* +--------+--------+--------+--------+--------+
* | key | sum1 | count1 | sum2 | count2 |
* +--------+--------+--------+--------+--------+
* ^
* |
- * avg(y) inputAggBufferOffset = 3
+ * avg(y) input buffer offset is 3
* }}}
*/
- protected val inputAggBufferOffset: Int
+ final def setInputBufferOffset(offset: Int): Unit = {
+ assert(inputBufferRow == null)
+ inputBufferRow = new SlicedInternalRow(offset, aggBufferAttributes.length)
+ }
- /**
- * Returns a copy of this ImperativeAggregate with an updated mutableAggBufferOffset.
- * This new copy's attributes may have different ids than the original.
- */
- def withNewInputAggBufferOffset(newInputAggBufferOffset: Int): ImperativeAggregate
+ final def initialize(mutableAggBuffer: MutableRow): Unit = {
+ doInitialize(mutableBufferRow.target(mutableAggBuffer))
--- End diff --
Should we avoid define new interface like
```
doInitialize
doUpdate
...
```
1. First, from user's point's view, the 'do' prefix doesn't make much sense.
2. Can we define a wrapper class for `ImperativeAggregate` so that we don't need to change the interface of `ImperativeAggregate`, For example:
```
class ImperativeAggregateWrapper {
final def setInputBufferOffset(offset: Int): Unit
val agg: ImperativeAggregate
def initialize(...): Unit = agg.initialize(...)
}
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14562: [SPARK-16973][SQL] remove the buffer offsets in Imperati...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14562
**[Test build #63906 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/63906/consoleFull)** for PR 14562 at commit [`e50aece`](https://github.com/apache/spark/commit/e50aece3eddff91aacc0d3b97646df1fb453cec4).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14562: [SPARK-16973][SQL] remove the buffer offsets in Imperati...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14562
**[Test build #63552 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/63552/consoleFull)** for PR 14562 at commit [`8e0531b`](https://github.com/apache/spark/commit/8e0531b6385a4480186a904b721b6e284fec3f43).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14562: [SPARK-16973][SQL] remove the buffer offsets in Imperati...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14562
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14562: [SPARK-16973][SQL] remove the buffer offsets in Imperati...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14562
**[Test build #63524 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/63524/consoleFull)** for PR 14562 at commit [`1444176`](https://github.com/apache/spark/commit/1444176451cce717eb5ceec37fbe1f4f7202c010).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14562: [SPARK-16973][SQL] remove the buffer offsets in I...
Posted by hvanhovell <gi...@git.apache.org>.
Github user hvanhovell commented on a diff in the pull request:
https://github.com/apache/spark/pull/14562#discussion_r75040157
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/interfaces.scala ---
@@ -219,111 +219,125 @@ sealed abstract class AggregateFunction extends Expression with ImplicitCastInpu
}
/**
- * API for aggregation functions that are expressed in terms of imperative initialize(), update(),
- * and merge() functions which operate on Row-based aggregation buffers.
- *
- * Within these functions, code should access fields of the mutable aggregation buffer by adding the
- * bufferSchema-relative field number to `mutableAggBufferOffset` then using this new field number
- * to access the buffer Row. This is necessary because this aggregation function's buffer is
- * embedded inside of a larger shared aggregation buffer when an aggregation operator evaluates
- * multiple aggregate functions at the same time.
- *
- * We need to perform similar field number arithmetic when merging multiple intermediate
- * aggregate buffers together in `merge()` (in this case, use `inputAggBufferOffset` when accessing
- * the input buffer).
- *
- * Correct ImperativeAggregate evaluation depends on the correctness of `mutableAggBufferOffset` and
- * `inputAggBufferOffset`, but not on the correctness of the attribute ids in `aggBufferAttributes`
- * and `inputAggBufferAttributes`.
+ * API for aggregation functions that are expressed in terms of imperative doInitialize(),
+ * doUpdate(), doMerge() and doComplete() functions which operate on Row-based aggregation buffers.
*/
abstract class ImperativeAggregate extends AggregateFunction with CodegenFallback {
+ // Although `mutableBufferRow` and `inputBufferRow` are 2 mutable fields in `ImperativeAggregate`,
+ // they can only be set once, thus make `ImperativeAggregate` kind of immutable and stateless.
+
+ /**
+ * The aggregation operator keeps a large shared mutable buffer row for all aggregate functions,
+ * each aggregate function should only access a slice of this shared buffer.
+ */
+ private var mutableBufferRow: SlicedMutableRow = _
+
+ /**
+ * During partial aggregation, the input buffer row to be merged is shared among all aggregate
+ * functions, each aggregate function should only access a slice of this input buffer.
+ */
+ private var inputBufferRow: SlicedInternalRow = _
+
/**
- * The offset of this function's first buffer value in the underlying shared mutable aggregation
- * buffer.
+ * Set the offset of this function's start buffer value in the underlying shared mutable
+ * aggregation buffer.
*
* For example, we have two aggregate functions `avg(x)` and `avg(y)`, which share the same
- * aggregation buffer. In this shared buffer, the position of the first buffer value of `avg(x)`
- * will be 0 and the position of the first buffer value of `avg(y)` will be 2:
+ * aggregation buffer. In this shared buffer, the position of the start buffer value of `avg(x)`
+ * will be 0 and the position of the start buffer value of `avg(y)` will be 2:
* {{{
- * avg(x) mutableAggBufferOffset = 0
+ * avg(x) mutable buffer offset is 0
* |
* v
* +--------+--------+--------+--------+
* | sum1 | count1 | sum2 | count2 |
* +--------+--------+--------+--------+
* ^
* |
- * avg(y) mutableAggBufferOffset = 2
+ * avg(y) mutable buffer offset is 2
* }}}
*/
- protected val mutableAggBufferOffset: Int
-
- /**
- * Returns a copy of this ImperativeAggregate with an updated mutableAggBufferOffset.
- * This new copy's attributes may have different ids than the original.
- */
- def withNewMutableAggBufferOffset(newMutableAggBufferOffset: Int): ImperativeAggregate
+ final def setMutableBufferOffset(offset: Int): Unit = {
+ assert(mutableBufferRow == null)
--- End diff --
Require implies that the caller has passed a bad argument. Assert checks if invariants hold (the class is not in an unexpected state). I think this should be an assert.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14562: [SPARK-16973][SQL] remove the buffer offsets in I...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/14562#discussion_r74182501
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/slicedRows.scala ---
@@ -0,0 +1,108 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.expressions
+
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.util.{ArrayData, MapData}
+import org.apache.spark.sql.types.{DataType, Decimal}
+import org.apache.spark.unsafe.types.{CalendarInterval, UTF8String}
+
+/**
+ * A sliced row provides a view of the underlying row, and forwards all read/write requests to the
+ * underlying row, after shifting the index by offset.
+ *
+ * Note that, the underlying row field in this class is mutable, so that we can reuse the same
+ * instance of this class while processing many rows.
+ */
+trait BaseSlicedInternalRow extends InternalRow {
+ def baseRow: InternalRow
+ def offset: Int
+
+ protected def toBaseIndex(index: Int): Int = {
+ assert(index >= 0, "index (" + index + ") should >= 0")
+ assert(index < numFields, "index (" + index + ") should < " + numFields)
+ index + offset
+ }
+
+ override def isNullAt(i: Int): Boolean = baseRow.isNullAt(toBaseIndex(i))
+ override def getBoolean(i: Int): Boolean = baseRow.getBoolean(toBaseIndex(i))
+ override def getByte(i: Int): Byte = baseRow.getByte(toBaseIndex(i))
+ override def getShort(i: Int): Short = baseRow.getShort(toBaseIndex(i))
+ override def getInt(i: Int): Int = baseRow.getInt(toBaseIndex(i))
+ override def getLong(i: Int): Long = baseRow.getLong(toBaseIndex(i))
+ override def getFloat(i: Int): Float = baseRow.getFloat(toBaseIndex(i))
+ override def getDouble(i: Int): Double = baseRow.getDouble(toBaseIndex((i)))
+ override def getDecimal(i: Int, precision: Int, scale: Int): Decimal =
+ baseRow.getDecimal(toBaseIndex(i), precision, scale)
+ override def getUTF8String(i: Int): UTF8String = baseRow.getUTF8String(toBaseIndex(i))
+ override def getBinary(i: Int): Array[Byte] = baseRow.getBinary(toBaseIndex(i))
+ override def getInterval(i: Int): CalendarInterval = baseRow.getInterval(toBaseIndex(i))
+ override def getStruct(i: Int, numFields: Int): InternalRow =
+ baseRow.getStruct(toBaseIndex(i), numFields)
+ override def getArray(i: Int): ArrayData = baseRow.getArray(toBaseIndex(i))
+ override def getMap(i: Int): MapData = baseRow.getMap(toBaseIndex(i))
+ override def get(i: Int, dataType: DataType): AnyRef = baseRow.get(toBaseIndex(i), dataType)
+
+ override def anyNull: Boolean = {
+ val len = offset + numFields
+ var i = offset
+ while (i < len) {
+ if (isNullAt(i)) { return true }
+ i += 1
+ }
+ false
+ }
+
+ override def copy(): InternalRow = {
+ throw new UnsupportedOperationException("Cannot copy a SlicedMutableRow")
+ }
+}
+
+case class SlicedInternalRow(offset: Int, numFields: Int) extends BaseSlicedInternalRow {
--- End diff --
does this need to be a case class?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14562: [SPARK-16973][SQL] remove the buffer offsets in Imperati...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14562
**[Test build #63908 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/63908/consoleFull)** for PR 14562 at commit [`5c8f324`](https://github.com/apache/spark/commit/5c8f324a9fdc58ef2bb145c842bdff74febd99bb).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14562: [SPARK-16973][SQL] remove the buffer offsets in I...
Posted by clockfly <gi...@git.apache.org>.
Github user clockfly commented on a diff in the pull request:
https://github.com/apache/spark/pull/14562#discussion_r75039312
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/interfaces.scala ---
@@ -219,111 +219,125 @@ sealed abstract class AggregateFunction extends Expression with ImplicitCastInpu
}
/**
- * API for aggregation functions that are expressed in terms of imperative initialize(), update(),
- * and merge() functions which operate on Row-based aggregation buffers.
- *
- * Within these functions, code should access fields of the mutable aggregation buffer by adding the
- * bufferSchema-relative field number to `mutableAggBufferOffset` then using this new field number
- * to access the buffer Row. This is necessary because this aggregation function's buffer is
- * embedded inside of a larger shared aggregation buffer when an aggregation operator evaluates
- * multiple aggregate functions at the same time.
- *
- * We need to perform similar field number arithmetic when merging multiple intermediate
- * aggregate buffers together in `merge()` (in this case, use `inputAggBufferOffset` when accessing
- * the input buffer).
- *
- * Correct ImperativeAggregate evaluation depends on the correctness of `mutableAggBufferOffset` and
- * `inputAggBufferOffset`, but not on the correctness of the attribute ids in `aggBufferAttributes`
- * and `inputAggBufferAttributes`.
+ * API for aggregation functions that are expressed in terms of imperative doInitialize(),
+ * doUpdate(), doMerge() and doComplete() functions which operate on Row-based aggregation buffers.
*/
abstract class ImperativeAggregate extends AggregateFunction with CodegenFallback {
+ // Although `mutableBufferRow` and `inputBufferRow` are 2 mutable fields in `ImperativeAggregate`,
+ // they can only be set once, thus make `ImperativeAggregate` kind of immutable and stateless.
+
+ /**
+ * The aggregation operator keeps a large shared mutable buffer row for all aggregate functions,
+ * each aggregate function should only access a slice of this shared buffer.
+ */
+ private var mutableBufferRow: SlicedMutableRow = _
+
+ /**
+ * During partial aggregation, the input buffer row to be merged is shared among all aggregate
+ * functions, each aggregate function should only access a slice of this input buffer.
+ */
+ private var inputBufferRow: SlicedInternalRow = _
+
/**
- * The offset of this function's first buffer value in the underlying shared mutable aggregation
- * buffer.
+ * Set the offset of this function's start buffer value in the underlying shared mutable
+ * aggregation buffer.
*
* For example, we have two aggregate functions `avg(x)` and `avg(y)`, which share the same
- * aggregation buffer. In this shared buffer, the position of the first buffer value of `avg(x)`
- * will be 0 and the position of the first buffer value of `avg(y)` will be 2:
+ * aggregation buffer. In this shared buffer, the position of the start buffer value of `avg(x)`
+ * will be 0 and the position of the start buffer value of `avg(y)` will be 2:
* {{{
- * avg(x) mutableAggBufferOffset = 0
+ * avg(x) mutable buffer offset is 0
* |
* v
* +--------+--------+--------+--------+
* | sum1 | count1 | sum2 | count2 |
* +--------+--------+--------+--------+
* ^
* |
- * avg(y) mutableAggBufferOffset = 2
+ * avg(y) mutable buffer offset is 2
* }}}
*/
- protected val mutableAggBufferOffset: Int
-
- /**
- * Returns a copy of this ImperativeAggregate with an updated mutableAggBufferOffset.
- * This new copy's attributes may have different ids than the original.
- */
- def withNewMutableAggBufferOffset(newMutableAggBufferOffset: Int): ImperativeAggregate
+ final def setMutableBufferOffset(offset: Int): Unit = {
+ assert(mutableBufferRow == null)
+ mutableBufferRow = new SlicedMutableRow(offset, aggBufferAttributes.length)
+ }
/**
- * The offset of this function's start buffer value in the underlying shared input aggregation
+ * Set the offset of this function's start buffer value in the underlying shared input aggregation
* buffer. An input aggregation buffer is used when we merge two aggregation buffers together in
- * the `update()` function and is immutable (we merge an input aggregation buffer and a mutable
+ * the `merge()` function and is immutable (we merge an input aggregation buffer and a mutable
* aggregation buffer and then store the new buffer values to the mutable aggregation buffer).
*
* An input aggregation buffer may contain extra fields, such as grouping keys, at its start, so
- * mutableAggBufferOffset and inputAggBufferOffset are often different.
+ * mutable buffer offset and input buffer offset are often different.
*
* For example, say we have a grouping expression, `key`, and two aggregate functions,
- * `avg(x)` and `avg(y)`. In the shared input aggregation buffer, the position of the first
- * buffer value of `avg(x)` will be 1 and the position of the first buffer value of `avg(y)`
+ * `avg(x)` and `avg(y)`. In the shared input aggregation buffer, the position of the start
+ * buffer value of `avg(x)` will be 1 and the position of the start buffer value of `avg(y)`
* will be 3 (position 0 is used for the value of `key`):
* {{{
- * avg(x) inputAggBufferOffset = 1
+ * avg(x) input buffer offset is 1
* |
* v
* +--------+--------+--------+--------+--------+
* | key | sum1 | count1 | sum2 | count2 |
* +--------+--------+--------+--------+--------+
* ^
* |
- * avg(y) inputAggBufferOffset = 3
+ * avg(y) input buffer offset is 3
* }}}
*/
- protected val inputAggBufferOffset: Int
+ final def setInputBufferOffset(offset: Int): Unit = {
+ assert(inputBufferRow == null)
+ inputBufferRow = new SlicedInternalRow(offset, aggBufferAttributes.length)
+ }
- /**
- * Returns a copy of this ImperativeAggregate with an updated mutableAggBufferOffset.
- * This new copy's attributes may have different ids than the original.
- */
- def withNewInputAggBufferOffset(newInputAggBufferOffset: Int): ImperativeAggregate
+ final def initialize(mutableAggBuffer: MutableRow): Unit = {
+ doInitialize(mutableBufferRow.target(mutableAggBuffer))
+ }
+
+ final def update(mutableAggBuffer: MutableRow, inputRow: InternalRow): Unit = {
+ doUpdate(mutableBufferRow.target(mutableAggBuffer), inputRow)
+ }
+
+ final def merge(mutableAggBuffer: MutableRow, inputAggBuffer: InternalRow): Unit = {
+ doMerge(mutableBufferRow.target(mutableAggBuffer), inputBufferRow.target(inputAggBuffer))
+ }
+
+ final override def eval(aggBuffer: InternalRow): Any = {
+ assert(aggBuffer.isInstanceOf[MutableRow])
+ doEval(mutableBufferRow.target(aggBuffer.asInstanceOf[MutableRow]))
+ }
+
+ final def newInstance(): ImperativeAggregate = {
+ makeCopy(mapProductIterator(_.asInstanceOf[AnyRef])).asInstanceOf[ImperativeAggregate]
--- End diff --
Where is `mapProductIterator` defined?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14562: [SPARK-16973][SQL] remove the buffer offsets in Imperati...
Posted by clockfly <gi...@git.apache.org>.
Github user clockfly commented on the issue:
https://github.com/apache/spark/pull/14562
I think my biggest concern is about the performance and abstraction.
1. Seems there are too many function call to update each row. Is this cost too high?
2. The new methods added in ImperativeAggregate may create confusion, like
There are both
```
def update
def doUpdate
def merge
def doMerge
def initialize
def doInitialize
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14562: [SPARK-16973][SQL] remove the buffer offsets in I...
Posted by clockfly <gi...@git.apache.org>.
Github user clockfly commented on a diff in the pull request:
https://github.com/apache/spark/pull/14562#discussion_r75042423
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/interfaces.scala ---
@@ -219,111 +219,125 @@ sealed abstract class AggregateFunction extends Expression with ImplicitCastInpu
}
/**
- * API for aggregation functions that are expressed in terms of imperative initialize(), update(),
- * and merge() functions which operate on Row-based aggregation buffers.
- *
- * Within these functions, code should access fields of the mutable aggregation buffer by adding the
- * bufferSchema-relative field number to `mutableAggBufferOffset` then using this new field number
- * to access the buffer Row. This is necessary because this aggregation function's buffer is
- * embedded inside of a larger shared aggregation buffer when an aggregation operator evaluates
- * multiple aggregate functions at the same time.
- *
- * We need to perform similar field number arithmetic when merging multiple intermediate
- * aggregate buffers together in `merge()` (in this case, use `inputAggBufferOffset` when accessing
- * the input buffer).
- *
- * Correct ImperativeAggregate evaluation depends on the correctness of `mutableAggBufferOffset` and
- * `inputAggBufferOffset`, but not on the correctness of the attribute ids in `aggBufferAttributes`
- * and `inputAggBufferAttributes`.
+ * API for aggregation functions that are expressed in terms of imperative doInitialize(),
+ * doUpdate(), doMerge() and doComplete() functions which operate on Row-based aggregation buffers.
*/
abstract class ImperativeAggregate extends AggregateFunction with CodegenFallback {
+ // Although `mutableBufferRow` and `inputBufferRow` are 2 mutable fields in `ImperativeAggregate`,
+ // they can only be set once, thus make `ImperativeAggregate` kind of immutable and stateless.
+
+ /**
+ * The aggregation operator keeps a large shared mutable buffer row for all aggregate functions,
+ * each aggregate function should only access a slice of this shared buffer.
+ */
+ private var mutableBufferRow: SlicedMutableRow = _
+
+ /**
+ * During partial aggregation, the input buffer row to be merged is shared among all aggregate
+ * functions, each aggregate function should only access a slice of this input buffer.
+ */
+ private var inputBufferRow: SlicedInternalRow = _
+
/**
- * The offset of this function's first buffer value in the underlying shared mutable aggregation
- * buffer.
+ * Set the offset of this function's start buffer value in the underlying shared mutable
+ * aggregation buffer.
*
* For example, we have two aggregate functions `avg(x)` and `avg(y)`, which share the same
- * aggregation buffer. In this shared buffer, the position of the first buffer value of `avg(x)`
- * will be 0 and the position of the first buffer value of `avg(y)` will be 2:
+ * aggregation buffer. In this shared buffer, the position of the start buffer value of `avg(x)`
+ * will be 0 and the position of the start buffer value of `avg(y)` will be 2:
* {{{
- * avg(x) mutableAggBufferOffset = 0
+ * avg(x) mutable buffer offset is 0
* |
* v
* +--------+--------+--------+--------+
* | sum1 | count1 | sum2 | count2 |
* +--------+--------+--------+--------+
* ^
* |
- * avg(y) mutableAggBufferOffset = 2
+ * avg(y) mutable buffer offset is 2
* }}}
*/
- protected val mutableAggBufferOffset: Int
-
- /**
- * Returns a copy of this ImperativeAggregate with an updated mutableAggBufferOffset.
- * This new copy's attributes may have different ids than the original.
- */
- def withNewMutableAggBufferOffset(newMutableAggBufferOffset: Int): ImperativeAggregate
+ final def setMutableBufferOffset(offset: Int): Unit = {
+ assert(mutableBufferRow == null)
--- End diff --
Instead of setting an attribute that is supposed to be immutable, can we use `copy` to copy the whole class?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14562: [SPARK-16973][SQL] remove the buffer offsets in Imperati...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14562
**[Test build #63438 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/63438/consoleFull)** for PR 14562 at commit [`9658056`](https://github.com/apache/spark/commit/9658056f420fc4280cc5ac656e7d45fc28a262de).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `case class CollectList(child: Expression) extends Collect `
* `case class CollectSet(child: Expression) extends Collect `
* `trait BaseSlicedInternalRow extends InternalRow `
* `case class SlicedInternalRow(offset: Int, numFields: Int) extends BaseSlicedInternalRow `
* `case class SlicedMutableRow(offset: Int, numFields: Int)`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14562: [SPARK-16973][SQL] remove the buffer offsets in Imperati...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14562
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14562: [SPARK-16973][SQL] remove the buffer offsets in Imperati...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14562
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/63438/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14562: [SPARK-16973][SQL] remove the buffer offsets in Imperati...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14562
**[Test build #63524 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/63524/consoleFull)** for PR 14562 at commit [`1444176`](https://github.com/apache/spark/commit/1444176451cce717eb5ceec37fbe1f4f7202c010).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `class SlicedInternalRow(protected val offset: Int, val numFields: Int)`
* `class SlicedMutableRow(protected val offset: Int, val numFields: Int)`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14562: [SPARK-16973][SQL] remove the buffer offsets in Imperati...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14562
**[Test build #63552 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/63552/consoleFull)** for PR 14562 at commit [`8e0531b`](https://github.com/apache/spark/commit/8e0531b6385a4480186a904b721b6e284fec3f43).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14562: [SPARK-16973][SQL] remove the buffer offsets in Imperati...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14562
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/63906/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14562: [SPARK-16973][SQL] remove the buffer offsets in Imperati...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14562
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14562: [SPARK-16973][SQL] remove the buffer offsets in I...
Posted by clockfly <gi...@git.apache.org>.
Github user clockfly commented on a diff in the pull request:
https://github.com/apache/spark/pull/14562#discussion_r75039705
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/slicedRows.scala ---
@@ -0,0 +1,110 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.expressions
+
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.util.{ArrayData, MapData}
+import org.apache.spark.sql.types.{DataType, Decimal}
+import org.apache.spark.unsafe.types.{CalendarInterval, UTF8String}
+
+/**
+ * A sliced row provides a view of the underlying row, and forwards all read/write requests to the
+ * underlying row, after shifting the index by offset.
+ *
+ * Note that, the underlying row field in this class is mutable, so that we can reuse the same
+ * instance of this class while processing many rows.
+ */
+trait BaseSlicedInternalRow extends InternalRow {
+ protected def baseRow: InternalRow
--- End diff --
Maybe rawRow better?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14562: [SPARK-16973][SQL] remove the buffer offsets in Imperati...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14562
**[Test build #63906 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/63906/consoleFull)** for PR 14562 at commit [`e50aece`](https://github.com/apache/spark/commit/e50aece3eddff91aacc0d3b97646df1fb453cec4).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `abstract class Collect extends ImperativeAggregateImpl `
* `sealed abstract class AggregateFunction extends Expression `
* `case class ImperativeAggregate(`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14562: [SPARK-16973][SQL] remove the buffer offsets in I...
Posted by clockfly <gi...@git.apache.org>.
Github user clockfly commented on a diff in the pull request:
https://github.com/apache/spark/pull/14562#discussion_r75039623
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/slicedRows.scala ---
@@ -0,0 +1,110 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.expressions
+
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.util.{ArrayData, MapData}
+import org.apache.spark.sql.types.{DataType, Decimal}
+import org.apache.spark.unsafe.types.{CalendarInterval, UTF8String}
+
+/**
+ * A sliced row provides a view of the underlying row, and forwards all read/write requests to the
+ * underlying row, after shifting the index by offset.
+ *
+ * Note that, the underlying row field in this class is mutable, so that we can reuse the same
+ * instance of this class while processing many rows.
+ */
+trait BaseSlicedInternalRow extends InternalRow {
+ protected def baseRow: InternalRow
+ protected def offset: Int
+
+ protected def toBaseIndex(index: Int): Int = {
--- End diff --
Or rawIndexOf?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14562: [SPARK-16973][SQL] remove the buffer offsets in Imperati...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14562
**[Test build #63438 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/63438/consoleFull)** for PR 14562 at commit [`9658056`](https://github.com/apache/spark/commit/9658056f420fc4280cc5ac656e7d45fc28a262de).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14562: [SPARK-16973][SQL] remove the buffer offsets in Imperati...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14562
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14562: [SPARK-16973][SQL] remove the buffer offsets in I...
Posted by clockfly <gi...@git.apache.org>.
Github user clockfly commented on a diff in the pull request:
https://github.com/apache/spark/pull/14562#discussion_r75041558
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/interfaces.scala ---
@@ -219,111 +219,125 @@ sealed abstract class AggregateFunction extends Expression with ImplicitCastInpu
}
/**
- * API for aggregation functions that are expressed in terms of imperative initialize(), update(),
- * and merge() functions which operate on Row-based aggregation buffers.
- *
- * Within these functions, code should access fields of the mutable aggregation buffer by adding the
- * bufferSchema-relative field number to `mutableAggBufferOffset` then using this new field number
- * to access the buffer Row. This is necessary because this aggregation function's buffer is
- * embedded inside of a larger shared aggregation buffer when an aggregation operator evaluates
- * multiple aggregate functions at the same time.
- *
- * We need to perform similar field number arithmetic when merging multiple intermediate
- * aggregate buffers together in `merge()` (in this case, use `inputAggBufferOffset` when accessing
- * the input buffer).
- *
- * Correct ImperativeAggregate evaluation depends on the correctness of `mutableAggBufferOffset` and
- * `inputAggBufferOffset`, but not on the correctness of the attribute ids in `aggBufferAttributes`
- * and `inputAggBufferAttributes`.
+ * API for aggregation functions that are expressed in terms of imperative doInitialize(),
+ * doUpdate(), doMerge() and doComplete() functions which operate on Row-based aggregation buffers.
*/
abstract class ImperativeAggregate extends AggregateFunction with CodegenFallback {
+ // Although `mutableBufferRow` and `inputBufferRow` are 2 mutable fields in `ImperativeAggregate`,
+ // they can only be set once, thus make `ImperativeAggregate` kind of immutable and stateless.
+
+ /**
+ * The aggregation operator keeps a large shared mutable buffer row for all aggregate functions,
+ * each aggregate function should only access a slice of this shared buffer.
+ */
+ private var mutableBufferRow: SlicedMutableRow = _
+
+ /**
+ * During partial aggregation, the input buffer row to be merged is shared among all aggregate
+ * functions, each aggregate function should only access a slice of this input buffer.
+ */
+ private var inputBufferRow: SlicedInternalRow = _
+
/**
- * The offset of this function's first buffer value in the underlying shared mutable aggregation
- * buffer.
+ * Set the offset of this function's start buffer value in the underlying shared mutable
+ * aggregation buffer.
*
* For example, we have two aggregate functions `avg(x)` and `avg(y)`, which share the same
- * aggregation buffer. In this shared buffer, the position of the first buffer value of `avg(x)`
- * will be 0 and the position of the first buffer value of `avg(y)` will be 2:
+ * aggregation buffer. In this shared buffer, the position of the start buffer value of `avg(x)`
+ * will be 0 and the position of the start buffer value of `avg(y)` will be 2:
* {{{
- * avg(x) mutableAggBufferOffset = 0
+ * avg(x) mutable buffer offset is 0
* |
* v
* +--------+--------+--------+--------+
* | sum1 | count1 | sum2 | count2 |
* +--------+--------+--------+--------+
* ^
* |
- * avg(y) mutableAggBufferOffset = 2
+ * avg(y) mutable buffer offset is 2
* }}}
*/
- protected val mutableAggBufferOffset: Int
-
- /**
- * Returns a copy of this ImperativeAggregate with an updated mutableAggBufferOffset.
- * This new copy's attributes may have different ids than the original.
- */
- def withNewMutableAggBufferOffset(newMutableAggBufferOffset: Int): ImperativeAggregate
+ final def setMutableBufferOffset(offset: Int): Unit = {
+ assert(mutableBufferRow == null)
+ mutableBufferRow = new SlicedMutableRow(offset, aggBufferAttributes.length)
+ }
/**
- * The offset of this function's start buffer value in the underlying shared input aggregation
+ * Set the offset of this function's start buffer value in the underlying shared input aggregation
* buffer. An input aggregation buffer is used when we merge two aggregation buffers together in
- * the `update()` function and is immutable (we merge an input aggregation buffer and a mutable
+ * the `merge()` function and is immutable (we merge an input aggregation buffer and a mutable
* aggregation buffer and then store the new buffer values to the mutable aggregation buffer).
*
* An input aggregation buffer may contain extra fields, such as grouping keys, at its start, so
- * mutableAggBufferOffset and inputAggBufferOffset are often different.
+ * mutable buffer offset and input buffer offset are often different.
*
* For example, say we have a grouping expression, `key`, and two aggregate functions,
- * `avg(x)` and `avg(y)`. In the shared input aggregation buffer, the position of the first
- * buffer value of `avg(x)` will be 1 and the position of the first buffer value of `avg(y)`
+ * `avg(x)` and `avg(y)`. In the shared input aggregation buffer, the position of the start
+ * buffer value of `avg(x)` will be 1 and the position of the start buffer value of `avg(y)`
* will be 3 (position 0 is used for the value of `key`):
* {{{
- * avg(x) inputAggBufferOffset = 1
+ * avg(x) input buffer offset is 1
* |
* v
* +--------+--------+--------+--------+--------+
* | key | sum1 | count1 | sum2 | count2 |
* +--------+--------+--------+--------+--------+
* ^
* |
- * avg(y) inputAggBufferOffset = 3
+ * avg(y) input buffer offset is 3
* }}}
*/
- protected val inputAggBufferOffset: Int
+ final def setInputBufferOffset(offset: Int): Unit = {
+ assert(inputBufferRow == null)
+ inputBufferRow = new SlicedInternalRow(offset, aggBufferAttributes.length)
+ }
- /**
- * Returns a copy of this ImperativeAggregate with an updated mutableAggBufferOffset.
- * This new copy's attributes may have different ids than the original.
- */
- def withNewInputAggBufferOffset(newInputAggBufferOffset: Int): ImperativeAggregate
+ final def initialize(mutableAggBuffer: MutableRow): Unit = {
+ doInitialize(mutableBufferRow.target(mutableAggBuffer))
+ }
+
+ final def update(mutableAggBuffer: MutableRow, inputRow: InternalRow): Unit = {
+ doUpdate(mutableBufferRow.target(mutableAggBuffer), inputRow)
--- End diff --
I am worried it is a bit costly to call `target` for every row.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14562: [SPARK-16973][SQL] remove the buffer offsets in I...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan closed the pull request at:
https://github.com/apache/spark/pull/14562
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14562: [SPARK-16973][SQL] remove the buffer offsets in I...
Posted by clockfly <gi...@git.apache.org>.
Github user clockfly commented on a diff in the pull request:
https://github.com/apache/spark/pull/14562#discussion_r75038089
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/interfaces.scala ---
@@ -219,111 +219,125 @@ sealed abstract class AggregateFunction extends Expression with ImplicitCastInpu
}
/**
- * API for aggregation functions that are expressed in terms of imperative initialize(), update(),
- * and merge() functions which operate on Row-based aggregation buffers.
- *
- * Within these functions, code should access fields of the mutable aggregation buffer by adding the
- * bufferSchema-relative field number to `mutableAggBufferOffset` then using this new field number
- * to access the buffer Row. This is necessary because this aggregation function's buffer is
- * embedded inside of a larger shared aggregation buffer when an aggregation operator evaluates
- * multiple aggregate functions at the same time.
- *
- * We need to perform similar field number arithmetic when merging multiple intermediate
- * aggregate buffers together in `merge()` (in this case, use `inputAggBufferOffset` when accessing
- * the input buffer).
- *
- * Correct ImperativeAggregate evaluation depends on the correctness of `mutableAggBufferOffset` and
- * `inputAggBufferOffset`, but not on the correctness of the attribute ids in `aggBufferAttributes`
- * and `inputAggBufferAttributes`.
+ * API for aggregation functions that are expressed in terms of imperative doInitialize(),
+ * doUpdate(), doMerge() and doComplete() functions which operate on Row-based aggregation buffers.
*/
abstract class ImperativeAggregate extends AggregateFunction with CodegenFallback {
+ // Although `mutableBufferRow` and `inputBufferRow` are 2 mutable fields in `ImperativeAggregate`,
+ // they can only be set once, thus make `ImperativeAggregate` kind of immutable and stateless.
+
+ /**
+ * The aggregation operator keeps a large shared mutable buffer row for all aggregate functions,
+ * each aggregate function should only access a slice of this shared buffer.
+ */
+ private var mutableBufferRow: SlicedMutableRow = _
+
+ /**
+ * During partial aggregation, the input buffer row to be merged is shared among all aggregate
+ * functions, each aggregate function should only access a slice of this input buffer.
+ */
+ private var inputBufferRow: SlicedInternalRow = _
+
/**
- * The offset of this function's first buffer value in the underlying shared mutable aggregation
- * buffer.
+ * Set the offset of this function's start buffer value in the underlying shared mutable
+ * aggregation buffer.
*
* For example, we have two aggregate functions `avg(x)` and `avg(y)`, which share the same
- * aggregation buffer. In this shared buffer, the position of the first buffer value of `avg(x)`
- * will be 0 and the position of the first buffer value of `avg(y)` will be 2:
+ * aggregation buffer. In this shared buffer, the position of the start buffer value of `avg(x)`
+ * will be 0 and the position of the start buffer value of `avg(y)` will be 2:
* {{{
- * avg(x) mutableAggBufferOffset = 0
+ * avg(x) mutable buffer offset is 0
* |
* v
* +--------+--------+--------+--------+
* | sum1 | count1 | sum2 | count2 |
* +--------+--------+--------+--------+
* ^
* |
- * avg(y) mutableAggBufferOffset = 2
+ * avg(y) mutable buffer offset is 2
* }}}
*/
- protected val mutableAggBufferOffset: Int
-
- /**
- * Returns a copy of this ImperativeAggregate with an updated mutableAggBufferOffset.
- * This new copy's attributes may have different ids than the original.
- */
- def withNewMutableAggBufferOffset(newMutableAggBufferOffset: Int): ImperativeAggregate
+ final def setMutableBufferOffset(offset: Int): Unit = {
+ assert(mutableBufferRow == null)
--- End diff --
Do you want to do a runtime check? Then how about using `require`?
`assert` may be removed by compiler.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14562: [SPARK-16973][SQL] remove the buffer offsets in Imperati...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14562
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/63524/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14562: [SPARK-16973][SQL] remove the buffer offsets in Imperati...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14562
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14562: [SPARK-16973][SQL] remove the buffer offsets in Imperati...
Posted by hvanhovell <gi...@git.apache.org>.
Github user hvanhovell commented on the issue:
https://github.com/apache/spark/pull/14562
Does this have performance implications? We are adding a layer of indirection to a hot code path.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14562: [SPARK-16973][SQL] remove the buffer offsets in I...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/14562#discussion_r74186888
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/slicedRows.scala ---
@@ -0,0 +1,108 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.expressions
+
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.util.{ArrayData, MapData}
+import org.apache.spark.sql.types.{DataType, Decimal}
+import org.apache.spark.unsafe.types.{CalendarInterval, UTF8String}
+
+/**
+ * A sliced row provides a view of the underlying row, and forwards all read/write requests to the
+ * underlying row, after shifting the index by offset.
+ *
+ * Note that, the underlying row field in this class is mutable, so that we can reuse the same
+ * instance of this class while processing many rows.
+ */
+trait BaseSlicedInternalRow extends InternalRow {
+ def baseRow: InternalRow
+ def offset: Int
+
+ protected def toBaseIndex(index: Int): Int = {
+ assert(index >= 0, "index (" + index + ") should >= 0")
+ assert(index < numFields, "index (" + index + ") should < " + numFields)
+ index + offset
+ }
+
+ override def isNullAt(i: Int): Boolean = baseRow.isNullAt(toBaseIndex(i))
+ override def getBoolean(i: Int): Boolean = baseRow.getBoolean(toBaseIndex(i))
+ override def getByte(i: Int): Byte = baseRow.getByte(toBaseIndex(i))
+ override def getShort(i: Int): Short = baseRow.getShort(toBaseIndex(i))
+ override def getInt(i: Int): Int = baseRow.getInt(toBaseIndex(i))
+ override def getLong(i: Int): Long = baseRow.getLong(toBaseIndex(i))
+ override def getFloat(i: Int): Float = baseRow.getFloat(toBaseIndex(i))
+ override def getDouble(i: Int): Double = baseRow.getDouble(toBaseIndex((i)))
+ override def getDecimal(i: Int, precision: Int, scale: Int): Decimal =
+ baseRow.getDecimal(toBaseIndex(i), precision, scale)
+ override def getUTF8String(i: Int): UTF8String = baseRow.getUTF8String(toBaseIndex(i))
+ override def getBinary(i: Int): Array[Byte] = baseRow.getBinary(toBaseIndex(i))
+ override def getInterval(i: Int): CalendarInterval = baseRow.getInterval(toBaseIndex(i))
+ override def getStruct(i: Int, numFields: Int): InternalRow =
+ baseRow.getStruct(toBaseIndex(i), numFields)
+ override def getArray(i: Int): ArrayData = baseRow.getArray(toBaseIndex(i))
+ override def getMap(i: Int): MapData = baseRow.getMap(toBaseIndex(i))
+ override def get(i: Int, dataType: DataType): AnyRef = baseRow.get(toBaseIndex(i), dataType)
+
+ override def anyNull: Boolean = {
+ val len = offset + numFields
+ var i = offset
+ while (i < len) {
+ if (isNullAt(i)) { return true }
+ i += 1
+ }
+ false
+ }
+
+ override def copy(): InternalRow = {
+ throw new UnsupportedOperationException("Cannot copy a SlicedMutableRow")
+ }
+}
+
+case class SlicedInternalRow(offset: Int, numFields: Int) extends BaseSlicedInternalRow {
--- End diff --
hmmm, does case class has performance penalty? It doesn't need to be though.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14562: [SPARK-16973][SQL] remove the buffer offsets in I...
Posted by hvanhovell <gi...@git.apache.org>.
Github user hvanhovell commented on a diff in the pull request:
https://github.com/apache/spark/pull/14562#discussion_r74051693
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/slicedRows.scala ---
@@ -0,0 +1,108 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.expressions
+
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.util.{ArrayData, MapData}
+import org.apache.spark.sql.types.{DataType, Decimal}
+import org.apache.spark.unsafe.types.{CalendarInterval, UTF8String}
+
+/**
+ * A sliced row provides a view of the underlying row, and forwards all read/write requests to the
+ * underlying row, after shifting the index by offset.
+ *
+ * Note that, the underlying row field in this class is mutable, so that we can reuse the same
+ * instance of this class while processing many rows.
+ */
+trait BaseSlicedInternalRow extends InternalRow {
+ def baseRow: InternalRow
+ def offset: Int
+
+ protected def toBaseIndex(index: Int): Int = {
+ assert(index >= 0, "index (" + index + ") should >= 0")
+ assert(index < numFields, "index (" + index + ") should < " + numFields)
+ index + offset
+ }
+
+ override def isNullAt(i: Int): Boolean = baseRow.isNullAt(toBaseIndex(i))
+ override def getBoolean(i: Int): Boolean = baseRow.getBoolean(toBaseIndex(i))
+ override def getByte(i: Int): Byte = baseRow.getByte(toBaseIndex(i))
+ override def getShort(i: Int): Short = baseRow.getShort(toBaseIndex(i))
+ override def getInt(i: Int): Int = baseRow.getInt(toBaseIndex(i))
+ override def getLong(i: Int): Long = baseRow.getLong(toBaseIndex(i))
+ override def getFloat(i: Int): Float = baseRow.getFloat(toBaseIndex(i))
+ override def getDouble(i: Int): Double = baseRow.getDouble(toBaseIndex((i)))
+ override def getDecimal(i: Int, precision: Int, scale: Int): Decimal =
+ baseRow.getDecimal(toBaseIndex(i), precision, scale)
+ override def getUTF8String(i: Int): UTF8String = baseRow.getUTF8String(toBaseIndex(i))
+ override def getBinary(i: Int): Array[Byte] = baseRow.getBinary(toBaseIndex(i))
+ override def getInterval(i: Int): CalendarInterval = baseRow.getInterval(toBaseIndex(i))
+ override def getStruct(i: Int, numFields: Int): InternalRow =
+ baseRow.getStruct(toBaseIndex(i), numFields)
+ override def getArray(i: Int): ArrayData = baseRow.getArray(toBaseIndex(i))
+ override def getMap(i: Int): MapData = baseRow.getMap(toBaseIndex(i))
+ override def get(i: Int, dataType: DataType): AnyRef = baseRow.get(toBaseIndex(i), dataType)
+
+ override def anyNull: Boolean = {
+ val len = offset + numFields
+ var i = offset
+ while (i < len) {
+ if (isNullAt(i)) { return true }
+ i += 1
+ }
+ false
+ }
+
+ override def copy(): InternalRow = {
+ throw new UnsupportedOperationException("Cannot copy a SlicedMutableRow")
--- End diff --
SlicedMutableRow -> SlicedInternalRow?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14562: [SPARK-16973][SQL] remove the buffer offsets in Imperati...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/14562
cc @yhuai @liancheng @clockfly
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org