You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by cloud-fan <gi...@git.apache.org> on 2016/04/23 17:21:08 UTC
[GitHub] spark pull request: [SPARK-14850][ML] specialize array data for Ve...
GitHub user cloud-fan opened a pull request:
https://github.com/apache/spark/pull/12640
[SPARK-14850][ML] specialize array data for VectorUDT/MatrixUDT
## What changes were proposed in this pull request?
This PR adds `IntArrayData` and `DoubleArrayData`, so that we don't need to box primitive arrays and unbox them back for VectorUDT/MatrixUDT.
## How was this patch tested?
existing tests.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/cloud-fan/spark ml
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/12640.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #12640
----
commit 4ce27ada75674d85344e5c2a5d6755035bc8276f
Author: Wenchen Fan <we...@databricks.com>
Date: 2016-04-23T15:10:47Z
specialize array data
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14850][ML] convert primitive array from...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/12640#discussion_r61539393
--- Diff: sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/UnsafeArrayData.java ---
@@ -336,4 +336,62 @@ public UnsafeArrayData copy() {
arrayCopy.pointTo(arrayDataCopy, Platform.BYTE_ARRAY_OFFSET, sizeInBytes);
return arrayCopy;
}
+
+ public int[] toPrimitiveIntArray() {
+ int[] result = new int[numElements];
+ Platform.copyMemory(baseObject, baseOffset + 4 + 4 * numElements,
+ result, Platform.INT_ARRAY_OFFSET, 4 * numElements);
+ return result;
+ }
+
+ public double[] toPrimitiveDoubleArray() {
+ double[] result = new double[numElements];
+ Platform.copyMemory(baseObject, baseOffset + 4 + 4 * numElements,
+ result, Platform.DOUBLE_ARRAY_OFFSET, 8 * numElements);
+ return result;
+ }
+
+ public static UnsafeArrayData fromPrimitiveArray(int[] arr) {
+ int offsetRegionSize = 4 * arr.length;
+ int valueRegionSize = 4 * arr.length;
+ int totalSize = 4 + offsetRegionSize + valueRegionSize;
+ byte[] data = new byte[totalSize];
--- End diff --
After think about it, we can't support very long array in `UnsafeArrayData`, as the offset region only use 4 bytes to encode the value offset. I'm going to add a validation so that we won't overflow if given a big array.
Actually we can improve the unsafe format for array data, which is similar to unsafe row:
```
[null bits] [values] [variable length portion]
```
One difference is that, the `values` region in unsafe row is 8-byte per field, but for array, it should depend on the data size, e.g. 1 byte for boolean. Then we can just memory copy primitive array into `values` region and no need to care about the offset region.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14850][ML] convert primitive array from...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on the pull request:
https://github.com/apache/spark/pull/12640#issuecomment-215931603
LGTM pending Jenkins
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14850][ML] convert primitive array from...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on the pull request:
https://github.com/apache/spark/pull/12640#issuecomment-215940963
LGTM. Merged into master. Thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14850][ML] convert primitive array from...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12640#issuecomment-215939422
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14850][ML] convert primitive array from...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12640#issuecomment-215441061
**[Test build #57253 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57253/consoleFull)** for PR 12640 at commit [`c6c3584`](https://github.com/apache/spark/commit/c6c3584c6cc7c500ca31cb21a00391c06caf3aa6).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14850][ML] convert primitive array from...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12640#issuecomment-215355291
**[Test build #57237 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57237/consoleFull)** for PR 12640 at commit [`a7b7694`](https://github.com/apache/spark/commit/a7b769459619eb8f5968fcde0a56caf8c7a0f499).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14850][ML] specialize array data for Ve...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12640#issuecomment-213919810
**[Test build #56843 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/56843/consoleFull)** for PR 12640 at commit [`13d7eb7`](https://github.com/apache/spark/commit/13d7eb7edf88798201587552f86a2ee2e56059e7).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `abstract class SpecializedArrayData extends ArrayData `
* `class IntArrayData(val values: Array[Int]) extends SpecializedArrayData `
* `class DoubleArrayData(val values: Array[Double]) extends SpecializedArrayData `
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14850][ML] specialize array data for Ve...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the pull request:
https://github.com/apache/spark/pull/12640#issuecomment-214984577
@davies , it's a good point!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14850][ML] specialize array data for Ve...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12640#issuecomment-213784113
**[Test build #56804 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/56804/consoleFull)** for PR 12640 at commit [`55d3178`](https://github.com/apache/spark/commit/55d31788836fe94ba38eec11e99881c0c8562a15).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `abstract class SpecializedArrayData extends ArrayData `
* `class IntArrayData(val values: Array[Int]) extends SpecializedArrayData `
* `class DoubleArrayData(val values: Array[Double]) extends SpecializedArrayData `
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14850][ML] convert primitive array from...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/12640#discussion_r61386945
--- Diff: sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/UnsafeArrayData.java ---
@@ -336,4 +336,62 @@ public UnsafeArrayData copy() {
arrayCopy.pointTo(arrayDataCopy, Platform.BYTE_ARRAY_OFFSET, sizeInBytes);
return arrayCopy;
}
+
+ public int[] toPrimitiveIntArray() {
--- End diff --
It would be hard to tell the difference between `toPrimitiveIntArray` and `toIntArray` by name and signature because both returns primitive arrays. How about `toIntArrayUnchecked`? Please add JavaDoc to explain the difference.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14850][ML] convert primitive array from...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12640#issuecomment-215352327
**[Test build #57230 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57230/consoleFull)** for PR 12640 at commit [`f4d2cbb`](https://github.com/apache/spark/commit/f4d2cbbefabdd7e42317835cc168ea92c26e040c).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14850][ML] convert primitive array from...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12640#issuecomment-215641970
**[Test build #57320 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57320/consoleFull)** for PR 12640 at commit [`537e363`](https://github.com/apache/spark/commit/537e3636da53523a14458006808dfb0303635470).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14850][ML] specialize array data for Ve...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on the pull request:
https://github.com/apache/spark/pull/12640#issuecomment-215318542
Had an offline discussion with @cloud-fan and we will try converting from/to UnsafeArrayData directly using memory copy and test its performance.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14850][ML] convert primitive array from...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/12640#discussion_r61387931
--- Diff: sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/UnsafeArrayData.java ---
@@ -336,4 +336,62 @@ public UnsafeArrayData copy() {
arrayCopy.pointTo(arrayDataCopy, Platform.BYTE_ARRAY_OFFSET, sizeInBytes);
return arrayCopy;
}
+
+ public int[] toPrimitiveIntArray() {
+ int[] result = new int[numElements];
+ Platform.copyMemory(baseObject, baseOffset + 4 + 4 * numElements,
+ result, Platform.INT_ARRAY_OFFSET, 4 * numElements);
+ return result;
+ }
+
+ public double[] toPrimitiveDoubleArray() {
+ double[] result = new double[numElements];
+ Platform.copyMemory(baseObject, baseOffset + 4 + 4 * numElements,
+ result, Platform.DOUBLE_ARRAY_OFFSET, 8 * numElements);
+ return result;
+ }
+
+ public static UnsafeArrayData fromPrimitiveArray(int[] arr) {
+ int offsetRegionSize = 4 * arr.length;
+ int valueRegionSize = 4 * arr.length;
+ int totalSize = 4 + offsetRegionSize + valueRegionSize;
+ byte[] data = new byte[totalSize];
--- End diff --
This is not an unsafe array. Should we allocate a direct buffer instead?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14850][ML] convert primitive array from...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on the pull request:
https://github.com/apache/spark/pull/12640#issuecomment-215343973
@cloud-fan Could you also update the benchmark?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14850][ML] convert primitive array from...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12640#issuecomment-215932258
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14850][ML] convert primitive array from...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12640#issuecomment-215473555
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14850][ML] convert primitive array from...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/12640#discussion_r61387937
--- Diff: sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/UnsafeArrayData.java ---
@@ -336,4 +336,62 @@ public UnsafeArrayData copy() {
arrayCopy.pointTo(arrayDataCopy, Platform.BYTE_ARRAY_OFFSET, sizeInBytes);
return arrayCopy;
}
+
+ public int[] toPrimitiveIntArray() {
+ int[] result = new int[numElements];
+ Platform.copyMemory(baseObject, baseOffset + 4 + 4 * numElements,
+ result, Platform.INT_ARRAY_OFFSET, 4 * numElements);
+ return result;
+ }
+
+ public double[] toPrimitiveDoubleArray() {
+ double[] result = new double[numElements];
+ Platform.copyMemory(baseObject, baseOffset + 4 + 4 * numElements,
+ result, Platform.DOUBLE_ARRAY_OFFSET, 8 * numElements);
+ return result;
+ }
+
+ public static UnsafeArrayData fromPrimitiveArray(int[] arr) {
+ int offsetRegionSize = 4 * arr.length;
+ int valueRegionSize = 4 * arr.length;
+ int totalSize = 4 + offsetRegionSize + valueRegionSize;
+ byte[] data = new byte[totalSize];
+
+ Platform.putInt(data, Platform.BYTE_ARRAY_OFFSET, arr.length);
+
+ int elementOffsetStart = 4 + offsetRegionSize;
+ for (int i = 0; i < arr.length; i++) {
+ Platform.putInt(data, Platform.BYTE_ARRAY_OFFSET + 4 + i * 4, elementOffsetStart + i * 4);
--- End diff --
This has two multiplications per element, which is expensive. Try:
~~~java
int regionOffset = Platform.BYTE_ARRAY_OFFSET + 4
int elementOffset = elementOffsetStart
for (...) {
Platform.putInt(data, regionOffset, elementOffset);
regionOffset += 4;
elementOffset += 4;
}
~~~
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14850][ML] specialize array data for Ve...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12640#issuecomment-213784744
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14850][ML] convert primitive array from...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12640#issuecomment-215932497
**[Test build #57403 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57403/consoleFull)** for PR 12640 at commit [`b10845c`](https://github.com/apache/spark/commit/b10845cb57a574c3337d3658d1c0cb42a22910a8).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14850][ML] convert primitive array from...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12640#issuecomment-215352568
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/57230/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14850][ML] specialize array data for Ve...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12640#issuecomment-213784751
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/56804/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14850][ML] convert primitive array from...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/12640#discussion_r61509114
--- Diff: sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/UnsafeArrayData.java ---
@@ -336,4 +336,78 @@ public UnsafeArrayData copy() {
arrayCopy.pointTo(arrayDataCopy, Platform.BYTE_ARRAY_OFFSET, sizeInBytes);
return arrayCopy;
}
+
+ /**
+ * A faster version of `toIntArray`, which use memory copy instead of iterating all elements.
+ * Note that, this method is dangerous if this array contains null elements. We don't write
+ * null elements into the data region and memory copy will crash as the data size doesn't match.
+ */
+ public int[] toIntArrayUnchecked() {
+ int[] result = new int[numElements];
+ Platform.copyMemory(baseObject, baseOffset + 4 + 4L * numElements,
+ result, Platform.INT_ARRAY_OFFSET, 4L * numElements);
+ return result;
+ }
+
+ /**
+ * A faster version of `toDoubleArray`, which use memory copy instead of iterating all elements.
+ * Note that, this method is dangerous if this array contains null elements. We don't write
+ * null elements into the data region and memory copy will crash as the data size doesn't match.
+ */
+ public double[] toDoubleArrayUnchecked() {
+ double[] result = new double[numElements];
+ Platform.copyMemory(baseObject, baseOffset + 4 + 4L * numElements,
+ result, Platform.DOUBLE_ARRAY_OFFSET, 8L * numElements);
+ return result;
+ }
+
+ public static UnsafeArrayData fromPrimitiveArray(int[] arr) {
+ int offsetRegionSize = 4 * arr.length;
--- End diff --
Please check that `4 * arr.length` won't overflow.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14850][ML] specialize array data for Ve...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12640#issuecomment-213919847
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/56843/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14850][ML] convert primitive array from...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12640#issuecomment-215473255
**[Test build #57253 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57253/consoleFull)** for PR 12640 at commit [`c6c3584`](https://github.com/apache/spark/commit/c6c3584c6cc7c500ca31cb21a00391c06caf3aa6).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14850][ML] convert primitive array from...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12640#issuecomment-215370951
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14850][ML] convert primitive array from...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12640#issuecomment-215370706
**[Test build #57237 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57237/consoleFull)** for PR 12640 at commit [`a7b7694`](https://github.com/apache/spark/commit/a7b769459619eb8f5968fcde0a56caf8c7a0f499).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `public final class UnsafeArrayData extends ArrayData `
* `public final class UnsafeMapData extends MapData `
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14850][ML] convert primitive array from...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/12640#discussion_r61509122
--- Diff: sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/UnsafeArrayData.java ---
@@ -336,4 +336,62 @@ public UnsafeArrayData copy() {
arrayCopy.pointTo(arrayDataCopy, Platform.BYTE_ARRAY_OFFSET, sizeInBytes);
return arrayCopy;
}
+
+ public int[] toPrimitiveIntArray() {
+ int[] result = new int[numElements];
+ Platform.copyMemory(baseObject, baseOffset + 4 + 4 * numElements,
+ result, Platform.INT_ARRAY_OFFSET, 4 * numElements);
+ return result;
+ }
+
+ public double[] toPrimitiveDoubleArray() {
+ double[] result = new double[numElements];
+ Platform.copyMemory(baseObject, baseOffset + 4 + 4 * numElements,
+ result, Platform.DOUBLE_ARRAY_OFFSET, 8 * numElements);
+ return result;
+ }
+
+ public static UnsafeArrayData fromPrimitiveArray(int[] arr) {
+ int offsetRegionSize = 4 * arr.length;
+ int valueRegionSize = 4 * arr.length;
+ int totalSize = 4 + offsetRegionSize + valueRegionSize;
+ byte[] data = new byte[totalSize];
--- End diff --
I see. We might have very large vectors, e.g., 10 million values. But it is not very common.
@cloud-fan Can we use `long[]` or `double[]` to back up the buffer? So we can store more elements. Right now the upper bound is about 3e8, which might be sufficient. But if a simple change would increase the limit, that would be better..
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14850][ML] convert primitive array from...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12640#issuecomment-215352565
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14850][ML] convert primitive array from...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12640#issuecomment-215920860
**[Test build #57387 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57387/consoleFull)** for PR 12640 at commit [`b10845c`](https://github.com/apache/spark/commit/b10845cb57a574c3337d3658d1c0cb42a22910a8).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14850][ML] specialize array data for Ve...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the pull request:
https://github.com/apache/spark/pull/12640#issuecomment-213764953
I ran the simple benchmark in JIRA:
```
sc.parallelize(0 until 1e4.toInt, 1).map { i =>
(i, Vectors.dense(Array.fill(1e6.toInt)(1.0)))
}.toDF.rdd.count()
```
The result for master is: 199306 ms
The result after tiis PR: 69342 ms
It's still much slower than the 1.4 version, but we have unsafe format since 1.5, and for this simple benchmark that has nearly no execution but only serialization, it's reasonable to be slower.
cc @mengxr
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14850][ML] specialize array data for Ve...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12640#issuecomment-215331169
**[Test build #57230 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57230/consoleFull)** for PR 12640 at commit [`f4d2cbb`](https://github.com/apache/spark/commit/f4d2cbbefabdd7e42317835cc168ea92c26e040c).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14850][ML] convert primitive array from...
Posted by davies <gi...@git.apache.org>.
Github user davies commented on a diff in the pull request:
https://github.com/apache/spark/pull/12640#discussion_r61485789
--- Diff: mllib/src/test/scala/org/apache/spark/mllib/linalg/UDTSerializationBenchmark.scala ---
@@ -0,0 +1,70 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.mllib.linalg
+
+import org.apache.spark.sql.catalyst.encoders.ExpressionEncoder
+import org.apache.spark.util.Benchmark
+
+/**
+ * Serialization benchmark for VectorUDT.
+ */
+object UDTSerializationBenchmark {
+
+ def main(args: Array[String]): Unit = {
+ val iters = 1e2.toInt
+ val numRows = 1e3.toInt
+
+ val encoder = ExpressionEncoder[Vector].defaultBinding
+
+ val vectors = (1 to numRows).map { i =>
+ Vectors.dense(Array.fill(1e5.toInt)(1.0 * i))
+ }.toArray
+ val rows = vectors.map(encoder.toRow)
+
+ val benchmark = new Benchmark("VectorUDT de/serialization", numRows, iters)
+
+ benchmark.addCase("serialize") { _ =>
+ var sum = 0
+ var i = 0
+ while (i < numRows) {
+ sum += encoder.toRow(vectors(i)).numFields
+ i += 1
+ }
+ }
+
+ benchmark.addCase("deserialize") { _ =>
+ var sum = 0
+ var i = 0
+ while (i < numRows) {
+ sum += encoder.fromRow(rows(i)).numActives
+ i += 1
+ }
+ }
+
+ /*
+ Java HotSpot(TM) 64-Bit Server VM 1.8.0_60-b27 on Mac OS X 10.11.4
+ Intel(R) Core(TM) i7-4960HQ CPU @ 2.60GHz
+
+ VectorUDT de/serialization: Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
+ -------------------------------------------------------------------------------------------
+ serialize 380 / 392 0.0 379730.0 1.0X
+ deserialize 138 / 142 0.0 137816.6 2.8X
+ */
+ benchmark.run()
--- End diff --
I think so, could you run the benchmark with more iterations to make sure that the C2 compiler could kick in (especially in Java 8)?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14850][ML] convert primitive array from...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12640#issuecomment-215939423
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/57403/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14850][ML] convert primitive array from...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12640#issuecomment-215374404
**[Test build #57241 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57241/consoleFull)** for PR 12640 at commit [`d445022`](https://github.com/apache/spark/commit/d4450221c1688ddb0b7f71ce48970d2937f252d2).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14850][ML] convert primitive array from...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12640#issuecomment-215385781
**[Test build #57239 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57239/consoleFull)** for PR 12640 at commit [`f6964f9`](https://github.com/apache/spark/commit/f6964f9e8d0db473354941f31cf182628d0a3299).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14850][ML] convert primitive array from...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/12640#discussion_r61386974
--- Diff: sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/UnsafeArrayData.java ---
@@ -336,4 +336,62 @@ public UnsafeArrayData copy() {
arrayCopy.pointTo(arrayDataCopy, Platform.BYTE_ARRAY_OFFSET, sizeInBytes);
return arrayCopy;
}
+
+ public int[] toPrimitiveIntArray() {
+ int[] result = new int[numElements];
+ Platform.copyMemory(baseObject, baseOffset + 4 + 4 * numElements,
--- End diff --
* `4 *` -> `4L *` to avoid overflow. Please check other places as well.
* I don't quite understand the `offsetRegionSize`. Is it reserved for marking null values or handling variable-length elements in the future? This is quite expensive for primitive arrays.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14850][ML] specialize array data for Ve...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on the pull request:
https://github.com/apache/spark/pull/12640#issuecomment-215314086
@cloud-fan This is still much slower than 1.4 and adding more subclasses of ArrayData may prevent JIT inline methods like `getInt` and `getDouble`. Is it easy to convert to `UnsafeArrayData` directly with memory copy?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14850][ML] convert primitive array from...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/12640#discussion_r61505845
--- Diff: mllib/src/test/scala/org/apache/spark/mllib/linalg/UDTSerializationBenchmark.scala ---
@@ -0,0 +1,70 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.mllib.linalg
+
+import org.apache.spark.sql.catalyst.encoders.ExpressionEncoder
+import org.apache.spark.util.Benchmark
+
+/**
+ * Serialization benchmark for VectorUDT.
+ */
+object UDTSerializationBenchmark {
+
+ def main(args: Array[String]): Unit = {
+ val iters = 1e2.toInt
+ val numRows = 1e3.toInt
+
+ val encoder = ExpressionEncoder[Vector].defaultBinding
+
+ val vectors = (1 to numRows).map { i =>
+ Vectors.dense(Array.fill(1e5.toInt)(1.0 * i))
+ }.toArray
+ val rows = vectors.map(encoder.toRow)
+
+ val benchmark = new Benchmark("VectorUDT de/serialization", numRows, iters)
+
+ benchmark.addCase("serialize") { _ =>
+ var sum = 0
+ var i = 0
+ while (i < numRows) {
+ sum += encoder.toRow(vectors(i)).numFields
--- End diff --
Can we call `VectorUDT.serialize` directly instead of `encoder.toRows`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14850][ML] convert primitive array from...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/12640#discussion_r61399541
--- Diff: mllib/src/test/scala/org/apache/spark/mllib/linalg/UDTSerializationBenchmark.scala ---
@@ -0,0 +1,70 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.mllib.linalg
+
+import org.apache.spark.sql.catalyst.encoders.ExpressionEncoder
+import org.apache.spark.util.Benchmark
+
+/**
+ * Serialization benchmark for VectorUDT.
+ */
+object UDTSerializationBenchmark {
+
+ def main(args: Array[String]): Unit = {
+ val iters = 1e2.toInt
+ val numRows = 1e3.toInt
+
+ val encoder = ExpressionEncoder[Vector].defaultBinding
+
+ val vectors = (1 to numRows).map { i =>
+ Vectors.dense(Array.fill(1e5.toInt)(1.0 * i))
+ }.toArray
+ val rows = vectors.map(encoder.toRow)
+
+ val benchmark = new Benchmark("VectorUDT de/serialization", numRows, iters)
+
+ benchmark.addCase("serialize") { _ =>
+ var sum = 0
+ var i = 0
+ while (i < numRows) {
+ sum += encoder.toRow(vectors(i)).numFields
+ i += 1
+ }
+ }
+
+ benchmark.addCase("deserialize") { _ =>
+ var sum = 0
+ var i = 0
+ while (i < numRows) {
+ sum += encoder.fromRow(rows(i)).numActives
+ i += 1
+ }
+ }
+
+ /*
+ Java HotSpot(TM) 64-Bit Server VM 1.8.0_60-b27 on Mac OS X 10.11.4
+ Intel(R) Core(TM) i7-4960HQ CPU @ 2.60GHz
+
+ VectorUDT de/serialization: Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
+ -------------------------------------------------------------------------------------------
+ serialize 380 / 392 0.0 379730.0 1.0X
+ deserialize 138 / 142 0.0 137816.6 2.8X
+ */
+ benchmark.run()
--- End diff --
did a micro benchmark, the `toDoubleArray` and the new `toPrimitiveIntArray` don't have much difference. Maybe JVM can optimize simple while loop?
```
def toDoubleArray(): Array[Double] = {
val size = numElements()
val values = new Array[Double](size)
var i = 0
while (i < size) {
values(i) = getDouble(i)
i += 1
}
values
}
```
cc @davies
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14850][ML] specialize array data for Ve...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12640#issuecomment-213763628
**[Test build #56804 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/56804/consoleFull)** for PR 12640 at commit [`55d3178`](https://github.com/apache/spark/commit/55d31788836fe94ba38eec11e99881c0c8562a15).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14850][ML] specialize array data for Ve...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12640#issuecomment-213757580
**[Test build #56803 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/56803/consoleFull)** for PR 12640 at commit [`4ce27ad`](https://github.com/apache/spark/commit/4ce27ada75674d85344e5c2a5d6755035bc8276f).
* This patch **fails Scala style tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `abstract class SpecializedArrayData extends ArrayData `
* `class IntArrayData(val values: Array[Int]) extends SpecializedArrayData `
* `class DoubleArrayData(val values: Array[Double]) extends SpecializedArrayData `
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14850][ML] specialize array data for Ve...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/12640#discussion_r61375365
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/ArrayData.scala ---
@@ -29,6 +29,82 @@ abstract class ArrayData extends SpecializedGetters with Serializable {
def array: Array[Any]
+ override def equals(o: Any): Boolean = {
+ if (!o.isInstanceOf[ArrayData]) {
+ return false
+ }
+
+ val other = o.asInstanceOf[ArrayData]
+ if (other eq null) {
+ return false
+ }
+
+ val len = numElements()
+ if (len != other.numElements()) {
+ return false
+ }
+
+ var i = 0
+ while (i < len) {
+ if (isNullAt(i) != other.isNullAt(i)) {
+ return false
+ }
+ if (!isNullAt(i)) {
+ val o1 = array(i)
+ val o2 = other.array(i)
+ o1 match {
+ case b1: Array[Byte] =>
+ if (!o2.isInstanceOf[Array[Byte]] ||
+ !java.util.Arrays.equals(b1, o2.asInstanceOf[Array[Byte]])) {
+ return false
+ }
+ case f1: Float if java.lang.Float.isNaN(f1) =>
+ if (!o2.isInstanceOf[Float] || ! java.lang.Float.isNaN(o2.asInstanceOf[Float])) {
+ return false
+ }
+ case d1: Double if java.lang.Double.isNaN(d1) =>
+ if (!o2.isInstanceOf[Double] || ! java.lang.Double.isNaN(o2.asInstanceOf[Double])) {
+ return false
+ }
+ case _ => if (o1 != o2) {
+ return false
+ }
+ }
+ }
+ i += 1
+ }
+ true
+ }
+
+ override def hashCode: Int = {
+ var result: Int = 37
+ var i = 0
+ val len = numElements()
+ while (i < len) {
--- End diff --
This could be very expensive for large arrays because it scans all elements, which is unnecessary to generate the hashCode.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14850][ML] convert primitive array from...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12640#issuecomment-215656335
**[Test build #57320 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57320/consoleFull)** for PR 12640 at commit [`537e363`](https://github.com/apache/spark/commit/537e3636da53523a14458006808dfb0303635470).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14850][ML] convert primitive array from...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12640#issuecomment-215386110
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14850][ML] convert primitive array from...
Posted by davies <gi...@git.apache.org>.
Github user davies commented on a diff in the pull request:
https://github.com/apache/spark/pull/12640#discussion_r61542453
--- Diff: sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/UnsafeArrayData.java ---
@@ -336,4 +336,62 @@ public UnsafeArrayData copy() {
arrayCopy.pointTo(arrayDataCopy, Platform.BYTE_ARRAY_OFFSET, sizeInBytes);
return arrayCopy;
}
+
+ public int[] toPrimitiveIntArray() {
+ int[] result = new int[numElements];
+ Platform.copyMemory(baseObject, baseOffset + 4 + 4 * numElements,
+ result, Platform.INT_ARRAY_OFFSET, 4 * numElements);
+ return result;
+ }
+
+ public double[] toPrimitiveDoubleArray() {
+ double[] result = new double[numElements];
+ Platform.copyMemory(baseObject, baseOffset + 4 + 4 * numElements,
+ result, Platform.DOUBLE_ARRAY_OFFSET, 8 * numElements);
+ return result;
+ }
+
+ public static UnsafeArrayData fromPrimitiveArray(int[] arr) {
+ int offsetRegionSize = 4 * arr.length;
+ int valueRegionSize = 4 * arr.length;
+ int totalSize = 4 + offsetRegionSize + valueRegionSize;
+ byte[] data = new byte[totalSize];
--- End diff --
+1
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14850][ML] convert primitive array from...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12640#issuecomment-215939391
**[Test build #57403 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57403/consoleFull)** for PR 12640 at commit [`b10845c`](https://github.com/apache/spark/commit/b10845cb57a574c3337d3658d1c0cb42a22910a8).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14850][ML] convert primitive array from...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12640#issuecomment-215921311
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/57387/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14850][ML] convert primitive array from...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the pull request:
https://github.com/apache/spark/pull/12640#issuecomment-215930973
retest this please.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14850][ML] convert primitive array from...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12640#issuecomment-215656516
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14850][ML] convert primitive array from...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12640#issuecomment-215374427
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14850][ML] convert primitive array from...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12640#issuecomment-215932250
**[Test build #57399 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57399/consoleFull)** for PR 12640 at commit [`b10845c`](https://github.com/apache/spark/commit/b10845cb57a574c3337d3658d1c0cb42a22910a8).
* This patch **fails MiMa tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14850][ML] convert primitive array from...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12640#issuecomment-215921297
**[Test build #57387 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57387/consoleFull)** for PR 12640 at commit [`b10845c`](https://github.com/apache/spark/commit/b10845cb57a574c3337d3658d1c0cb42a22910a8).
* This patch **fails MiMa tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14850][ML] convert primitive array from...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/12640#discussion_r61529874
--- Diff: mllib/src/test/scala/org/apache/spark/mllib/linalg/UDTSerializationBenchmark.scala ---
@@ -0,0 +1,70 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.mllib.linalg
+
+import org.apache.spark.sql.catalyst.encoders.ExpressionEncoder
+import org.apache.spark.util.Benchmark
+
+/**
+ * Serialization benchmark for VectorUDT.
+ */
+object UDTSerializationBenchmark {
+
+ def main(args: Array[String]): Unit = {
+ val iters = 1e2.toInt
+ val numRows = 1e3.toInt
+
+ val encoder = ExpressionEncoder[Vector].defaultBinding
+
+ val vectors = (1 to numRows).map { i =>
+ Vectors.dense(Array.fill(1e5.toInt)(1.0 * i))
+ }.toArray
+ val rows = vectors.map(encoder.toRow)
+
+ val benchmark = new Benchmark("VectorUDT de/serialization", numRows, iters)
+
+ benchmark.addCase("serialize") { _ =>
+ var sum = 0
+ var i = 0
+ while (i < numRows) {
+ sum += encoder.toRow(vectors(i)).numFields
--- End diff --
it's different, `VectorUDT.serialize` only turn user object to catalyst data, but the real serialization should also include convert catalyst data into unsafe format.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14850][ML] convert primitive array from...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12640#issuecomment-215363656
**[Test build #57239 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57239/consoleFull)** for PR 12640 at commit [`f6964f9`](https://github.com/apache/spark/commit/f6964f9e8d0db473354941f31cf182628d0a3299).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14850][ML] specialize array data for Ve...
Posted by davies <gi...@git.apache.org>.
Github user davies commented on the pull request:
https://github.com/apache/spark/pull/12640#issuecomment-214979959
@cloud-fan Could you also improve the conversion between DoubleArrayData and UnsafeArrayData using memory copy?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14850][ML] convert primitive array from...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/12640#discussion_r61389927
--- Diff: sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/UnsafeArrayData.java ---
@@ -336,4 +336,62 @@ public UnsafeArrayData copy() {
arrayCopy.pointTo(arrayDataCopy, Platform.BYTE_ARRAY_OFFSET, sizeInBytes);
return arrayCopy;
}
+
+ public int[] toPrimitiveIntArray() {
+ int[] result = new int[numElements];
+ Platform.copyMemory(baseObject, baseOffset + 4 + 4 * numElements,
+ result, Platform.INT_ARRAY_OFFSET, 4 * numElements);
+ return result;
+ }
+
+ public double[] toPrimitiveDoubleArray() {
+ double[] result = new double[numElements];
+ Platform.copyMemory(baseObject, baseOffset + 4 + 4 * numElements,
+ result, Platform.DOUBLE_ARRAY_OFFSET, 8 * numElements);
+ return result;
+ }
+
+ public static UnsafeArrayData fromPrimitiveArray(int[] arr) {
+ int offsetRegionSize = 4 * arr.length;
+ int valueRegionSize = 4 * arr.length;
+ int totalSize = 4 + offsetRegionSize + valueRegionSize;
+ byte[] data = new byte[totalSize];
--- End diff --
I think it doesn't matter, unsafe row also uses on-heap byte array
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14850][ML] convert primitive array from...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12640#issuecomment-215370955
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/57237/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14850][ML] convert primitive array from...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/12640#discussion_r61397517
--- Diff: mllib/src/test/scala/org/apache/spark/mllib/linalg/UDTSerializationBenchmark.scala ---
@@ -0,0 +1,70 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.mllib.linalg
+
+import org.apache.spark.sql.catalyst.encoders.ExpressionEncoder
+import org.apache.spark.util.Benchmark
+
+/**
+ * Serialization benchmark for VectorUDT.
+ */
+object UDTSerializationBenchmark {
+
+ def main(args: Array[String]): Unit = {
+ val iters = 1e2.toInt
+ val numRows = 1e3.toInt
+
+ val encoder = ExpressionEncoder[Vector].defaultBinding
+
+ val vectors = (1 to numRows).map { i =>
+ Vectors.dense(Array.fill(1e5.toInt)(1.0 * i))
+ }.toArray
+ val rows = vectors.map(encoder.toRow)
+
+ val benchmark = new Benchmark("VectorUDT de/serialization", numRows, iters)
+
+ benchmark.addCase("serialize") { _ =>
+ var sum = 0
+ var i = 0
+ while (i < numRows) {
+ sum += encoder.toRow(vectors(i)).numFields
+ i += 1
+ }
+ }
+
+ benchmark.addCase("deserialize") { _ =>
+ var sum = 0
+ var i = 0
+ while (i < numRows) {
+ sum += encoder.fromRow(rows(i)).numActives
+ i += 1
+ }
+ }
+
+ /*
+ Java HotSpot(TM) 64-Bit Server VM 1.8.0_60-b27 on Mac OS X 10.11.4
+ Intel(R) Core(TM) i7-4960HQ CPU @ 2.60GHz
+
+ VectorUDT de/serialization: Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
+ -------------------------------------------------------------------------------------------
+ serialize 380 / 392 0.0 379730.0 1.0X
+ deserialize 138 / 142 0.0 137816.6 2.8X
+ */
+ benchmark.run()
--- End diff --
result on master:
```
VectorUDT de/serialization: Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
-------------------------------------------------------------------------------------------
serialize 1414 / 1462 0.0 1414104.1 1.0X
deserialize 169 / 178 0.0 169323.7 8.4X
```
The deserialize is much faster now, but the deserialize isn't , investigating
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14850][ML] specialize array data for Ve...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12640#issuecomment-213910584
**[Test build #56843 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/56843/consoleFull)** for PR 12640 at commit [`13d7eb7`](https://github.com/apache/spark/commit/13d7eb7edf88798201587552f86a2ee2e56059e7).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14850][ML] specialize array data for Ve...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12640#issuecomment-213757582
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/56803/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14850][ML] convert primitive array from...
Posted by kiszk <gi...@git.apache.org>.
Github user kiszk commented on the pull request:
https://github.com/apache/spark/pull/12640#issuecomment-215338341
@cloud-fan , @mengxr, it would be worth to add ```final``` to a declaration of ```UnsafeArrayData ``` for encouraging method inlining by JIT compiler, as follows:
``` public final class UnsafeArrayData extends ArrayData```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14850][ML] convert primitive array from...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/12640#discussion_r61541291
--- Diff: sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/UnsafeArrayData.java ---
@@ -336,4 +336,62 @@ public UnsafeArrayData copy() {
arrayCopy.pointTo(arrayDataCopy, Platform.BYTE_ARRAY_OFFSET, sizeInBytes);
return arrayCopy;
}
+
+ public int[] toPrimitiveIntArray() {
+ int[] result = new int[numElements];
+ Platform.copyMemory(baseObject, baseOffset + 4 + 4 * numElements,
+ result, Platform.INT_ARRAY_OFFSET, 4 * numElements);
+ return result;
+ }
+
+ public double[] toPrimitiveDoubleArray() {
+ double[] result = new double[numElements];
+ Platform.copyMemory(baseObject, baseOffset + 4 + 4 * numElements,
+ result, Platform.DOUBLE_ARRAY_OFFSET, 8 * numElements);
+ return result;
+ }
+
+ public static UnsafeArrayData fromPrimitiveArray(int[] arr) {
+ int offsetRegionSize = 4 * arr.length;
+ int valueRegionSize = 4 * arr.length;
+ int totalSize = 4 + offsetRegionSize + valueRegionSize;
+ byte[] data = new byte[totalSize];
--- End diff --
cc @davies @mengxr
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14850][ML] convert primitive array from...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12640#issuecomment-215372214
**[Test build #57241 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57241/consoleFull)** for PR 12640 at commit [`d445022`](https://github.com/apache/spark/commit/d4450221c1688ddb0b7f71ce48970d2937f252d2).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14850][ML] convert primitive array from...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/12640#discussion_r61540468
--- Diff: mllib/src/test/scala/org/apache/spark/mllib/linalg/UDTSerializationBenchmark.scala ---
@@ -0,0 +1,70 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.mllib.linalg
+
+import org.apache.spark.sql.catalyst.encoders.ExpressionEncoder
+import org.apache.spark.util.Benchmark
+
+/**
+ * Serialization benchmark for VectorUDT.
+ */
+object UDTSerializationBenchmark {
+
+ def main(args: Array[String]): Unit = {
+ val iters = 1e2.toInt
+ val numRows = 1e3.toInt
+
+ val encoder = ExpressionEncoder[Vector].defaultBinding
+
+ val vectors = (1 to numRows).map { i =>
+ Vectors.dense(Array.fill(1e5.toInt)(1.0 * i))
+ }.toArray
+ val rows = vectors.map(encoder.toRow)
+
+ val benchmark = new Benchmark("VectorUDT de/serialization", numRows, iters)
+
+ benchmark.addCase("serialize") { _ =>
+ var sum = 0
+ var i = 0
+ while (i < numRows) {
+ sum += encoder.toRow(vectors(i)).numFields
+ i += 1
+ }
+ }
+
+ benchmark.addCase("deserialize") { _ =>
+ var sum = 0
+ var i = 0
+ while (i < numRows) {
+ sum += encoder.fromRow(rows(i)).numActives
+ i += 1
+ }
+ }
+
+ /*
+ Java HotSpot(TM) 64-Bit Server VM 1.8.0_60-b27 on Mac OS X 10.11.4
+ Intel(R) Core(TM) i7-4960HQ CPU @ 2.60GHz
+
+ VectorUDT de/serialization: Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
+ -------------------------------------------------------------------------------------------
+ serialize 380 / 392 0.0 379730.0 1.0X
+ deserialize 138 / 142 0.0 137816.6 2.8X
+ */
+ benchmark.run()
--- End diff --
I rerun the benchmark with 5 times higher iterations, but the result shows no difference.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14850][ML] convert primitive array from...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12640#issuecomment-215921309
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14850][ML] specialize array data for Ve...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12640#issuecomment-213757486
**[Test build #56803 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/56803/consoleFull)** for PR 12640 at commit [`4ce27ad`](https://github.com/apache/spark/commit/4ce27ada75674d85344e5c2a5d6755035bc8276f).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14850][ML] convert primitive array from...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12640#issuecomment-215931405
**[Test build #57399 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57399/consoleFull)** for PR 12640 at commit [`b10845c`](https://github.com/apache/spark/commit/b10845cb57a574c3337d3658d1c0cb42a22910a8).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14850][ML] convert primitive array from...
Posted by davies <gi...@git.apache.org>.
Github user davies commented on a diff in the pull request:
https://github.com/apache/spark/pull/12640#discussion_r61486213
--- Diff: sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/UnsafeArrayData.java ---
@@ -336,4 +336,62 @@ public UnsafeArrayData copy() {
arrayCopy.pointTo(arrayDataCopy, Platform.BYTE_ARRAY_OFFSET, sizeInBytes);
return arrayCopy;
}
+
+ public int[] toPrimitiveIntArray() {
+ int[] result = new int[numElements];
+ Platform.copyMemory(baseObject, baseOffset + 4 + 4 * numElements,
+ result, Platform.INT_ARRAY_OFFSET, 4 * numElements);
+ return result;
+ }
+
+ public double[] toPrimitiveDoubleArray() {
+ double[] result = new double[numElements];
+ Platform.copyMemory(baseObject, baseOffset + 4 + 4 * numElements,
+ result, Platform.DOUBLE_ARRAY_OFFSET, 8 * numElements);
+ return result;
+ }
+
+ public static UnsafeArrayData fromPrimitiveArray(int[] arr) {
+ int offsetRegionSize = 4 * arr.length;
+ int valueRegionSize = 4 * arr.length;
+ int totalSize = 4 + offsetRegionSize + valueRegionSize;
+ byte[] data = new byte[totalSize];
--- End diff --
@mengxr We only use off-heap memory (direct buffer) for page (tens of MB), otherwise always use on-heap arrays. Off heap memory may be less efficient to handle smaller allocates.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14850][ML] convert primitive array from...
Posted by davies <gi...@git.apache.org>.
Github user davies commented on a diff in the pull request:
https://github.com/apache/spark/pull/12640#discussion_r61540619
--- Diff: mllib/src/test/scala/org/apache/spark/mllib/linalg/UDTSerializationBenchmark.scala ---
@@ -0,0 +1,70 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.mllib.linalg
+
+import org.apache.spark.sql.catalyst.encoders.ExpressionEncoder
+import org.apache.spark.util.Benchmark
+
+/**
+ * Serialization benchmark for VectorUDT.
+ */
+object UDTSerializationBenchmark {
+
+ def main(args: Array[String]): Unit = {
+ val iters = 1e2.toInt
+ val numRows = 1e3.toInt
+
+ val encoder = ExpressionEncoder[Vector].defaultBinding
+
+ val vectors = (1 to numRows).map { i =>
+ Vectors.dense(Array.fill(1e5.toInt)(1.0 * i))
+ }.toArray
+ val rows = vectors.map(encoder.toRow)
+
+ val benchmark = new Benchmark("VectorUDT de/serialization", numRows, iters)
+
+ benchmark.addCase("serialize") { _ =>
+ var sum = 0
+ var i = 0
+ while (i < numRows) {
+ sum += encoder.toRow(vectors(i)).numFields
+ i += 1
+ }
+ }
+
+ benchmark.addCase("deserialize") { _ =>
+ var sum = 0
+ var i = 0
+ while (i < numRows) {
+ sum += encoder.fromRow(rows(i)).numActives
+ i += 1
+ }
+ }
+
+ /*
+ Java HotSpot(TM) 64-Bit Server VM 1.8.0_60-b27 on Mac OS X 10.11.4
+ Intel(R) Core(TM) i7-4960HQ CPU @ 2.60GHz
+
+ VectorUDT de/serialization: Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
+ -------------------------------------------------------------------------------------------
+ serialize 380 / 392 0.0 379730.0 1.0X
+ deserialize 138 / 142 0.0 137816.6 2.8X
+ */
+ benchmark.run()
--- End diff --
Because we ran the test multiple times, and pick the best one, so that's fine.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14850][ML] convert primitive array from...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12640#issuecomment-215473565
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/57253/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14850][ML] convert primitive array from...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12640#issuecomment-215386113
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/57239/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14850][ML] convert primitive array from...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12640#issuecomment-215656518
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/57320/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14850][ML] convert primitive array from...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/12640#discussion_r61383378
--- Diff: sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/UnsafeArrayData.java ---
@@ -336,4 +336,62 @@ public UnsafeArrayData copy() {
arrayCopy.pointTo(arrayDataCopy, Platform.BYTE_ARRAY_OFFSET, sizeInBytes);
return arrayCopy;
}
+
+ public int[] toPrimitiveIntArray() {
--- End diff --
I didn't override the `toIntArray`, but create this special method instead. This operation is dangerous, if some elements are null, we won't return 0, but may crash instead. The reason is we don't write null values, if an element is null, we simply mark it as null in the offset region and skip it. For example, the data size of unsafe int array may be less than `4 * numElements` and the memory copy may crash.
Ideally I think we need to improve unsafe array format to handle primitive array better.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14850][ML] convert primitive array from...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12640#issuecomment-215374433
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/57241/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14850][ML] convert primitive array from...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the pull request:
https://github.com/apache/spark/pull/12640#issuecomment-215932443
retest this please.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14850][ML] specialize array data for Ve...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12640#issuecomment-213919846
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14850][ML] convert primitive array from...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/12640
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14850][ML] specialize array data for Ve...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12640#issuecomment-213757581
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14850][ML] convert primitive array from...
Posted by tedyu <gi...@git.apache.org>.
Github user tedyu commented on a diff in the pull request:
https://github.com/apache/spark/pull/12640#discussion_r61669718
--- Diff: sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/UnsafeArrayData.java ---
@@ -336,4 +336,64 @@ public UnsafeArrayData copy() {
arrayCopy.pointTo(arrayDataCopy, Platform.BYTE_ARRAY_OFFSET, sizeInBytes);
return arrayCopy;
}
+
+ public static UnsafeArrayData fromPrimitiveArray(int[] arr) {
+ if (arr.length > (Integer.MAX_VALUE - 4) / 8) {
+ throw new UnsupportedOperationException("Cannot convert this array to unsafe format as " +
--- End diff --
Include (Integer.MAX_VALUE - 4) / 8 in the message so that the user knows the limit ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14850][ML] convert primitive array from...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/12640#discussion_r61389700
--- Diff: sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/UnsafeArrayData.java ---
@@ -336,4 +336,62 @@ public UnsafeArrayData copy() {
arrayCopy.pointTo(arrayDataCopy, Platform.BYTE_ARRAY_OFFSET, sizeInBytes);
return arrayCopy;
}
+
+ public int[] toPrimitiveIntArray() {
+ int[] result = new int[numElements];
+ Platform.copyMemory(baseObject, baseOffset + 4 + 4 * numElements,
+ result, Platform.INT_ARRAY_OFFSET, 4 * numElements);
+ return result;
+ }
+
+ public double[] toPrimitiveDoubleArray() {
+ double[] result = new double[numElements];
+ Platform.copyMemory(baseObject, baseOffset + 4 + 4 * numElements,
+ result, Platform.DOUBLE_ARRAY_OFFSET, 8 * numElements);
+ return result;
+ }
+
+ public static UnsafeArrayData fromPrimitiveArray(int[] arr) {
+ int offsetRegionSize = 4 * arr.length;
+ int valueRegionSize = 4 * arr.length;
+ int totalSize = 4 + offsetRegionSize + valueRegionSize;
+ byte[] data = new byte[totalSize];
+
+ Platform.putInt(data, Platform.BYTE_ARRAY_OFFSET, arr.length);
+
+ int elementOffsetStart = 4 + offsetRegionSize;
+ for (int i = 0; i < arr.length; i++) {
+ Platform.putInt(data, Platform.BYTE_ARRAY_OFFSET + 4 + i * 4, elementOffsetStart + i * 4);
--- End diff --
I just found this problem while benchmarking, working on it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14850][ML] convert primitive array from...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12640#issuecomment-215932260
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/57399/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org