You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by kiszk <gi...@git.apache.org> on 2016/05/04 09:33:19 UTC
[GitHub] spark pull request: [SPARK-15117][SQL][WIP] Generate Java code tha...
GitHub user kiszk opened a pull request:
https://github.com/apache/spark/pull/12894
[SPARK-15117][SQL][WIP] Generate Java code that gets a value in each compressed column of CachedBatch when DataFrame.cache() is called
## What changes were proposed in this pull request?
Waiting #11956 to be merged.
This PR generates Java code to get a value of each compressed column from CachedBatch when DataFrame.cache() is called. This is done in whole stage code generation.
When DataFrame.cache() is called, data is stored as column-oriented storage (columnar cache) in CachedBatch. This PR avoid conversion from column-oriented storage to row-oriented storage. This PR handles several primitive types (boolean/byte/short/int/long) that may be stored into a column witha compressed data format.
This PR consists of two parts.
1. Pass data in CachedBatch to generated code by using ```decompress()``` method. CachedBatch consists of multiple ByteBuffer arrays. A ByteBuffer may have compressed data. If the array is compressed, decompress it and pass it to generated code. If the array is not compressed, it is just passed to generated code.
2. Generate code both for row-oriented storage and column-oriented storage only if
- InMemoryColumnarTableScan exists in a plan sub-tree. A decision is performed by checking an given iterator is ColumnaIterator at runtime
- Sort or join does not exist in a plan sub-tree.
This PR generates Java code for columnar cache only if types in all columns, which are accessed in operations, are primitive
This PR improves performance of aggregate sum by 2.0x - 5.2x. This benchmark is available at [here](https://github.com/kiszk/spark/blob/SPARK-14098/sql/core/src/test/scala/org/apache/spark/sql/DataFrameCacheBenchmark.scala)
Performance results:
````
Java HotSpot(TM) 64-Bit Server VM 1.8.0_66-b17 on Linux 2.6.32-504.el6.x86_64
Intel(R) Xeon(R) CPU E5-2667 v2 @ 3.30GHz
Running benchmark: Int Sum with PassThrough cache
Running case: InternalRow codegen
Running case: ColumnVector codegen
Int Sum with PassThrough cache: Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
-------------------------------------------------------------------------------------------
InternalRow codegen 450 / 459 69.9 14.3 1.0X
ColumnVector codegen 86 / 92 363.9 2.7 5.2X
Running benchmark: Int Sum with RunLength cache
Running case: InternalRow codegen
Running case: ColumnVector codegen
Int Sum with RunLength cache: Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
-------------------------------------------------------------------------------------------
InternalRow codegen 475 / 499 66.3 15.1 1.0X
ColumnVector codegen 169 / 177 185.8 5.4 2.8X
Running benchmark: Int Sum with Dictionary cache
Running case: InternalRow codegen
Running case: ColumnVector codegen
Int Sum with Dictionary cache: Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
-------------------------------------------------------------------------------------------
InternalRow codegen 595 / 643 52.9 18.9 1.0X
ColumnVector codegen 297 / 306 106.1 9.4 2.0X
Running benchmark: Int Sum with Delta cache
Running case: InternalRow codegen
Running case: ColumnVector codegen
Int Sum with Delta cache: Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
-------------------------------------------------------------------------------------------
InternalRow codegen 450 / 455 69.9 14.3 1.0X
ColumnVector codegen 211 / 216 149.1 6.7 2.1X
Running benchmark: Long Sum with PassThrough cache
Running case: InternalRow codegen
Running case: ColumnVector codegen
Long Sum with PassThrough cache: Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
-------------------------------------------------------------------------------------------
InternalRow codegen 353 / 378 44.6 22.4 1.0X
ColumnVector codegen 94 / 104 167.8 6.0 3.8X
````
Motivating example:
````
val df = sc.parallelize(0 to 9, 1).toDF()
df.cache().filter("value <= 5").show()
````
Generated code
````java
/* 001 */ public Object generate(Object[] references) {
/* 002 */ return new GeneratedIterator(references);
/* 003 */ }
/* 004 */
/* 005 */ /** Codegened pipeline for:
/* 006 */ * Filter (value#1 <= 5)
/* 007 */ +- INPUT
/* 008 */ */
/* 009 */ final class GeneratedIterator extends org.apache.spark.sql.execution.BufferedRowIterator {
/* 010 */ private Object[] references;
/* 011 */ private scala.collection.Iterator inputadapter_input;
/* 012 */ private org.apache.spark.sql.execution.metric.LongSQLMetric filter_numOutputRows;
/* 013 */ private org.apache.spark.sql.execution.metric.LongSQLMetricValue filter_metricValue;
/* 014 */ private UnsafeRow filter_result;
/* 015 */ private org.apache.spark.sql.catalyst.expressions.codegen.BufferHolder filter_holder;
/* 016 */ private org.apache.spark.sql.catalyst.expressions.codegen.UnsafeRowWriter filter_rowWriter;
/* 017 */ private scala.collection.Iterator inputadapter_input1;
/* 018 */ private int columnar_batchIdx;
/* 019 */ private int columnar_numRows;
/* 020 */ private org.apache.spark.sql.execution.vectorized.ColumnVector inputadapter_col0;
/* 021 */ private UnsafeRow inputadapter_result;
/* 022 */ private org.apache.spark.sql.catalyst.expressions.codegen.BufferHolder inputadapter_holder;
/* 023 */ private org.apache.spark.sql.catalyst.expressions.codegen.UnsafeRowWriter inputadapter_rowWriter;
/* 024 */ private org.apache.spark.sql.execution.metric.LongSQLMetric filter_numOutputRows1;
/* 025 */ private org.apache.spark.sql.execution.metric.LongSQLMetricValue filter_metricValue1;
/* 026 */ private UnsafeRow filter_result1;
/* 027 */ private org.apache.spark.sql.catalyst.expressions.codegen.BufferHolder filter_holder1;
/* 028 */ private org.apache.spark.sql.catalyst.expressions.codegen.UnsafeRowWriter filter_rowWriter1;
/* 029 */ private org.apache.spark.sql.execution.columnar.ColumnarIterator columnar_itr;
/* 030 */
/* 031 */ public GeneratedIterator(Object[] references) {
/* 032 */ this.references = references;
/* 033 */ }
/* 034 */
/* 035 */ public void init(int index, scala.collection.Iterator inputs[]) {
/* 036 */ partitionIndex = index;
/* 037 */ inputadapter_input = inputs[0];
/* 038 */ this.filter_numOutputRows = (org.apache.spark.sql.execution.metric.LongSQLMetric) references[0];
/* 039 */ filter_metricValue = (org.apache.spark.sql.execution.metric.LongSQLMetricValue) filter_numOutputRows.localValue();
/* 040 */ filter_result = new UnsafeRow(1);
/* 041 */ this.filter_holder = new org.apache.spark.sql.catalyst.expressions.codegen.BufferHolder(filter_result, 0);
/* 042 */ this.filter_rowWriter = new org.apache.spark.sql.catalyst.expressions.codegen.UnsafeRowWriter(filter_holder, 1);
/* 043 */ inputadapter_input1 = inputs[0];
/* 044 */ columnar_batchIdx = 0;
/* 045 */ columnar_numRows = 0;
/* 046 */ inputadapter_col0 = null;
/* 047 */ inputadapter_result = new UnsafeRow(1);
/* 048 */ this.inputadapter_holder = new org.apache.spark.sql.catalyst.expressions.codegen.BufferHolder(inputadapter_result, 0);
/* 049 */ this.inputadapter_rowWriter = new org.apache.spark.sql.catalyst.expressions.codegen.UnsafeRowWriter(inputadapter_holder, 1);
/* 050 */ this.filter_numOutputRows1 = (org.apache.spark.sql.execution.metric.LongSQLMetric) references[1];
/* 051 */ filter_metricValue1 = (org.apache.spark.sql.execution.metric.LongSQLMetricValue) filter_numOutputRows1.localValue();
/* 052 */ filter_result1 = new UnsafeRow(1);
/* 053 */ this.filter_holder1 = new org.apache.spark.sql.catalyst.expressions.codegen.BufferHolder(filter_result1, 0);
/* 054 */ this.filter_rowWriter1 = new org.apache.spark.sql.catalyst.expressions.codegen.UnsafeRowWriter(filter_holder1, 1);
/* 055 */ columnar_itr = null;
/* 056 */ }
/* 057 */
/* 058 */ private void processBatch() throws java.io.IOException {
/* 059 */ /*** PRODUCE: Filter (value#1 <= 5) */
/* 060 */
/* 061 */ /*** PRODUCE: INPUT */
/* 062 */
/* 063 */ while (true) {
/* 064 */ if (columnar_batchIdx == 0) {
/* 065 */ columnar_numRows = columnar_itr.initForColumnar();
/* 066 */ if (columnar_numRows < 0) {
/* 067 */ cleanup();
/* 068 */ break;
/* 069 */ }
/* 070 */ inputadapter_col0 = columnar_itr.getColumn(0);
/* 071 */ }
/* 072 */
/* 073 */ while (columnar_batchIdx < columnar_numRows) {
/* 074 */ int inputadapter_rowIdx = columnar_batchIdx++;
/* 075 */ /*** CONSUME: Filter (value#1 <= 5) */
/* 076 */
/* 077 */ /* columnVector[inputadapter_col0, inputadapter_rowIdx, int] */
/* 078 */ int inputadapter_value1 = inputadapter_col0.getInt(inputadapter_rowIdx);
/* 079 */
/* 080 */ /* (input[0, int] <= 5) */
/* 081 */ boolean filter_value4 = false;
/* 082 */ filter_value4 = inputadapter_value1 <= 5;
/* 083 */ if (!filter_value4) continue;
/* 084 */
/* 085 */ filter_metricValue1.add(1);
/* 086 */
/* 087 */ /*** CONSUME: WholeStageCodegen */
/* 088 */
/* 089 */ filter_rowWriter1.write(0, inputadapter_value1);
/* 090 */ append(filter_result1);
/* 091 */ if (shouldStop()) return;
/* 092 */ }
/* 093 */ columnar_batchIdx = 0;
/* 094 */ }
/* 095 */ }
/* 096 */
/* 097 */ private void processRow() throws java.io.IOException {
/* 098 */ /*** PRODUCE: Filter (value#1 <= 5) */
/* 099 */
/* 100 */ /*** PRODUCE: INPUT */
/* 101 */
/* 102 */ while (inputadapter_input.hasNext()) {
/* 103 */ InternalRow inputadapter_row = (InternalRow) inputadapter_input.next();
/* 104 */ /*** CONSUME: Filter (value#1 <= 5) */
/* 105 */
/* 106 */ /* input[0, int] */
/* 107 */ int inputadapter_value = inputadapter_row.getInt(0);
/* 108 */
/* 109 */ /* (input[0, int] <= 5) */
/* 110 */ boolean filter_value = false;
/* 111 */ filter_value = inputadapter_value <= 5;
/* 112 */ if (!filter_value) continue;
/* 112 */ if (!filter_value) continue;
/* 113 */
/* 114 */ filter_metricValue.add(1);
/* 115 */
/* 116 */ /*** CONSUME: WholeStageCodegen */
/* 117 */
/* 118 */ filter_rowWriter.write(0, inputadapter_value);
/* 119 */ append(filter_result);
/* 120 */ if (shouldStop()) return;
/* 121 */ }
/* 122 */ }
/* 123 */
/* 124 */ private void cleanup() {
/* 125 */ inputadapter_col0 = null;
/* 126 */
/* 127 */ columnar_itr = null;
/* 128 */ }
/* 129 */
/* 130 */ protected void processNext() throws java.io.IOException {
/* 131 */ if ((columnar_batchIdx != 0) ||
/* 132 */ (inputadapter_input1 instanceof org.apache.spark.sql.execution.columnar.ColumnarIterator &&
/* 133 */ (columnar_itr = (org.apache.spark.sql.execution.columnar.ColumnarIterator)inputadapter_input1).isSupportColumnarCodeGen())) {
/* 134 */ processBatch();
/* 135 */ } else {
/* 136 */ processRow();
/* 137 */ }
/* 138 */ }
/* 139 */ }
````
## How was this patch tested?
Tested existing test suites
added test suites for operations to dataframe generated by df.cache().
(If this patch involves UI changes, please attach a screenshot; otherwise, remove this)
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/kiszk/spark SPARK-15117
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/12894.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #12894
----
commit 458caedb41919003d956e99a962a5e453bb2ee8b
Author: Kazuaki Ishizaki <is...@jp.ibm.com>
Date: 2016-03-25T08:17:38Z
add a property 'spark.sql.inMemoryColumnarStorage.codegen' to controll code generation
commit 16d3daacfd9ee7cf4461bed08923381a07ff9142
Author: Kazuaki Ishizaki <is...@jp.ibm.com>
Date: 2016-03-25T08:22:45Z
add CachedBatch.column() to prepare a columnar storage that is accessed by generated code
commit 431ca45ff0810ddb61e19c154047e0e634af7bde
Author: Kazuaki Ishizaki <is...@jp.ibm.com>
Date: 2016-03-25T08:26:20Z
add utility methods for putting a primitive value to ByteBuffer
commit 6aaa30b8962b1ce8a64577c35782ba878a121827
Author: Kazuaki Ishizaki <is...@jp.ibm.com>
Date: 2016-03-25T08:32:32Z
add ByteBufferColumnVector, which wraps ByteByffer for columnar storage, as an implementation of ColumnVector
commit e70a23555f21bb036c422d77ca12dca133d3534e
Author: Kazuaki Ishizaki <is...@jp.ibm.com>
Date: 2016-03-25T08:34:59Z
add APIs to get a CachedBatch
commit 7701225c71e05718d7eb631261828618d03d005c
Author: Kazuaki Ishizaki <is...@jp.ibm.com>
Date: 2016-03-25T08:36:10Z
add decompress() method
commit 89d20300178cabb65342938fcb0c8a4c9274807d
Author: Kazuaki Ishizaki <is...@jp.ibm.com>
Date: 2016-03-25T08:36:49Z
add decompress() method
commit 4cdea9988dfaac15410ece4e8f3079f670e0197c
Author: Kazuaki Ishizaki <is...@jp.ibm.com>
Date: 2016-03-25T08:37:20Z
add ColumnVectorReference class
commit 7d23dc547e6da3695811140231bec835cf3190e5
Author: Kazuaki Ishizaki <is...@jp.ibm.com>
Date: 2016-03-25T08:38:52Z
Do not call createHashMap() until actually executed at first
commit 2d8937f649fc0f99df4b2aa659444b2418fa45f4
Author: Kazuaki Ishizaki <is...@jp.ibm.com>
Date: 2016-03-25T08:40:45Z
generate two version of Java codes for row-oriented and column-oriented storages if InMemoryColumnar exists in a tree plan
commit 7e66c56d59ea4c2b066c99a74725273c657fd7ae
Author: Kazuaki Ishizaki <is...@jp.ibm.com>
Date: 2016-03-25T08:58:29Z
apply SPARK-14092 to generated code for columnar storage
commit 7049b73bb3723ab0aa52160081bf93327d8007e9
Author: Kazuaki Ishizaki <is...@jp.ibm.com>
Date: 2016-03-25T10:04:13Z
fix compilation error at runtime
commit e8c41ff47e27a6daecf4c32d1c7177d2e3c5ce27
Author: Kazuaki Ishizaki <is...@jp.ibm.com>
Date: 2016-03-25T12:11:15Z
drop debug print
commit 0ada4cbca0b6b88f49235f9b4a63d117b1ba9407
Author: Kazuaki Ishizaki <is...@jp.ibm.com>
Date: 2016-03-31T03:10:11Z
merge with the latest
commit 2e36abdd917d1c2d9ff6614b9eb802fcdb60b723
Author: Kazuaki Ishizaki <is...@jp.ibm.com>
Date: 2016-04-10T19:21:53Z
rebase
commit 8c0c55c68f6acbd041480b16168829ea386f6708
Author: Kazuaki Ishizaki <is...@jp.ibm.com>
Date: 2016-04-12T19:57:47Z
fix scalastyle error
commit 90a767bb08fd57b9058030c025624efbbebdd8c2
Author: Kazuaki Ishizaki <is...@jp.ibm.com>
Date: 2016-04-12T20:17:22Z
fix scalastyle error
commit 5b4d39b37fd560f5b4333cae0051a25045ff3440
Author: Kazuaki Ishizaki <is...@jp.ibm.com>
Date: 2016-04-13T16:20:57Z
make source code shorter
commit ceb342d59cbe445ebb6f080a982a69b53c602f38
Author: Kazuaki Ishizaki <is...@jp.ibm.com>
Date: 2016-04-13T16:22:57Z
avoid memory leak due to twice allocations of hashMap
commit 49337c51a255e86ffbd66cd235a81960f88cd227
Author: Kazuaki Ishizaki <is...@jp.ibm.com>
Date: 2016-04-13T18:28:05Z
fix assertion error
commit b69ec83f49657102cb2c1f2cd1cd8e156346baab
Author: Kazuaki Ishizaki <is...@jp.ibm.com>
Date: 2016-04-14T19:12:01Z
make decompress() simple
commit 387797fac9a19b3d7f2121d42b8e2b275ad0d3a7
Author: Kazuaki Ishizaki <is...@jp.ibm.com>
Date: 2016-04-16T19:14:12Z
support nulls in decompress() for PassThru
commit 675466f75441b417b1edb529b1d0113e928e3972
Author: Kazuaki Ishizaki <is...@jp.ibm.com>
Date: 2016-04-16T19:14:42Z
make it simple
commit effd59a7984934ea6aa7773aa6711e210efb10f0
Author: Kazuaki Ishizaki <is...@jp.ibm.com>
Date: 2016-04-16T19:19:36Z
Walk over an iterator when a loop is finished since an aggregation is once called this stuff
This fixes failure of auto_join24
commit 237b003c1d7f108c80fc5f6892cd161cbaf3fc50
Author: Kazuaki Ishizaki <is...@jp.ibm.com>
Date: 2016-04-17T13:59:15Z
remove unused code
commit 460f6ac967e6cd983651df6b527279caa17cc92c
Author: Kazuaki Ishizaki <is...@jp.ibm.com>
Date: 2016-04-17T14:00:45Z
Duplicate ByteBuffer in a column for CachedBatch
commit 28aeeadafb2848447cb5df0a8dc32d5331f29888
Author: Kazuaki Ishizaki <is...@jp.ibm.com>
Date: 2016-04-17T14:08:24Z
Move string constants to WholeStageCodegen
commit b9fd4773965f3b5b616f2042fad41eaec96855c9
Author: Kazuaki Ishizaki <is...@jp.ibm.com>
Date: 2016-04-17T14:10:11Z
avoid generating code for CachedBatch since these routine generates code to directly access InternalRow
commit a193b54b5492982fd99a9e6e2d1eec08dc5f896a
Author: Kazuaki Ishizaki <is...@jp.ibm.com>
Date: 2016-04-17T14:10:50Z
Make code simple
commit 3d0d30e9dc9e210c116072489358ada43db44461
Author: Kazuaki Ishizaki <is...@jp.ibm.com>
Date: 2016-04-17T14:15:19Z
Add benchmark suites with results
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #12894: [SPARK-15117][SQL][WIP] Generate Java code that gets a v...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/12894
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/60286/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #12894: [SPARK-15117][SQL][WIP] Generate Java code that gets a v...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/12894
**[Test build #60286 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/60286/consoleFull)** for PR 12894 at commit [`f847ef4`](https://github.com/apache/spark/commit/f847ef4f50cbae3c751837484932dd90c5a83313).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15117][SQL][WIP] Generate Java code tha...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12894#issuecomment-216814214
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #12894: [SPARK-15117][SQL][WIP] Generate Java code that gets a v...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/12894
@kiszk, I do understand it is painful to keep PR up-to-date but shouldn't we probably have the Jenkins pass at the last even if it has conflicts?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #12894: [SPARK-15117][SQL][WIP] Generate Java code that gets a v...
Posted by kiszk <gi...@git.apache.org>.
Github user kiszk commented on the issue:
https://github.com/apache/spark/pull/12894
@HyukjinKwon Thank you for pointing out this.
This PR will be replaced with https://issues.apache.org/jira/browse/SPARK-20823
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15117][SQL][WIP] Generate Java code tha...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12894#issuecomment-216813304
**[Test build #57746 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57746/consoleFull)** for PR 12894 at commit [`a79ace5`](https://github.com/apache/spark/commit/a79ace55749da364f9fb9199f165f946b15ea8c2).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #12894: [SPARK-15117][SQL][WIP] Generate Java code that g...
Posted by kiszk <gi...@git.apache.org>.
Github user kiszk closed the pull request at:
https://github.com/apache/spark/pull/12894
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15117][SQL][WIP] Generate Java code tha...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12894#issuecomment-216811827
**[Test build #57745 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57745/consoleFull)** for PR 12894 at commit [`1660a64`](https://github.com/apache/spark/commit/1660a6419e003f49795edea919359fe8fa656284).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #12894: [SPARK-15117][SQL][WIP] Generate Java code that gets a v...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/12894
**[Test build #60286 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/60286/consoleFull)** for PR 12894 at commit [`f847ef4`](https://github.com/apache/spark/commit/f847ef4f50cbae3c751837484932dd90c5a83313).
* This patch **fails to build**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15117][SQL][WIP] Generate Java code tha...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12894#issuecomment-216814216
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/57746/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15117][SQL][WIP] Generate Java code tha...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12894#issuecomment-216828758
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/57745/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15117][SQL][WIP] Generate Java code tha...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12894#issuecomment-216814206
**[Test build #57746 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57746/consoleFull)** for PR 12894 at commit [`a79ace5`](https://github.com/apache/spark/commit/a79ace55749da364f9fb9199f165f946b15ea8c2).
* This patch **fails to build**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `public final class ByteBufferColumnVector extends ColumnVector `
* `case class CachedBatch(numRows: Int, buffers: Array[Array[Byte]], stats: InternalRow) `
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15117][SQL][WIP] Generate Java code tha...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12894#issuecomment-216828621
**[Test build #57745 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57745/consoleFull)** for PR 12894 at commit [`1660a64`](https://github.com/apache/spark/commit/1660a6419e003f49795edea919359fe8fa656284).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15117][SQL][WIP] Generate Java code tha...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12894#issuecomment-216828755
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #12894: [SPARK-15117][SQL][WIP] Generate Java code that gets a v...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/12894
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org