You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2019/12/26 08:26:24 UTC
[GitHub] [spark] yaooqinn opened a new pull request #27013:
[SPARK-30356][SQL] Codegen support for the function str_to_map
yaooqinn opened a new pull request #27013: [SPARK-30356][SQL] Codegen support for the function str_to_map
URL: https://github.com/apache/spark/pull/27013
### What changes were proposed in this pull request?
rm `CodegenFallBack` from `StringToMap`
### Why are the changes needed?
improve codegen coverage and gain better perfomance
### Does this PR introduce any user-facing change?
<!--
If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible.
If no, write 'No'.
-->
no
### How was this patch tested?
<!--
If tests were added, say they were added here. Please make sure to add some test cases that check the changes thoroughly including negative and positive cases if possible.
If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future.
If tests were not added, please describe why they were not added and/or why it was difficult to add.
-->
1. pass ComplexTypeSuite
2. manually review generated code
```java
-- !query 12
explain codegen select v, str_to_map(v) from values ('abc:a:a,:'), (null), (''), ('1:2') t(v)
-- !query 12 schema
struct<plan:string>
-- !query 12 output
Found 1 WholeStageCodegen subtrees.
== Subtree 1 / 1 (maxMethodCodeSize:511; maxConstantPoolSize:188(0.29% used); numInnerClasses:0) ==
*Project [v#x, str_to_map(v#x, ,, :) AS str_to_map(v, ,, :)#x]
+- *LocalTableScan [v#x]
Generated code:
/* 001 */ public Object generate(Object[] references) {
/* 002 */ return new GeneratedIteratorForCodegenStage1(references);
/* 003 */ }
/* 004 */
/* 005 */ // codegenStageId=1
/* 006 */ final class GeneratedIteratorForCodegenStage1 extends org.apache.spark.sql.execution.BufferedRowIterator {
/* 007 */ private Object[] references;
/* 008 */ private scala.collection.Iterator[] inputs;
/* 009 */ private scala.collection.Iterator localtablescan_input_0;
/* 010 */ private org.apache.spark.sql.catalyst.expressions.codegen.UnsafeRowWriter[] project_mutableStateArray_0 = new org.apache.spark.sql.catalyst.expressions.codegen.UnsafeRowWriter[1];
/* 011 */ private org.apache.spark.sql.catalyst.expressions.codegen.UnsafeArrayWriter[] project_mutableStateArray_1 = new org.apache.spark.sql.catalyst.expressions.codegen.UnsafeArrayWriter[2];
/* 012 */
/* 013 */ public GeneratedIteratorForCodegenStage1(Object[] references) {
/* 014 */ this.references = references;
/* 015 */ }
/* 016 */
/* 017 */ public void init(int index, scala.collection.Iterator[] inputs) {
/* 018 */ partitionIndex = index;
/* 019 */ this.inputs = inputs;
/* 020 */ localtablescan_input_0 = inputs[0];
/* 021 */ project_mutableStateArray_0[0] = new org.apache.spark.sql.catalyst.expressions.codegen.UnsafeRowWriter(2, 64);
/* 022 */ project_mutableStateArray_1[0] = new org.apache.spark.sql.catalyst.expressions.codegen.UnsafeArrayWriter(project_mutableStateArray_0[0], 8);
/* 023 */ project_mutableStateArray_1[1] = new org.apache.spark.sql.catalyst.expressions.codegen.UnsafeArrayWriter(project_mutableStateArray_0[0], 8);
/* 024 */
/* 025 */ }
/* 026 */
/* 027 */ private void project_doConsume_0(InternalRow localtablescan_row_0, UTF8String project_expr_0_0, boolean project_exprIsNull_0_0) throws java.io.IOException {
/* 028 */ boolean project_isNull_1 = true;
/* 029 */ MapData project_value_1 = null;
/* 030 */
/* 031 */ if (!project_exprIsNull_0_0) {
/* 032 */ project_isNull_1 = false; // resultCode could change nullability.
/* 033 */
/* 034 */ int project_i_0 = 0;
/* 035 */ UTF8String[] project_kvs_0 = project_expr_0_0.split(((UTF8String) references[2] /* literal */), -1);
/* 036 */ while (project_i_0 < project_kvs_0.length) {
/* 037 */ UTF8String[] kv = project_kvs_0[project_i_0].split(((UTF8String) references[3] /* literal */), 2);
/* 038 */ UTF8String key = kv[0];
/* 039 */ UTF8String value = null;
/* 040 */ if (kv.length == 2) {
/* 041 */ value = kv[1];
/* 042 */ }
/* 043 */ ((org.apache.spark.sql.catalyst.util.ArrayBasedMapBuilder) references[1] /* mapBuilder */).put(key, value);
/* 044 */ project_i_0++;
/* 045 */ }
/* 046 */ project_value_1 = ((org.apache.spark.sql.catalyst.util.ArrayBasedMapBuilder) references[1] /* mapBuilder */).build();
/* 047 */
/* 048 */ }
/* 049 */ project_mutableStateArray_0[0].reset();
/* 050 */
/* 051 */ project_mutableStateArray_0[0].zeroOutNullBytes();
/* 052 */
/* 053 */ if (project_exprIsNull_0_0) {
/* 054 */ project_mutableStateArray_0[0].setNullAt(0);
/* 055 */ } else {
/* 056 */ project_mutableStateArray_0[0].write(0, project_expr_0_0);
/* 057 */ }
/* 058 */
/* 059 */ if (project_isNull_1) {
/* 060 */ project_mutableStateArray_0[0].setNullAt(1);
/* 061 */ } else {
/* 062 */ final MapData project_tmpInput_0 = project_value_1;
/* 063 */ if (project_tmpInput_0 instanceof UnsafeMapData) {
/* 064 */ project_mutableStateArray_0[0].write(1, (UnsafeMapData) project_tmpInput_0);
/* 065 */ } else {
/* 066 */ // Remember the current cursor so that we can calculate how many bytes are
/* 067 */ // written later.
/* 068 */ final int project_previousCursor_0 = project_mutableStateArray_0[0].cursor();
/* 069 */
/* 070 */ // preserve 8 bytes to write the key array numBytes later.
/* 071 */ project_mutableStateArray_0[0].grow(8);
/* 072 */ project_mutableStateArray_0[0].increaseCursor(8);
/* 073 */
/* 074 */ // Remember the current cursor so that we can write numBytes of key array later.
/* 075 */ final int project_tmpCursor_0 = project_mutableStateArray_0[0].cursor();
/* 076 */
/* 077 */ final ArrayData project_tmpInput_1 = project_tmpInput_0.keyArray();
/* 078 */ if (project_tmpInput_1 instanceof UnsafeArrayData) {
/* 079 */ project_mutableStateArray_0[0].write((UnsafeArrayData) project_tmpInput_1);
/* 080 */ } else {
/* 081 */ final int project_numElements_0 = project_tmpInput_1.numElements();
/* 082 */ project_mutableStateArray_1[0].initialize(project_numElements_0);
/* 083 */
/* 084 */ for (int project_index_0 = 0; project_index_0 < project_numElements_0; project_index_0++) {
/* 085 */ project_mutableStateArray_1[0].write(project_index_0, project_tmpInput_1.getUTF8String(project_index_0));
/* 086 */ }
/* 087 */ }
/* 088 */
/* 089 */ // Write the numBytes of key array into the first 8 bytes.
/* 090 */ Platform.putLong(
/* 091 */ project_mutableStateArray_0[0].getBuffer(),
/* 092 */ project_tmpCursor_0 - 8,
/* 093 */ project_mutableStateArray_0[0].cursor() - project_tmpCursor_0);
/* 094 */
/* 095 */ final ArrayData project_tmpInput_2 = project_tmpInput_0.valueArray();
/* 096 */ if (project_tmpInput_2 instanceof UnsafeArrayData) {
/* 097 */ project_mutableStateArray_0[0].write((UnsafeArrayData) project_tmpInput_2);
/* 098 */ } else {
/* 099 */ final int project_numElements_1 = project_tmpInput_2.numElements();
/* 100 */ project_mutableStateArray_1[1].initialize(project_numElements_1);
/* 101 */
/* 102 */ for (int project_index_1 = 0; project_index_1 < project_numElements_1; project_index_1++) {
/* 103 */ if (project_tmpInput_2.isNullAt(project_index_1)) {
/* 104 */ project_mutableStateArray_1[1].setNull8Bytes(project_index_1);
/* 105 */ } else {
/* 106 */ project_mutableStateArray_1[1].write(project_index_1, project_tmpInput_2.getUTF8String(project_index_1));
/* 107 */ }
/* 108 */
/* 109 */ }
/* 110 */ }
/* 111 */
/* 112 */ project_mutableStateArray_0[0].setOffsetAndSizeFromPreviousCursor(1, project_previousCursor_0);
/* 113 */ }
/* 114 */ }
/* 115 */ append((project_mutableStateArray_0[0].getRow()));
/* 116 */
/* 117 */ }
/* 118 */
/* 119 */ protected void processNext() throws java.io.IOException {
/* 120 */ while ( localtablescan_input_0.hasNext()) {
/* 121 */ InternalRow localtablescan_row_0 = (InternalRow) localtablescan_input_0.next();
/* 122 */ ((org.apache.spark.sql.execution.metric.SQLMetric) references[0] /* numOutputRows */).add(1);
/* 123 */ boolean localtablescan_isNull_0 = localtablescan_row_0.isNullAt(0);
/* 124 */ UTF8String localtablescan_value_0 = localtablescan_isNull_0 ?
/* 125 */ null : (localtablescan_row_0.getUTF8String(0));
/* 126 */
/* 127 */ project_doConsume_0(localtablescan_row_0, localtablescan_value_0, localtablescan_isNull_0);
/* 128 */ if (shouldStop()) return;
/* 129 */ }
/* 130 */ }
/* 131 */
/* 132 */ }
-- !query 13
select v, str_to_map(v) from values ('abc:a:a,:'), (null), (''), ('1:2') t(v)
-- !query 13 schema
struct<v:string,str_to_map(v, ,, :):map<string,string>>
-- !query 13 output
{"":null}
1:2 {"1":"2"}
NULL NULL
abc:a:a,: {"":"","abc":"a:a"}
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27013: [SPARK-30356][SQL]
Codegen support for the function str_to_map
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27013: [SPARK-30356][SQL] Codegen support for the function str_to_map
URL: https://github.com/apache/spark/pull/27013#issuecomment-569011363
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27013: [SPARK-30356][SQL]
Codegen support for the function str_to_map
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27013: [SPARK-30356][SQL] Codegen support for the function str_to_map
URL: https://github.com/apache/spark/pull/27013#issuecomment-569258699
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27013:
[SPARK-30356][SQL] Codegen support for the function str_to_map
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27013: [SPARK-30356][SQL] Codegen support for the function str_to_map
URL: https://github.com/apache/spark/pull/27013#issuecomment-569011369
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/20595/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27013: [SPARK-30356][SQL]
Codegen support for the function str_to_map
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27013: [SPARK-30356][SQL] Codegen support for the function str_to_map
URL: https://github.com/apache/spark/pull/27013#issuecomment-569202941
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27013:
[SPARK-30356][SQL] Codegen support for the function str_to_map
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27013: [SPARK-30356][SQL] Codegen support for the function str_to_map
URL: https://github.com/apache/spark/pull/27013#issuecomment-569200140
**[Test build #115843 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/115843/testReport)** for PR 27013 at commit [`61d9ed4`](https://github.com/apache/spark/commit/61d9ed47ce70c8b068a2cfec8c6cd6b1bd3bd449).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27013: [SPARK-30356][SQL]
Codegen support for the function str_to_map
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27013: [SPARK-30356][SQL] Codegen support for the function str_to_map
URL: https://github.com/apache/spark/pull/27013#issuecomment-569042035
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/20600/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27013:
[SPARK-30356][SQL] Codegen support for the function str_to_map
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27013: [SPARK-30356][SQL] Codegen support for the function str_to_map
URL: https://github.com/apache/spark/pull/27013#issuecomment-569258705
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/115849/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27013:
[SPARK-30356][SQL] Codegen support for the function str_to_map
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27013: [SPARK-30356][SQL] Codegen support for the function str_to_map
URL: https://github.com/apache/spark/pull/27013#issuecomment-569170388
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/20625/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27013: [SPARK-30356][SQL]
Codegen support for the function str_to_map
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27013: [SPARK-30356][SQL] Codegen support for the function str_to_map
URL: https://github.com/apache/spark/pull/27013#issuecomment-569043013
**[Test build #115808 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/115808/testReport)** for PR 27013 at commit [`485d31f`](https://github.com/apache/spark/commit/485d31f98d68b632c07e58193080a1ac07778d5a).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27013: [SPARK-30356][SQL]
Codegen support for the function str_to_map
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27013: [SPARK-30356][SQL] Codegen support for the function str_to_map
URL: https://github.com/apache/spark/pull/27013#issuecomment-569218847
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/20642/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27013: [SPARK-30356][SQL]
Codegen support for the function str_to_map
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27013: [SPARK-30356][SQL] Codegen support for the function str_to_map
URL: https://github.com/apache/spark/pull/27013#issuecomment-569200140
**[Test build #115843 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/115843/testReport)** for PR 27013 at commit [`61d9ed4`](https://github.com/apache/spark/commit/61d9ed47ce70c8b068a2cfec8c6cd6b1bd3bd449).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27013: [SPARK-30356][SQL]
Codegen support for the function str_to_map
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27013: [SPARK-30356][SQL] Codegen support for the function str_to_map
URL: https://github.com/apache/spark/pull/27013#issuecomment-569216492
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] yaooqinn commented on issue #27013: [SPARK-30356][SQL]
Codegen support for the function str_to_map
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on issue #27013: [SPARK-30356][SQL] Codegen support for the function str_to_map
URL: https://github.com/apache/spark/pull/27013#issuecomment-569217786
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27013: [SPARK-30356][SQL]
Codegen support for the function str_to_map
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27013: [SPARK-30356][SQL] Codegen support for the function str_to_map
URL: https://github.com/apache/spark/pull/27013#issuecomment-569011369
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/20595/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27013:
[SPARK-30356][SQL] Codegen support for the function str_to_map
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27013: [SPARK-30356][SQL] Codegen support for the function str_to_map
URL: https://github.com/apache/spark/pull/27013#issuecomment-569077320
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/115808/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27013: [SPARK-30356][SQL]
Codegen support for the function str_to_map
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27013: [SPARK-30356][SQL] Codegen support for the function str_to_map
URL: https://github.com/apache/spark/pull/27013#issuecomment-569258705
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/115849/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27013:
[SPARK-30356][SQL] Codegen support for the function str_to_map
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27013: [SPARK-30356][SQL] Codegen support for the function str_to_map
URL: https://github.com/apache/spark/pull/27013#issuecomment-569170150
**[Test build #115832 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/115832/testReport)** for PR 27013 at commit [`07f1038`](https://github.com/apache/spark/commit/07f1038ef8b17b85fd154e6da1c9156840638547).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] yaooqinn commented on issue #27013: [SPARK-30356][SQL]
Codegen support for the function str_to_map
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on issue #27013: [SPARK-30356][SQL] Codegen support for the function str_to_map
URL: https://github.com/apache/spark/pull/27013#issuecomment-569197891
```sql
spark-sql> explain codegen select v, str_to_map(v) from values ('abc:a:a,:'), (null), (''), ('1:2') t(v)
> ;
Found 0 WholeStageCodegen subtrees.
Time taken: 0.104 seconds, Fetched 1 row(s)
```
does this mean that whole stage codegen to be off?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27013: [SPARK-30356][SQL]
Codegen support for the function str_to_map
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27013: [SPARK-30356][SQL] Codegen support for the function str_to_map
URL: https://github.com/apache/spark/pull/27013#issuecomment-569011017
**[Test build #115802 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/115802/testReport)** for PR 27013 at commit [`081a28a`](https://github.com/apache/spark/commit/081a28a83b934dfb50e5576d75a772b27dd0fd4f).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27013: [SPARK-30356][SQL]
Codegen support for the function str_to_map
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27013: [SPARK-30356][SQL] Codegen support for the function str_to_map
URL: https://github.com/apache/spark/pull/27013#issuecomment-569077315
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27013: [SPARK-30356][SQL]
Codegen support for the function str_to_map
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27013: [SPARK-30356][SQL] Codegen support for the function str_to_map
URL: https://github.com/apache/spark/pull/27013#issuecomment-569200374
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/20636/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27013:
[SPARK-30356][SQL] Codegen support for the function str_to_map
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27013: [SPARK-30356][SQL] Codegen support for the function str_to_map
URL: https://github.com/apache/spark/pull/27013#issuecomment-569077315
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on issue #27013: [SPARK-30356][SQL]
Codegen support for the function str_to_map
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on issue #27013: [SPARK-30356][SQL] Codegen support for the function str_to_map
URL: https://github.com/apache/spark/pull/27013#issuecomment-569201345
Yea, I know we disable whole stage codegen when seeing `CodegenFallBack` now, I'm wondering if this is reasonable.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] yaooqinn commented on issue #27013: [SPARK-30356][SQL]
Codegen support for the function str_to_map
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on issue #27013: [SPARK-30356][SQL] Codegen support for the function str_to_map
URL: https://github.com/apache/spark/pull/27013#issuecomment-569172462
thanks for suggestion @HeartSaVioR
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27013: [SPARK-30356][SQL]
Codegen support for the function str_to_map
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27013: [SPARK-30356][SQL] Codegen support for the function str_to_map
URL: https://github.com/apache/spark/pull/27013#issuecomment-569202596
**[Test build #115832 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/115832/testReport)** for PR 27013 at commit [`07f1038`](https://github.com/apache/spark/commit/07f1038ef8b17b85fd154e6da1c9156840638547).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27013:
[SPARK-30356][SQL] Codegen support for the function str_to_map
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27013: [SPARK-30356][SQL] Codegen support for the function str_to_map
URL: https://github.com/apache/spark/pull/27013#issuecomment-569216492
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27013: [SPARK-30356][SQL]
Codegen support for the function str_to_map
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27013: [SPARK-30356][SQL] Codegen support for the function str_to_map
URL: https://github.com/apache/spark/pull/27013#issuecomment-569258371
**[Test build #115849 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/115849/testReport)** for PR 27013 at commit [`61d9ed4`](https://github.com/apache/spark/commit/61d9ed47ce70c8b068a2cfec8c6cd6b1bd3bd449).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan closed pull request #27013: [SPARK-30356][SQL]
Codegen support for the function str_to_map
Posted by GitBox <gi...@apache.org>.
cloud-fan closed pull request #27013: [SPARK-30356][SQL] Codegen support for the function str_to_map
URL: https://github.com/apache/spark/pull/27013
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27013:
[SPARK-30356][SQL] Codegen support for the function str_to_map
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27013: [SPARK-30356][SQL] Codegen support for the function str_to_map
URL: https://github.com/apache/spark/pull/27013#issuecomment-569258699
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #27013:
[SPARK-30356][SQL] Codegen support for the function str_to_map
Posted by GitBox <gi...@apache.org>.
viirya commented on a change in pull request #27013: [SPARK-30356][SQL] Codegen support for the function str_to_map
URL: https://github.com/apache/spark/pull/27013#discussion_r361523532
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/complexTypeCreator.scala
##########
@@ -451,5 +452,25 @@ case class StringToMap(text: Expression, pairDelim: Expression, keyValueDelim: E
mapBuilder.build()
}
+ override protected def doGenCode(ctx: CodegenContext, ev: ExprCode): ExprCode = {
+ val builderTerm = ctx.addReferenceObj("mapBuilder", mapBuilder)
+ val keyValues = ctx.freshName("kvs")
+ val idx = ctx.freshName("i")
+ val kv = ctx.freshName("kv")
+
+ nullSafeCodeGen(ctx, ev, (text, pd, kvd) =>
+ s"""
+ |int $idx = 0;
+ |UTF8String[] $keyValues = $text.split($pd, -1);
+ |while ($idx < $keyValues.length) {
+ | UTF8String[] $kv = $keyValues[$idx].split($kvd, 2);
Review comment:
nit: we don't need create fresh name for kv.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27013:
[SPARK-30356][SQL] Codegen support for the function str_to_map
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27013: [SPARK-30356][SQL] Codegen support for the function str_to_map
URL: https://github.com/apache/spark/pull/27013#issuecomment-569200367
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27013:
[SPARK-30356][SQL] Codegen support for the function str_to_map
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27013: [SPARK-30356][SQL] Codegen support for the function str_to_map
URL: https://github.com/apache/spark/pull/27013#issuecomment-569218840
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27013: [SPARK-30356][SQL]
Codegen support for the function str_to_map
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27013: [SPARK-30356][SQL] Codegen support for the function str_to_map
URL: https://github.com/apache/spark/pull/27013#issuecomment-569077320
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/115808/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #27013:
[SPARK-30356][SQL] Codegen support for the function str_to_map
Posted by GitBox <gi...@apache.org>.
maropu commented on a change in pull request #27013: [SPARK-30356][SQL] Codegen support for the function str_to_map
URL: https://github.com/apache/spark/pull/27013#discussion_r361544610
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/complexTypeCreator.scala
##########
@@ -403,10 +403,11 @@ case class CreateNamedStruct(children: Seq[Expression]) extends Expression {
{"a":"1","b":"2","c":"3"}
> SELECT _FUNC_('a');
{"a":null}
- """)
+ """,
+ since = "2.0.1")
Review comment:
Ah, I see.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27013:
[SPARK-30356][SQL] Codegen support for the function str_to_map
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27013: [SPARK-30356][SQL] Codegen support for the function str_to_map
URL: https://github.com/apache/spark/pull/27013#issuecomment-569042030
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu commented on issue #27013: [SPARK-30356][SQL]
Codegen support for the function str_to_map
Posted by GitBox <gi...@apache.org>.
maropu commented on issue #27013: [SPARK-30356][SQL] Codegen support for the function str_to_map
URL: https://github.com/apache/spark/pull/27013#issuecomment-569041607
Looks ok to me if the tests passed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] yaooqinn commented on issue #27013: [SPARK-30356][SQL]
Codegen support for the function str_to_map
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on issue #27013: [SPARK-30356][SQL] Codegen support for the function str_to_map
URL: https://github.com/apache/spark/pull/27013#issuecomment-569012520
cc: @cloud-fan @maropu @HyukjinKwon, thanks for reviewing and happy holidays! 🎇
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27013: [SPARK-30356][SQL]
Codegen support for the function str_to_map
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27013: [SPARK-30356][SQL] Codegen support for the function str_to_map
URL: https://github.com/apache/spark/pull/27013#issuecomment-569218840
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #27013:
[SPARK-30356][SQL] Codegen support for the function str_to_map
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #27013: [SPARK-30356][SQL] Codegen support for the function str_to_map
URL: https://github.com/apache/spark/pull/27013#discussion_r361588857
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/complexTypeCreator.scala
##########
@@ -451,5 +452,24 @@ case class StringToMap(text: Expression, pairDelim: Expression, keyValueDelim: E
mapBuilder.build()
}
+ override protected def doGenCode(ctx: CodegenContext, ev: ExprCode): ExprCode = {
+ val builderTerm = ctx.addReferenceObj("mapBuilder", mapBuilder)
+ val keyValues = ctx.freshName("kvs")
+ val idx = ctx.freshName("i")
+
+ nullSafeCodeGen(ctx, ev, (text, pd, kvd) =>
+ s"""
+ |int $idx = 0;
+ |UTF8String[] $keyValues = $text.split($pd, -1);
+ |while ($idx < $keyValues.length) {
Review comment:
shall we use for loop? then we don't need to declare `int idx` outside of the loop.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27013: [SPARK-30356][SQL]
Codegen support for the function str_to_map
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27013: [SPARK-30356][SQL] Codegen support for the function str_to_map
URL: https://github.com/apache/spark/pull/27013#issuecomment-569216496
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/115843/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #27013:
[SPARK-30356][SQL] Codegen support for the function str_to_map
Posted by GitBox <gi...@apache.org>.
viirya commented on a change in pull request #27013: [SPARK-30356][SQL] Codegen support for the function str_to_map
URL: https://github.com/apache/spark/pull/27013#discussion_r361525299
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/complexTypeCreator.scala
##########
@@ -403,10 +403,11 @@ case class CreateNamedStruct(children: Seq[Expression]) extends Expression {
{"a":"1","b":"2","c":"3"}
> SELECT _FUNC_('a');
{"a":null}
- """)
+ """,
+ since = "2.0.1")
Review comment:
When I go to confirm this `since` tag, I looked at the original PR #13990.
The author implemented codegen version actually, but did remove it based on the comment: https://github.com/apache/spark/pull/13990#discussion_r69660241
I think the comment makes sense to me. To add or not codegen looks not a big deal. This implementation looks simple, anyway.
I have no special option to add it or not. cc @cloud-fan
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27013: [SPARK-30356][SQL]
Codegen support for the function str_to_map
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27013: [SPARK-30356][SQL] Codegen support for the function str_to_map
URL: https://github.com/apache/spark/pull/27013#issuecomment-569050939
**[Test build #115802 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/115802/testReport)** for PR 27013 at commit [`081a28a`](https://github.com/apache/spark/commit/081a28a83b934dfb50e5576d75a772b27dd0fd4f).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27013:
[SPARK-30356][SQL] Codegen support for the function str_to_map
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27013: [SPARK-30356][SQL] Codegen support for the function str_to_map
URL: https://github.com/apache/spark/pull/27013#issuecomment-569200374
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/20636/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27013:
[SPARK-30356][SQL] Codegen support for the function str_to_map
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27013: [SPARK-30356][SQL] Codegen support for the function str_to_map
URL: https://github.com/apache/spark/pull/27013#issuecomment-569051163
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on issue #27013: [SPARK-30356][SQL]
Codegen support for the function str_to_map
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on issue #27013: [SPARK-30356][SQL] Codegen support for the function str_to_map
URL: https://github.com/apache/spark/pull/27013#issuecomment-569171637
Could we elaborate more on what the patch is doing in the section of `What changes were proposed in this pull request?` in PR description? Looks like the patch is not just simply removing CodegenFallBack from StringToMap, but also adding codegen implementation.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27013:
[SPARK-30356][SQL] Codegen support for the function str_to_map
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27013: [SPARK-30356][SQL] Codegen support for the function str_to_map
URL: https://github.com/apache/spark/pull/27013#issuecomment-569202941
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27013:
[SPARK-30356][SQL] Codegen support for the function str_to_map
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27013: [SPARK-30356][SQL] Codegen support for the function str_to_map
URL: https://github.com/apache/spark/pull/27013#issuecomment-569216496
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/115843/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27013: [SPARK-30356][SQL]
Codegen support for the function str_to_map
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27013: [SPARK-30356][SQL] Codegen support for the function str_to_map
URL: https://github.com/apache/spark/pull/27013#issuecomment-569042030
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27013:
[SPARK-30356][SQL] Codegen support for the function str_to_map
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27013: [SPARK-30356][SQL] Codegen support for the function str_to_map
URL: https://github.com/apache/spark/pull/27013#issuecomment-569218847
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/20642/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27013:
[SPARK-30356][SQL] Codegen support for the function str_to_map
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27013: [SPARK-30356][SQL] Codegen support for the function str_to_map
URL: https://github.com/apache/spark/pull/27013#issuecomment-569218607
**[Test build #115849 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/115849/testReport)** for PR 27013 at commit [`61d9ed4`](https://github.com/apache/spark/commit/61d9ed47ce70c8b068a2cfec8c6cd6b1bd3bd449).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27013: [SPARK-30356][SQL]
Codegen support for the function str_to_map
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27013: [SPARK-30356][SQL] Codegen support for the function str_to_map
URL: https://github.com/apache/spark/pull/27013#issuecomment-569051163
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27013: [SPARK-30356][SQL]
Codegen support for the function str_to_map
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27013: [SPARK-30356][SQL] Codegen support for the function str_to_map
URL: https://github.com/apache/spark/pull/27013#issuecomment-569170388
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/20625/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27013:
[SPARK-30356][SQL] Codegen support for the function str_to_map
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27013: [SPARK-30356][SQL] Codegen support for the function str_to_map
URL: https://github.com/apache/spark/pull/27013#issuecomment-569051169
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/115802/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27013:
[SPARK-30356][SQL] Codegen support for the function str_to_map
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27013: [SPARK-30356][SQL] Codegen support for the function str_to_map
URL: https://github.com/apache/spark/pull/27013#issuecomment-569170384
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27013:
[SPARK-30356][SQL] Codegen support for the function str_to_map
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27013: [SPARK-30356][SQL] Codegen support for the function str_to_map
URL: https://github.com/apache/spark/pull/27013#issuecomment-569011017
**[Test build #115802 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/115802/testReport)** for PR 27013 at commit [`081a28a`](https://github.com/apache/spark/commit/081a28a83b934dfb50e5576d75a772b27dd0fd4f).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27013: [SPARK-30356][SQL]
Codegen support for the function str_to_map
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27013: [SPARK-30356][SQL] Codegen support for the function str_to_map
URL: https://github.com/apache/spark/pull/27013#issuecomment-569170150
**[Test build #115832 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/115832/testReport)** for PR 27013 at commit [`07f1038`](https://github.com/apache/spark/commit/07f1038ef8b17b85fd154e6da1c9156840638547).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27013:
[SPARK-30356][SQL] Codegen support for the function str_to_map
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27013: [SPARK-30356][SQL] Codegen support for the function str_to_map
URL: https://github.com/apache/spark/pull/27013#issuecomment-569011363
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27013: [SPARK-30356][SQL]
Codegen support for the function str_to_map
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27013: [SPARK-30356][SQL] Codegen support for the function str_to_map
URL: https://github.com/apache/spark/pull/27013#issuecomment-569170384
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #27013:
[SPARK-30356][SQL] Codegen support for the function str_to_map
Posted by GitBox <gi...@apache.org>.
maropu commented on a change in pull request #27013: [SPARK-30356][SQL] Codegen support for the function str_to_map
URL: https://github.com/apache/spark/pull/27013#discussion_r361544610
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/complexTypeCreator.scala
##########
@@ -403,10 +403,11 @@ case class CreateNamedStruct(children: Seq[Expression]) extends Expression {
{"a":"1","b":"2","c":"3"}
> SELECT _FUNC_('a');
{"a":null}
- """)
+ """,
+ since = "2.0.1")
Review comment:
Ah, I see. Thanks for the pointer. I have no strong opinion, too, so I'll leave it to the @cloud-fan descision.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27013: [SPARK-30356][SQL]
Codegen support for the function str_to_map
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27013: [SPARK-30356][SQL] Codegen support for the function str_to_map
URL: https://github.com/apache/spark/pull/27013#issuecomment-569218607
**[Test build #115849 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/115849/testReport)** for PR 27013 at commit [`61d9ed4`](https://github.com/apache/spark/commit/61d9ed47ce70c8b068a2cfec8c6cd6b1bd3bd449).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27013: [SPARK-30356][SQL]
Codegen support for the function str_to_map
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27013: [SPARK-30356][SQL] Codegen support for the function str_to_map
URL: https://github.com/apache/spark/pull/27013#issuecomment-569077087
**[Test build #115808 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/115808/testReport)** for PR 27013 at commit [`485d31f`](https://github.com/apache/spark/commit/485d31f98d68b632c07e58193080a1ac07778d5a).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27013: [SPARK-30356][SQL]
Codegen support for the function str_to_map
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27013: [SPARK-30356][SQL] Codegen support for the function str_to_map
URL: https://github.com/apache/spark/pull/27013#issuecomment-569200367
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27013: [SPARK-30356][SQL]
Codegen support for the function str_to_map
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27013: [SPARK-30356][SQL] Codegen support for the function str_to_map
URL: https://github.com/apache/spark/pull/27013#issuecomment-569051169
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/115802/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27013:
[SPARK-30356][SQL] Codegen support for the function str_to_map
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27013: [SPARK-30356][SQL] Codegen support for the function str_to_map
URL: https://github.com/apache/spark/pull/27013#issuecomment-569042035
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/20600/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27013:
[SPARK-30356][SQL] Codegen support for the function str_to_map
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27013: [SPARK-30356][SQL] Codegen support for the function str_to_map
URL: https://github.com/apache/spark/pull/27013#issuecomment-569202946
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/115832/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] yaooqinn commented on a change in pull request #27013:
[SPARK-30356][SQL] Codegen support for the function str_to_map
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on a change in pull request #27013: [SPARK-30356][SQL] Codegen support for the function str_to_map
URL: https://github.com/apache/spark/pull/27013#discussion_r361595611
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/complexTypeCreator.scala
##########
@@ -451,5 +452,24 @@ case class StringToMap(text: Expression, pairDelim: Expression, keyValueDelim: E
mapBuilder.build()
}
+ override protected def doGenCode(ctx: CodegenContext, ev: ExprCode): ExprCode = {
+ val builderTerm = ctx.addReferenceObj("mapBuilder", mapBuilder)
+ val keyValues = ctx.freshName("kvs")
+ val idx = ctx.freshName("i")
+
+ nullSafeCodeGen(ctx, ev, (text, pd, kvd) =>
+ s"""
+ |int $idx = 0;
+ |UTF8String[] $keyValues = $text.split($pd, -1);
+ |while ($idx < $keyValues.length) {
Review comment:
make `idx` as `freshName` because https://github.com/apache/spark/blob/c4115793555d35a10c79038ce51dc5f803f3b838/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/GenerateMutableProjection.scala#L132 has defined `i` in scope, seems only happen in test cases?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27013:
[SPARK-30356][SQL] Codegen support for the function str_to_map
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27013: [SPARK-30356][SQL] Codegen support for the function str_to_map
URL: https://github.com/apache/spark/pull/27013#issuecomment-569043013
**[Test build #115808 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/115808/testReport)** for PR 27013 at commit [`485d31f`](https://github.com/apache/spark/commit/485d31f98d68b632c07e58193080a1ac07778d5a).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27013: [SPARK-30356][SQL]
Codegen support for the function str_to_map
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27013: [SPARK-30356][SQL] Codegen support for the function str_to_map
URL: https://github.com/apache/spark/pull/27013#issuecomment-569202946
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/115832/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on issue #27013: [SPARK-30356][SQL]
Codegen support for the function str_to_map
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on issue #27013: [SPARK-30356][SQL] Codegen support for the function str_to_map
URL: https://github.com/apache/spark/pull/27013#issuecomment-569273296
thanks, merging to master!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27013: [SPARK-30356][SQL]
Codegen support for the function str_to_map
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27013: [SPARK-30356][SQL] Codegen support for the function str_to_map
URL: https://github.com/apache/spark/pull/27013#issuecomment-569216385
**[Test build #115843 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/115843/testReport)** for PR 27013 at commit [`61d9ed4`](https://github.com/apache/spark/commit/61d9ed47ce70c8b068a2cfec8c6cd6b1bd3bd449).
* This patch **fails due to an unknown error code, -9**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on issue #27013: [SPARK-30356][SQL]
Codegen support for the function str_to_map
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on issue #27013: [SPARK-30356][SQL] Codegen support for the function str_to_map
URL: https://github.com/apache/spark/pull/27013#issuecomment-569197547
not related to this PR, `CodegenFallBack` means the entire expression tree can't be codegened, but seems OK to still do whole stage codegen?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org