You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/10/30 03:56:39 UTC
[GitHub] [spark] Peng-Lei opened a new pull request #34446: [SPARK-37161][SQL] RowToColumnConverter support AnsiIntervalType
Peng-Lei opened a new pull request #34446:
URL: https://github.com/apache/spark/pull/34446
### What changes were proposed in this pull request?
Add RowToColumnConverter for AnsiIntervalType
### Why are the changes needed?
currently, we have RowToColumnConverter for all data types except AnsiIntervalType
```
private def getConverterForType(dataType: DataType, nullable: Boolean): TypeConverter = {
val core = dataType match {
case BinaryType => BinaryConverter
case BooleanType => BooleanConverter
case ByteType => ByteConverter
case ShortType => ShortConverter
case IntegerType | DateType => IntConverter
case FloatType => FloatConverter
case LongType | TimestampType => LongConverter
case DoubleType => DoubleConverter
case StringType => StringConverter
case CalendarIntervalType => CalendarConverter
case at: ArrayType => ArrayConverter(getConverterForType(at.elementType, at.containsNull))
case st: StructType => new StructConverter(st.fields.map(
(f) => getConverterForType(f.dataType, f.nullable)))
case dt: DecimalType => new DecimalConverter(dt)
case mt: MapType => MapConverter(getConverterForType(mt.keyType, nullable = false),
getConverterForType(mt.valueType, mt.valueContainsNull))
case unknown => throw QueryExecutionErrors.unsupportedDataTypeError(unknown.toString)
}
```
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Add ut test
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34446: [SPARK-37161][SQL] RowToColumnConverter support AnsiIntervalType
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34446:
URL: https://github.com/apache/spark/pull/34446#issuecomment-955179170
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144774/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34446: [SPARK-37161][SQL] RowToColumnConverter support AnsiIntervalType
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34446:
URL: https://github.com/apache/spark/pull/34446#issuecomment-955143242
**[Test build #144774 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144774/testReport)** for PR 34446 at commit [`5d878a6`](https://github.com/apache/spark/commit/5d878a6e9548bcc2e49fc38dd8f8901f4f5a333c).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34446: [SPARK-37161][SQL] RowToColumnConverter support AnsiIntervalType
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34446:
URL: https://github.com/apache/spark/pull/34446#issuecomment-955157742
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49244/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34446: [SPARK-37161][SQL] RowToColumnConverter support AnsiIntervalType
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34446:
URL: https://github.com/apache/spark/pull/34446#issuecomment-955186308
**[Test build #144780 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144780/testReport)** for PR 34446 at commit [`d14996a`](https://github.com/apache/spark/commit/d14996a7a8e145c4a69a5ad13bce3c92cf802c23).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34446: [SPARK-37161][SQL] RowToColumnConverter support AnsiIntervalType
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34446:
URL: https://github.com/apache/spark/pull/34446#issuecomment-955143242
**[Test build #144774 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144774/testReport)** for PR 34446 at commit [`5d878a6`](https://github.com/apache/spark/commit/5d878a6e9548bcc2e49fc38dd8f8901f4f5a333c).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gengliangwang commented on a change in pull request #34446: [SPARK-37161][SQL] RowToColumnConverter support AnsiIntervalType
Posted by GitBox <gi...@apache.org>.
gengliangwang commented on a change in pull request #34446:
URL: https://github.com/apache/spark/pull/34446#discussion_r740174603
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/Columnar.scala
##########
@@ -260,9 +260,9 @@ private object RowToColumnConverter {
case BooleanType => BooleanConverter
case ByteType => ByteConverter
case ShortType => ShortConverter
- case IntegerType | DateType => IntConverter
+ case IntegerType | DateType | _: YearMonthIntervalType => IntConverter
case FloatType => FloatConverter
- case LongType | TimestampType => LongConverter
+ case LongType | TimestampType | _: DayTimeIntervalType => LongConverter
Review comment:
Yes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34446: [SPARK-37161][SQL] RowToColumnConverter support AnsiIntervalType
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34446:
URL: https://github.com/apache/spark/pull/34446#issuecomment-955200283
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49249/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34446: [SPARK-37161][SQL] RowToColumnConverter support AnsiIntervalType
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34446:
URL: https://github.com/apache/spark/pull/34446#issuecomment-955314540
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144780/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Peng-Lei commented on a change in pull request #34446: [SPARK-37161][SQL] RowToColumnConverter support AnsiIntervalType
Posted by GitBox <gi...@apache.org>.
Peng-Lei commented on a change in pull request #34446:
URL: https://github.com/apache/spark/pull/34446#discussion_r739910466
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/vectorized/ColumnarBatchSuite.scala
##########
@@ -1688,6 +1688,45 @@ class ColumnarBatchSuite extends SparkFunSuite {
}
}
+ test("SPARK-37161: RowToColumnConverter for AnsiIntervalType") {
+ DataTypeTestUtils.yearMonthIntervalTypes.foreach { dt =>
+ val schema = new StructType().add(dt.typeName, dt)
+ val converter = new RowToColumnConverter(schema)
+ val columns = OnHeapColumnVector.allocateColumns(10, schema)
+ val batch = new ColumnarBatch(columns.toArray, 3)
Review comment:
Update.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Peng-Lei commented on a change in pull request #34446: [SPARK-37161][SQL] RowToColumnConverter support AnsiIntervalType
Posted by GitBox <gi...@apache.org>.
Peng-Lei commented on a change in pull request #34446:
URL: https://github.com/apache/spark/pull/34446#discussion_r739635620
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/Columnar.scala
##########
@@ -260,9 +260,9 @@ private object RowToColumnConverter {
case BooleanType => BooleanConverter
case ByteType => ByteConverter
case ShortType => ShortConverter
- case IntegerType | DateType => IntConverter
+ case IntegerType | DateType | YearMonthIntervalType(_, _) => IntConverter
Review comment:
Thank you very much for the reminder. Done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34446: [SPARK-37161][SQL] RowToColumnConverter support AnsiIntervalType
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34446:
URL: https://github.com/apache/spark/pull/34446#issuecomment-955153867
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49244/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34446: [SPARK-37161][SQL] RowToColumnConverter support AnsiIntervalType
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34446:
URL: https://github.com/apache/spark/pull/34446#issuecomment-955177851
**[Test build #144774 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144774/testReport)** for PR 34446 at commit [`5d878a6`](https://github.com/apache/spark/commit/5d878a6e9548bcc2e49fc38dd8f8901f4f5a333c).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34446: [SPARK-37161][SQL] RowToColumnConverter support AnsiIntervalType
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34446:
URL: https://github.com/apache/spark/pull/34446#issuecomment-955850632
**[Test build #144794 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144794/testReport)** for PR 34446 at commit [`7999aa4`](https://github.com/apache/spark/commit/7999aa42742475ec0a6f82cb55e5ef9a2216be4e).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #34446: [SPARK-37161][SQL] RowToColumnConverter support AnsiIntervalType
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #34446:
URL: https://github.com/apache/spark/pull/34446#discussion_r740054627
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/Columnar.scala
##########
@@ -260,9 +260,9 @@ private object RowToColumnConverter {
case BooleanType => BooleanConverter
case ByteType => ByteConverter
case ShortType => ShortConverter
- case IntegerType | DateType => IntConverter
+ case IntegerType | DateType | _: YearMonthIntervalType => IntConverter
case FloatType => FloatConverter
- case LongType | TimestampType => LongConverter
+ case LongType | TimestampType | _: DayTimeIntervalType => LongConverter
Review comment:
@gengliangwang shall we handle timestamp NTZ here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34446: [SPARK-37161][SQL] RowToColumnConverter support AnsiIntervalType
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34446:
URL: https://github.com/apache/spark/pull/34446#issuecomment-955869407
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49264/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk commented on a change in pull request #34446: [SPARK-37161][SQL] RowToColumnConverter support AnsiIntervalType
Posted by GitBox <gi...@apache.org>.
MaxGekk commented on a change in pull request #34446:
URL: https://github.com/apache/spark/pull/34446#discussion_r739653452
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/vectorized/ColumnarBatchSuite.scala
##########
@@ -1688,6 +1688,45 @@ class ColumnarBatchSuite extends SparkFunSuite {
}
}
+ test("SPARK-37161: RowToColumnConverter for AnsiIntervalType") {
+ DataTypeTestUtils.yearMonthIntervalTypes.foreach { dt =>
+ val schema = new StructType().add(dt.typeName, dt)
+ val converter = new RowToColumnConverter(schema)
+ val columns = OnHeapColumnVector.allocateColumns(10, schema)
+ val batch = new ColumnarBatch(columns.toArray, 3)
Review comment:
Where is `batch` used? and why it is needed?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34446: [SPARK-37161][SQL] RowToColumnConverter support AnsiIntervalType
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34446:
URL: https://github.com/apache/spark/pull/34446#issuecomment-955149302
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49244/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk commented on a change in pull request #34446: [SPARK-37161][SQL] RowToColumnConverter support AnsiIntervalType
Posted by GitBox <gi...@apache.org>.
MaxGekk commented on a change in pull request #34446:
URL: https://github.com/apache/spark/pull/34446#discussion_r739631666
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/vectorized/ColumnarBatchSuite.scala
##########
@@ -1688,6 +1688,45 @@ class ColumnarBatchSuite extends SparkFunSuite {
}
}
+ test("RowToColumnConverter for AnsiIntervalType") {
Review comment:
Add JIRA ID:
```suggestion
test("SPARK-37161: RowToColumnConverter for AnsiIntervalType") {
```
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/Columnar.scala
##########
@@ -260,9 +260,9 @@ private object RowToColumnConverter {
case BooleanType => BooleanConverter
case ByteType => ByteConverter
case ShortType => ShortConverter
- case IntegerType | DateType => IntConverter
+ case IntegerType | DateType | YearMonthIntervalType(_, _) => IntConverter
Review comment:
Please, use _: YearMonthIntervalType. In this ways, your code won't depend on the number of parameters. Also, see https://github.com/databricks/scala-style-guide#pattern-matching :
_"If the only goal is to match on the type of the object, do NOT expand fully all the arguments, as it makes refactoring more difficult and the code more error prone."_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34446: [SPARK-37161][SQL] RowToColumnConverter support AnsiIntervalType
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34446:
URL: https://github.com/apache/spark/pull/34446#issuecomment-955287925
**[Test build #144780 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144780/testReport)** for PR 34446 at commit [`d14996a`](https://github.com/apache/spark/commit/d14996a7a8e145c4a69a5ad13bce3c92cf802c23).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34446: [SPARK-37161][SQL] RowToColumnConverter support AnsiIntervalType
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34446:
URL: https://github.com/apache/spark/pull/34446#issuecomment-955196694
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49249/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34446: [SPARK-37161][SQL] RowToColumnConverter support AnsiIntervalType
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34446:
URL: https://github.com/apache/spark/pull/34446#issuecomment-955179170
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144774/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34446: [SPARK-37161][SQL] RowToColumnConverter support AnsiIntervalType
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34446:
URL: https://github.com/apache/spark/pull/34446#issuecomment-955186308
**[Test build #144780 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144780/testReport)** for PR 34446 at commit [`d14996a`](https://github.com/apache/spark/commit/d14996a7a8e145c4a69a5ad13bce3c92cf802c23).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34446: [SPARK-37161][SQL] RowToColumnConverter support AnsiIntervalType
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34446:
URL: https://github.com/apache/spark/pull/34446#issuecomment-955897123
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49264/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34446: [SPARK-37161][SQL] RowToColumnConverter support AnsiIntervalType
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34446:
URL: https://github.com/apache/spark/pull/34446#issuecomment-955850632
**[Test build #144794 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144794/testReport)** for PR 34446 at commit [`7999aa4`](https://github.com/apache/spark/commit/7999aa42742475ec0a6f82cb55e5ef9a2216be4e).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34446: [SPARK-37161][SQL] RowToColumnConverter support AnsiIntervalType
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34446:
URL: https://github.com/apache/spark/pull/34446#issuecomment-955960746
**[Test build #144794 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144794/testReport)** for PR 34446 at commit [`7999aa4`](https://github.com/apache/spark/commit/7999aa42742475ec0a6f82cb55e5ef9a2216be4e).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34446: [SPARK-37161][SQL] RowToColumnConverter support AnsiIntervalType
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34446:
URL: https://github.com/apache/spark/pull/34446#issuecomment-955200283
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49249/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Peng-Lei commented on a change in pull request #34446: [SPARK-37161][SQL] RowToColumnConverter support AnsiIntervalType
Posted by GitBox <gi...@apache.org>.
Peng-Lei commented on a change in pull request #34446:
URL: https://github.com/apache/spark/pull/34446#discussion_r739907618
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/vectorized/ColumnarBatchSuite.scala
##########
@@ -1688,6 +1688,45 @@ class ColumnarBatchSuite extends SparkFunSuite {
}
}
+ test("SPARK-37161: RowToColumnConverter for AnsiIntervalType") {
+ DataTypeTestUtils.yearMonthIntervalTypes.foreach { dt =>
+ val schema = new StructType().add(dt.typeName, dt)
+ val converter = new RowToColumnConverter(schema)
+ val columns = OnHeapColumnVector.allocateColumns(10, schema)
+ val batch = new ColumnarBatch(columns.toArray, 3)
Review comment:
Sorry, a mistake. It should be `val batch = new ColumnarBatch(columns.toArray, **10**)`, It's meant to free up resources(`batch.close()`) instead of `columns.foreach(_.close())`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Peng-Lei commented on pull request #34446: [SPARK-37161][SQL] RowToColumnConverter support AnsiIntervalType
Posted by GitBox <gi...@apache.org>.
Peng-Lei commented on pull request #34446:
URL: https://github.com/apache/spark/pull/34446#issuecomment-955140778
@MaxGekk Could you take a look? Thank you very much. This PR is follow [#PR](https://github.com/apache/spark/pull/33733)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34446: [SPARK-37161][SQL] RowToColumnConverter support AnsiIntervalType
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34446:
URL: https://github.com/apache/spark/pull/34446#issuecomment-955191996
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49249/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34446: [SPARK-37161][SQL] RowToColumnConverter support AnsiIntervalType
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34446:
URL: https://github.com/apache/spark/pull/34446#issuecomment-955968635
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144794/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34446: [SPARK-37161][SQL] RowToColumnConverter support AnsiIntervalType
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34446:
URL: https://github.com/apache/spark/pull/34446#issuecomment-955314540
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144780/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34446: [SPARK-37161][SQL] RowToColumnConverter support AnsiIntervalType
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34446:
URL: https://github.com/apache/spark/pull/34446#issuecomment-955968635
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144794/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk commented on pull request #34446: [SPARK-37161][SQL] RowToColumnConverter support AnsiIntervalType
Posted by GitBox <gi...@apache.org>.
MaxGekk commented on pull request #34446:
URL: https://github.com/apache/spark/pull/34446#issuecomment-955978742
+1, LGTM. Merging to master.
Thank you, @Peng-Lei .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk closed pull request #34446: [SPARK-37161][SQL] RowToColumnConverter support AnsiIntervalType
Posted by GitBox <gi...@apache.org>.
MaxGekk closed pull request #34446:
URL: https://github.com/apache/spark/pull/34446
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34446: [SPARK-37161][SQL] RowToColumnConverter support AnsiIntervalType
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34446:
URL: https://github.com/apache/spark/pull/34446#issuecomment-955157742
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49244/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34446: [SPARK-37161][SQL] RowToColumnConverter support AnsiIntervalType
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34446:
URL: https://github.com/apache/spark/pull/34446#issuecomment-955897123
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49264/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34446: [SPARK-37161][SQL] RowToColumnConverter support AnsiIntervalType
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34446:
URL: https://github.com/apache/spark/pull/34446#issuecomment-955891516
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49264/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org