You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2020/04/17 08:21:57 UTC
[GitHub] [spark] turboFei opened a new pull request #28239: SPARK-31467: Fix
test issue with table named in hive/SQLQuerySuite
turboFei opened a new pull request #28239: SPARK-31467: Fix test issue with table named in hive/SQLQuerySuite
URL: https://github.com/apache/spark/pull/28239
### What changes were proposed in this pull request?
### Why are the changes needed?
### Does this PR introduce any user-facing change?
No.
### How was this patch tested?
Not needed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] turboFei commented on issue #28239:
[SPARK-31467][SQL][TEST] Refactor the tests in hive/SQLQuerySuite to
prevent TableAlreadyExistsException
Posted by GitBox <gi...@apache.org>.
turboFei commented on issue #28239: [SPARK-31467][SQL][TEST] Refactor the tests in hive/SQLQuerySuite to prevent TableAlreadyExistsException
URL: https://github.com/apache/spark/pull/28239#issuecomment-616014159
> I think the change looks reasonable. Did you check them only in `SQLQuerySuite`? I mean, have you checked if the logs of the other test suites have no `TableAlreadyExistsException`?
Hi, I found this issue when adding UT into hive/SQLQuerySuite.
There is no `TableAlreadyExistsException` in the logs before add UT.
I just checked the sql/SQLQuerySuite and refactor it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] turboFei removed a comment on issue #28239: SPARK-31467:
Refactor the tests in hive/SQLQuerySuite to prevent
TableAlreadyExistsException
Posted by GitBox <gi...@apache.org>.
turboFei removed a comment on issue #28239: SPARK-31467: Refactor the tests in hive/SQLQuerySuite to prevent TableAlreadyExistsException
URL: https://github.com/apache/spark/pull/28239#issuecomment-615117898
It seems that there are many tests do not clean up used tables/views.
I will try to refactor this suite.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28239:
[SPARK-31467][SQL][TEST] Refactor the sql tests to prevent
TableAlreadyExistsException
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #28239: [SPARK-31467][SQL][TEST] Refactor the sql tests to prevent TableAlreadyExistsException
URL: https://github.com/apache/spark/pull/28239#issuecomment-616062013
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/121468/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28239:
[SPARK-31467][SQL][TEST] Refactor the tests in hive/SQLQuerySuite to
prevent TableAlreadyExistsException
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #28239: [SPARK-31467][SQL][TEST] Refactor the tests in hive/SQLQuerySuite to prevent TableAlreadyExistsException
URL: https://github.com/apache/spark/pull/28239#issuecomment-616014349
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28239:
[SPARK-31467][SQL][TEST] Refactor the sql tests to prevent
TableAlreadyExistsException
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #28239: [SPARK-31467][SQL][TEST] Refactor the sql tests to prevent TableAlreadyExistsException
URL: https://github.com/apache/spark/pull/28239#issuecomment-616087513
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/26157/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] turboFei commented on a change in pull request #28239:
[SPARK-31467][SQL][TEST] Refactor the tests in hive/SQLQuerySuite to
prevent TableAlreadyExistsException
Posted by GitBox <gi...@apache.org>.
turboFei commented on a change in pull request #28239: [SPARK-31467][SQL][TEST] Refactor the tests in hive/SQLQuerySuite to prevent TableAlreadyExistsException
URL: https://github.com/apache/spark/pull/28239#discussion_r410793370
##########
File path: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/SQLQuerySuite.scala
##########
@@ -898,33 +913,39 @@ class SQLQuerySuite extends QueryTest with SQLTestUtils with TestHiveSingleton {
}
test("resolve udtf in projection #1") {
- val ds = (1 to 5).map(i => s"""{"a":[$i, ${i + 1}]}""").toDS()
- read.json(ds).createOrReplaceTempView("data")
- val df = sql("SELECT explode(a) AS val FROM data")
- val col = df("val")
+ withTempView("data") {
+ val ds = (1 to 5).map(i => s"""{"a":[$i, ${i + 1}]}""").toDS()
+ read.json(ds).createOrReplaceTempView("data")
+ val df = sql("SELECT explode(a) AS val FROM data")
+ val col = df("val")
Review comment:
yes, should we remove this line?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu commented on issue #28239: SPARK-31467: Refactor the
tests in hive/SQLQuerySuite to prevent TableAlreadyExistsException
Posted by GitBox <gi...@apache.org>.
maropu commented on issue #28239: SPARK-31467: Refactor the tests in hive/SQLQuerySuite to prevent TableAlreadyExistsException
URL: https://github.com/apache/spark/pull/28239#issuecomment-615965046
ok to test
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28239:
[SPARK-31467][SQL][TEST] Refactor the tests in hive/SQLQuerySuite to
prevent TableAlreadyExistsException
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #28239: [SPARK-31467][SQL][TEST] Refactor the tests in hive/SQLQuerySuite to prevent TableAlreadyExistsException
URL: https://github.com/apache/spark/pull/28239#issuecomment-615999012
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/121460/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28239:
[SPARK-31467][SQL][TEST] Refactor the tests in hive/SQLQuerySuite to
prevent TableAlreadyExistsException
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #28239: [SPARK-31467][SQL][TEST] Refactor the tests in hive/SQLQuerySuite to prevent TableAlreadyExistsException
URL: https://github.com/apache/spark/pull/28239#issuecomment-615114395
Can one of the admins verify this patch?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #28239: [SPARK-31467][SQL][TEST]
Refactor the sql tests to prevent TableAlreadyExistsException
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #28239: [SPARK-31467][SQL][TEST] Refactor the sql tests to prevent TableAlreadyExistsException
URL: https://github.com/apache/spark/pull/28239#issuecomment-616061653
**[Test build #121470 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/121470/testReport)** for PR 28239 at commit [`bb52763`](https://github.com/apache/spark/commit/bb52763b6e3ed55285e25811bf23b92b4d8c4394).
* This patch **fails due to an unknown error code, -9**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28239:
[SPARK-31467][SQL][TEST] Refactor the sql tests to prevent
TableAlreadyExistsException
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #28239: [SPARK-31467][SQL][TEST] Refactor the sql tests to prevent TableAlreadyExistsException
URL: https://github.com/apache/spark/pull/28239#issuecomment-616062009
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] turboFei commented on a change in pull request #28239:
[SPARK-31467][SQL][TEST] Refactor the tests in hive/SQLQuerySuite to
prevent TableAlreadyExistsException
Posted by GitBox <gi...@apache.org>.
turboFei commented on a change in pull request #28239: [SPARK-31467][SQL][TEST] Refactor the tests in hive/SQLQuerySuite to prevent TableAlreadyExistsException
URL: https://github.com/apache/spark/pull/28239#discussion_r410795468
##########
File path: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/SQLQuerySuite.scala
##########
@@ -974,47 +995,57 @@ class SQLQuerySuite extends QueryTest with SQLTestUtils with TestHiveSingleton {
}
test("SPARK-5203 union with different decimal precision") {
- Seq.empty[(java.math.BigDecimal, java.math.BigDecimal)]
- .toDF("d1", "d2")
- .select($"d1".cast(DecimalType(10, 5)).as("d"))
- .createOrReplaceTempView("dn")
+ withTempView("dn") {
+ Seq.empty[(java.math.BigDecimal, java.math.BigDecimal)]
+ .toDF("d1", "d2")
+ .select($"d1".cast(DecimalType(10, 5)).as("d"))
+ .createOrReplaceTempView("dn")
- sql("select d from dn union all select d * 2 from dn")
- .queryExecution.analyzed
+ sql("select d from dn union all select d * 2 from dn")
+ .queryExecution.analyzed
+ }
}
test("Star Expansion - script transform") {
- assume(TestUtils.testCommandAvailable("/bin/bash"))
- val data = (1 to 100000).map { i => (i, i, i) }
- data.toDF("d1", "d2", "d3").createOrReplaceTempView("script_trans")
- assert(100000 === sql("SELECT TRANSFORM (*) USING 'cat' FROM script_trans").count())
+ withTempView("script_trans") {
+ assume(TestUtils.testCommandAvailable("/bin/bash"))
+ val data = (1 to 100000).map { i => (i, i, i) }
+ data.toDF("d1", "d2", "d3").createOrReplaceTempView("script_trans")
+ assert(100000 === sql("SELECT TRANSFORM (*) USING 'cat' FROM script_trans").count())
+ }
}
test("test script transform for stdout") {
- assume(TestUtils.testCommandAvailable("/bin/bash"))
- val data = (1 to 100000).map { i => (i, i, i) }
- data.toDF("d1", "d2", "d3").createOrReplaceTempView("script_trans")
- assert(100000 ===
- sql("SELECT TRANSFORM (d1, d2, d3) USING 'cat' AS (a,b,c) FROM script_trans").count())
+ withTempView("script_trans") {
+ assume(TestUtils.testCommandAvailable("/bin/bash"))
+ val data = (1 to 100000).map { i => (i, i, i) }
+ data.toDF("d1", "d2", "d3").createOrReplaceTempView("script_trans")
+ assert(100000 ===
+ sql("SELECT TRANSFORM (d1, d2, d3) USING 'cat' AS (a,b,c) FROM script_trans").count())
+ }
}
test("test script transform for stderr") {
- assume(TestUtils.testCommandAvailable("/bin/bash"))
- val data = (1 to 100000).map { i => (i, i, i) }
- data.toDF("d1", "d2", "d3").createOrReplaceTempView("script_trans")
- assert(0 ===
- sql("SELECT TRANSFORM (d1, d2, d3) USING 'cat 1>&2' AS (a,b,c) FROM script_trans").count())
+ withTempView("script_trans") {
+ assume(TestUtils.testCommandAvailable("/bin/bash"))
+ val data = (1 to 100000).map { i => (i, i, i) }
+ data.toDF("d1", "d2", "d3").createOrReplaceTempView("script_trans")
+ assert(0 ===
+ sql("SELECT TRANSFORM (d1, d2, d3) USING 'cat 1>&2' AS (a,b,c) FROM script_trans").count())
+ }
}
test("test script transform data type") {
- assume(TestUtils.testCommandAvailable("/bin/bash"))
- val data = (1 to 5).map { i => (i, i) }
- data.toDF("key", "value").createOrReplaceTempView("test")
- checkAnswer(
- sql("""FROM
- |(FROM test SELECT TRANSFORM(key, value) USING 'cat' AS (`thing1` int, thing2 string)) t
- |SELECT thing1 + 1
+ withTempView("test") {
+ assume(TestUtils.testCommandAvailable("/bin/bash"))
+ val data = (1 to 5).map { i => (i, i) }
+ data.toDF("key", "value").createOrReplaceTempView("test")
+ checkAnswer(
+ sql("""FROM
+ |(FROM test SELECT TRANSFORM(key, value) USING 'cat' AS (`thing1` int, thing2 string))
Review comment:
thanks
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28239:
[SPARK-31467][SQL][TEST] Refactor the tests in hive/SQLQuerySuite to
prevent TableAlreadyExistsException
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #28239: [SPARK-31467][SQL][TEST] Refactor the tests in hive/SQLQuerySuite to prevent TableAlreadyExistsException
URL: https://github.com/apache/spark/pull/28239#issuecomment-616011284
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28239:
[SPARK-31467][SQL][TEST] Refactor the tests in hive/SQLQuerySuite to
prevent TableAlreadyExistsException
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #28239: [SPARK-31467][SQL][TEST] Refactor the tests in hive/SQLQuerySuite to prevent TableAlreadyExistsException
URL: https://github.com/apache/spark/pull/28239#issuecomment-615999011
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #28239:
[SPARK-31467][SQL][TEST] Refactor the sql tests to prevent
TableAlreadyExistsException
Posted by GitBox <gi...@apache.org>.
maropu commented on a change in pull request #28239: [SPARK-31467][SQL][TEST] Refactor the sql tests to prevent TableAlreadyExistsException
URL: https://github.com/apache/spark/pull/28239#discussion_r410865751
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/json/JsonSuite.scala
##########
@@ -833,171 +861,182 @@ abstract class JsonSuite extends QueryTest with SharedSparkSession with TestJson
}
test("Applying schemas") {
- val dir = Utils.createTempDir()
- dir.delete()
- val path = dir.getCanonicalPath
- primitiveFieldAndType.map(record => record.replaceAll("\n", " ")).write.text(path)
+ withTempView("jsonTable1", "jsonTable2") {
+ val dir = Utils.createTempDir()
+ dir.delete()
+ val path = dir.getCanonicalPath
+ primitiveFieldAndType.map(record => record.replaceAll("\n", " ")).write.text(path)
- val schema = StructType(
- StructField("bigInteger", DecimalType.SYSTEM_DEFAULT, true) ::
- StructField("boolean", BooleanType, true) ::
- StructField("double", DoubleType, true) ::
- StructField("integer", IntegerType, true) ::
- StructField("long", LongType, true) ::
- StructField("null", StringType, true) ::
- StructField("string", StringType, true) :: Nil)
+ val schema = StructType(
+ StructField("bigInteger", DecimalType.SYSTEM_DEFAULT, true) ::
+ StructField("boolean", BooleanType, true) ::
+ StructField("double", DoubleType, true) ::
+ StructField("integer", IntegerType, true) ::
+ StructField("long", LongType, true) ::
+ StructField("null", StringType, true) ::
+ StructField("string", StringType, true) :: Nil)
- val jsonDF1 = spark.read.schema(schema).json(path)
+ val jsonDF1 = spark.read.schema(schema).json(path)
- assert(schema === jsonDF1.schema)
+ assert(schema === jsonDF1.schema)
- jsonDF1.createOrReplaceTempView("jsonTable1")
+ jsonDF1.createOrReplaceTempView("jsonTable1")
- checkAnswer(
- sql("select * from jsonTable1"),
- Row(new java.math.BigDecimal("92233720368547758070"),
- true,
- 1.7976931348623157,
- 10,
- 21474836470L,
- null,
- "this is a simple string.")
- )
+ checkAnswer(
+ sql("select * from jsonTable1"),
+ Row(new java.math.BigDecimal("92233720368547758070"),
+ true,
+ 1.7976931348623157,
+ 10,
+ 21474836470L,
+ null,
+ "this is a simple string.")
+ )
- val jsonDF2 = spark.read.schema(schema).json(primitiveFieldAndType)
+ val jsonDF2 = spark.read.schema(schema).json(primitiveFieldAndType)
- assert(schema === jsonDF2.schema)
+ assert(schema === jsonDF2.schema)
- jsonDF2.createOrReplaceTempView("jsonTable2")
+ jsonDF2.createOrReplaceTempView("jsonTable2")
- checkAnswer(
- sql("select * from jsonTable2"),
- Row(new java.math.BigDecimal("92233720368547758070"),
- true,
- 1.7976931348623157,
- 10,
- 21474836470L,
- null,
- "this is a simple string.")
- )
+ checkAnswer(
+ sql("select * from jsonTable2"),
+ Row(new java.math.BigDecimal("92233720368547758070"),
+ true,
+ 1.7976931348623157,
+ 10,
+ 21474836470L,
+ null,
+ "this is a simple string.")
+ )
+ }
}
test("Applying schemas with MapType") {
- val schemaWithSimpleMap = StructType(
- StructField("map", MapType(StringType, IntegerType, true), false) :: Nil)
- val jsonWithSimpleMap = spark.read.schema(schemaWithSimpleMap).json(mapType1)
-
- jsonWithSimpleMap.createOrReplaceTempView("jsonWithSimpleMap")
+ withTempView("jsonWithSimpleMap", "jsonWithComplexMap") {
+ val schemaWithSimpleMap = StructType(
+ StructField("map", MapType(StringType, IntegerType, true), false) :: Nil)
+ val jsonWithSimpleMap = spark.read.schema(schemaWithSimpleMap).json(mapType1)
- checkAnswer(
- sql("select `map` from jsonWithSimpleMap"),
- Row(Map("a" -> 1)) ::
- Row(Map("b" -> 2)) ::
- Row(Map("c" -> 3)) ::
- Row(Map("c" -> 1, "d" -> 4)) ::
- Row(Map("e" -> null)) :: Nil
- )
+ jsonWithSimpleMap.createOrReplaceTempView("jsonWithSimpleMap")
- withSQLConf(SQLConf.SUPPORT_QUOTED_REGEX_COLUMN_NAME.key -> "false") {
checkAnswer(
- sql("select `map`['c'] from jsonWithSimpleMap"),
- Row(null) ::
- Row(null) ::
- Row(3) ::
- Row(1) ::
- Row(null) :: Nil
+ sql("select `map` from jsonWithSimpleMap"),
+ Row(Map("a" -> 1)) ::
+ Row(Map("b" -> 2)) ::
+ Row(Map("c" -> 3)) ::
+ Row(Map("c" -> 1, "d" -> 4)) ::
+ Row(Map("e" -> null)) :: Nil
)
- }
- val innerStruct = StructType(
- StructField("field1", ArrayType(IntegerType, true), true) ::
- StructField("field2", IntegerType, true) :: Nil)
- val schemaWithComplexMap = StructType(
- StructField("map", MapType(StringType, innerStruct, true), false) :: Nil)
+ withSQLConf(SQLConf.SUPPORT_QUOTED_REGEX_COLUMN_NAME.key -> "false") {
+ checkAnswer(
+ sql("select `map`['c'] from jsonWithSimpleMap"),
+ Row(null) ::
+ Row(null) ::
+ Row(3) ::
+ Row(1) ::
+ Row(null) :: Nil
+ )
+ }
- val jsonWithComplexMap = spark.read.schema(schemaWithComplexMap).json(mapType2)
+ val innerStruct = StructType(
+ StructField("field1", ArrayType(IntegerType, true), true) ::
+ StructField("field2", IntegerType, true) :: Nil)
+ val schemaWithComplexMap = StructType(
+ StructField("map", MapType(StringType, innerStruct, true), false) :: Nil)
- jsonWithComplexMap.createOrReplaceTempView("jsonWithComplexMap")
+ val jsonWithComplexMap = spark.read.schema(schemaWithComplexMap).json(mapType2)
- checkAnswer(
- sql("select `map` from jsonWithComplexMap"),
- Row(Map("a" -> Row(Seq(1, 2, 3, null), null))) ::
- Row(Map("b" -> Row(null, 2))) ::
- Row(Map("c" -> Row(Seq(), 4))) ::
- Row(Map("c" -> Row(null, 3), "d" -> Row(Seq(null), null))) ::
- Row(Map("e" -> null)) ::
- Row(Map("f" -> Row(null, null))) :: Nil
- )
+ jsonWithComplexMap.createOrReplaceTempView("jsonWithComplexMap")
- withSQLConf(SQLConf.SUPPORT_QUOTED_REGEX_COLUMN_NAME.key -> "false") {
checkAnswer(
- sql("select `map`['a'].field1, `map`['c'].field2 from jsonWithComplexMap"),
- Row(Seq(1, 2, 3, null), null) ::
- Row(null, null) ::
- Row(null, 4) ::
- Row(null, 3) ::
- Row(null, null) ::
- Row(null, null) :: Nil
+ sql("select `map` from jsonWithComplexMap"),
+ Row(Map("a" -> Row(Seq(1, 2, 3, null), null))) ::
+ Row(Map("b" -> Row(null, 2))) ::
+ Row(Map("c" -> Row(Seq(), 4))) ::
+ Row(Map("c" -> Row(null, 3), "d" -> Row(Seq(null), null))) ::
+ Row(Map("e" -> null)) ::
+ Row(Map("f" -> Row(null, null))) :: Nil
)
+
+ withSQLConf(SQLConf.SUPPORT_QUOTED_REGEX_COLUMN_NAME.key -> "false") {
+ checkAnswer(
+ sql("select `map`['a'].field1, `map`['c'].field2 from jsonWithComplexMap"),
+ Row(Seq(1, 2, 3, null), null) ::
+ Row(null, null) ::
+ Row(null, 4) ::
+ Row(null, 3) ::
+ Row(null, null) ::
+ Row(null, null) :: Nil
+ )
+ }
}
}
test("SPARK-2096 Correctly parse dot notations") {
- val jsonDF = spark.read.json(complexFieldAndType2)
- jsonDF.createOrReplaceTempView("jsonTable")
+ withTempView("jsonTable") {
+ val jsonDF = spark.read.json(complexFieldAndType2)
+ jsonDF.createOrReplaceTempView("jsonTable")
- checkAnswer(
- sql("select arrayOfStruct[0].field1, arrayOfStruct[0].field2 from jsonTable"),
- Row(true, "str1")
- )
- checkAnswer(
- sql(
- """
- |select complexArrayOfStruct[0].field1[1].inner2[0], complexArrayOfStruct[1].field2[0][1]
- |from jsonTable
+ checkAnswer(
+ sql("select arrayOfStruct[0].field1, arrayOfStruct[0].field2 from jsonTable"),
+ Row(true, "str1")
+ )
+ checkAnswer(
+ sql(
+ """
+ |select complexArrayOfStruct[0].field1[1].inner2[0],
Review comment:
nit: wrong indents.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #28239: [SPARK-31467][SQL][TEST]
Refactor the tests in hive/SQLQuerySuite to prevent
TableAlreadyExistsException
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #28239: [SPARK-31467][SQL][TEST] Refactor the tests in hive/SQLQuerySuite to prevent TableAlreadyExistsException
URL: https://github.com/apache/spark/pull/28239#issuecomment-616014236
**[Test build #121467 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/121467/testReport)** for PR 28239 at commit [`565631b`](https://github.com/apache/spark/commit/565631b3b97ede0ca81d3141ca771842322702cb).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28239:
[SPARK-31467][SQL][TEST] Refactor the sql tests to prevent
TableAlreadyExistsException
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #28239: [SPARK-31467][SQL][TEST] Refactor the sql tests to prevent TableAlreadyExistsException
URL: https://github.com/apache/spark/pull/28239#issuecomment-616212513
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28239:
[SPARK-31467][SQL][TEST] Refactor the sql tests to prevent
TableAlreadyExistsException
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #28239: [SPARK-31467][SQL][TEST] Refactor the sql tests to prevent TableAlreadyExistsException
URL: https://github.com/apache/spark/pull/28239#issuecomment-616140714
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/121473/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28239:
[SPARK-31467][SQL][TEST] Refactor the tests in hive/SQLQuerySuite to
prevent TableAlreadyExistsException
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #28239: [SPARK-31467][SQL][TEST] Refactor the tests in hive/SQLQuerySuite to prevent TableAlreadyExistsException
URL: https://github.com/apache/spark/pull/28239#issuecomment-615999011
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] turboFei commented on issue #28239:
[SPARK-31467][SQL][TEST] Refactor the sql tests to prevent
TableAlreadyExistsException
Posted by GitBox <gi...@apache.org>.
turboFei commented on issue #28239: [SPARK-31467][SQL][TEST] Refactor the sql tests to prevent TableAlreadyExistsException
URL: https://github.com/apache/spark/pull/28239#issuecomment-616154444
Have refactor the indents and double check it.
cc @maropu @dongjoon-hyun
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #28239: [SPARK-31467][SQL][TEST]
Refactor the sql tests to prevent TableAlreadyExistsException
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #28239: [SPARK-31467][SQL][TEST] Refactor the sql tests to prevent TableAlreadyExistsException
URL: https://github.com/apache/spark/pull/28239#issuecomment-616212212
**[Test build #121477 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/121477/testReport)** for PR 28239 at commit [`453c5a5`](https://github.com/apache/spark/commit/453c5a5e0717d2681fc2e0ed4f48ca093d4020a0).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28239:
[SPARK-31467][SQL][TEST] Refactor the sql tests to prevent
TableAlreadyExistsException
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #28239: [SPARK-31467][SQL][TEST] Refactor the sql tests to prevent TableAlreadyExistsException
URL: https://github.com/apache/spark/pull/28239#issuecomment-616062013
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/121468/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #28239:
[SPARK-31467][SQL][TEST] Refactor the tests in hive/SQLQuerySuite to
prevent TableAlreadyExistsException
Posted by GitBox <gi...@apache.org>.
maropu commented on a change in pull request #28239: [SPARK-31467][SQL][TEST] Refactor the tests in hive/SQLQuerySuite to prevent TableAlreadyExistsException
URL: https://github.com/apache/spark/pull/28239#discussion_r410762784
##########
File path: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/SQLQuerySuite.scala
##########
@@ -974,47 +995,57 @@ class SQLQuerySuite extends QueryTest with SQLTestUtils with TestHiveSingleton {
}
test("SPARK-5203 union with different decimal precision") {
- Seq.empty[(java.math.BigDecimal, java.math.BigDecimal)]
- .toDF("d1", "d2")
- .select($"d1".cast(DecimalType(10, 5)).as("d"))
- .createOrReplaceTempView("dn")
+ withTempView("dn") {
+ Seq.empty[(java.math.BigDecimal, java.math.BigDecimal)]
+ .toDF("d1", "d2")
+ .select($"d1".cast(DecimalType(10, 5)).as("d"))
+ .createOrReplaceTempView("dn")
- sql("select d from dn union all select d * 2 from dn")
- .queryExecution.analyzed
+ sql("select d from dn union all select d * 2 from dn")
+ .queryExecution.analyzed
+ }
}
test("Star Expansion - script transform") {
- assume(TestUtils.testCommandAvailable("/bin/bash"))
- val data = (1 to 100000).map { i => (i, i, i) }
- data.toDF("d1", "d2", "d3").createOrReplaceTempView("script_trans")
- assert(100000 === sql("SELECT TRANSFORM (*) USING 'cat' FROM script_trans").count())
+ withTempView("script_trans") {
+ assume(TestUtils.testCommandAvailable("/bin/bash"))
+ val data = (1 to 100000).map { i => (i, i, i) }
+ data.toDF("d1", "d2", "d3").createOrReplaceTempView("script_trans")
+ assert(100000 === sql("SELECT TRANSFORM (*) USING 'cat' FROM script_trans").count())
+ }
}
test("test script transform for stdout") {
- assume(TestUtils.testCommandAvailable("/bin/bash"))
- val data = (1 to 100000).map { i => (i, i, i) }
- data.toDF("d1", "d2", "d3").createOrReplaceTempView("script_trans")
- assert(100000 ===
- sql("SELECT TRANSFORM (d1, d2, d3) USING 'cat' AS (a,b,c) FROM script_trans").count())
+ withTempView("script_trans") {
+ assume(TestUtils.testCommandAvailable("/bin/bash"))
+ val data = (1 to 100000).map { i => (i, i, i) }
+ data.toDF("d1", "d2", "d3").createOrReplaceTempView("script_trans")
+ assert(100000 ===
+ sql("SELECT TRANSFORM (d1, d2, d3) USING 'cat' AS (a,b,c) FROM script_trans").count())
+ }
}
test("test script transform for stderr") {
- assume(TestUtils.testCommandAvailable("/bin/bash"))
- val data = (1 to 100000).map { i => (i, i, i) }
- data.toDF("d1", "d2", "d3").createOrReplaceTempView("script_trans")
- assert(0 ===
- sql("SELECT TRANSFORM (d1, d2, d3) USING 'cat 1>&2' AS (a,b,c) FROM script_trans").count())
+ withTempView("script_trans") {
+ assume(TestUtils.testCommandAvailable("/bin/bash"))
+ val data = (1 to 100000).map { i => (i, i, i) }
+ data.toDF("d1", "d2", "d3").createOrReplaceTempView("script_trans")
+ assert(0 ===
+ sql("SELECT TRANSFORM (d1, d2, d3) USING 'cat 1>&2' AS (a,b,c) FROM script_trans").count())
+ }
}
test("test script transform data type") {
- assume(TestUtils.testCommandAvailable("/bin/bash"))
- val data = (1 to 5).map { i => (i, i) }
- data.toDF("key", "value").createOrReplaceTempView("test")
- checkAnswer(
- sql("""FROM
- |(FROM test SELECT TRANSFORM(key, value) USING 'cat' AS (`thing1` int, thing2 string)) t
- |SELECT thing1 + 1
+ withTempView("test") {
+ assume(TestUtils.testCommandAvailable("/bin/bash"))
+ val data = (1 to 5).map { i => (i, i) }
+ data.toDF("key", "value").createOrReplaceTempView("test")
+ checkAnswer(
+ sql("""FROM
+ |(FROM test SELECT TRANSFORM(key, value) USING 'cat' AS (`thing1` int, thing2 string))
Review comment:
nit: wrong indents and you don't need to move `t` into the next line.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28239:
[SPARK-31467][SQL][TEST] Refactor the tests in hive/SQLQuerySuite to
prevent TableAlreadyExistsException
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #28239: [SPARK-31467][SQL][TEST] Refactor the tests in hive/SQLQuerySuite to prevent TableAlreadyExistsException
URL: https://github.com/apache/spark/pull/28239#issuecomment-616011288
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/26150/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] turboFei commented on issue #28239: SPARK-31467: Fix issue
when adding UT uses table named `test` in hive/SQLQuerySuite
Posted by GitBox <gi...@apache.org>.
turboFei commented on issue #28239: SPARK-31467: Fix issue when adding UT uses table named `test` in hive/SQLQuerySuite
URL: https://github.com/apache/spark/pull/28239#issuecomment-615117898
It seems that there are many tests do not clean up used tables/views.
I will try to refactor this suite.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #28239:
[SPARK-31467][SQL][TEST] Refactor the sql tests to prevent
TableAlreadyExistsException
Posted by GitBox <gi...@apache.org>.
maropu commented on a change in pull request #28239: [SPARK-31467][SQL][TEST] Refactor the sql tests to prevent TableAlreadyExistsException
URL: https://github.com/apache/spark/pull/28239#discussion_r410865751
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/json/JsonSuite.scala
##########
@@ -833,171 +861,182 @@ abstract class JsonSuite extends QueryTest with SharedSparkSession with TestJson
}
test("Applying schemas") {
- val dir = Utils.createTempDir()
- dir.delete()
- val path = dir.getCanonicalPath
- primitiveFieldAndType.map(record => record.replaceAll("\n", " ")).write.text(path)
+ withTempView("jsonTable1", "jsonTable2") {
+ val dir = Utils.createTempDir()
+ dir.delete()
+ val path = dir.getCanonicalPath
+ primitiveFieldAndType.map(record => record.replaceAll("\n", " ")).write.text(path)
- val schema = StructType(
- StructField("bigInteger", DecimalType.SYSTEM_DEFAULT, true) ::
- StructField("boolean", BooleanType, true) ::
- StructField("double", DoubleType, true) ::
- StructField("integer", IntegerType, true) ::
- StructField("long", LongType, true) ::
- StructField("null", StringType, true) ::
- StructField("string", StringType, true) :: Nil)
+ val schema = StructType(
+ StructField("bigInteger", DecimalType.SYSTEM_DEFAULT, true) ::
+ StructField("boolean", BooleanType, true) ::
+ StructField("double", DoubleType, true) ::
+ StructField("integer", IntegerType, true) ::
+ StructField("long", LongType, true) ::
+ StructField("null", StringType, true) ::
+ StructField("string", StringType, true) :: Nil)
- val jsonDF1 = spark.read.schema(schema).json(path)
+ val jsonDF1 = spark.read.schema(schema).json(path)
- assert(schema === jsonDF1.schema)
+ assert(schema === jsonDF1.schema)
- jsonDF1.createOrReplaceTempView("jsonTable1")
+ jsonDF1.createOrReplaceTempView("jsonTable1")
- checkAnswer(
- sql("select * from jsonTable1"),
- Row(new java.math.BigDecimal("92233720368547758070"),
- true,
- 1.7976931348623157,
- 10,
- 21474836470L,
- null,
- "this is a simple string.")
- )
+ checkAnswer(
+ sql("select * from jsonTable1"),
+ Row(new java.math.BigDecimal("92233720368547758070"),
+ true,
+ 1.7976931348623157,
+ 10,
+ 21474836470L,
+ null,
+ "this is a simple string.")
+ )
- val jsonDF2 = spark.read.schema(schema).json(primitiveFieldAndType)
+ val jsonDF2 = spark.read.schema(schema).json(primitiveFieldAndType)
- assert(schema === jsonDF2.schema)
+ assert(schema === jsonDF2.schema)
- jsonDF2.createOrReplaceTempView("jsonTable2")
+ jsonDF2.createOrReplaceTempView("jsonTable2")
- checkAnswer(
- sql("select * from jsonTable2"),
- Row(new java.math.BigDecimal("92233720368547758070"),
- true,
- 1.7976931348623157,
- 10,
- 21474836470L,
- null,
- "this is a simple string.")
- )
+ checkAnswer(
+ sql("select * from jsonTable2"),
+ Row(new java.math.BigDecimal("92233720368547758070"),
+ true,
+ 1.7976931348623157,
+ 10,
+ 21474836470L,
+ null,
+ "this is a simple string.")
+ )
+ }
}
test("Applying schemas with MapType") {
- val schemaWithSimpleMap = StructType(
- StructField("map", MapType(StringType, IntegerType, true), false) :: Nil)
- val jsonWithSimpleMap = spark.read.schema(schemaWithSimpleMap).json(mapType1)
-
- jsonWithSimpleMap.createOrReplaceTempView("jsonWithSimpleMap")
+ withTempView("jsonWithSimpleMap", "jsonWithComplexMap") {
+ val schemaWithSimpleMap = StructType(
+ StructField("map", MapType(StringType, IntegerType, true), false) :: Nil)
+ val jsonWithSimpleMap = spark.read.schema(schemaWithSimpleMap).json(mapType1)
- checkAnswer(
- sql("select `map` from jsonWithSimpleMap"),
- Row(Map("a" -> 1)) ::
- Row(Map("b" -> 2)) ::
- Row(Map("c" -> 3)) ::
- Row(Map("c" -> 1, "d" -> 4)) ::
- Row(Map("e" -> null)) :: Nil
- )
+ jsonWithSimpleMap.createOrReplaceTempView("jsonWithSimpleMap")
- withSQLConf(SQLConf.SUPPORT_QUOTED_REGEX_COLUMN_NAME.key -> "false") {
checkAnswer(
- sql("select `map`['c'] from jsonWithSimpleMap"),
- Row(null) ::
- Row(null) ::
- Row(3) ::
- Row(1) ::
- Row(null) :: Nil
+ sql("select `map` from jsonWithSimpleMap"),
+ Row(Map("a" -> 1)) ::
+ Row(Map("b" -> 2)) ::
+ Row(Map("c" -> 3)) ::
+ Row(Map("c" -> 1, "d" -> 4)) ::
+ Row(Map("e" -> null)) :: Nil
)
- }
- val innerStruct = StructType(
- StructField("field1", ArrayType(IntegerType, true), true) ::
- StructField("field2", IntegerType, true) :: Nil)
- val schemaWithComplexMap = StructType(
- StructField("map", MapType(StringType, innerStruct, true), false) :: Nil)
+ withSQLConf(SQLConf.SUPPORT_QUOTED_REGEX_COLUMN_NAME.key -> "false") {
+ checkAnswer(
+ sql("select `map`['c'] from jsonWithSimpleMap"),
+ Row(null) ::
+ Row(null) ::
+ Row(3) ::
+ Row(1) ::
+ Row(null) :: Nil
+ )
+ }
- val jsonWithComplexMap = spark.read.schema(schemaWithComplexMap).json(mapType2)
+ val innerStruct = StructType(
+ StructField("field1", ArrayType(IntegerType, true), true) ::
+ StructField("field2", IntegerType, true) :: Nil)
+ val schemaWithComplexMap = StructType(
+ StructField("map", MapType(StringType, innerStruct, true), false) :: Nil)
- jsonWithComplexMap.createOrReplaceTempView("jsonWithComplexMap")
+ val jsonWithComplexMap = spark.read.schema(schemaWithComplexMap).json(mapType2)
- checkAnswer(
- sql("select `map` from jsonWithComplexMap"),
- Row(Map("a" -> Row(Seq(1, 2, 3, null), null))) ::
- Row(Map("b" -> Row(null, 2))) ::
- Row(Map("c" -> Row(Seq(), 4))) ::
- Row(Map("c" -> Row(null, 3), "d" -> Row(Seq(null), null))) ::
- Row(Map("e" -> null)) ::
- Row(Map("f" -> Row(null, null))) :: Nil
- )
+ jsonWithComplexMap.createOrReplaceTempView("jsonWithComplexMap")
- withSQLConf(SQLConf.SUPPORT_QUOTED_REGEX_COLUMN_NAME.key -> "false") {
checkAnswer(
- sql("select `map`['a'].field1, `map`['c'].field2 from jsonWithComplexMap"),
- Row(Seq(1, 2, 3, null), null) ::
- Row(null, null) ::
- Row(null, 4) ::
- Row(null, 3) ::
- Row(null, null) ::
- Row(null, null) :: Nil
+ sql("select `map` from jsonWithComplexMap"),
+ Row(Map("a" -> Row(Seq(1, 2, 3, null), null))) ::
+ Row(Map("b" -> Row(null, 2))) ::
+ Row(Map("c" -> Row(Seq(), 4))) ::
+ Row(Map("c" -> Row(null, 3), "d" -> Row(Seq(null), null))) ::
+ Row(Map("e" -> null)) ::
+ Row(Map("f" -> Row(null, null))) :: Nil
)
+
+ withSQLConf(SQLConf.SUPPORT_QUOTED_REGEX_COLUMN_NAME.key -> "false") {
+ checkAnswer(
+ sql("select `map`['a'].field1, `map`['c'].field2 from jsonWithComplexMap"),
+ Row(Seq(1, 2, 3, null), null) ::
+ Row(null, null) ::
+ Row(null, 4) ::
+ Row(null, 3) ::
+ Row(null, null) ::
+ Row(null, null) :: Nil
+ )
+ }
}
}
test("SPARK-2096 Correctly parse dot notations") {
- val jsonDF = spark.read.json(complexFieldAndType2)
- jsonDF.createOrReplaceTempView("jsonTable")
+ withTempView("jsonTable") {
+ val jsonDF = spark.read.json(complexFieldAndType2)
+ jsonDF.createOrReplaceTempView("jsonTable")
- checkAnswer(
- sql("select arrayOfStruct[0].field1, arrayOfStruct[0].field2 from jsonTable"),
- Row(true, "str1")
- )
- checkAnswer(
- sql(
- """
- |select complexArrayOfStruct[0].field1[1].inner2[0], complexArrayOfStruct[1].field2[0][1]
- |from jsonTable
+ checkAnswer(
+ sql("select arrayOfStruct[0].field1, arrayOfStruct[0].field2 from jsonTable"),
+ Row(true, "str1")
+ )
+ checkAnswer(
+ sql(
+ """
+ |select complexArrayOfStruct[0].field1[1].inner2[0],
Review comment:
nit: wrong indents.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #28239: [SPARK-31467][SQL][TEST]
Refactor the sql tests to prevent TableAlreadyExistsException
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #28239: [SPARK-31467][SQL][TEST] Refactor the sql tests to prevent TableAlreadyExistsException
URL: https://github.com/apache/spark/pull/28239#issuecomment-616031847
**[Test build #121466 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/121466/testReport)** for PR 28239 at commit [`d699f58`](https://github.com/apache/spark/commit/d699f58b4eca6f8fe7ae1fb9b2ff0e92e0e4f013).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #28239:
[SPARK-31467][SQL][TEST] Refactor the sql tests to prevent
TableAlreadyExistsException
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #28239: [SPARK-31467][SQL][TEST] Refactor the sql tests to prevent TableAlreadyExistsException
URL: https://github.com/apache/spark/pull/28239#issuecomment-616011120
**[Test build #121466 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/121466/testReport)** for PR 28239 at commit [`d699f58`](https://github.com/apache/spark/commit/d699f58b4eca6f8fe7ae1fb9b2ff0e92e0e4f013).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28239:
[SPARK-31467][SQL][TEST] Refactor the sql tests to prevent
TableAlreadyExistsException
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #28239: [SPARK-31467][SQL][TEST] Refactor the sql tests to prevent TableAlreadyExistsException
URL: https://github.com/apache/spark/pull/28239#issuecomment-616032050
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28239:
[SPARK-31467][SQL][TEST] Refactor the sql tests to prevent
TableAlreadyExistsException
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #28239: [SPARK-31467][SQL][TEST] Refactor the sql tests to prevent TableAlreadyExistsException
URL: https://github.com/apache/spark/pull/28239#issuecomment-616087511
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] turboFei commented on issue #28239: SPARK-31467: Fix issue
when adding UT uses table named `test` in hive/SQLQuerySuite
Posted by GitBox <gi...@apache.org>.
turboFei commented on issue #28239: SPARK-31467: Fix issue when adding UT uses table named `test` in hive/SQLQuerySuite
URL: https://github.com/apache/spark/pull/28239#issuecomment-615132398
Just add withTempView for the tests that do not clean up used tables/views.
cc @HyukjinKwon
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28239:
SPARK-31467: Fix issue when adding UT uses table named `test` in
hive/SQLQuerySuite
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #28239: SPARK-31467: Fix issue when adding UT uses table named `test` in hive/SQLQuerySuite
URL: https://github.com/apache/spark/pull/28239#issuecomment-615113916
Can one of the admins verify this patch?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #28239: [SPARK-31467][SQL][TEST]
Refactor the tests in hive/SQLQuerySuite to prevent
TableAlreadyExistsException
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #28239: [SPARK-31467][SQL][TEST] Refactor the tests in hive/SQLQuerySuite to prevent TableAlreadyExistsException
URL: https://github.com/apache/spark/pull/28239#issuecomment-615998786
**[Test build #121460 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/121460/testReport)** for PR 28239 at commit [`cdde32f`](https://github.com/apache/spark/commit/cdde32f9fba62899ade205d8bdf93dc09fd34948).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #28239:
[SPARK-31467][SQL][TEST] Refactor the sql tests to prevent
TableAlreadyExistsException
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #28239: [SPARK-31467][SQL][TEST] Refactor the sql tests to prevent TableAlreadyExistsException
URL: https://github.com/apache/spark/pull/28239#issuecomment-616029708
**[Test build #121470 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/121470/testReport)** for PR 28239 at commit [`bb52763`](https://github.com/apache/spark/commit/bb52763b6e3ed55285e25811bf23b92b4d8c4394).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28239:
[SPARK-31467][SQL][TEST] Refactor the sql tests to prevent
TableAlreadyExistsException
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #28239: [SPARK-31467][SQL][TEST] Refactor the sql tests to prevent TableAlreadyExistsException
URL: https://github.com/apache/spark/pull/28239#issuecomment-616212515
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/121477/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28239:
[SPARK-31467][SQL][TEST] Refactor the sql tests to prevent
TableAlreadyExistsException
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #28239: [SPARK-31467][SQL][TEST] Refactor the sql tests to prevent TableAlreadyExistsException
URL: https://github.com/apache/spark/pull/28239#issuecomment-616029887
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/26154/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28239:
[SPARK-31467][SQL][TEST] Refactor the tests in hive/SQLQuerySuite to
prevent TableAlreadyExistsException
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #28239: [SPARK-31467][SQL][TEST] Refactor the tests in hive/SQLQuerySuite to prevent TableAlreadyExistsException
URL: https://github.com/apache/spark/pull/28239#issuecomment-615966697
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/26144/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28239:
[SPARK-31467][SQL][TEST] Refactor the sql tests to prevent
TableAlreadyExistsException
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #28239: [SPARK-31467][SQL][TEST] Refactor the sql tests to prevent TableAlreadyExistsException
URL: https://github.com/apache/spark/pull/28239#issuecomment-616029887
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/26154/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28239:
[SPARK-31467][SQL][TEST] Refactor the sql tests to prevent
TableAlreadyExistsException
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #28239: [SPARK-31467][SQL][TEST] Refactor the sql tests to prevent TableAlreadyExistsException
URL: https://github.com/apache/spark/pull/28239#issuecomment-616029885
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu commented on issue #28239: [SPARK-31467][SQL][TEST]
Refactor the sql tests to prevent TableAlreadyExistsException
Posted by GitBox <gi...@apache.org>.
maropu commented on issue #28239: [SPARK-31467][SQL][TEST] Refactor the sql tests to prevent TableAlreadyExistsException
URL: https://github.com/apache/spark/pull/28239#issuecomment-616088399
Ur, I see. cc: @dongjoon-hyun
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28239:
[SPARK-31467][SQL][TEST] Refactor the tests in hive/SQLQuerySuite to
prevent TableAlreadyExistsException
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #28239: [SPARK-31467][SQL][TEST] Refactor the tests in hive/SQLQuerySuite to prevent TableAlreadyExistsException
URL: https://github.com/apache/spark/pull/28239#issuecomment-615966690
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28239:
[SPARK-31467][SQL][TEST] Refactor the sql tests to prevent
TableAlreadyExistsException
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #28239: [SPARK-31467][SQL][TEST] Refactor the sql tests to prevent TableAlreadyExistsException
URL: https://github.com/apache/spark/pull/28239#issuecomment-616087513
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/26157/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28239:
[SPARK-31467][SQL][TEST] Refactor the sql tests to prevent
TableAlreadyExistsException
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #28239: [SPARK-31467][SQL][TEST] Refactor the sql tests to prevent TableAlreadyExistsException
URL: https://github.com/apache/spark/pull/28239#issuecomment-616032052
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/121466/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28239: SPARK-31467: Fix
test issue with table named in hive/SQLQuerySuite
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #28239: SPARK-31467: Fix test issue with table named in hive/SQLQuerySuite
URL: https://github.com/apache/spark/pull/28239#issuecomment-615114395
Can one of the admins verify this patch?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #28239: [SPARK-31467][SQL][TEST]
Refactor the sql tests to prevent TableAlreadyExistsException
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #28239: [SPARK-31467][SQL][TEST] Refactor the sql tests to prevent TableAlreadyExistsException
URL: https://github.com/apache/spark/pull/28239#issuecomment-616140284
**[Test build #121473 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/121473/testReport)** for PR 28239 at commit [`bb52763`](https://github.com/apache/spark/commit/bb52763b6e3ed55285e25811bf23b92b4d8c4394).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28239:
[SPARK-31467][SQL][TEST] Refactor the sql tests to prevent
TableAlreadyExistsException
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #28239: [SPARK-31467][SQL][TEST] Refactor the sql tests to prevent TableAlreadyExistsException
URL: https://github.com/apache/spark/pull/28239#issuecomment-616156339
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #28239: [SPARK-31467][SQL][TEST]
Refactor the sql tests to prevent TableAlreadyExistsException
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #28239: [SPARK-31467][SQL][TEST] Refactor the sql tests to prevent TableAlreadyExistsException
URL: https://github.com/apache/spark/pull/28239#issuecomment-616061650
**[Test build #121468 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/121468/testReport)** for PR 28239 at commit [`babffc7`](https://github.com/apache/spark/commit/babffc701603d7b1243d739f26794b1eb4f09a74).
* This patch **fails due to an unknown error code, -9**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28239:
[SPARK-31467][SQL][TEST] Refactor the sql tests to prevent
TableAlreadyExistsException
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #28239: [SPARK-31467][SQL][TEST] Refactor the sql tests to prevent TableAlreadyExistsException
URL: https://github.com/apache/spark/pull/28239#issuecomment-616156344
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/26161/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28239:
[SPARK-31467][SQL][TEST] Refactor the sql tests to prevent
TableAlreadyExistsException
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #28239: [SPARK-31467][SQL][TEST] Refactor the sql tests to prevent TableAlreadyExistsException
URL: https://github.com/apache/spark/pull/28239#issuecomment-616156344
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/26161/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28239:
[SPARK-31467][SQL][TEST] Refactor the sql tests to prevent
TableAlreadyExistsException
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #28239: [SPARK-31467][SQL][TEST] Refactor the sql tests to prevent TableAlreadyExistsException
URL: https://github.com/apache/spark/pull/28239#issuecomment-616212513
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] turboFei commented on issue #28239:
[SPARK-31467][SQL][TEST] Refactor the sql tests to prevent
TableAlreadyExistsException
Posted by GitBox <gi...@apache.org>.
turboFei commented on issue #28239: [SPARK-31467][SQL][TEST] Refactor the sql tests to prevent TableAlreadyExistsException
URL: https://github.com/apache/spark/pull/28239#issuecomment-616029679
I have used `grep` command to find all tests, which does clean up their used views, and refactor them.
cc @maropu
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28239:
[SPARK-31467][SQL][TEST] Refactor the tests in hive/SQLQuerySuite to
prevent TableAlreadyExistsException
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #28239: [SPARK-31467][SQL][TEST] Refactor the tests in hive/SQLQuerySuite to prevent TableAlreadyExistsException
URL: https://github.com/apache/spark/pull/28239#issuecomment-616014476
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/121467/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28239:
[SPARK-31467][SQL][TEST] Refactor the sql tests to prevent
TableAlreadyExistsException
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #28239: [SPARK-31467][SQL][TEST] Refactor the sql tests to prevent TableAlreadyExistsException
URL: https://github.com/apache/spark/pull/28239#issuecomment-616015377
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/26152/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28239:
[SPARK-31467][SQL][TEST] Refactor the sql tests to prevent
TableAlreadyExistsException
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #28239: [SPARK-31467][SQL][TEST] Refactor the sql tests to prevent TableAlreadyExistsException
URL: https://github.com/apache/spark/pull/28239#issuecomment-616061768
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/121470/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28239:
[SPARK-31467][SQL][TEST] Refactor the tests in hive/SQLQuerySuite to
prevent TableAlreadyExistsException
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #28239: [SPARK-31467][SQL][TEST] Refactor the tests in hive/SQLQuerySuite to prevent TableAlreadyExistsException
URL: https://github.com/apache/spark/pull/28239#issuecomment-616014350
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/26151/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #28239: [SPARK-31467][SQL][TEST]
Refactor the sql tests to prevent TableAlreadyExistsException
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #28239: [SPARK-31467][SQL][TEST] Refactor the sql tests to prevent TableAlreadyExistsException
URL: https://github.com/apache/spark/pull/28239#issuecomment-616015147
**[Test build #121468 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/121468/testReport)** for PR 28239 at commit [`babffc7`](https://github.com/apache/spark/commit/babffc701603d7b1243d739f26794b1eb4f09a74).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28239:
[SPARK-31467][SQL][TEST] Refactor the tests in hive/SQLQuerySuite to
prevent TableAlreadyExistsException
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #28239: [SPARK-31467][SQL][TEST] Refactor the tests in hive/SQLQuerySuite to prevent TableAlreadyExistsException
URL: https://github.com/apache/spark/pull/28239#issuecomment-616014474
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28239:
[SPARK-31467][SQL][TEST] Refactor the sql tests to prevent
TableAlreadyExistsException
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #28239: [SPARK-31467][SQL][TEST] Refactor the sql tests to prevent TableAlreadyExistsException
URL: https://github.com/apache/spark/pull/28239#issuecomment-616061768
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/121470/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu commented on issue #28239: [SPARK-31467][SQL][TEST]
Refactor the sql tests to prevent TableAlreadyExistsException
Posted by GitBox <gi...@apache.org>.
maropu commented on issue #28239: [SPARK-31467][SQL][TEST] Refactor the sql tests to prevent TableAlreadyExistsException
URL: https://github.com/apache/spark/pull/28239#issuecomment-616086332
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28239:
[SPARK-31467][SQL][TEST] Refactor the tests in hive/SQLQuerySuite to
prevent TableAlreadyExistsException
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #28239: [SPARK-31467][SQL][TEST] Refactor the tests in hive/SQLQuerySuite to prevent TableAlreadyExistsException
URL: https://github.com/apache/spark/pull/28239#issuecomment-616014474
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28239:
[SPARK-31467][SQL][TEST] Refactor the sql tests to prevent
TableAlreadyExistsException
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #28239: [SPARK-31467][SQL][TEST] Refactor the sql tests to prevent TableAlreadyExistsException
URL: https://github.com/apache/spark/pull/28239#issuecomment-616015377
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/26152/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #28239: [SPARK-31467][SQL][TEST]
Refactor the tests in hive/SQLQuerySuite to prevent
TableAlreadyExistsException
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #28239: [SPARK-31467][SQL][TEST] Refactor the tests in hive/SQLQuerySuite to prevent TableAlreadyExistsException
URL: https://github.com/apache/spark/pull/28239#issuecomment-616014471
**[Test build #121467 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/121467/testReport)** for PR 28239 at commit [`565631b`](https://github.com/apache/spark/commit/565631b3b97ede0ca81d3141ca771842322702cb).
* This patch **fails Scala style tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #28239:
[SPARK-31467][SQL][TEST] Refactor the tests in hive/SQLQuerySuite to
prevent TableAlreadyExistsException
Posted by GitBox <gi...@apache.org>.
maropu commented on a change in pull request #28239: [SPARK-31467][SQL][TEST] Refactor the tests in hive/SQLQuerySuite to prevent TableAlreadyExistsException
URL: https://github.com/apache/spark/pull/28239#discussion_r410762784
##########
File path: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/SQLQuerySuite.scala
##########
@@ -974,47 +995,57 @@ class SQLQuerySuite extends QueryTest with SQLTestUtils with TestHiveSingleton {
}
test("SPARK-5203 union with different decimal precision") {
- Seq.empty[(java.math.BigDecimal, java.math.BigDecimal)]
- .toDF("d1", "d2")
- .select($"d1".cast(DecimalType(10, 5)).as("d"))
- .createOrReplaceTempView("dn")
+ withTempView("dn") {
+ Seq.empty[(java.math.BigDecimal, java.math.BigDecimal)]
+ .toDF("d1", "d2")
+ .select($"d1".cast(DecimalType(10, 5)).as("d"))
+ .createOrReplaceTempView("dn")
- sql("select d from dn union all select d * 2 from dn")
- .queryExecution.analyzed
+ sql("select d from dn union all select d * 2 from dn")
+ .queryExecution.analyzed
+ }
}
test("Star Expansion - script transform") {
- assume(TestUtils.testCommandAvailable("/bin/bash"))
- val data = (1 to 100000).map { i => (i, i, i) }
- data.toDF("d1", "d2", "d3").createOrReplaceTempView("script_trans")
- assert(100000 === sql("SELECT TRANSFORM (*) USING 'cat' FROM script_trans").count())
+ withTempView("script_trans") {
+ assume(TestUtils.testCommandAvailable("/bin/bash"))
+ val data = (1 to 100000).map { i => (i, i, i) }
+ data.toDF("d1", "d2", "d3").createOrReplaceTempView("script_trans")
+ assert(100000 === sql("SELECT TRANSFORM (*) USING 'cat' FROM script_trans").count())
+ }
}
test("test script transform for stdout") {
- assume(TestUtils.testCommandAvailable("/bin/bash"))
- val data = (1 to 100000).map { i => (i, i, i) }
- data.toDF("d1", "d2", "d3").createOrReplaceTempView("script_trans")
- assert(100000 ===
- sql("SELECT TRANSFORM (d1, d2, d3) USING 'cat' AS (a,b,c) FROM script_trans").count())
+ withTempView("script_trans") {
+ assume(TestUtils.testCommandAvailable("/bin/bash"))
+ val data = (1 to 100000).map { i => (i, i, i) }
+ data.toDF("d1", "d2", "d3").createOrReplaceTempView("script_trans")
+ assert(100000 ===
+ sql("SELECT TRANSFORM (d1, d2, d3) USING 'cat' AS (a,b,c) FROM script_trans").count())
+ }
}
test("test script transform for stderr") {
- assume(TestUtils.testCommandAvailable("/bin/bash"))
- val data = (1 to 100000).map { i => (i, i, i) }
- data.toDF("d1", "d2", "d3").createOrReplaceTempView("script_trans")
- assert(0 ===
- sql("SELECT TRANSFORM (d1, d2, d3) USING 'cat 1>&2' AS (a,b,c) FROM script_trans").count())
+ withTempView("script_trans") {
+ assume(TestUtils.testCommandAvailable("/bin/bash"))
+ val data = (1 to 100000).map { i => (i, i, i) }
+ data.toDF("d1", "d2", "d3").createOrReplaceTempView("script_trans")
+ assert(0 ===
+ sql("SELECT TRANSFORM (d1, d2, d3) USING 'cat 1>&2' AS (a,b,c) FROM script_trans").count())
+ }
}
test("test script transform data type") {
- assume(TestUtils.testCommandAvailable("/bin/bash"))
- val data = (1 to 5).map { i => (i, i) }
- data.toDF("key", "value").createOrReplaceTempView("test")
- checkAnswer(
- sql("""FROM
- |(FROM test SELECT TRANSFORM(key, value) USING 'cat' AS (`thing1` int, thing2 string)) t
- |SELECT thing1 + 1
+ withTempView("test") {
+ assume(TestUtils.testCommandAvailable("/bin/bash"))
+ val data = (1 to 5).map { i => (i, i) }
+ data.toDF("key", "value").createOrReplaceTempView("test")
+ checkAnswer(
+ sql("""FROM
+ |(FROM test SELECT TRANSFORM(key, value) USING 'cat' AS (`thing1` int, thing2 string))
Review comment:
nit: wrong indents.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #28239:
[SPARK-31467][SQL][TEST] Refactor the sql tests to prevent
TableAlreadyExistsException
Posted by GitBox <gi...@apache.org>.
maropu commented on a change in pull request #28239: [SPARK-31467][SQL][TEST] Refactor the sql tests to prevent TableAlreadyExistsException
URL: https://github.com/apache/spark/pull/28239#discussion_r410866961
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/json/JsonSuite.scala
##########
@@ -833,171 +861,182 @@ abstract class JsonSuite extends QueryTest with SharedSparkSession with TestJson
}
test("Applying schemas") {
- val dir = Utils.createTempDir()
- dir.delete()
- val path = dir.getCanonicalPath
- primitiveFieldAndType.map(record => record.replaceAll("\n", " ")).write.text(path)
+ withTempView("jsonTable1", "jsonTable2") {
+ val dir = Utils.createTempDir()
+ dir.delete()
+ val path = dir.getCanonicalPath
+ primitiveFieldAndType.map(record => record.replaceAll("\n", " ")).write.text(path)
- val schema = StructType(
- StructField("bigInteger", DecimalType.SYSTEM_DEFAULT, true) ::
- StructField("boolean", BooleanType, true) ::
- StructField("double", DoubleType, true) ::
- StructField("integer", IntegerType, true) ::
- StructField("long", LongType, true) ::
- StructField("null", StringType, true) ::
- StructField("string", StringType, true) :: Nil)
+ val schema = StructType(
+ StructField("bigInteger", DecimalType.SYSTEM_DEFAULT, true) ::
+ StructField("boolean", BooleanType, true) ::
+ StructField("double", DoubleType, true) ::
+ StructField("integer", IntegerType, true) ::
+ StructField("long", LongType, true) ::
+ StructField("null", StringType, true) ::
+ StructField("string", StringType, true) :: Nil)
- val jsonDF1 = spark.read.schema(schema).json(path)
+ val jsonDF1 = spark.read.schema(schema).json(path)
- assert(schema === jsonDF1.schema)
+ assert(schema === jsonDF1.schema)
- jsonDF1.createOrReplaceTempView("jsonTable1")
+ jsonDF1.createOrReplaceTempView("jsonTable1")
- checkAnswer(
- sql("select * from jsonTable1"),
- Row(new java.math.BigDecimal("92233720368547758070"),
- true,
- 1.7976931348623157,
- 10,
- 21474836470L,
- null,
- "this is a simple string.")
- )
+ checkAnswer(
+ sql("select * from jsonTable1"),
+ Row(new java.math.BigDecimal("92233720368547758070"),
+ true,
+ 1.7976931348623157,
+ 10,
+ 21474836470L,
+ null,
+ "this is a simple string.")
+ )
- val jsonDF2 = spark.read.schema(schema).json(primitiveFieldAndType)
+ val jsonDF2 = spark.read.schema(schema).json(primitiveFieldAndType)
- assert(schema === jsonDF2.schema)
+ assert(schema === jsonDF2.schema)
- jsonDF2.createOrReplaceTempView("jsonTable2")
+ jsonDF2.createOrReplaceTempView("jsonTable2")
- checkAnswer(
- sql("select * from jsonTable2"),
- Row(new java.math.BigDecimal("92233720368547758070"),
- true,
- 1.7976931348623157,
- 10,
- 21474836470L,
- null,
- "this is a simple string.")
- )
+ checkAnswer(
+ sql("select * from jsonTable2"),
+ Row(new java.math.BigDecimal("92233720368547758070"),
+ true,
+ 1.7976931348623157,
+ 10,
+ 21474836470L,
+ null,
+ "this is a simple string.")
+ )
+ }
}
test("Applying schemas with MapType") {
- val schemaWithSimpleMap = StructType(
- StructField("map", MapType(StringType, IntegerType, true), false) :: Nil)
- val jsonWithSimpleMap = spark.read.schema(schemaWithSimpleMap).json(mapType1)
-
- jsonWithSimpleMap.createOrReplaceTempView("jsonWithSimpleMap")
+ withTempView("jsonWithSimpleMap", "jsonWithComplexMap") {
+ val schemaWithSimpleMap = StructType(
+ StructField("map", MapType(StringType, IntegerType, true), false) :: Nil)
+ val jsonWithSimpleMap = spark.read.schema(schemaWithSimpleMap).json(mapType1)
- checkAnswer(
- sql("select `map` from jsonWithSimpleMap"),
- Row(Map("a" -> 1)) ::
- Row(Map("b" -> 2)) ::
- Row(Map("c" -> 3)) ::
- Row(Map("c" -> 1, "d" -> 4)) ::
- Row(Map("e" -> null)) :: Nil
- )
+ jsonWithSimpleMap.createOrReplaceTempView("jsonWithSimpleMap")
- withSQLConf(SQLConf.SUPPORT_QUOTED_REGEX_COLUMN_NAME.key -> "false") {
checkAnswer(
- sql("select `map`['c'] from jsonWithSimpleMap"),
- Row(null) ::
- Row(null) ::
- Row(3) ::
- Row(1) ::
- Row(null) :: Nil
+ sql("select `map` from jsonWithSimpleMap"),
+ Row(Map("a" -> 1)) ::
+ Row(Map("b" -> 2)) ::
+ Row(Map("c" -> 3)) ::
+ Row(Map("c" -> 1, "d" -> 4)) ::
+ Row(Map("e" -> null)) :: Nil
)
- }
- val innerStruct = StructType(
- StructField("field1", ArrayType(IntegerType, true), true) ::
- StructField("field2", IntegerType, true) :: Nil)
- val schemaWithComplexMap = StructType(
- StructField("map", MapType(StringType, innerStruct, true), false) :: Nil)
+ withSQLConf(SQLConf.SUPPORT_QUOTED_REGEX_COLUMN_NAME.key -> "false") {
+ checkAnswer(
+ sql("select `map`['c'] from jsonWithSimpleMap"),
+ Row(null) ::
+ Row(null) ::
+ Row(3) ::
+ Row(1) ::
+ Row(null) :: Nil
+ )
+ }
- val jsonWithComplexMap = spark.read.schema(schemaWithComplexMap).json(mapType2)
+ val innerStruct = StructType(
+ StructField("field1", ArrayType(IntegerType, true), true) ::
+ StructField("field2", IntegerType, true) :: Nil)
+ val schemaWithComplexMap = StructType(
+ StructField("map", MapType(StringType, innerStruct, true), false) :: Nil)
- jsonWithComplexMap.createOrReplaceTempView("jsonWithComplexMap")
+ val jsonWithComplexMap = spark.read.schema(schemaWithComplexMap).json(mapType2)
- checkAnswer(
- sql("select `map` from jsonWithComplexMap"),
- Row(Map("a" -> Row(Seq(1, 2, 3, null), null))) ::
- Row(Map("b" -> Row(null, 2))) ::
- Row(Map("c" -> Row(Seq(), 4))) ::
- Row(Map("c" -> Row(null, 3), "d" -> Row(Seq(null), null))) ::
- Row(Map("e" -> null)) ::
- Row(Map("f" -> Row(null, null))) :: Nil
- )
+ jsonWithComplexMap.createOrReplaceTempView("jsonWithComplexMap")
- withSQLConf(SQLConf.SUPPORT_QUOTED_REGEX_COLUMN_NAME.key -> "false") {
checkAnswer(
- sql("select `map`['a'].field1, `map`['c'].field2 from jsonWithComplexMap"),
- Row(Seq(1, 2, 3, null), null) ::
- Row(null, null) ::
- Row(null, 4) ::
- Row(null, 3) ::
- Row(null, null) ::
- Row(null, null) :: Nil
+ sql("select `map` from jsonWithComplexMap"),
+ Row(Map("a" -> Row(Seq(1, 2, 3, null), null))) ::
+ Row(Map("b" -> Row(null, 2))) ::
+ Row(Map("c" -> Row(Seq(), 4))) ::
+ Row(Map("c" -> Row(null, 3), "d" -> Row(Seq(null), null))) ::
+ Row(Map("e" -> null)) ::
+ Row(Map("f" -> Row(null, null))) :: Nil
)
+
+ withSQLConf(SQLConf.SUPPORT_QUOTED_REGEX_COLUMN_NAME.key -> "false") {
+ checkAnswer(
+ sql("select `map`['a'].field1, `map`['c'].field2 from jsonWithComplexMap"),
+ Row(Seq(1, 2, 3, null), null) ::
+ Row(null, null) ::
+ Row(null, 4) ::
+ Row(null, 3) ::
+ Row(null, null) ::
+ Row(null, null) :: Nil
+ )
+ }
}
}
test("SPARK-2096 Correctly parse dot notations") {
- val jsonDF = spark.read.json(complexFieldAndType2)
- jsonDF.createOrReplaceTempView("jsonTable")
+ withTempView("jsonTable") {
+ val jsonDF = spark.read.json(complexFieldAndType2)
+ jsonDF.createOrReplaceTempView("jsonTable")
- checkAnswer(
- sql("select arrayOfStruct[0].field1, arrayOfStruct[0].field2 from jsonTable"),
- Row(true, "str1")
- )
- checkAnswer(
- sql(
- """
- |select complexArrayOfStruct[0].field1[1].inner2[0], complexArrayOfStruct[1].field2[0][1]
- |from jsonTable
+ checkAnswer(
+ sql("select arrayOfStruct[0].field1, arrayOfStruct[0].field2 from jsonTable"),
+ Row(true, "str1")
+ )
+ checkAnswer(
+ sql(
+ """
+ |select complexArrayOfStruct[0].field1[1].inner2[0],
+ |complexArrayOfStruct[1].field2[0][1]
+ |from jsonTable
""".stripMargin),
Review comment:
nit: wrong indents.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28239:
[SPARK-31467][SQL][TEST] Refactor the sql tests to prevent
TableAlreadyExistsException
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #28239: [SPARK-31467][SQL][TEST] Refactor the sql tests to prevent TableAlreadyExistsException
URL: https://github.com/apache/spark/pull/28239#issuecomment-616032052
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/121466/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #28239:
[SPARK-31467][SQL][TEST] Refactor the sql tests to prevent
TableAlreadyExistsException
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #28239: [SPARK-31467][SQL][TEST] Refactor the sql tests to prevent TableAlreadyExistsException
URL: https://github.com/apache/spark/pull/28239#issuecomment-616156014
**[Test build #121477 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/121477/testReport)** for PR 28239 at commit [`453c5a5`](https://github.com/apache/spark/commit/453c5a5e0717d2681fc2e0ed4f48ca093d4020a0).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] turboFei commented on issue #28239:
[SPARK-31467][SQL][TEST] Refactor the sql tests to prevent
TableAlreadyExistsException
Posted by GitBox <gi...@apache.org>.
turboFei commented on issue #28239: [SPARK-31467][SQL][TEST] Refactor the sql tests to prevent TableAlreadyExistsException
URL: https://github.com/apache/spark/pull/28239#issuecomment-616017124
Will try to refactor other suites.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28239:
[SPARK-31467][SQL][TEST] Refactor the sql tests to prevent
TableAlreadyExistsException
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #28239: [SPARK-31467][SQL][TEST] Refactor the sql tests to prevent TableAlreadyExistsException
URL: https://github.com/apache/spark/pull/28239#issuecomment-616029885
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28239:
[SPARK-31467][SQL][TEST] Refactor the sql tests to prevent
TableAlreadyExistsException
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #28239: [SPARK-31467][SQL][TEST] Refactor the sql tests to prevent TableAlreadyExistsException
URL: https://github.com/apache/spark/pull/28239#issuecomment-616212515
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/121477/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu edited a comment on issue #28239:
[SPARK-31467][SQL][TEST] Refactor the sql tests to prevent
TableAlreadyExistsException
Posted by GitBox <gi...@apache.org>.
maropu edited a comment on issue #28239: [SPARK-31467][SQL][TEST] Refactor the sql tests to prevent TableAlreadyExistsException
URL: https://github.com/apache/spark/pull/28239#issuecomment-615968152
I think (trivial though), the change looks ok. Did you check them only in `SQLQuerySuite`? I mean, have you checked if the logs of the other test suites have no `TableAlreadyExistsException`?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28239:
[SPARK-31467][SQL][TEST] Refactor the sql tests to prevent
TableAlreadyExistsException
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #28239: [SPARK-31467][SQL][TEST] Refactor the sql tests to prevent TableAlreadyExistsException
URL: https://github.com/apache/spark/pull/28239#issuecomment-616140714
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/121473/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28239:
[SPARK-31467][SQL][TEST] Refactor the tests in hive/SQLQuerySuite to
prevent TableAlreadyExistsException
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #28239: [SPARK-31467][SQL][TEST] Refactor the tests in hive/SQLQuerySuite to prevent TableAlreadyExistsException
URL: https://github.com/apache/spark/pull/28239#issuecomment-615966697
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/26144/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28239:
[SPARK-31467][SQL][TEST] Refactor the sql tests to prevent
TableAlreadyExistsException
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #28239: [SPARK-31467][SQL][TEST] Refactor the sql tests to prevent TableAlreadyExistsException
URL: https://github.com/apache/spark/pull/28239#issuecomment-616087511
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] turboFei edited a comment on issue #28239:
[SPARK-31467][SQL][TEST] Refactor the tests in hive/SQLQuerySuite to
prevent TableAlreadyExistsException
Posted by GitBox <gi...@apache.org>.
turboFei edited a comment on issue #28239: [SPARK-31467][SQL][TEST] Refactor the tests in hive/SQLQuerySuite to prevent TableAlreadyExistsException
URL: https://github.com/apache/spark/pull/28239#issuecomment-616014159
> I think the change looks reasonable. Did you check them only in `SQLQuerySuite`? I mean, have you checked if the logs of the other test suites have no `TableAlreadyExistsException`?
Hi, I found this issue when adding UT into hive/SQLQuerySuite.
There is no `TableAlreadyExistsException` in the logs before add UT.
I just check the sql/SQLQuerySuite and refactor it too.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28239:
[SPARK-31467][SQL][TEST] Refactor the tests in hive/SQLQuerySuite to
prevent TableAlreadyExistsException
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #28239: [SPARK-31467][SQL][TEST] Refactor the tests in hive/SQLQuerySuite to prevent TableAlreadyExistsException
URL: https://github.com/apache/spark/pull/28239#issuecomment-616011288
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/26150/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28239:
[SPARK-31467][SQL][TEST] Refactor the tests in hive/SQLQuerySuite to
prevent TableAlreadyExistsException
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #28239: [SPARK-31467][SQL][TEST] Refactor the tests in hive/SQLQuerySuite to prevent TableAlreadyExistsException
URL: https://github.com/apache/spark/pull/28239#issuecomment-615999012
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/121460/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28239:
[SPARK-31467][SQL][TEST] Refactor the tests in hive/SQLQuerySuite to
prevent TableAlreadyExistsException
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #28239: [SPARK-31467][SQL][TEST] Refactor the tests in hive/SQLQuerySuite to prevent TableAlreadyExistsException
URL: https://github.com/apache/spark/pull/28239#issuecomment-616011284
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28239: SPARK-31467: Fix
test issue with table named in hive/SQLQuerySuite
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #28239: SPARK-31467: Fix test issue with table named in hive/SQLQuerySuite
URL: https://github.com/apache/spark/pull/28239#issuecomment-615113916
Can one of the admins verify this patch?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28239:
[SPARK-31467][SQL][TEST] Refactor the sql tests to prevent
TableAlreadyExistsException
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #28239: [SPARK-31467][SQL][TEST] Refactor the sql tests to prevent TableAlreadyExistsException
URL: https://github.com/apache/spark/pull/28239#issuecomment-616062009
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28239:
[SPARK-31467][SQL][TEST] Refactor the sql tests to prevent
TableAlreadyExistsException
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #28239: [SPARK-31467][SQL][TEST] Refactor the sql tests to prevent TableAlreadyExistsException
URL: https://github.com/apache/spark/pull/28239#issuecomment-616015376
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28239:
[SPARK-31467][SQL][TEST] Refactor the tests in hive/SQLQuerySuite to
prevent TableAlreadyExistsException
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #28239: [SPARK-31467][SQL][TEST] Refactor the tests in hive/SQLQuerySuite to prevent TableAlreadyExistsException
URL: https://github.com/apache/spark/pull/28239#issuecomment-616014349
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #28239: [SPARK-31467][SQL][TEST]
Refactor the sql tests to prevent TableAlreadyExistsException
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #28239: [SPARK-31467][SQL][TEST] Refactor the sql tests to prevent TableAlreadyExistsException
URL: https://github.com/apache/spark/pull/28239#issuecomment-616156014
**[Test build #121477 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/121477/testReport)** for PR 28239 at commit [`453c5a5`](https://github.com/apache/spark/commit/453c5a5e0717d2681fc2e0ed4f48ca093d4020a0).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28239:
[SPARK-31467][SQL][TEST] Refactor the sql tests to prevent
TableAlreadyExistsException
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #28239: [SPARK-31467][SQL][TEST] Refactor the sql tests to prevent TableAlreadyExistsException
URL: https://github.com/apache/spark/pull/28239#issuecomment-616061766
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28239:
[SPARK-31467][SQL][TEST] Refactor the tests in hive/SQLQuerySuite to
prevent TableAlreadyExistsException
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #28239: [SPARK-31467][SQL][TEST] Refactor the tests in hive/SQLQuerySuite to prevent TableAlreadyExistsException
URL: https://github.com/apache/spark/pull/28239#issuecomment-616014350
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/26151/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] turboFei commented on a change in pull request #28239:
[SPARK-31467][SQL][TEST] Refactor the tests in hive/SQLQuerySuite to
prevent TableAlreadyExistsException
Posted by GitBox <gi...@apache.org>.
turboFei commented on a change in pull request #28239: [SPARK-31467][SQL][TEST] Refactor the tests in hive/SQLQuerySuite to prevent TableAlreadyExistsException
URL: https://github.com/apache/spark/pull/28239#discussion_r410795576
##########
File path: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/SQLQuerySuite.scala
##########
@@ -898,33 +913,39 @@ class SQLQuerySuite extends QueryTest with SQLTestUtils with TestHiveSingleton {
}
test("resolve udtf in projection #1") {
- val ds = (1 to 5).map(i => s"""{"a":[$i, ${i + 1}]}""").toDS()
- read.json(ds).createOrReplaceTempView("data")
- val df = sql("SELECT explode(a) AS val FROM data")
- val col = df("val")
+ withTempView("data") {
+ val ds = (1 to 5).map(i => s"""{"a":[$i, ${i + 1}]}""").toDS()
+ read.json(ds).createOrReplaceTempView("data")
+ val df = sql("SELECT explode(a) AS val FROM data")
+ val col = df("val")
Review comment:
removed
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu commented on issue #28239: [SPARK-31467][SQL][TEST]
Refactor the tests in hive/SQLQuerySuite to prevent
TableAlreadyExistsException
Posted by GitBox <gi...@apache.org>.
maropu commented on issue #28239: [SPARK-31467][SQL][TEST] Refactor the tests in hive/SQLQuerySuite to prevent TableAlreadyExistsException
URL: https://github.com/apache/spark/pull/28239#issuecomment-615968152
I think the change looks reasonable. Did you check them only in `SQLQuerySuite`? I mean, have you checked if the logs of the other test suites have no `TableAlreadyExistsException`?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #28239:
[SPARK-31467][SQL][TEST] Refactor the tests in hive/SQLQuerySuite to
prevent TableAlreadyExistsException
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #28239: [SPARK-31467][SQL][TEST] Refactor the tests in hive/SQLQuerySuite to prevent TableAlreadyExistsException
URL: https://github.com/apache/spark/pull/28239#issuecomment-615966442
**[Test build #121460 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/121460/testReport)** for PR 28239 at commit [`cdde32f`](https://github.com/apache/spark/commit/cdde32f9fba62899ade205d8bdf93dc09fd34948).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #28239: [SPARK-31467][SQL][TEST]
Refactor the sql tests to prevent TableAlreadyExistsException
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #28239: [SPARK-31467][SQL][TEST] Refactor the sql tests to prevent TableAlreadyExistsException
URL: https://github.com/apache/spark/pull/28239#issuecomment-616087130
**[Test build #121473 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/121473/testReport)** for PR 28239 at commit [`bb52763`](https://github.com/apache/spark/commit/bb52763b6e3ed55285e25811bf23b92b4d8c4394).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] turboFei commented on issue #28239:
[SPARK-31467][SQL][TEST] Refactor the sql tests to prevent
TableAlreadyExistsException
Posted by GitBox <gi...@apache.org>.
turboFei commented on issue #28239: [SPARK-31467][SQL][TEST] Refactor the sql tests to prevent TableAlreadyExistsException
URL: https://github.com/apache/spark/pull/28239#issuecomment-616087763
Oh, I mean that if we createOrReplaceTempView and does not clean up it after this test.
If another test create a table with same name, `TableAlreadyExistsException` would be thrown.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #28239:
[SPARK-31467][SQL][TEST] Refactor the sql tests to prevent
TableAlreadyExistsException
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #28239: [SPARK-31467][SQL][TEST] Refactor the sql tests to prevent TableAlreadyExistsException
URL: https://github.com/apache/spark/pull/28239#issuecomment-616015147
**[Test build #121468 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/121468/testReport)** for PR 28239 at commit [`babffc7`](https://github.com/apache/spark/commit/babffc701603d7b1243d739f26794b1eb4f09a74).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #28239:
[SPARK-31467][SQL][TEST] Refactor the sql tests to prevent
TableAlreadyExistsException
Posted by GitBox <gi...@apache.org>.
maropu commented on a change in pull request #28239: [SPARK-31467][SQL][TEST] Refactor the sql tests to prevent TableAlreadyExistsException
URL: https://github.com/apache/spark/pull/28239#discussion_r410864812
##########
File path: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/SQLQuerySuite.scala
##########
@@ -1236,43 +1273,47 @@ class SQLQuerySuite extends QueryTest with SQLTestUtils with TestHiveSingleton {
ignore("SPARK-10310: " +
"script transformation using default input/output SerDe and record reader/writer") {
- spark
- .range(5)
- .selectExpr("id AS a", "id AS b")
- .createOrReplaceTempView("test")
+ withTempView("test") {
+ spark
+ .range(5)
+ .selectExpr("id AS a", "id AS b")
+ .createOrReplaceTempView("test")
- val scriptFilePath = getTestResourcePath("data")
- checkAnswer(
- sql(
- s"""FROM(
- | FROM test SELECT TRANSFORM(a, b)
- | USING 'python $scriptFilePath/scripts/test_transform.py "\t"'
- | AS (c STRING, d STRING)
- |) t
- |SELECT c
+ val scriptFilePath = getTestResourcePath("data")
+ checkAnswer(
+ sql(
+ s"""FROM(
+ | FROM test SELECT TRANSFORM(a, b)
+ | USING 'python $scriptFilePath/scripts/test_transform.py "\t"'
+ | AS (c STRING, d STRING)
+ |) t
+ |SELECT c
""".stripMargin),
- (0 until 5).map(i => Row(i + "#")))
+ (0 until 5).map(i => Row(i + "#")))
+ }
}
ignore("SPARK-10310: script transformation using LazySimpleSerDe") {
- spark
- .range(5)
- .selectExpr("id AS a", "id AS b")
- .createOrReplaceTempView("test")
-
- val scriptFilePath = getTestResourcePath("data")
- val df = sql(
- s"""FROM test
- |SELECT TRANSFORM(a, b)
- |ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe'
- |WITH SERDEPROPERTIES('field.delim' = '|')
- |USING 'python $scriptFilePath/scripts/test_transform.py "|"'
- |AS (c STRING, d STRING)
- |ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe'
- |WITH SERDEPROPERTIES('field.delim' = '|')
+ withTempView("data") {
Review comment:
`date` -> `test`?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #28239: [SPARK-31467][SQL][TEST]
Refactor the tests in hive/SQLQuerySuite to prevent
TableAlreadyExistsException
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #28239: [SPARK-31467][SQL][TEST] Refactor the tests in hive/SQLQuerySuite to prevent TableAlreadyExistsException
URL: https://github.com/apache/spark/pull/28239#issuecomment-615966442
**[Test build #121460 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/121460/testReport)** for PR 28239 at commit [`cdde32f`](https://github.com/apache/spark/commit/cdde32f9fba62899ade205d8bdf93dc09fd34948).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #28239: [SPARK-31467][SQL][TEST]
Refactor the tests in hive/SQLQuerySuite to prevent
TableAlreadyExistsException
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #28239: [SPARK-31467][SQL][TEST] Refactor the tests in hive/SQLQuerySuite to prevent TableAlreadyExistsException
URL: https://github.com/apache/spark/pull/28239#issuecomment-616011120
**[Test build #121466 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/121466/testReport)** for PR 28239 at commit [`d699f58`](https://github.com/apache/spark/commit/d699f58b4eca6f8fe7ae1fb9b2ff0e92e0e4f013).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28239:
[SPARK-31467][SQL][TEST] Refactor the sql tests to prevent
TableAlreadyExistsException
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #28239: [SPARK-31467][SQL][TEST] Refactor the sql tests to prevent TableAlreadyExistsException
URL: https://github.com/apache/spark/pull/28239#issuecomment-616032050
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #28239:
[SPARK-31467][SQL][TEST] Refactor the sql tests to prevent
TableAlreadyExistsException
Posted by GitBox <gi...@apache.org>.
maropu commented on a change in pull request #28239: [SPARK-31467][SQL][TEST] Refactor the sql tests to prevent TableAlreadyExistsException
URL: https://github.com/apache/spark/pull/28239#discussion_r410864812
##########
File path: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/SQLQuerySuite.scala
##########
@@ -1236,43 +1273,47 @@ class SQLQuerySuite extends QueryTest with SQLTestUtils with TestHiveSingleton {
ignore("SPARK-10310: " +
"script transformation using default input/output SerDe and record reader/writer") {
- spark
- .range(5)
- .selectExpr("id AS a", "id AS b")
- .createOrReplaceTempView("test")
+ withTempView("test") {
+ spark
+ .range(5)
+ .selectExpr("id AS a", "id AS b")
+ .createOrReplaceTempView("test")
- val scriptFilePath = getTestResourcePath("data")
- checkAnswer(
- sql(
- s"""FROM(
- | FROM test SELECT TRANSFORM(a, b)
- | USING 'python $scriptFilePath/scripts/test_transform.py "\t"'
- | AS (c STRING, d STRING)
- |) t
- |SELECT c
+ val scriptFilePath = getTestResourcePath("data")
+ checkAnswer(
+ sql(
+ s"""FROM(
+ | FROM test SELECT TRANSFORM(a, b)
+ | USING 'python $scriptFilePath/scripts/test_transform.py "\t"'
+ | AS (c STRING, d STRING)
+ |) t
+ |SELECT c
""".stripMargin),
- (0 until 5).map(i => Row(i + "#")))
+ (0 until 5).map(i => Row(i + "#")))
+ }
}
ignore("SPARK-10310: script transformation using LazySimpleSerDe") {
- spark
- .range(5)
- .selectExpr("id AS a", "id AS b")
- .createOrReplaceTempView("test")
-
- val scriptFilePath = getTestResourcePath("data")
- val df = sql(
- s"""FROM test
- |SELECT TRANSFORM(a, b)
- |ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe'
- |WITH SERDEPROPERTIES('field.delim' = '|')
- |USING 'python $scriptFilePath/scripts/test_transform.py "|"'
- |AS (c STRING, d STRING)
- |ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe'
- |WITH SERDEPROPERTIES('field.delim' = '|')
+ withTempView("data") {
Review comment:
`data` -> `test`?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28239:
[SPARK-31467][SQL][TEST] Refactor the sql tests to prevent
TableAlreadyExistsException
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #28239: [SPARK-31467][SQL][TEST] Refactor the sql tests to prevent TableAlreadyExistsException
URL: https://github.com/apache/spark/pull/28239#issuecomment-616015376
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #28239:
[SPARK-31467][SQL][TEST] Refactor the tests in hive/SQLQuerySuite to
prevent TableAlreadyExistsException
Posted by GitBox <gi...@apache.org>.
maropu commented on a change in pull request #28239: [SPARK-31467][SQL][TEST] Refactor the tests in hive/SQLQuerySuite to prevent TableAlreadyExistsException
URL: https://github.com/apache/spark/pull/28239#discussion_r410762469
##########
File path: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/SQLQuerySuite.scala
##########
@@ -898,33 +913,39 @@ class SQLQuerySuite extends QueryTest with SQLTestUtils with TestHiveSingleton {
}
test("resolve udtf in projection #1") {
- val ds = (1 to 5).map(i => s"""{"a":[$i, ${i + 1}]}""").toDS()
- read.json(ds).createOrReplaceTempView("data")
- val df = sql("SELECT explode(a) AS val FROM data")
- val col = df("val")
+ withTempView("data") {
+ val ds = (1 to 5).map(i => s"""{"a":[$i, ${i + 1}]}""").toDS()
+ read.json(ds).createOrReplaceTempView("data")
+ val df = sql("SELECT explode(a) AS val FROM data")
+ val col = df("val")
Review comment:
(Not related to this pr though), `col` not used?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #28239:
[SPARK-31467][SQL][TEST] Refactor the tests in hive/SQLQuerySuite to
prevent TableAlreadyExistsException
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #28239: [SPARK-31467][SQL][TEST] Refactor the tests in hive/SQLQuerySuite to prevent TableAlreadyExistsException
URL: https://github.com/apache/spark/pull/28239#issuecomment-616014236
**[Test build #121467 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/121467/testReport)** for PR 28239 at commit [`565631b`](https://github.com/apache/spark/commit/565631b3b97ede0ca81d3141ca771842322702cb).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28239:
[SPARK-31467][SQL][TEST] Refactor the tests in hive/SQLQuerySuite to
prevent TableAlreadyExistsException
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #28239: [SPARK-31467][SQL][TEST] Refactor the tests in hive/SQLQuerySuite to prevent TableAlreadyExistsException
URL: https://github.com/apache/spark/pull/28239#issuecomment-615966690
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] turboFei edited a comment on issue #28239:
[SPARK-31467][SQL][TEST] Refactor the sql tests to prevent
TableAlreadyExistsException
Posted by GitBox <gi...@apache.org>.
turboFei edited a comment on issue #28239: [SPARK-31467][SQL][TEST] Refactor the sql tests to prevent TableAlreadyExistsException
URL: https://github.com/apache/spark/pull/28239#issuecomment-616096011
thanks, will fix all indents related with `""" """`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28239:
[SPARK-31467][SQL][TEST] Refactor the sql tests to prevent
TableAlreadyExistsException
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #28239: [SPARK-31467][SQL][TEST] Refactor the sql tests to prevent TableAlreadyExistsException
URL: https://github.com/apache/spark/pull/28239#issuecomment-616156339
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28239:
[SPARK-31467][SQL][TEST] Refactor the sql tests to prevent
TableAlreadyExistsException
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #28239: [SPARK-31467][SQL][TEST] Refactor the sql tests to prevent TableAlreadyExistsException
URL: https://github.com/apache/spark/pull/28239#issuecomment-616140708
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28239:
[SPARK-31467][SQL][TEST] Refactor the tests in hive/SQLQuerySuite to
prevent TableAlreadyExistsException
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #28239: [SPARK-31467][SQL][TEST] Refactor the tests in hive/SQLQuerySuite to prevent TableAlreadyExistsException
URL: https://github.com/apache/spark/pull/28239#issuecomment-616014476
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/121467/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28239:
[SPARK-31467][SQL][TEST] Refactor the sql tests to prevent
TableAlreadyExistsException
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #28239: [SPARK-31467][SQL][TEST] Refactor the sql tests to prevent TableAlreadyExistsException
URL: https://github.com/apache/spark/pull/28239#issuecomment-616140708
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #28239: [SPARK-31467][SQL][TEST]
Refactor the sql tests to prevent TableAlreadyExistsException
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #28239: [SPARK-31467][SQL][TEST] Refactor the sql tests to prevent TableAlreadyExistsException
URL: https://github.com/apache/spark/pull/28239#issuecomment-616029708
**[Test build #121470 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/121470/testReport)** for PR 28239 at commit [`bb52763`](https://github.com/apache/spark/commit/bb52763b6e3ed55285e25811bf23b92b4d8c4394).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #28239:
[SPARK-31467][SQL][TEST] Refactor the sql tests to prevent
TableAlreadyExistsException
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #28239: [SPARK-31467][SQL][TEST] Refactor the sql tests to prevent TableAlreadyExistsException
URL: https://github.com/apache/spark/pull/28239#issuecomment-616087130
**[Test build #121473 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/121473/testReport)** for PR 28239 at commit [`bb52763`](https://github.com/apache/spark/commit/bb52763b6e3ed55285e25811bf23b92b4d8c4394).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] turboFei commented on issue #28239:
[SPARK-31467][SQL][TEST] Refactor the sql tests to prevent
TableAlreadyExistsException
Posted by GitBox <gi...@apache.org>.
turboFei commented on issue #28239: [SPARK-31467][SQL][TEST] Refactor the sql tests to prevent TableAlreadyExistsException
URL: https://github.com/apache/spark/pull/28239#issuecomment-616096011
thanks, would fix all indents related with `""" """`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu commented on issue #28239: [SPARK-31467][SQL][TEST]
Refactor the sql tests to prevent TableAlreadyExistsException
Posted by GitBox <gi...@apache.org>.
maropu commented on issue #28239: [SPARK-31467][SQL][TEST] Refactor the sql tests to prevent TableAlreadyExistsException
URL: https://github.com/apache/spark/pull/28239#issuecomment-616087306
I think `createOrReplaceTempView` doesn't throw `TableAlreadyExistsException` without `withTempView`. Does the title describe a different issue?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28239:
[SPARK-31467][SQL][TEST] Refactor the sql tests to prevent
TableAlreadyExistsException
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #28239: [SPARK-31467][SQL][TEST] Refactor the sql tests to prevent TableAlreadyExistsException
URL: https://github.com/apache/spark/pull/28239#issuecomment-616061766
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org