You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by viirya <gi...@git.apache.org> on 2018/05/19 13:20:19 UTC
[GitHub] spark pull request #21371: [SPARK-24250][SQL][FollowUp] Fix compile error by...
GitHub user viirya opened a pull request:
https://github.com/apache/spark/pull/21371
[SPARK-24250][SQL][FollowUp] Fix compile error by adding needed import back
## What changes were proposed in this pull request?
Two recent commits changing SQLConf at the same time and accidentally make compile error in current code base. This patch goes to add necessary import back.
## How was this patch tested?
N/A.
Please review http://spark.apache.org/contributing.html before opening a pull request.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/viirya/spark-1 SPARK-24250-hot-fix
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/21371.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #21371
----
commit 7b193f6bd55838210936716f2c428ae2cc26489d
Author: Liang-Chi Hsieh <vi...@...>
Date: 2018-05-19T13:16:58Z
Add needed import back.
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21371: [SPARK-24250][SQL][FollowUp] Fix compile error and flaky...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21371
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21371: [SPARK-24250][SQL][FollowUp] Fix compile error and flaky...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21371
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/90849/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21371: [SPARK-24250][SQL][FollowUp] Fix compile error by adding...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21371
**[Test build #90836 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90836/testReport)** for PR 21371 at commit [`7b193f6`](https://github.com/apache/spark/commit/7b193f6bd55838210936716f2c428ae2cc26489d).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21371: [SPARK-24250][SQL][FollowUp] Fix compile error and flaky...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/21371
retest this please.
On Sun, May 20, 2018, 12:24 PM Apache Spark QA <no...@github.com>
wrote:
> *Test build #90844 has finished
> <https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90844/testReport>*
> for PR 21371 at commit a823841
> <https://github.com/apache/spark/commit/a823841fd419541137fd02d9a02822b1fa36f5ae>
> .
>
> - This patch *fails Spark unit tests*.
> - This patch merges cleanly.
> - This patch adds no public classes.
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/apache/spark/pull/21371#issuecomment-390454799>, or mute
> the thread
> <https://github.com/notifications/unsubscribe-auth/AAEM93shyk6XfCzlQXX_b7qcigEBKJ6Zks5t0OHpgaJpZM4UFtQ3>
> .
>
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21371: [SPARK-24250][SQL][FollowUp] Fix compile error by adding...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21371
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21371: [SPARK-24250][SQL][FollowUp] Fix compile error and flaky...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21371
**[Test build #90848 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90848/testReport)** for PR 21371 at commit [`a823841`](https://github.com/apache/spark/commit/a823841fd419541137fd02d9a02822b1fa36f5ae).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21371: [SPARK-24250][SQL][FollowUp] Fix compile error and flaky...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21371
**[Test build #90844 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90844/testReport)** for PR 21371 at commit [`a823841`](https://github.com/apache/spark/commit/a823841fd419541137fd02d9a02822b1fa36f5ae).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21371: [SPARK-24250][SQL][FollowUp] Fix compile error an...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/21371#discussion_r189455399
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/json/JsonInferSchema.scala ---
@@ -66,8 +69,12 @@ private[sql] object JsonInferSchema {
s"Parse Mode: ${FailFastMode.name}.", e)
}
}
- }
- }.fold(StructType(Nil))(
+ }.fold(StructType(Nil))(
+ compatibleRootType(columnNameOfCorruptRecord, parseMode))
+ Iterator(typeInPartition)
+ }.collect()
--- End diff --
`RDD.fold` won't allocate an array for keeping the result. It merges the data types right after a task is finished and return a data type.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21371: [SPARK-24250][SQL][FollowUp] Fix compile error an...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/21371#discussion_r189454251
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/json/JsonInferSchema.scala ---
@@ -66,8 +69,12 @@ private[sql] object JsonInferSchema {
s"Parse Mode: ${FailFastMode.name}.", e)
}
}
- }
- }.fold(StructType(Nil))(
+ }.fold(StructType(Nil))(
+ compatibleRootType(columnNameOfCorruptRecord, parseMode))
+ Iterator(typeInPartition)
+ }.collect()
--- End diff --
> good catch! but wondering how the test passed in my PR...
It is somehow flaky. If all types are folded at executor sides, when they are going to fold at local, it just merging `StructType()` and `StructType(StructField("id"), StructField("ID"))`. So you can still get current schema back.
But if unfortunately, you have one partition with only `id` column, you need to merge `StructType(StructField("id"))` and `StructType(StructField("ID")` in local. Then the problem will happen.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21371: [SPARK-24250][SQL][FollowUp] Fix compile error by adding...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21371
**[Test build #90836 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90836/testReport)** for PR 21371 at commit [`7b193f6`](https://github.com/apache/spark/commit/7b193f6bd55838210936716f2c428ae2cc26489d).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21371: [SPARK-24250][SQL][FollowUp] Fix compile error and flaky...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/21371
finally...
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21371: [SPARK-24250][SQL][FollowUp] Fix compile error by adding...
Posted by hvanhovell <gi...@git.apache.org>.
Github user hvanhovell commented on the issue:
https://github.com/apache/spark/pull/21371
@viirya the test failure seems to genuine, can you take another look?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21371: [SPARK-24250][SQL][FollowUp] Fix compile error by adding...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/21371
@hvanhovell I ran locally and seems no such test failure? Checking.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21371: [SPARK-24250][SQL][FollowUp] Fix compile error and flaky...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21371
**[Test build #90849 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90849/testReport)** for PR 21371 at commit [`a823841`](https://github.com/apache/spark/commit/a823841fd419541137fd02d9a02822b1fa36f5ae).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21371: [SPARK-24250][SQL][FollowUp] Fix compile error by adding...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21371
**[Test build #90840 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90840/testReport)** for PR 21371 at commit [`7b193f6`](https://github.com/apache/spark/commit/7b193f6bd55838210936716f2c428ae2cc26489d).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21371: [SPARK-24250][SQL][FollowUp] Fix compile error and flaky...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21371
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/90844/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21371: [SPARK-24250][SQL][FollowUp] Fix compile error by adding...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21371
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21371: [SPARK-24250][SQL][FollowUp] Fix compile error by...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/21371#discussion_r189448749
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/json/JsonInferSchema.scala ---
@@ -66,8 +69,12 @@ private[sql] object JsonInferSchema {
s"Parse Mode: ${FailFastMode.name}.", e)
}
}
- }
- }.fold(StructType(Nil))(
+ }.fold(StructType(Nil))(
+ compatibleRootType(columnNameOfCorruptRecord, parseMode))
+ Iterator(typeInPartition)
+ }.collect()
--- End diff --
Previously two foldings with `compatibleRootType` on executors and local in one `RDD.fold` call all use default SQL config values. Now on executors SQL config can work, so inconsistency happens because local folding uses default SQL config.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21371: [SPARK-24250][SQL][FollowUp] Fix compile error and flaky...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/21371
retest this please...
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21371: [SPARK-24250][SQL][FollowUp] Fix compile error and flaky...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21371
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21371: [SPARK-24250][SQL][FollowUp] Fix compile error by adding...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21371
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/90836/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21371: [SPARK-24250][SQL][FollowUp] Fix compile error and flaky...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21371
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/90845/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21371: [SPARK-24250][SQL][FollowUp] Fix compile error by adding...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21371
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21371: [SPARK-24250][SQL][FollowUp] Fix compile error and flaky...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21371
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21371: [SPARK-24250][SQL][FollowUp] Fix compile error by adding...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/21371
cc @cloud-fan @HyukjinKwon
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21371: [SPARK-24250][SQL][FollowUp] Fix compile error an...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/21371#discussion_r189454283
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/json/JsonInferSchema.scala ---
@@ -66,8 +69,12 @@ private[sql] object JsonInferSchema {
s"Parse Mode: ${FailFastMode.name}.", e)
}
}
- }
- }.fold(StructType(Nil))(
+ }.fold(StructType(Nil))(
+ compatibleRootType(columnNameOfCorruptRecord, parseMode))
+ Iterator(typeInPartition)
+ }.collect()
--- End diff --
> the collect here may cause OOM while the previous code doesn't have this problem. Let's revert my PR first and think about how to fix it.
We just iterate rows and fold a data type. So I think it shouldn't be OOM since we don't load all data at once?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21371: [SPARK-24250][SQL][FollowUp] Fix compile error by adding...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21371
**[Test build #90841 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90841/testReport)** for PR 21371 at commit [`d010c17`](https://github.com/apache/spark/commit/d010c17f94e031e18a2edab682648bcaef692d5b).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21371: [SPARK-24250][SQL][FollowUp] Fix compile error by adding...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21371
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/90840/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21371: [SPARK-24250][SQL][FollowUp] Fix compile error and flaky...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21371
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21371: [SPARK-24250][SQL][FollowUp] Fix compile error and flaky...
Posted by kiszk <gi...@git.apache.org>.
Github user kiszk commented on the issue:
https://github.com/apache/spark/pull/21371
retest this please
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21371: [SPARK-24250][SQL][FollowUp] Fix compile error by adding...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21371
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/90841/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21371: [SPARK-24250][SQL][FollowUp] Fix compile error an...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/21371#discussion_r189454021
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/json/JsonInferSchema.scala ---
@@ -66,8 +69,12 @@ private[sql] object JsonInferSchema {
s"Parse Mode: ${FailFastMode.name}.", e)
}
}
- }
- }.fold(StructType(Nil))(
+ }.fold(StructType(Nil))(
+ compatibleRootType(columnNameOfCorruptRecord, parseMode))
+ Iterator(typeInPartition)
+ }.collect()
--- End diff --
the collect here may cause OOM while the previous code doesn't have this problem. Let's revert my PR first and think about how to fix it.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21371: [SPARK-24250][SQL][FollowUp] Fix compile error by...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/21371#discussion_r189446146
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/internal/ExecutorSideSQLConfSuite.scala ---
@@ -59,7 +59,8 @@ class ExecutorSideSQLConfSuite extends SparkFunSuite with SQLTestUtils {
val pathString = path.getCanonicalPath
spark.range(10).select('id.as("ID")).write.json(pathString)
spark.range(10).write.mode("append").json(pathString)
- assert(spark.read.json(pathString).columns.toSet == Set("id", "ID"))
+ val df = spark.read.option("samplingRatio", 1).json(pathString)
--- End diff --
oh, I just found `samplingRatio` is 1.0 by default. The flaky test should not because of this.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21371: [SPARK-24250][SQL][FollowUp] Fix compile error and flaky...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21371
**[Test build #90850 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90850/testReport)** for PR 21371 at commit [`a823841`](https://github.com/apache/spark/commit/a823841fd419541137fd02d9a02822b1fa36f5ae).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21371: [SPARK-24250][SQL][FollowUp] Fix compile error an...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/21371#discussion_r189453815
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/json/JsonInferSchema.scala ---
@@ -66,8 +69,12 @@ private[sql] object JsonInferSchema {
s"Parse Mode: ${FailFastMode.name}.", e)
}
}
- }
- }.fold(StructType(Nil))(
+ }.fold(StructType(Nil))(
+ compatibleRootType(columnNameOfCorruptRecord, parseMode))
+ Iterator(typeInPartition)
+ }.collect()
--- End diff --
good catch! but wondering how the test passed in my PR...
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21371: [SPARK-24250][SQL][FollowUp] Fix compile error and flaky...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21371
**[Test build #90845 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90845/testReport)** for PR 21371 at commit [`a823841`](https://github.com/apache/spark/commit/a823841fd419541137fd02d9a02822b1fa36f5ae).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21371: [SPARK-24250][SQL][FollowUp] Fix compile error and flaky...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21371
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/90848/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21371: [SPARK-24250][SQL][FollowUp] Fix compile error and flaky...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21371
**[Test build #90849 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90849/testReport)** for PR 21371 at commit [`a823841`](https://github.com/apache/spark/commit/a823841fd419541137fd02d9a02822b1fa36f5ae).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21371: [SPARK-24250][SQL][FollowUp] Fix compile error by adding...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21371
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3374/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21371: [SPARK-24250][SQL][FollowUp] Fix compile error and flaky...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21371
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3377/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21371: [SPARK-24250][SQL][FollowUp] Fix compile error and flaky...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21371
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3378/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21371: [SPARK-24250][SQL][FollowUp] Fix compile error and flaky...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21371
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3381/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21371: [SPARK-24250][SQL][FollowUp] Fix compile error by adding...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21371
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3367/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21371: [SPARK-24250][SQL][FollowUp] Fix compile error by adding...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/21371
cc @dbtsai
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21371: [SPARK-24250][SQL][FollowUp] Fix compile error and flaky...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21371
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21371: [SPARK-24250][SQL][FollowUp] Fix compile error by adding...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21371
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21371: [SPARK-24250][SQL][FollowUp] Fix compile error by adding...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21371
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3373/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21371: [SPARK-24250][SQL][FollowUp] Fix compile error by adding...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21371
**[Test build #90844 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90844/testReport)** for PR 21371 at commit [`a823841`](https://github.com/apache/spark/commit/a823841fd419541137fd02d9a02822b1fa36f5ae).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21371: [SPARK-24250][SQL][FollowUp] Fix compile error and flaky...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21371
**[Test build #90850 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90850/testReport)** for PR 21371 at commit [`a823841`](https://github.com/apache/spark/commit/a823841fd419541137fd02d9a02822b1fa36f5ae).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21371: [SPARK-24250][SQL][FollowUp] Fix compile error an...
Posted by viirya <gi...@git.apache.org>.
Github user viirya closed the pull request at:
https://github.com/apache/spark/pull/21371
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21371: [SPARK-24250][SQL][FollowUp] Fix compile error by adding...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/21371
retest this please.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21371: [SPARK-24250][SQL][FollowUp] Fix compile error and flaky...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/21371
As the commit is reverted, this can be closed for now.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21371: [SPARK-24250][SQL][FollowUp] Fix compile error by adding...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/21371
@hvanhovell the test is flaky and I propose a fixing for it now. Please take a look if you have time. Thanks.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21371: [SPARK-24250][SQL][FollowUp] Fix compile error an...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/21371#discussion_r189455480
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/json/JsonInferSchema.scala ---
@@ -66,8 +69,12 @@ private[sql] object JsonInferSchema {
s"Parse Mode: ${FailFastMode.name}.", e)
}
}
- }
- }.fold(StructType(Nil))(
+ }.fold(StructType(Nil))(
+ compatibleRootType(columnNameOfCorruptRecord, parseMode))
+ Iterator(typeInPartition)
+ }.collect()
--- End diff --
Oh, I see. `RDD.fold` does folding immediately when a data type is returned from one task. But we now allocate an array to hold all data types.
I think we can use `RDD.toLocalIterator` instead of `collect`?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21371: [SPARK-24250][SQL][FollowUp] Fix compile error and flaky...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21371
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21371: [SPARK-24250][SQL][FollowUp] Fix compile error by adding...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21371
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21371: [SPARK-24250][SQL][FollowUp] Fix compile error by adding...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21371
**[Test build #90840 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90840/testReport)** for PR 21371 at commit [`7b193f6`](https://github.com/apache/spark/commit/7b193f6bd55838210936716f2c428ae2cc26489d).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21371: [SPARK-24250][SQL][FollowUp] Fix compile error by adding...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21371
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21371: [SPARK-24250][SQL][FollowUp] Fix compile error by adding...
Posted by mgaido91 <gi...@git.apache.org>.
Github user mgaido91 commented on the issue:
https://github.com/apache/spark/pull/21371
LGTM
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21371: [SPARK-24250][SQL][FollowUp] Fix compile error and flaky...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21371
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21371: [SPARK-24250][SQL][FollowUp] Fix compile error and flaky...
Posted by kiszk <gi...@git.apache.org>.
Github user kiszk commented on the issue:
https://github.com/apache/spark/pull/21371
retest this please
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21371: [SPARK-24250][SQL][FollowUp] Fix compile error by adding...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21371
**[Test build #90841 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90841/testReport)** for PR 21371 at commit [`d010c17`](https://github.com/apache/spark/commit/d010c17f94e031e18a2edab682648bcaef692d5b).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21371: [SPARK-24250][SQL][FollowUp] Fix compile error and flaky...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21371
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21371: [SPARK-24250][SQL][FollowUp] Fix compile error and flaky...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21371
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/90850/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21371: [SPARK-24250][SQL][FollowUp] Fix compile error and flaky...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21371
**[Test build #90845 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90845/testReport)** for PR 21371 at commit [`a823841`](https://github.com/apache/spark/commit/a823841fd419541137fd02d9a02822b1fa36f5ae).
* This patch **fails due to an unknown error code, -9**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21371: [SPARK-24250][SQL][FollowUp] Fix compile error and flaky...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21371
**[Test build #90848 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90848/testReport)** for PR 21371 at commit [`a823841`](https://github.com/apache/spark/commit/a823841fd419541137fd02d9a02822b1fa36f5ae).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org