You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by wangyum <gi...@git.apache.org> on 2018/08/30 18:15:19 UTC
[GitHub] spark pull request #22287: [SPARK-25135][SQL] FileFormatWriter should respec...
GitHub user wangyum opened a pull request:
https://github.com/apache/spark/pull/22287
[SPARK-25135][SQL] FileFormatWriter should respect the schema of Hive
## What changes were proposed in this pull request?
This pr fix `FileFormatWriter's dataSchema` should respect the schema of Hive. Otherwise there will be two issues.
1. Throwing an exception(This can be reproduce by added test case):
```scala
java.util.NoSuchElementException: None.get
at scala.None$.get(Option.scala:347)
at scala.None$.get(Option.scala:345)
at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$3$$anonfun$4.apply(FileFormatWriter.scala:87)
at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$3$$anonfun$4.apply(FileFormatWriter.scala:87)
```
2. The schema of the Hive table is not the same as the schema of the parquet file.
## How was this patch tested?
- Unit tests for FileFormatWriter should respect the schema of Hive.
- Manual tests for didn't break UI issues fixed by [SPARK-22834](https://issues.apache.org/jira/browse/SPARK-22834):
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/wangyum/spark SPARK-25135-view
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/22287.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #22287
----
commit b54953a8224aa0a7759289a83e876e3bfc166cb6
Author: Yuming Wang <yu...@...>
Date: 2018-08-30T17:46:02Z
FileFormatWriter should respect the input query schema in HIVE
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22287: [SPARK-25135][SQL] FileFormatWriter should respect the s...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22287
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22287: [SPARK-25135][SQL] FileFormatWriter should respect the s...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22287
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/2704/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22287: [SPARK-25135][SQL] FileFormatWriter should respect the s...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22287
**[Test build #95483 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95483/testReport)** for PR 22287 at commit [`b54953a`](https://github.com/apache/spark/commit/b54953a8224aa0a7759289a83e876e3bfc166cb6).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22287: [SPARK-25135][SQL] FileFormatWriter should respect the s...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22287
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/95483/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22287: [SPARK-25135][SQL] FileFormatWriter should respect the s...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22287
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #22287: [SPARK-25135][SQL] FileFormatWriter should respec...
Posted by wangyum <gi...@git.apache.org>.
Github user wangyum closed the pull request at:
https://github.com/apache/spark/pull/22287
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22287: [SPARK-25135][SQL] FileFormatWriter should respect the s...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22287
**[Test build #95483 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95483/testReport)** for PR 22287 at commit [`b54953a`](https://github.com/apache/spark/commit/b54953a8224aa0a7759289a83e876e3bfc166cb6).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org