You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by dongjoon-hyun <gi...@git.apache.org> on 2017/08/20 04:04:05 UTC
[GitHub] spark pull request #19004: [SPARK-21791][SQL] ORC should support column name...
GitHub user dongjoon-hyun opened a pull request:
https://github.com/apache/spark/pull/19004
[SPARK-21791][SQL] ORC should support column names with dot
## What changes were proposed in this pull request?
Currently, Apache Spark ORC data source doesn't support field names with dot properly. For a feature parity with Parquet, we had better support this. Since Apache ORC 1.4 supports this by #18953, this PR adds a test case to prevent a future regression.
**PARQUET**
```scala
scala> Seq(Some(1), None).toDF("col.dots").write.parquet("/tmp/parquet_dot")
scala> spark.read.parquet("/tmp/parquet_dot").show
+--------+
|col.dots|
+--------+
| 1|
| null|
+--------+
```
**ORC**
```scala
scala> Seq(Some(1), None).toDF("col.dots").write.orc("/tmp/orc_dot")
scala> spark.read.orc("/tmp/orc_dot").show
org.apache.spark.sql.catalyst.parser.ParseException:
mismatched input '.' expecting ':'(line 1, pos 10)
== SQL ==
struct<col.dots:int>
----------^^^
```
## How was this patch tested?
After merging #18953, I will enable this test case.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/dongjoon-hyun/spark SPARK-21791
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/19004.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #19004
----
commit 5995bd5447ed8d0d5ffcbfbf1132a1177892edd4
Author: Dongjoon Hyun <do...@apache.org>
Date: 2017-08-20T03:58:32Z
[SPARK-21791][SQL] ORC should support column names with dot
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19004: [SPARK-21791][SQL] ORC should support column name...
Posted by dongjoon-hyun <gi...@git.apache.org>.
Github user dongjoon-hyun closed the pull request at:
https://github.com/apache/spark/pull/19004
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19004: [SPARK-21791][SQL] ORC should support column names with ...
Posted by dongjoon-hyun <gi...@git.apache.org>.
Github user dongjoon-hyun commented on the issue:
https://github.com/apache/spark/pull/19004
Since this is a test case only PR to confirm that, I'll include this into https://github.com/apache/spark/pull/18953 to save committer's review efforts.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19004: [SPARK-21791][SQL] ORC should support column names with ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19004
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/80883/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19004: [SPARK-21791][SQL] ORC should support column names with ...
Posted by dongjoon-hyun <gi...@git.apache.org>.
Github user dongjoon-hyun commented on the issue:
https://github.com/apache/spark/pull/19004
cc @cloud-fan , @gatorsmile , @hvanhovell , @sameeragarwal , @rxin .
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19004: [SPARK-21791][SQL] ORC should support column names with ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19004
**[Test build #80883 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80883/testReport)** for PR 19004 at commit [`5995bd5`](https://github.com/apache/spark/commit/5995bd5447ed8d0d5ffcbfbf1132a1177892edd4).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `class OrcSourceSuite extends OrcSuite with SQLTestUtils `
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19004: [SPARK-21791][SQL] ORC should support column names with ...
Posted by dongjoon-hyun <gi...@git.apache.org>.
Github user dongjoon-hyun commented on the issue:
https://github.com/apache/spark/pull/19004
Also, this PR is included into https://github.com/apache/spark/pull/17980, too.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19004: [SPARK-21791][SQL] ORC should support column names with ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19004
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19004: [SPARK-21791][SQL] ORC should support column names with ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19004
**[Test build #80883 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80883/testReport)** for PR 19004 at commit [`5995bd5`](https://github.com/apache/spark/commit/5995bd5447ed8d0d5ffcbfbf1132a1177892edd4).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org