You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by dongjoon-hyun <gi...@git.apache.org> on 2017/10/16 00:55:28 UTC
[GitHub] spark pull request #19502: [SPARK-22282][SQL] Remove OrcRelation.ORC_COMPRES...
GitHub user dongjoon-hyun opened a pull request:
https://github.com/apache/spark/pull/19502
[SPARK-22282][SQL] Remove OrcRelation.ORC_COMPRESSION
## What changes were proposed in this pull request?
This PR aims to replace `OrcRelation.ORC_COMPRESSION` with `org.apache.orc.OrcConf.COMPRESS`.
Since [SPARK-21422](https://issues.apache.org/jira/browse/SPARK-21422), we can use `OrcConf.COMPRESS` instead of Hive's.
```scala
// The references of Hive's classes will be minimized.
val ORC_COMPRESSION = "orc.compress"
```
## How was this patch tested?
Pass the Jenkins with the existing and updated test cases.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/dongjoon-hyun/spark SPARK-22282
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/19502.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #19502
----
commit 7416e4ad262c01ac5e52ff0e0e4077c8ad98d7e1
Author: Dongjoon Hyun <do...@apache.org>
Date: 2017-10-16T00:53:04Z
[SPARK-22282][SQL] Remove OrcRelation.ORC_COMPRESSION
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19502: [SPARK-22282][SQL] Rename OrcRelation to OrcFileFormat a...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19502
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19502: [SPARK-22282][SQL] Rename OrcRelation to OrcFileFormat a...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19502
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82778/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19502: [SPARK-22282][SQL] Rename OrcRelation to OrcFileFormat a...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19502
**[Test build #82783 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82783/testReport)** for PR 19502 at commit [`02792d9`](https://github.com/apache/spark/commit/02792d953872ed6d718aa9ceb247affee5004b66).
* This patch **fails due to an unknown error code, -9**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19502: [SPARK-22282][SQL] Rename OrcRelation to OrcFileFormat a...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19502
**[Test build #82778 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82778/testReport)** for PR 19502 at commit [`a554112`](https://github.com/apache/spark/commit/a554112daa60a130074709c77bcdffbd443a7c2f).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19502: [SPARK-22282][SQL] Rename OrcRelation to OrcFileFormat a...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/19502
Thanks! Merged to master.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19502: [SPARK-22282][SQL] Rename OrcRelation to OrcFileFormat a...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19502
**[Test build #82775 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82775/testReport)** for PR 19502 at commit [`3d18d75`](https://github.com/apache/spark/commit/3d18d751e82791e53cd9f17a528777aaf355390d).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19502: [SPARK-22282][SQL] Rename OrcRelation to OrcFileF...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/19502#discussion_r144752155
--- Diff: sql/hive/src/test/scala/org/apache/spark/sql/hive/orc/OrcSourceSuite.scala ---
@@ -19,6 +19,7 @@ package org.apache.spark.sql.hive.orc
import java.io.File
+import org.apache.orc.OrcConf.COMPRESS
import org.scalatest.BeforeAndAfterAll
--- End diff --
Another test in `OrcSourceSuite`:
https://github.com/apache/spark/blob/d8f45408635d4fccac557cb1e877dfe9267fb326/sql/hive/src/test/scala/org/apache/spark/sql/hive/orc/OrcSourceSuite.scala#L153
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19502: [SPARK-22282][SQL] Rename OrcRelation to OrcFileF...
Posted by dongjoon-hyun <gi...@git.apache.org>.
Github user dongjoon-hyun commented on a diff in the pull request:
https://github.com/apache/spark/pull/19502#discussion_r144755323
--- Diff: sql/hive/src/test/scala/org/apache/spark/sql/hive/orc/OrcQuerySuite.scala ---
@@ -180,7 +181,7 @@ class OrcQuerySuite extends QueryTest with BeforeAndAfterAll with OrcTest {
// Respect `orc.compress`.
--- End diff --
Sure. Thank you for review, @viirya .
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19502: [SPARK-22282][SQL] Rename OrcRelation to OrcFileFormat a...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19502
**[Test build #82783 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82783/testReport)** for PR 19502 at commit [`02792d9`](https://github.com/apache/spark/commit/02792d953872ed6d718aa9ceb247affee5004b66).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19502: [SPARK-22282][SQL] Rename OrcRelation to OrcFileFormat a...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19502
**[Test build #82778 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82778/testReport)** for PR 19502 at commit [`a554112`](https://github.com/apache/spark/commit/a554112daa60a130074709c77bcdffbd443a7c2f).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19502: [SPARK-22282][SQL] Remove OrcRelation.ORC_COMPRES...
Posted by dongjoon-hyun <gi...@git.apache.org>.
Github user dongjoon-hyun commented on a diff in the pull request:
https://github.com/apache/spark/pull/19502#discussion_r144743034
--- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/orc/OrcFileFormat.scala ---
@@ -256,9 +257,6 @@ private[orc] class OrcOutputWriter(
}
private[orc] object OrcRelation extends HiveInspectors {
--- End diff --
Sure. No problem. Thank you for review, @HyukjinKwon !
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19502: [SPARK-22282][SQL] Rename OrcRelation to OrcFileFormat a...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19502
**[Test build #82775 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82775/testReport)** for PR 19502 at commit [`3d18d75`](https://github.com/apache/spark/commit/3d18d751e82791e53cd9f17a528777aaf355390d).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19502: [SPARK-22282][SQL] Rename OrcRelation to OrcFileFormat a...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19502
**[Test build #82781 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82781/testReport)** for PR 19502 at commit [`b7335ac`](https://github.com/apache/spark/commit/b7335acbf456d123c8af2196e3f923fecc92d5a2).
* This patch **fails due to an unknown error code, -9**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19502: [SPARK-22282][SQL] Rename OrcRelation to OrcFileF...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/19502#discussion_r144745102
--- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/orc/OrcOptions.scala ---
@@ -42,7 +44,7 @@ private[orc] class OrcOptions(
val compressionCodec: String = {
// `compression`, `orc.compress`, and `spark.sql.orc.compression.codec` are
// in order of precedence from highest to lowest.
--- End diff --
Please update the comment.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19502: [SPARK-22282][SQL] Rename OrcRelation to OrcFileFormat a...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19502
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82775/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19502: [SPARK-22282][SQL] Rename OrcRelation to OrcFileFormat a...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19502
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82782/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19502: [SPARK-22282][SQL] Rename OrcRelation to OrcFileF...
Posted by dongjoon-hyun <gi...@git.apache.org>.
Github user dongjoon-hyun commented on a diff in the pull request:
https://github.com/apache/spark/pull/19502#discussion_r144755349
--- Diff: sql/hive/src/test/scala/org/apache/spark/sql/hive/orc/OrcSourceSuite.scala ---
@@ -19,6 +19,7 @@ package org.apache.spark.sql.hive.orc
import java.io.File
+import org.apache.orc.OrcConf.COMPRESS
import org.scalatest.BeforeAndAfterAll
--- End diff --
This is case sensitive test case. We should not change this.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19502: [SPARK-22282][SQL] Rename OrcRelation to OrcFileF...
Posted by dongjoon-hyun <gi...@git.apache.org>.
Github user dongjoon-hyun commented on a diff in the pull request:
https://github.com/apache/spark/pull/19502#discussion_r144749283
--- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/orc/OrcOptions.scala ---
@@ -42,7 +44,7 @@ private[orc] class OrcOptions(
val compressionCodec: String = {
// `compression`, `orc.compress`, and `spark.sql.orc.compression.codec` are
// in order of precedence from highest to lowest.
--- End diff --
I see.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19502: [SPARK-22282][SQL] Rename OrcRelation to OrcFileFormat a...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19502
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19502: [SPARK-22282][SQL] Rename OrcRelation to OrcFileFormat a...
Posted by dongjoon-hyun <gi...@git.apache.org>.
Github user dongjoon-hyun commented on the issue:
https://github.com/apache/spark/pull/19502
Thank you, @gatorsmile , @HyukjinKwon , and @viirya !
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19502: [SPARK-22282][SQL] Rename OrcRelation to OrcFileFormat a...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19502
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19502: [SPARK-22282][SQL] Rename OrcRelation to OrcFileF...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/19502#discussion_r144746184
--- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/orc/OrcOptions.scala ---
@@ -42,7 +44,7 @@ private[orc] class OrcOptions(
val compressionCodec: String = {
// `compression`, `orc.compress`, and `spark.sql.orc.compression.codec` are
// in order of precedence from highest to lowest.
--- End diff --
`COMPRESS.getAttribute` is not in our code base. Please change it to
> // `compression`, `orc.compress`(i.e., OrcConf.COMPRESS), and `spark.sql.orc.compression.codec` are
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19502: [SPARK-22282][SQL] Remove OrcRelation.ORC_COMPRES...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/19502#discussion_r144742687
--- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/orc/OrcFileFormat.scala ---
@@ -256,9 +257,6 @@ private[orc] class OrcOutputWriter(
}
private[orc] object OrcRelation extends HiveInspectors {
--- End diff --
@dongjoon-hyun, mind changing this to `OrcFileFormat` while we are here (see https://github.com/apache/spark/pull/14529)?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19502: [SPARK-22282][SQL] Rename OrcRelation to OrcFileFormat a...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19502
**[Test build #82782 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82782/testReport)** for PR 19502 at commit [`2b0a092`](https://github.com/apache/spark/commit/2b0a09258a35c07905bd1a0bb4e3bf0dba6b5e66).
* This patch **fails due to an unknown error code, -9**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19502: [SPARK-22282][SQL] Rename OrcRelation to OrcFileFormat a...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19502
**[Test build #82782 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82782/testReport)** for PR 19502 at commit [`2b0a092`](https://github.com/apache/spark/commit/2b0a09258a35c07905bd1a0bb4e3bf0dba6b5e66).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19502: [SPARK-22282][SQL] Rename OrcRelation to OrcFileF...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/19502#discussion_r144751920
--- Diff: sql/hive/src/test/scala/org/apache/spark/sql/hive/orc/OrcQuerySuite.scala ---
@@ -180,7 +181,7 @@ class OrcQuerySuite extends QueryTest with BeforeAndAfterAll with OrcTest {
// Respect `orc.compress`.
--- End diff --
Also the test name `Respect orc.compress option when compression is unset`.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19502: [SPARK-22282][SQL] Rename OrcRelation to OrcFileF...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/19502#discussion_r144756503
--- Diff: sql/hive/src/test/scala/org/apache/spark/sql/hive/orc/OrcSourceSuite.scala ---
@@ -19,6 +19,7 @@ package org.apache.spark.sql.hive.orc
import java.io.File
+import org.apache.orc.OrcConf.COMPRESS
import org.scalatest.BeforeAndAfterAll
--- End diff --
Oh, I was thinking something like `COMPRESS.getAttribute.toUpperCase`.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19502: [SPARK-22282][SQL] Rename OrcRelation to OrcFileF...
Posted by dongjoon-hyun <gi...@git.apache.org>.
Github user dongjoon-hyun commented on a diff in the pull request:
https://github.com/apache/spark/pull/19502#discussion_r144745832
--- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/orc/OrcOptions.scala ---
@@ -42,7 +44,7 @@ private[orc] class OrcOptions(
val compressionCodec: String = {
// `compression`, `orc.compress`, and `spark.sql.orc.compression.codec` are
// in order of precedence from highest to lowest.
--- End diff --
Thank you for review, @gatorsmile .
The name `orc.compress` and precedence order is not changed. Which part do you want to change?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19502: [SPARK-22282][SQL] Rename OrcRelation to OrcFileF...
Posted by dongjoon-hyun <gi...@git.apache.org>.
Github user dongjoon-hyun commented on a diff in the pull request:
https://github.com/apache/spark/pull/19502#discussion_r144756632
--- Diff: sql/hive/src/test/scala/org/apache/spark/sql/hive/orc/OrcSourceSuite.scala ---
@@ -19,6 +19,7 @@ package org.apache.spark.sql.hive.orc
import java.io.File
+import org.apache.orc.OrcConf.COMPRESS
import org.scalatest.BeforeAndAfterAll
--- End diff --
If you want, no problem at all. :)
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19502: [SPARK-22282][SQL] Rename OrcRelation to OrcFileFormat a...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19502
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82781/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19502: [SPARK-22282][SQL] Rename OrcRelation to OrcFileF...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/19502#discussion_r144756868
--- Diff: sql/hive/src/test/scala/org/apache/spark/sql/hive/orc/OrcSourceSuite.scala ---
@@ -19,6 +19,7 @@ package org.apache.spark.sql.hive.orc
import java.io.File
+import org.apache.orc.OrcConf.COMPRESS
import org.scalatest.BeforeAndAfterAll
--- End diff --
Just like to be consistent. Thanks. :)
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19502: [SPARK-22282][SQL] Rename OrcRelation to OrcFileF...
Posted by dongjoon-hyun <gi...@git.apache.org>.
Github user dongjoon-hyun commented on a diff in the pull request:
https://github.com/apache/spark/pull/19502#discussion_r144756453
--- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/orc/OrcFileFormat.scala ---
@@ -255,10 +256,7 @@ private[orc] class OrcOutputWriter(
}
}
-private[orc] object OrcRelation extends HiveInspectors {
- // The references of Hive's classes will be minimized.
- val ORC_COMPRESSION = "orc.compress"
--- End diff --
Yep. I agree. I don't think Apache ORC changes this in the future since this is a primitive configuration.
BTW, Thank you for pointing this doc. I'll fix some typos here.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19502: [SPARK-22282][SQL] Rename OrcRelation to OrcFileFormat a...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19502
**[Test build #82771 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82771/testReport)** for PR 19502 at commit [`7416e4a`](https://github.com/apache/spark/commit/7416e4ad262c01ac5e52ff0e0e4077c8ad98d7e1).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19502: [SPARK-22282][SQL] Rename OrcRelation to OrcFileF...
Posted by dongjoon-hyun <gi...@git.apache.org>.
Github user dongjoon-hyun commented on a diff in the pull request:
https://github.com/apache/spark/pull/19502#discussion_r144757242
--- Diff: sql/hive/src/test/scala/org/apache/spark/sql/hive/orc/OrcSourceSuite.scala ---
@@ -19,6 +19,7 @@ package org.apache.spark.sql.hive.orc
import java.io.File
+import org.apache.orc.OrcConf.COMPRESS
import org.scalatest.BeforeAndAfterAll
--- End diff --
Yep!
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19502: [SPARK-22282][SQL] Remove OrcRelation.ORC_COMPRESSION
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19502
**[Test build #82771 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82771/testReport)** for PR 19502 at commit [`7416e4a`](https://github.com/apache/spark/commit/7416e4ad262c01ac5e52ff0e0e4077c8ad98d7e1).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19502: [SPARK-22282][SQL] Rename OrcRelation to OrcFileFormat a...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19502
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19502: [SPARK-22282][SQL] Rename OrcRelation to OrcFileFormat a...
Posted by dongjoon-hyun <gi...@git.apache.org>.
Github user dongjoon-hyun commented on the issue:
https://github.com/apache/spark/pull/19502
@HyukjinKwon , @gatorsmile , @viirya .
This is ready for review again. Thanks!
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19502: [SPARK-22282][SQL] Rename OrcRelation to OrcFileFormat a...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19502
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19502: [SPARK-22282][SQL] Rename OrcRelation to OrcFileF...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/19502#discussion_r144752376
--- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/orc/OrcFileFormat.scala ---
@@ -255,10 +256,7 @@ private[orc] class OrcOutputWriter(
}
}
-private[orc] object OrcRelation extends HiveInspectors {
- // The references of Hive's classes will be minimized.
- val ORC_COMPRESSION = "orc.compress"
--- End diff --
We have documented `orc.compress` as option explicitly in https://github.com/apache/spark/blob/d8f45408635d4fccac557cb1e877dfe9267fb326/sql/core/src/main/scala/org/apache/spark/sql/DataFrameWriter.scala#L523
Now we depends on the configuration name from an external library. But I think the configuration name should not be changed at all. So looks should be fine.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19502: [SPARK-22282][SQL] Rename OrcRelation to OrcFileF...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/19502
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19502: [SPARK-22282][SQL] Rename OrcRelation to OrcFileFormat a...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19502
**[Test build #82788 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82788/testReport)** for PR 19502 at commit [`02792d9`](https://github.com/apache/spark/commit/02792d953872ed6d718aa9ceb247affee5004b66).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19502: [SPARK-22282][SQL] Rename OrcRelation to OrcFileFormat a...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19502
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19502: [SPARK-22282][SQL] Rename OrcRelation to OrcFileFormat a...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19502
**[Test build #82781 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82781/testReport)** for PR 19502 at commit [`b7335ac`](https://github.com/apache/spark/commit/b7335acbf456d123c8af2196e3f923fecc92d5a2).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19502: [SPARK-22282][SQL] Rename OrcRelation to OrcFileFormat a...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19502
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82783/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19502: [SPARK-22282][SQL] Rename OrcRelation to OrcFileFormat a...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/19502
retest this please
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19502: [SPARK-22282][SQL] Rename OrcRelation to OrcFileFormat a...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19502
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82788/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19502: [SPARK-22282][SQL] Remove OrcRelation.ORC_COMPRESSION
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/19502
LGTM btw.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19502: [SPARK-22282][SQL] Rename OrcRelation to OrcFileF...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/19502#discussion_r144756726
--- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/orc/OrcFileFormat.scala ---
@@ -255,10 +256,7 @@ private[orc] class OrcOutputWriter(
}
}
-private[orc] object OrcRelation extends HiveInspectors {
- // The references of Hive's classes will be minimized.
- val ORC_COMPRESSION = "orc.compress"
--- End diff --
Oh, right, the `parquet`.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19502: [SPARK-22282][SQL] Rename OrcRelation to OrcFileFormat a...
Posted by dongjoon-hyun <gi...@git.apache.org>.
Github user dongjoon-hyun commented on the issue:
https://github.com/apache/spark/pull/19502
Thank you, @HyukjinKwon !
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19502: [SPARK-22282][SQL] Rename OrcRelation to OrcFileFormat a...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19502
**[Test build #82788 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82788/testReport)** for PR 19502 at commit [`02792d9`](https://github.com/apache/spark/commit/02792d953872ed6d718aa9ceb247affee5004b66).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19502: [SPARK-22282][SQL] Rename OrcRelation to OrcFileFormat a...
Posted by dongjoon-hyun <gi...@git.apache.org>.
Github user dongjoon-hyun commented on the issue:
https://github.com/apache/spark/pull/19502
Thank you for retesting, @HyukjinKwon !
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19502: [SPARK-22282][SQL] Rename OrcRelation to OrcFileFormat a...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19502
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82771/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19502: [SPARK-22282][SQL] Rename OrcRelation to OrcFileFormat a...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19502
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19502: [SPARK-22282][SQL] Rename OrcRelation to OrcFileF...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/19502#discussion_r144751856
--- Diff: sql/hive/src/test/scala/org/apache/spark/sql/hive/orc/OrcQuerySuite.scala ---
@@ -180,7 +181,7 @@ class OrcQuerySuite extends QueryTest with BeforeAndAfterAll with OrcTest {
// Respect `orc.compress`.
--- End diff --
Shall we change this comment too?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org