You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by gatorsmile <gi...@git.apache.org> on 2017/12/30 06:01:09 UTC
[GitHub] spark pull request #20120: [SPARK-22926] [SQL] Respect table-level conf comp...
GitHub user gatorsmile opened a pull request:
https://github.com/apache/spark/pull/20120
[SPARK-22926] [SQL] Respect table-level conf compression codec `Compression` in multiple scenarios
## What changes were proposed in this pull request?
Currently, we ignore table-specific `compression` conf when the Hive serde tables are converted to the data source tables. We also ignore it when users set `compression` in the TBLPROPERTIES clause instead of the OPTIONS clause, even if the tables are native data source tables. This PR is to fix the above issue.
The PR https://github.com/apache/spark/pull/20087 will make the Hive-serde writer aware of the table-level prop.
The PR https://github.com/apache/spark/pull/20076 will make Spark aware of the table-specific conf `parquet.compression` too
## How was this patch tested?
Added test cases
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/gatorsmile/spark compressionParquet
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/20120.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #20120
----
commit 4916d6c39585dd063e74a299e9f745a1787ac6e3
Author: gatorsmile <ga...@...>
Date: 2017-12-30T04:59:42Z
fix
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20120: [SPARK-22926] [SQL] Respect table-level conf compression...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20120
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85538/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20120: [SPARK-22926] [SQL] Respect table-level conf compression...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/20120
cc @cloud-fan
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20120: [SPARK-22926] [SQL] Respect table-level conf compression...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20120
**[Test build #85546 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85546/testReport)** for PR 20120 at commit [`7189562`](https://github.com/apache/spark/commit/7189562d9e37c6a0642cab037daba18c11170812).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20120: [SPARK-22926] [SQL] Respect table-level conf compression...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20120
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20120: [SPARK-22926] [SQL] Respect table-level conf compression...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/20120
retest this please
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20120: [SPARK-22926] [SQL] Respect table-level conf compression...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20120
**[Test build #85538 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85538/testReport)** for PR 20120 at commit [`7189562`](https://github.com/apache/spark/commit/7189562d9e37c6a0642cab037daba18c11170812).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20120: [SPARK-22926] [SQL] Respect table-level conf compression...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20120
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #20120: [SPARK-22926] [SQL] Respect table-level conf comp...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile closed the pull request at:
https://github.com/apache/spark/pull/20120
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20120: [SPARK-22926] [SQL] Respect table-level conf compression...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20120
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85546/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20120: [SPARK-22926] [SQL] Respect table-level conf compression...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20120
**[Test build #85538 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85538/testReport)** for PR 20120 at commit [`7189562`](https://github.com/apache/spark/commit/7189562d9e37c6a0642cab037daba18c11170812).
* This patch **fails due to an unknown error code, -9**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20120: [SPARK-22926] [SQL] Respect table-level conf compression...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20120
**[Test build #85546 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85546/testReport)** for PR 20120 at commit [`7189562`](https://github.com/apache/spark/commit/7189562d9e37c6a0642cab037daba18c11170812).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20120: [SPARK-22926] [SQL] Respect table-level conf compression...
Posted by dongjoon-hyun <gi...@git.apache.org>.
Github user dongjoon-hyun commented on the issue:
https://github.com/apache/spark/pull/20120
@gatorsmile . Did you add some document for this?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20120: [SPARK-22926] [SQL] Respect table-level conf compression...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/20120
I think we should document the difference between table options and properties. AFAIK we added table properties to data source tables since Spark 2.3, and previously table options is the only place for users to put some configs to change some behaviors.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org