You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2020/09/08 07:39:17 UTC
[GitHub] [spark] AngersZhuuuu opened a new pull request #29672: [SPARK-32818][SQL] Make `CONVERT_METASTORE_PARQUET` and `CONVERT_METASTORE_ORC` session level configurable
AngersZhuuuu opened a new pull request #29672:
URL: https://github.com/apache/spark/pull/29672
### What changes were proposed in this pull request?
There are some cases that we have history hive table, in the past, it use `TEXTFILE` serde, later it's change to `ORC` serde, since during metastore convert, we will use table level serde to scan all partition, if some old partition's file format is not same , will failed to convert , we need to close metastore convert and restart spark program then run.
But we know that enable hive metastore convert can speed up data r/w, and avoid many minor problem.
It's bad for ad-hoc engine such as long running spark thrift server. so I think it's necessary to make these two config can be changed by use in session level
### Why are the changes needed?
Make use range more flexiable
### Does this PR introduce _any_ user-facing change?
People can set` CONVERT_METASTORE_PARQUET` and `CONVERT_METASTORE_ORC` in session level
### How was this patch tested?
WIP
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu edited a comment on pull request #29672: [SPARK-32818][SQL] Make `CONVERT_METASTORE_PARQUET` and `CONVERT_METASTORE_ORC` session level configurable
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu edited a comment on pull request #29672:
URL: https://github.com/apache/spark/pull/29672#issuecomment-688682740
Spark can't support `SET FILEFORMAT` now, so it's hard to construct UT, should we support this?
cc @wangyum @maropu @dongjoon-hyun
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29672: [SPARK-32818][SQL] Make `CONVERT_METASTORE_PARQUET` and `CONVERT_METASTORE_ORC` session level configurable
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29672:
URL: https://github.com/apache/spark/pull/29672#issuecomment-688766797
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29672: [SPARK-32818][SQL] Make `CONVERT_METASTORE_PARQUET` and `CONVERT_METASTORE_ORC` session level configurable
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29672:
URL: https://github.com/apache/spark/pull/29672#issuecomment-689407145
**[Test build #128440 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128440/testReport)** for PR 29672 at commit [`c2ff589`](https://github.com/apache/spark/commit/c2ff589a813fe7db38245f5f03cc8df019613960).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu closed pull request #29672: [SPARK-32818][SQL] Make `CONVERT_METASTORE_PARQUET` and `CONVERT_METASTORE_ORC` session level configurable
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu closed pull request #29672:
URL: https://github.com/apache/spark/pull/29672
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29672: [SPARK-32818][SQL] Make `CONVERT_METASTORE_PARQUET` and `CONVERT_METASTORE_ORC` session level configurable
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29672:
URL: https://github.com/apache/spark/pull/29672#issuecomment-688682624
**[Test build #128389 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128389/testReport)** for PR 29672 at commit [`c2ff589`](https://github.com/apache/spark/commit/c2ff589a813fe7db38245f5f03cc8df019613960).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on a change in pull request #29672: [SPARK-32818][SQL] Make `CONVERT_METASTORE_PARQUET` and `CONVERT_METASTORE_ORC` session level configurable
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on a change in pull request #29672:
URL: https://github.com/apache/spark/pull/29672#discussion_r486135994
##########
File path: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/SQLQuerySuite.scala
##########
@@ -2560,6 +2560,33 @@ abstract class SQLQuerySuiteBase extends QueryTest with SQLTestUtils with TestHi
}
}
}
+
+ test("SPARK-32818: make metastore convert session level configurable") {
Review comment:
> Does this test fail w/o this patch? Looks it can pass even in master.
I miss the point that SQLConf.get is related to current thread's SparkSession.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29672: [SPARK-32818][SQL] Make `CONVERT_METASTORE_PARQUET` and `CONVERT_METASTORE_ORC` session level configurable
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29672:
URL: https://github.com/apache/spark/pull/29672#issuecomment-688683206
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29672: [SPARK-32818][SQL] Make `CONVERT_METASTORE_PARQUET` and `CONVERT_METASTORE_ORC` session level configurable
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29672:
URL: https://github.com/apache/spark/pull/29672#issuecomment-688682624
**[Test build #128389 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128389/testReport)** for PR 29672 at commit [`c2ff589`](https://github.com/apache/spark/commit/c2ff589a813fe7db38245f5f03cc8df019613960).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29672: [SPARK-32818][SQL] Make `CONVERT_METASTORE_PARQUET` and `CONVERT_METASTORE_ORC` session level configurable
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29672:
URL: https://github.com/apache/spark/pull/29672#issuecomment-689564562
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on pull request #29672: [SPARK-32818][SQL] Make `CONVERT_METASTORE_PARQUET` and `CONVERT_METASTORE_ORC` session level configurable
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on pull request #29672:
URL: https://github.com/apache/spark/pull/29672#issuecomment-689564788
Do we need to implement `SET FILEFORMAT`?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29672: [SPARK-32818][SQL] Make `CONVERT_METASTORE_PARQUET` and `CONVERT_METASTORE_ORC` session level configurable
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29672:
URL: https://github.com/apache/spark/pull/29672#issuecomment-689479877
**[Test build #128440 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128440/testReport)** for PR 29672 at commit [`c2ff589`](https://github.com/apache/spark/commit/c2ff589a813fe7db38245f5f03cc8df019613960).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29672: [SPARK-32818][SQL] Make `CONVERT_METASTORE_PARQUET` and `CONVERT_METASTORE_ORC` session level configurable
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29672:
URL: https://github.com/apache/spark/pull/29672#issuecomment-689407758
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29672: [SPARK-32818][SQL] Make `CONVERT_METASTORE_PARQUET` and `CONVERT_METASTORE_ORC` session level configurable
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29672:
URL: https://github.com/apache/spark/pull/29672#issuecomment-689568176
**[Test build #128454 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128454/testReport)** for PR 29672 at commit [`fc221f5`](https://github.com/apache/spark/commit/fc221f5f5c9a73b4627ba24517c925393acf21ba).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on pull request #29672: [SPARK-32818][SQL] Make `CONVERT_METASTORE_PARQUET` and `CONVERT_METASTORE_ORC` session level configurable
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on pull request #29672:
URL: https://github.com/apache/spark/pull/29672#issuecomment-689404108
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29672: [SPARK-32818][SQL] Make `CONVERT_METASTORE_PARQUET` and `CONVERT_METASTORE_ORC` session level configurable
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29672:
URL: https://github.com/apache/spark/pull/29672#issuecomment-688766797
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu edited a comment on pull request #29672: [SPARK-32818][SQL] Make `CONVERT_METASTORE_PARQUET` and `CONVERT_METASTORE_ORC` session level configurable
Posted by GitBox <gi...@apache.org>.
maropu edited a comment on pull request #29672:
URL: https://github.com/apache/spark/pull/29672#issuecomment-689508791
> Spark can't support SET FILEFORMAT now, so it's hard to construct UT, should we support this?
Not sure about why we cannot add tests; how about creating a hive table, then checking an optimizing plan (or a physical plan) for scanning it?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29672: [SPARK-32818][SQL] Make `CONVERT_METASTORE_PARQUET` and `CONVERT_METASTORE_ORC` session level configurable
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29672:
URL: https://github.com/apache/spark/pull/29672#issuecomment-689480811
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29672: [SPARK-32818][SQL] Make `CONVERT_METASTORE_PARQUET` and `CONVERT_METASTORE_ORC` session level configurable
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29672:
URL: https://github.com/apache/spark/pull/29672#issuecomment-689564562
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu commented on pull request #29672: [SPARK-32818][SQL] Make `CONVERT_METASTORE_PARQUET` and `CONVERT_METASTORE_ORC` session level configurable
Posted by GitBox <gi...@apache.org>.
maropu commented on pull request #29672:
URL: https://github.com/apache/spark/pull/29672#issuecomment-689508791
> Spark can't support SET FILEFORMAT now, so it's hard to construct UT, should we support this?
Not sure about why we cannot add tests; how about just creating a hive table, then checking an optimizing plan (or a physical plan) for scanning it?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29672: [SPARK-32818][SQL] Make `CONVERT_METASTORE_PARQUET` and `CONVERT_METASTORE_ORC` session level configurable
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29672:
URL: https://github.com/apache/spark/pull/29672#issuecomment-689669060
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on pull request #29672: [SPARK-32818][SQL] Make `CONVERT_METASTORE_PARQUET` and `CONVERT_METASTORE_ORC` session level configurable
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on pull request #29672:
URL: https://github.com/apache/spark/pull/29672#issuecomment-689564461
> > Spark can't support SET FILEFORMAT now, so it's hard to construct UT, should we support this?
>
> Not sure about why we cannot add tests; how about creating a hive table, then checking an optimizing plan (or a physical plan) for scanning it?
It's ok to checking plan,. But mostly, I want to reproduce the case I mentioned in desc.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29672: [SPARK-32818][SQL] Make `CONVERT_METASTORE_PARQUET` and `CONVERT_METASTORE_ORC` session level configurable
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29672:
URL: https://github.com/apache/spark/pull/29672#issuecomment-688683206
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29672: [SPARK-32818][SQL] Make `CONVERT_METASTORE_PARQUET` and `CONVERT_METASTORE_ORC` session level configurable
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29672:
URL: https://github.com/apache/spark/pull/29672#issuecomment-689669060
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29672: [SPARK-32818][SQL] Make `CONVERT_METASTORE_PARQUET` and `CONVERT_METASTORE_ORC` session level configurable
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29672:
URL: https://github.com/apache/spark/pull/29672#issuecomment-689667064
**[Test build #128454 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128454/testReport)** for PR 29672 at commit [`fc221f5`](https://github.com/apache/spark/commit/fc221f5f5c9a73b4627ba24517c925393acf21ba).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29672: [SPARK-32818][SQL] Make `CONVERT_METASTORE_PARQUET` and `CONVERT_METASTORE_ORC` session level configurable
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29672:
URL: https://github.com/apache/spark/pull/29672#issuecomment-689407758
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29672: [SPARK-32818][SQL] Make `CONVERT_METASTORE_PARQUET` and `CONVERT_METASTORE_ORC` session level configurable
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29672:
URL: https://github.com/apache/spark/pull/29672#issuecomment-688766010
**[Test build #128389 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128389/testReport)** for PR 29672 at commit [`c2ff589`](https://github.com/apache/spark/commit/c2ff589a813fe7db38245f5f03cc8df019613960).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29672: [SPARK-32818][SQL] Make `CONVERT_METASTORE_PARQUET` and `CONVERT_METASTORE_ORC` session level configurable
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29672:
URL: https://github.com/apache/spark/pull/29672#issuecomment-688766802
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/128389/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29672: [SPARK-32818][SQL] Make `CONVERT_METASTORE_PARQUET` and `CONVERT_METASTORE_ORC` session level configurable
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29672:
URL: https://github.com/apache/spark/pull/29672#issuecomment-689480811
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #29672: [SPARK-32818][SQL] Make `CONVERT_METASTORE_PARQUET` and `CONVERT_METASTORE_ORC` session level configurable
Posted by GitBox <gi...@apache.org>.
maropu commented on a change in pull request #29672:
URL: https://github.com/apache/spark/pull/29672#discussion_r486071663
##########
File path: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/SQLQuerySuite.scala
##########
@@ -2560,6 +2560,33 @@ abstract class SQLQuerySuiteBase extends QueryTest with SQLTestUtils with TestHi
}
}
}
+
+ test("SPARK-32818: make metastore convert session level configurable") {
Review comment:
Does this test fail w/o this patch? Looks it can pass even in master.
##########
File path: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/SQLQuerySuite.scala
##########
@@ -2560,6 +2560,33 @@ abstract class SQLQuerySuiteBase extends QueryTest with SQLTestUtils with TestHi
}
}
}
+
+ test("SPARK-32818: make metastore convert session level configurable") {
+ withSQLConf(HiveUtils.CONVERT_METASTORE_ORC.key -> "true") {
+ withTable("t") {
+ sql("CREATE TABLE t (i INT) PARTITIONED BY (p STRING) STORED AS ORC")
+ sql("INSERT INTO t PARTITION(p='20200901') SELECT 1")
Review comment:
I think we don't need the partition definition for this test and how about writing it like this;
```
sql("CREATE TABLE t (i INT) USING hive")
sql("SELECT * FROM t")
```
##########
File path: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/SQLQuerySuite.scala
##########
@@ -33,7 +33,7 @@ import org.apache.spark.sql.catalyst.analysis.{EliminateSubqueryAliases, Functio
import org.apache.spark.sql.catalyst.catalog.{CatalogTableType, CatalogUtils, HiveTableRelation}
import org.apache.spark.sql.catalyst.parser.ParseException
import org.apache.spark.sql.catalyst.plans.logical.{LogicalPlan, SubqueryAlias}
-import org.apache.spark.sql.execution.TestUncaughtExceptionHandler
+import org.apache.spark.sql.execution.{FileSourceScanExec, SparkPlan, TestUncaughtExceptionHandler}
Review comment:
nit: `SparkPlan` not used.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29672: [SPARK-32818][SQL] Make `CONVERT_METASTORE_PARQUET` and `CONVERT_METASTORE_ORC` session level configurable
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29672:
URL: https://github.com/apache/spark/pull/29672#issuecomment-689568176
**[Test build #128454 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128454/testReport)** for PR 29672 at commit [`fc221f5`](https://github.com/apache/spark/commit/fc221f5f5c9a73b4627ba24517c925393acf21ba).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on pull request #29672: [SPARK-32818][SQL] Make `CONVERT_METASTORE_PARQUET` and `CONVERT_METASTORE_ORC` session level configurable
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on pull request #29672:
URL: https://github.com/apache/spark/pull/29672#issuecomment-688682740
Spark can't support `SET FILEFORMAT` now, so it's hard to construct UT, should we support this?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29672: [SPARK-32818][SQL] Make `CONVERT_METASTORE_PARQUET` and `CONVERT_METASTORE_ORC` session level configurable
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29672:
URL: https://github.com/apache/spark/pull/29672#issuecomment-689407145
**[Test build #128440 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128440/testReport)** for PR 29672 at commit [`c2ff589`](https://github.com/apache/spark/commit/c2ff589a813fe7db38245f5f03cc8df019613960).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org