You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2020/02/12 20:08:56 UTC
[GitHub] [spark] MaxGekk opened a new pull request #27552: [WIP] Enable Java
8 time API in Thrift server
MaxGekk opened a new pull request #27552: [WIP] Enable Java 8 time API in Thrift server
URL: https://github.com/apache/spark/pull/27552
<!--
Thanks for sending a pull request! Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://spark.apache.org/contributing.html
2. Ensure you have added or run the appropriate tests for your PR: https://spark.apache.org/developer-tools.html
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][SPARK-XXXX] Your PR title ...'.
4. Be sure to keep the PR description updated to reflect all changes.
5. Please write your PR title to summarize what this PR proposes.
6. If possible, provide a concise example to reproduce the issue for a faster review.
-->
### What changes were proposed in this pull request?
<!--
Please clarify what changes you are proposing. The purpose of this section is to outline the changes and how this PR fixes the issue.
If possible, please consider writing useful notes for better and faster reviews in your PR. See the examples below.
1. If you refactor some codes with changing classes, showing the class hierarchy will help reviewers.
2. If you fix some SQL features, you can provide some references of other DBMSes.
3. If there is design documentation, please add the link.
4. If there is a discussion in the mailing list, please add the link.
-->
### Why are the changes needed?
<!--
Please clarify why the changes are needed. For instance,
1. If you propose a new API, clarify the use case for a new API.
2. If you fix a bug, you can clarify why it is a bug.
-->
### Does this PR introduce any user-facing change?
<!--
If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible.
If no, write 'No'.
-->
### How was this patch tested?
<!--
If tests were added, say they were added here. Please make sure to add some test cases that check the changes thoroughly including negative and positive cases if possible.
If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future.
If tests were not added, please describe why they were not added and/or why it was difficult to add.
-->
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #27552:
[SPARK-30808][SQL] Enable Java 8 time API in Thrift server
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server
URL: https://github.com/apache/spark/pull/27552#discussion_r385531744
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/HiveResult.scala
##########
@@ -36,27 +37,43 @@ object HiveResult {
* Returns the result as a hive compatible sequence of strings. This is used in tests and
* `SparkSQLDriver` for CLI applications.
*/
- def hiveResultString(executedPlan: SparkPlan): Seq[String] = executedPlan match {
- case ExecutedCommandExec(_: DescribeCommandBase) =>
- // If it is a describe command for a Hive table, we want to have the output format
- // be similar with Hive.
- executedPlan.executeCollectPublic().map {
- case Row(name: String, dataType: String, comment) =>
- Seq(name, dataType,
- Option(comment.asInstanceOf[String]).getOrElse(""))
- .map(s => String.format(s"%-20s", s))
- .mkString("\t")
- }
- // SHOW TABLES in Hive only output table names, while ours output database, table name, isTemp.
- case command @ ExecutedCommandExec(s: ShowTablesCommand) if !s.isExtended =>
- command.executeCollect().map(_.getString(1))
- case other =>
- val result: Seq[Seq[Any]] = other.executeCollectPublic().map(_.toSeq).toSeq
- // We need the types so we can output struct field names
- val types = executedPlan.output.map(_.dataType)
- // Reformat to match hive tab delimited output.
- result.map(_.zip(types).map(e => toHiveString(e)))
- .map(_.mkString("\t"))
+ def hiveResultString(ds: Dataset[_]): Seq[String] = {
+ val executedPlan = ds.queryExecution.executedPlan
+ executedPlan match {
+ case ExecutedCommandExec(_: DescribeCommandBase) =>
+ // If it is a describe command for a Hive table, we want to have the output format
+ // be similar with Hive.
+ executedPlan.executeCollectPublic().map {
+ case Row(name: String, dataType: String, comment) =>
+ Seq(name, dataType,
+ Option(comment.asInstanceOf[String]).getOrElse(""))
+ .map(s => String.format(s"%-20s", s))
+ .mkString("\t")
+ }
+ // SHOW TABLES in Hive only output table names,
+ // while ours output database, table name, isTemp.
+ case command @ ExecutedCommandExec(s: ShowTablesCommand) if !s.isExtended =>
+ command.executeCollect().map(_.getString(1))
+ case _ =>
+ val sessionWithJava8DatetimeEnabled = {
+ val cloned = ds.sparkSession.cloneSession()
+ cloned.conf.set(SQLConf.DATETIME_JAVA8API_ENABLED.key, true)
Review comment:
Yea I tested it too and looks fine. Maybe some refactor of how to format old `Date/Timestamp` fixes it already.
@yaooqinn can you send a PR to revert it? Let's see if all tests pass.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27552: [SPARK-30808][SQL]
Enable Java 8 time API in Thrift server
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server
URL: https://github.com/apache/spark/pull/27552#issuecomment-586846187
**[Test build #118543 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118543/testReport)** for PR 27552 at commit [`e491f96`](https://github.com/apache/spark/commit/e491f966b23a734358b222ed8486a547467a22ed).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27552:
[SPARK-30808][SQL] Enable Java 8 time API in Thrift server
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server
URL: https://github.com/apache/spark/pull/27552#issuecomment-586904049
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118552/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27552:
[SPARK-30808][SQL] Enable Java 8 time API in Thrift server
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server
URL: https://github.com/apache/spark/pull/27552#issuecomment-586740826
**[Test build #118508 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118508/testReport)** for PR 27552 at commit [`deaef58`](https://github.com/apache/spark/commit/deaef5837b0d7bbbb1eb0f6b6e90aef4b1eefc7d).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] yaooqinn commented on a change in pull request #27552:
[SPARK-30808][SQL] Enable Java 8 time API in Thrift server
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on a change in pull request #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server
URL: https://github.com/apache/spark/pull/27552#discussion_r385492731
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/SQLQueryTestSuite.scala
##########
@@ -512,7 +511,7 @@ class SQLQueryTestSuite extends QueryTest with SharedSparkSession {
val schema = df.schema.catalogString
// Get answer, but also get rid of the #1234 expression ids that show up in explain plans
val answer = SQLExecution.withNewExecutionId(df.queryExecution, Some(sql)) {
- hiveResultString(df.queryExecution.executedPlan).map(replaceNotIncludedMsg)
+ hiveResultString(df).map(replaceNotIncludedMsg)
Review comment:
We do not have our own JDBC implementation yet, and the hive JDBC haven't supported this too
https://github.com/apache/hive/blob/ebd41526b9433a196ae0f934bc466cb5c155a202/jdbc/src/java/org/apache/hive/jdbc/HiveQueryResultSet.java#L402-L411
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27552: [SPARK-30808][SQL]
Enable Java 8 time API in Thrift server
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server
URL: https://github.com/apache/spark/pull/27552#issuecomment-586740826
**[Test build #118508 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118508/testReport)** for PR 27552 at commit [`deaef58`](https://github.com/apache/spark/commit/deaef5837b0d7bbbb1eb0f6b6e90aef4b1eefc7d).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27552: [SPARK-30808][SQL]
Enable Java 8 time API in Thrift server
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server
URL: https://github.com/apache/spark/pull/27552#issuecomment-587107201
**[Test build #118579 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118579/testReport)** for PR 27552 at commit [`880b1de`](https://github.com/apache/spark/commit/880b1dec95595b58688a3d5f7a2e0637a090cb47).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27552:
[SPARK-30808][SQL] Enable Java 8 time API in Thrift server
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server
URL: https://github.com/apache/spark/pull/27552#issuecomment-586741571
**[Test build #118509 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118509/testReport)** for PR 27552 at commit [`804e709`](https://github.com/apache/spark/commit/804e709860357c5957a1b44e85acc1a7f41ebef3).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27552:
[SPARK-30808][SQL] Enable Java 8 time API in Thrift server
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server
URL: https://github.com/apache/spark/pull/27552#issuecomment-586848299
**[Test build #118545 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118545/testReport)** for PR 27552 at commit [`95597a7`](https://github.com/apache/spark/commit/95597a7244e2c99a9ae267e5705748ad8403a169).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27552:
[SPARK-30808][SQL] Enable Java 8 time API in Thrift server
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server
URL: https://github.com/apache/spark/pull/27552#issuecomment-586926693
**[Test build #118569 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118569/testReport)** for PR 27552 at commit [`2129f30`](https://github.com/apache/spark/commit/2129f30c8f54c669405db302dbfb52f78631fc05).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27552: [SPARK-30808][SQL]
Enable Java 8 time API in Thrift server
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server
URL: https://github.com/apache/spark/pull/27552#issuecomment-586767250
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23266/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27552:
[SPARK-30808][SQL] Enable Java 8 time API in Thrift server
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server
URL: https://github.com/apache/spark/pull/27552#issuecomment-586740962
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23263/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27552:
[SPARK-30808][SQL] Enable Java 8 time API in Thrift server
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server
URL: https://github.com/apache/spark/pull/27552#issuecomment-586749224
Build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27552:
[SPARK-30808][SQL] Enable Java 8 time API in Thrift server
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server
URL: https://github.com/apache/spark/pull/27552#issuecomment-586927286
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23324/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27552: [SPARK-30808][SQL]
Enable Java 8 time API in Thrift server
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server
URL: https://github.com/apache/spark/pull/27552#issuecomment-586774294
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk commented on issue #27552: [SPARK-30808][SQL]
Enable Java 8 time API in Thrift server
Posted by GitBox <gi...@apache.org>.
MaxGekk commented on issue #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server
URL: https://github.com/apache/spark/pull/27552#issuecomment-586766722
jenkins, retest this, please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27552: [WIP] Enable
Java 8 time API in Thrift server
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27552: [WIP] Enable Java 8 time API in Thrift server
URL: https://github.com/apache/spark/pull/27552#issuecomment-585395017
**[Test build #118319 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118319/testReport)** for PR 27552 at commit [`916838a`](https://github.com/apache/spark/commit/916838a3d43aeac59cdc799fed0de8d279b0ad66).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27552: [SPARK-30808][SQL]
Enable Java 8 time API in Thrift server
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server
URL: https://github.com/apache/spark/pull/27552#issuecomment-585645012
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118347/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27552:
[SPARK-30808][SQL] Enable Java 8 time API in Thrift server
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server
URL: https://github.com/apache/spark/pull/27552#issuecomment-586749226
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118508/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk edited a comment on issue #27552:
[SPARK-30808][SQL] Enable Java 8 time API in Thrift server
Posted by GitBox <gi...@apache.org>.
MaxGekk edited a comment on issue #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server
URL: https://github.com/apache/spark/pull/27552#issuecomment-586947271
While debugging the `test ambiguousReferences resolved as hive` test from `org.apache.spark.sql.hive.execution.HiveResolutionSuite`, I see CreateDataSourceTableCommand is invoke twice for the same table name **t1** :
1. In `lazy val dataset = Dataset.ofRows(sparkSession, logical)` inside of `TestHiveQueryExecution`
2. In `hiveResultString()`:
```scala
val result: Seq[Seq[Any]] = Dataset.ofRows(ds.sparkSession, ds.queryExecution.logical)
.queryExecution
.executedPlan
.executeCollectPublic().map(_.toSeq).toSeq
```
This happens due to logical plan has side effects:
https://github.com/apache/spark/blob/e2d3983de78f5c80fac066b7ee8bedd0987110dd/sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala#L226
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27552: [SPARK-30808][SQL]
Enable Java 8 time API in Thrift server
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server
URL: https://github.com/apache/spark/pull/27552#issuecomment-586853991
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27552:
[SPARK-30808][SQL] Enable Java 8 time API in Thrift server
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server
URL: https://github.com/apache/spark/pull/27552#issuecomment-586866062
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27552:
[SPARK-30808][SQL] Enable Java 8 time API in Thrift server
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server
URL: https://github.com/apache/spark/pull/27552#issuecomment-586846187
**[Test build #118543 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118543/testReport)** for PR 27552 at commit [`e491f96`](https://github.com/apache/spark/commit/e491f966b23a734358b222ed8486a547467a22ed).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27552: [SPARK-30808][SQL]
Enable Java 8 time API in Thrift server
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server
URL: https://github.com/apache/spark/pull/27552#issuecomment-585649109
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27552:
[SPARK-30808][SQL] Enable Java 8 time API in Thrift server
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server
URL: https://github.com/apache/spark/pull/27552#issuecomment-585649121
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23109/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27552: [SPARK-30808][SQL]
Enable Java 8 time API in Thrift server
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server
URL: https://github.com/apache/spark/pull/27552#issuecomment-586933972
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27552: [SPARK-30808][SQL]
Enable Java 8 time API in Thrift server
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server
URL: https://github.com/apache/spark/pull/27552#issuecomment-586864227
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118549/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27552:
[SPARK-30808][SQL] Enable Java 8 time API in Thrift server
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server
URL: https://github.com/apache/spark/pull/27552#issuecomment-586853991
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27552: [SPARK-30808][SQL]
Enable Java 8 time API in Thrift server
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server
URL: https://github.com/apache/spark/pull/27552#issuecomment-586748608
**[Test build #118509 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118509/testReport)** for PR 27552 at commit [`804e709`](https://github.com/apache/spark/commit/804e709860357c5957a1b44e85acc1a7f41ebef3).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27552: [SPARK-30808][SQL]
Enable Java 8 time API in Thrift server
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server
URL: https://github.com/apache/spark/pull/27552#issuecomment-586749224
Build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27552: [SPARK-30808][SQL]
Enable Java 8 time API in Thrift server
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server
URL: https://github.com/apache/spark/pull/27552#issuecomment-586980064
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23333/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27552:
[SPARK-30808][SQL] Enable Java 8 time API in Thrift server
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server
URL: https://github.com/apache/spark/pull/27552#issuecomment-586774294
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27552:
[SPARK-30808][SQL] Enable Java 8 time API in Thrift server
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server
URL: https://github.com/apache/spark/pull/27552#issuecomment-585645012
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118347/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27552: [SPARK-30808][SQL]
Enable Java 8 time API in Thrift server
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server
URL: https://github.com/apache/spark/pull/27552#issuecomment-586767247
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27552:
[SPARK-30808][SQL] Enable Java 8 time API in Thrift server
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server
URL: https://github.com/apache/spark/pull/27552#issuecomment-585648495
**[Test build #118352 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118352/testReport)** for PR 27552 at commit [`580dd09`](https://github.com/apache/spark/commit/580dd09ca9e3b8df54fcf31806913e581e4da720).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27552:
[SPARK-30808][SQL] Enable Java 8 time API in Thrift server
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server
URL: https://github.com/apache/spark/pull/27552#issuecomment-586741695
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23264/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27552: [SPARK-30808][SQL]
Enable Java 8 time API in Thrift server
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server
URL: https://github.com/apache/spark/pull/27552#issuecomment-586903804
**[Test build #118552 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118552/testReport)** for PR 27552 at commit [`f9112b7`](https://github.com/apache/spark/commit/f9112b7ab9bb654d880a8bdf0ad28a78c3da0712).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27552: [WIP] Enable Java
8 time API in Thrift server
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27552: [WIP] Enable Java 8 time API in Thrift server
URL: https://github.com/apache/spark/pull/27552#issuecomment-585395511
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on issue #27552: [SPARK-30808][SQL]
Enable Java 8 time API in Thrift server
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on issue #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server
URL: https://github.com/apache/spark/pull/27552#issuecomment-586083994
@MaxGekk it's possible from a logical plan, e.g. `Dataset.ofRows(session, df.logicalPlan)`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27552: [SPARK-30808][SQL]
Enable Java 8 time API in Thrift server
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server
URL: https://github.com/apache/spark/pull/27552#issuecomment-586927280
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27552: [SPARK-30808][SQL]
Enable Java 8 time API in Thrift server
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server
URL: https://github.com/apache/spark/pull/27552#issuecomment-586859868
**[Test build #118549 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118549/testReport)** for PR 27552 at commit [`ce77628`](https://github.com/apache/spark/commit/ce77628cad487d7d86cb33577a699ac37bf8f5fd).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] yaooqinn commented on a change in pull request #27552:
[SPARK-30808][SQL] Enable Java 8 time API in Thrift server
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on a change in pull request #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server
URL: https://github.com/apache/spark/pull/27552#discussion_r385506472
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/HiveResult.scala
##########
@@ -36,27 +37,43 @@ object HiveResult {
* Returns the result as a hive compatible sequence of strings. This is used in tests and
* `SparkSQLDriver` for CLI applications.
*/
- def hiveResultString(executedPlan: SparkPlan): Seq[String] = executedPlan match {
- case ExecutedCommandExec(_: DescribeCommandBase) =>
- // If it is a describe command for a Hive table, we want to have the output format
- // be similar with Hive.
- executedPlan.executeCollectPublic().map {
- case Row(name: String, dataType: String, comment) =>
- Seq(name, dataType,
- Option(comment.asInstanceOf[String]).getOrElse(""))
- .map(s => String.format(s"%-20s", s))
- .mkString("\t")
- }
- // SHOW TABLES in Hive only output table names, while ours output database, table name, isTemp.
- case command @ ExecutedCommandExec(s: ShowTablesCommand) if !s.isExtended =>
- command.executeCollect().map(_.getString(1))
- case other =>
- val result: Seq[Seq[Any]] = other.executeCollectPublic().map(_.toSeq).toSeq
- // We need the types so we can output struct field names
- val types = executedPlan.output.map(_.dataType)
- // Reformat to match hive tab delimited output.
- result.map(_.zip(types).map(e => toHiveString(e)))
- .map(_.mkString("\t"))
+ def hiveResultString(ds: Dataset[_]): Seq[String] = {
+ val executedPlan = ds.queryExecution.executedPlan
+ executedPlan match {
+ case ExecutedCommandExec(_: DescribeCommandBase) =>
+ // If it is a describe command for a Hive table, we want to have the output format
+ // be similar with Hive.
+ executedPlan.executeCollectPublic().map {
+ case Row(name: String, dataType: String, comment) =>
+ Seq(name, dataType,
+ Option(comment.asInstanceOf[String]).getOrElse(""))
+ .map(s => String.format(s"%-20s", s))
+ .mkString("\t")
+ }
+ // SHOW TABLES in Hive only output table names,
+ // while ours output database, table name, isTemp.
+ case command @ ExecutedCommandExec(s: ShowTablesCommand) if !s.isExtended =>
+ command.executeCollect().map(_.getString(1))
+ case _ =>
+ val sessionWithJava8DatetimeEnabled = {
+ val cloned = ds.sparkSession.cloneSession()
+ cloned.conf.set(SQLConf.DATETIME_JAVA8API_ENABLED.key, true)
Review comment:
why is this always `true`?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #27552:
[SPARK-30808][SQL] Enable Java 8 time API in Thrift server
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server
URL: https://github.com/apache/spark/pull/27552#discussion_r379986770
##########
File path: sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/SparkExecuteStatementOperation.scala
##########
@@ -178,7 +179,12 @@ private[hive] class SparkExecuteStatementOperation(
}
curCol += 1
}
- resultRowSet.addRow(row.toArray.asInstanceOf[Array[Object]])
+ val resultRow = row.map {
Review comment:
can you add a code comment? Say that the results are passed to Hive and Hive only accepts `Timestamp`/`Date`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27552:
[SPARK-30808][SQL] Enable Java 8 time API in Thrift server
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server
URL: https://github.com/apache/spark/pull/27552#issuecomment-586767250
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23266/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan edited a comment on issue #27552:
[SPARK-30808][SQL] Enable Java 8 time API in Thrift server
Posted by GitBox <gi...@apache.org>.
cloud-fan edited a comment on issue #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server
URL: https://github.com/apache/spark/pull/27552#issuecomment-586964319
@MaxGekk I see the problem now. We should use `ds.logicalPlan` instead of `ds.queryExecution.logical`. As `ds.logicalPlan` can be `LocalRelation` to hold the command result.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27552: [SPARK-30808][SQL]
Enable Java 8 time API in Thrift server
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server
URL: https://github.com/apache/spark/pull/27552#issuecomment-586764227
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27552: [SPARK-30808][SQL]
Enable Java 8 time API in Thrift server
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server
URL: https://github.com/apache/spark/pull/27552#issuecomment-586927286
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23324/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27552: [SPARK-30808][SQL]
Enable Java 8 time API in Thrift server
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server
URL: https://github.com/apache/spark/pull/27552#issuecomment-586979510
**[Test build #118579 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118579/testReport)** for PR 27552 at commit [`880b1de`](https://github.com/apache/spark/commit/880b1dec95595b58688a3d5f7a2e0637a090cb47).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27552: [SPARK-30808][SQL]
Enable Java 8 time API in Thrift server
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server
URL: https://github.com/apache/spark/pull/27552#issuecomment-586749226
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118508/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk commented on issue #27552: [SPARK-30808][SQL]
Enable Java 8 time API in Thrift server
Posted by GitBox <gi...@apache.org>.
MaxGekk commented on issue #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server
URL: https://github.com/apache/spark/pull/27552#issuecomment-585992099
> We can create a new Dataset to fix it ...
@cloud-fan Is it possible to create/restore a Dataset from an executedPlan?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27552:
[SPARK-30808][SQL] Enable Java 8 time API in Thrift server
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server
URL: https://github.com/apache/spark/pull/27552#issuecomment-586859868
**[Test build #118549 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118549/testReport)** for PR 27552 at commit [`ce77628`](https://github.com/apache/spark/commit/ce77628cad487d7d86cb33577a699ac37bf8f5fd).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27552:
[SPARK-30808][SQL] Enable Java 8 time API in Thrift server
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server
URL: https://github.com/apache/spark/pull/27552#issuecomment-586740958
Build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27552:
[SPARK-30808][SQL] Enable Java 8 time API in Thrift server
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server
URL: https://github.com/apache/spark/pull/27552#issuecomment-586846544
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27552:
[SPARK-30808][SQL] Enable Java 8 time API in Thrift server
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server
URL: https://github.com/apache/spark/pull/27552#issuecomment-586933980
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118569/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk commented on issue #27552: [SPARK-30808][SQL]
Enable Java 8 time API in Thrift server
Posted by GitBox <gi...@apache.org>.
MaxGekk commented on issue #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server
URL: https://github.com/apache/spark/pull/27552#issuecomment-586941903
I would prefer to cut off this PR at this point https://github.com/apache/spark/pull/27552#issuecomment-585664688, and implement moving settings of `spark.sql.datetime.java8API.enabled` to hiveResult() separately.
Making dataset as lazy val doesn't help me, so, I stuck for now.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27552:
[SPARK-30808][SQL] Enable Java 8 time API in Thrift server
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server
URL: https://github.com/apache/spark/pull/27552#issuecomment-586980064
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23333/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #27552:
[SPARK-30808][SQL] Enable Java 8 time API in Thrift server
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server
URL: https://github.com/apache/spark/pull/27552#discussion_r380014884
##########
File path: sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/SparkSQLDriver.scala
##########
@@ -60,9 +60,10 @@ private[hive] class SparkSQLDriver(val context: SQLContext = SparkSQLEnv.sqlCont
// TODO unify the error code
try {
context.sparkContext.setJobDescription(command)
+ val df = context.sql(command)
val execution = context.sessionState.executePlan(context.sql(command).logicalPlan)
hiveResponse = SQLExecution.withNewExecutionId(execution) {
Review comment:
`SQLExecution.withNewExecutionId(df.queryExecution)`?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27552: [SPARK-30808][SQL]
Enable Java 8 time API in Thrift server
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server
URL: https://github.com/apache/spark/pull/27552#issuecomment-586926693
**[Test build #118569 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118569/testReport)** for PR 27552 at commit [`2129f30`](https://github.com/apache/spark/commit/2129f30c8f54c669405db302dbfb52f78631fc05).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27552: [SPARK-30808][SQL]
Enable Java 8 time API in Thrift server
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server
URL: https://github.com/apache/spark/pull/27552#issuecomment-586774258
**[Test build #118511 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118511/testReport)** for PR 27552 at commit [`80f4c89`](https://github.com/apache/spark/commit/80f4c891d3da7bbefcd5738cb61a8d4dabfa1a96).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27552: [SPARK-30808][SQL]
Enable Java 8 time API in Thrift server
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server
URL: https://github.com/apache/spark/pull/27552#issuecomment-586748629
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #27552:
[SPARK-30808][SQL] Enable Java 8 time API in Thrift server
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server
URL: https://github.com/apache/spark/pull/27552#discussion_r380014991
##########
File path: sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/SparkSQLEnv.scala
##########
@@ -24,6 +24,7 @@ import org.apache.spark.{SparkConf, SparkContext}
import org.apache.spark.internal.Logging
import org.apache.spark.sql.{SparkSession, SQLContext}
import org.apache.spark.sql.hive.{HiveExternalCatalog, HiveUtils}
+import org.apache.spark.sql.internal.SQLConf
Review comment:
unnecessary change?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27552: [SPARK-30808][SQL]
Enable Java 8 time API in Thrift server
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server
URL: https://github.com/apache/spark/pull/27552#issuecomment-585644893
**[Test build #118347 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118347/testReport)** for PR 27552 at commit [`d98cdbc`](https://github.com/apache/spark/commit/d98cdbcb85ba7df6d24429081da97f5619cfe54b).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27552: [WIP]
Enable Java 8 time API in Thrift server
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27552: [WIP] Enable Java 8 time API in Thrift server
URL: https://github.com/apache/spark/pull/27552#issuecomment-585395511
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #27552:
[SPARK-30808][SQL] Enable Java 8 time API in Thrift server
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server
URL: https://github.com/apache/spark/pull/27552#discussion_r385507806
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/HiveResult.scala
##########
@@ -36,27 +37,43 @@ object HiveResult {
* Returns the result as a hive compatible sequence of strings. This is used in tests and
* `SparkSQLDriver` for CLI applications.
*/
- def hiveResultString(executedPlan: SparkPlan): Seq[String] = executedPlan match {
- case ExecutedCommandExec(_: DescribeCommandBase) =>
- // If it is a describe command for a Hive table, we want to have the output format
- // be similar with Hive.
- executedPlan.executeCollectPublic().map {
- case Row(name: String, dataType: String, comment) =>
- Seq(name, dataType,
- Option(comment.asInstanceOf[String]).getOrElse(""))
- .map(s => String.format(s"%-20s", s))
- .mkString("\t")
- }
- // SHOW TABLES in Hive only output table names, while ours output database, table name, isTemp.
- case command @ ExecutedCommandExec(s: ShowTablesCommand) if !s.isExtended =>
- command.executeCollect().map(_.getString(1))
- case other =>
- val result: Seq[Seq[Any]] = other.executeCollectPublic().map(_.toSeq).toSeq
- // We need the types so we can output struct field names
- val types = executedPlan.output.map(_.dataType)
- // Reformat to match hive tab delimited output.
- result.map(_.zip(types).map(e => toHiveString(e)))
- .map(_.mkString("\t"))
+ def hiveResultString(ds: Dataset[_]): Seq[String] = {
+ val executedPlan = ds.queryExecution.executedPlan
+ executedPlan match {
+ case ExecutedCommandExec(_: DescribeCommandBase) =>
+ // If it is a describe command for a Hive table, we want to have the output format
+ // be similar with Hive.
+ executedPlan.executeCollectPublic().map {
+ case Row(name: String, dataType: String, comment) =>
+ Seq(name, dataType,
+ Option(comment.asInstanceOf[String]).getOrElse(""))
+ .map(s => String.format(s"%-20s", s))
+ .mkString("\t")
+ }
+ // SHOW TABLES in Hive only output table names,
+ // while ours output database, table name, isTemp.
+ case command @ ExecutedCommandExec(s: ShowTablesCommand) if !s.isExtended =>
+ command.executeCollect().map(_.getString(1))
+ case _ =>
+ val sessionWithJava8DatetimeEnabled = {
+ val cloned = ds.sparkSession.cloneSession()
+ cloned.conf.set(SQLConf.DATETIME_JAVA8API_ENABLED.key, true)
Review comment:
oh wait, we format `Date/Timestamp` by our own formatter, so this should be no problem.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27552: [WIP] Enable Java
8 time API in Thrift server
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27552: [WIP] Enable Java 8 time API in Thrift server
URL: https://github.com/apache/spark/pull/27552#issuecomment-585621425
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23104/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #27552:
[SPARK-30808][SQL] Enable Java 8 time API in Thrift server
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server
URL: https://github.com/apache/spark/pull/27552#discussion_r380014294
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/HiveResult.scala
##########
@@ -36,27 +37,41 @@ object HiveResult {
* Returns the result as a hive compatible sequence of strings. This is used in tests and
* `SparkSQLDriver` for CLI applications.
*/
- def hiveResultString(executedPlan: SparkPlan): Seq[String] = executedPlan match {
- case ExecutedCommandExec(_: DescribeCommandBase) =>
- // If it is a describe command for a Hive table, we want to have the output format
- // be similar with Hive.
- executedPlan.executeCollectPublic().map {
- case Row(name: String, dataType: String, comment) =>
- Seq(name, dataType,
- Option(comment.asInstanceOf[String]).getOrElse(""))
- .map(s => String.format(s"%-20s", s))
- .mkString("\t")
- }
- // SHOW TABLES in Hive only output table names, while ours output database, table name, isTemp.
- case command @ ExecutedCommandExec(s: ShowTablesCommand) if !s.isExtended =>
- command.executeCollect().map(_.getString(1))
- case other =>
- val result: Seq[Seq[Any]] = other.executeCollectPublic().map(_.toSeq).toSeq
- // We need the types so we can output struct field names
- val types = executedPlan.output.map(_.dataType)
- // Reformat to match hive tab delimited output.
- result.map(_.zip(types).map(e => toHiveString(e)))
- .map(_.mkString("\t"))
+ def hiveResultString(ds: Dataset[_]): Seq[String] = {
+ val executedPlan = ds.queryExecution.executedPlan
+ executedPlan match {
+ case ExecutedCommandExec(_: DescribeCommandBase) =>
+ // If it is a describe command for a Hive table, we want to have the output format
+ // be similar with Hive.
+ executedPlan.executeCollectPublic().map {
+ case Row(name: String, dataType: String, comment) =>
+ Seq(name, dataType,
+ Option(comment.asInstanceOf[String]).getOrElse(""))
+ .map(s => String.format(s"%-20s", s))
+ .mkString("\t")
+ }
+ // SHOW TABLES in Hive only output table names,
+ // while ours output database, table name, isTemp.
+ case command @ ExecutedCommandExec(s: ShowTablesCommand) if !s.isExtended =>
+ command.executeCollect().map(_.getString(1))
+ case _ =>
+ val sessionWithJava8DatetimeEnabled = {
+ val cloned = ds.sparkSession.cloneSession()
+ cloned.conf.set(SQLConf.DATETIME_JAVA8API_ENABLED.key, true)
+ cloned
+ }
+ sessionWithJava8DatetimeEnabled.withActive {
+ val result: Seq[Seq[Any]] = Dataset.ofRows(ds.sparkSession, ds.queryExecution.logical)
Review comment:
nit: let's add a code comment to explain why we can't collect the input `ds`. Its encoder was already created with java 8 datetime disabled.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] yaooqinn commented on a change in pull request #27552:
[SPARK-30808][SQL] Enable Java 8 time API in Thrift server
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on a change in pull request #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server
URL: https://github.com/apache/spark/pull/27552#discussion_r385526108
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/HiveResult.scala
##########
@@ -36,27 +37,43 @@ object HiveResult {
* Returns the result as a hive compatible sequence of strings. This is used in tests and
* `SparkSQLDriver` for CLI applications.
*/
- def hiveResultString(executedPlan: SparkPlan): Seq[String] = executedPlan match {
- case ExecutedCommandExec(_: DescribeCommandBase) =>
- // If it is a describe command for a Hive table, we want to have the output format
- // be similar with Hive.
- executedPlan.executeCollectPublic().map {
- case Row(name: String, dataType: String, comment) =>
- Seq(name, dataType,
- Option(comment.asInstanceOf[String]).getOrElse(""))
- .map(s => String.format(s"%-20s", s))
- .mkString("\t")
- }
- // SHOW TABLES in Hive only output table names, while ours output database, table name, isTemp.
- case command @ ExecutedCommandExec(s: ShowTablesCommand) if !s.isExtended =>
- command.executeCollect().map(_.getString(1))
- case other =>
- val result: Seq[Seq[Any]] = other.executeCollectPublic().map(_.toSeq).toSeq
- // We need the types so we can output struct field names
- val types = executedPlan.output.map(_.dataType)
- // Reformat to match hive tab delimited output.
- result.map(_.zip(types).map(e => toHiveString(e)))
- .map(_.mkString("\t"))
+ def hiveResultString(ds: Dataset[_]): Seq[String] = {
+ val executedPlan = ds.queryExecution.executedPlan
+ executedPlan match {
+ case ExecutedCommandExec(_: DescribeCommandBase) =>
+ // If it is a describe command for a Hive table, we want to have the output format
+ // be similar with Hive.
+ executedPlan.executeCollectPublic().map {
+ case Row(name: String, dataType: String, comment) =>
+ Seq(name, dataType,
+ Option(comment.asInstanceOf[String]).getOrElse(""))
+ .map(s => String.format(s"%-20s", s))
+ .mkString("\t")
+ }
+ // SHOW TABLES in Hive only output table names,
+ // while ours output database, table name, isTemp.
+ case command @ ExecutedCommandExec(s: ShowTablesCommand) if !s.isExtended =>
+ command.executeCollect().map(_.getString(1))
+ case _ =>
+ val sessionWithJava8DatetimeEnabled = {
+ val cloned = ds.sparkSession.cloneSession()
+ cloned.conf.set(SQLConf.DATETIME_JAVA8API_ENABLED.key, true)
Review comment:
I also pass these tests through `ThriftServerQueryTestSuite`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27552: [SPARK-30808][SQL]
Enable Java 8 time API in Thrift server
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server
URL: https://github.com/apache/spark/pull/27552#issuecomment-586859959
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118543/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27552:
[SPARK-30808][SQL] Enable Java 8 time API in Thrift server
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server
URL: https://github.com/apache/spark/pull/27552#issuecomment-586741690
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27552:
[SPARK-30808][SQL] Enable Java 8 time API in Thrift server
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server
URL: https://github.com/apache/spark/pull/27552#issuecomment-586753677
**[Test build #118510 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118510/testReport)** for PR 27552 at commit [`80f4c89`](https://github.com/apache/spark/commit/80f4c891d3da7bbefcd5738cb61a8d4dabfa1a96).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27552: [WIP]
Enable Java 8 time API in Thrift server
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27552: [WIP] Enable Java 8 time API in Thrift server
URL: https://github.com/apache/spark/pull/27552#issuecomment-585395525
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23077/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27552: [SPARK-30808][SQL]
Enable Java 8 time API in Thrift server
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server
URL: https://github.com/apache/spark/pull/27552#issuecomment-586764182
**[Test build #118510 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118510/testReport)** for PR 27552 at commit [`80f4c89`](https://github.com/apache/spark/commit/80f4c891d3da7bbefcd5738cb61a8d4dabfa1a96).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27552:
[SPARK-30808][SQL] Enable Java 8 time API in Thrift server
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server
URL: https://github.com/apache/spark/pull/27552#issuecomment-586774299
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118511/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27552: [SPARK-30808][SQL]
Enable Java 8 time API in Thrift server
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server
URL: https://github.com/apache/spark/pull/27552#issuecomment-586866068
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23306/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk commented on a change in pull request #27552:
[SPARK-30808][SQL] Enable Java 8 time API in Thrift server
Posted by GitBox <gi...@apache.org>.
MaxGekk commented on a change in pull request #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server
URL: https://github.com/apache/spark/pull/27552#discussion_r378739387
##########
File path: sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/SparkExecuteStatementOperation.scala
##########
@@ -178,7 +179,12 @@ private[hive] class SparkExecuteStatementOperation(
}
curCol += 1
}
- resultRowSet.addRow(row.toArray.asInstanceOf[Array[Object]])
+ val resultRow = row.map {
Review comment:
I have to convert Java 8 classes to `java.sql.Timestamp`/`Date` for correct formatting. I cannot use `TimestampFormatter` here because the conversions to strings can be performed in `ColumnBuffer` in `hive-serde`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27552:
[SPARK-30808][SQL] Enable Java 8 time API in Thrift server
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server
URL: https://github.com/apache/spark/pull/27552#issuecomment-586860312
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23303/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27552:
[SPARK-30808][SQL] Enable Java 8 time API in Thrift server
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server
URL: https://github.com/apache/spark/pull/27552#issuecomment-586904032
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27552: [SPARK-30808][SQL]
Enable Java 8 time API in Thrift server
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server
URL: https://github.com/apache/spark/pull/27552#issuecomment-586740958
Build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27552: [WIP]
Enable Java 8 time API in Thrift server
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27552: [WIP] Enable Java 8 time API in Thrift server
URL: https://github.com/apache/spark/pull/27552#issuecomment-585407373
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118319/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27552: [WIP]
Enable Java 8 time API in Thrift server
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27552: [WIP] Enable Java 8 time API in Thrift server
URL: https://github.com/apache/spark/pull/27552#issuecomment-585621425
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23104/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] yaooqinn commented on a change in pull request #27552:
[SPARK-30808][SQL] Enable Java 8 time API in Thrift server
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on a change in pull request #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server
URL: https://github.com/apache/spark/pull/27552#discussion_r385526108
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/HiveResult.scala
##########
@@ -36,27 +37,43 @@ object HiveResult {
* Returns the result as a hive compatible sequence of strings. This is used in tests and
* `SparkSQLDriver` for CLI applications.
*/
- def hiveResultString(executedPlan: SparkPlan): Seq[String] = executedPlan match {
- case ExecutedCommandExec(_: DescribeCommandBase) =>
- // If it is a describe command for a Hive table, we want to have the output format
- // be similar with Hive.
- executedPlan.executeCollectPublic().map {
- case Row(name: String, dataType: String, comment) =>
- Seq(name, dataType,
- Option(comment.asInstanceOf[String]).getOrElse(""))
- .map(s => String.format(s"%-20s", s))
- .mkString("\t")
- }
- // SHOW TABLES in Hive only output table names, while ours output database, table name, isTemp.
- case command @ ExecutedCommandExec(s: ShowTablesCommand) if !s.isExtended =>
- command.executeCollect().map(_.getString(1))
- case other =>
- val result: Seq[Seq[Any]] = other.executeCollectPublic().map(_.toSeq).toSeq
- // We need the types so we can output struct field names
- val types = executedPlan.output.map(_.dataType)
- // Reformat to match hive tab delimited output.
- result.map(_.zip(types).map(e => toHiveString(e)))
- .map(_.mkString("\t"))
+ def hiveResultString(ds: Dataset[_]): Seq[String] = {
+ val executedPlan = ds.queryExecution.executedPlan
+ executedPlan match {
+ case ExecutedCommandExec(_: DescribeCommandBase) =>
+ // If it is a describe command for a Hive table, we want to have the output format
+ // be similar with Hive.
+ executedPlan.executeCollectPublic().map {
+ case Row(name: String, dataType: String, comment) =>
+ Seq(name, dataType,
+ Option(comment.asInstanceOf[String]).getOrElse(""))
+ .map(s => String.format(s"%-20s", s))
+ .mkString("\t")
+ }
+ // SHOW TABLES in Hive only output table names,
+ // while ours output database, table name, isTemp.
+ case command @ ExecutedCommandExec(s: ShowTablesCommand) if !s.isExtended =>
+ command.executeCollect().map(_.getString(1))
+ case _ =>
+ val sessionWithJava8DatetimeEnabled = {
+ val cloned = ds.sparkSession.cloneSession()
+ cloned.conf.set(SQLConf.DATETIME_JAVA8API_ENABLED.key, true)
Review comment:
I also pass these test through `ThriftServerQueryTestSuite`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27552: [SPARK-30808][SQL]
Enable Java 8 time API in Thrift server
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server
URL: https://github.com/apache/spark/pull/27552#issuecomment-586741571
**[Test build #118509 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118509/testReport)** for PR 27552 at commit [`804e709`](https://github.com/apache/spark/commit/804e709860357c5957a1b44e85acc1a7f41ebef3).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27552: [SPARK-30808][SQL]
Enable Java 8 time API in Thrift server
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server
URL: https://github.com/apache/spark/pull/27552#issuecomment-586848299
**[Test build #118545 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118545/testReport)** for PR 27552 at commit [`95597a7`](https://github.com/apache/spark/commit/95597a7244e2c99a9ae267e5705748ad8403a169).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on issue #27552: [SPARK-30808][SQL]
Enable Java 8 time API in Thrift server
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on issue #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server
URL: https://github.com/apache/spark/pull/27552#issuecomment-587109451
thanks, merging to master/3.0!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on issue #27552: [SPARK-30808][SQL]
Enable Java 8 time API in Thrift server
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on issue #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server
URL: https://github.com/apache/spark/pull/27552#issuecomment-585663847
> but it doesn't work - the SQL config didn't change the behavior
It's probably because the `Dataset` is already there and its encoder is already created, which decides to return old timestamp/date. We can create a new `Dataset` to fix it, e.g. `df.filter(lit("true")).collect`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #27552:
[SPARK-30808][SQL] Enable Java 8 time API in Thrift server
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server
URL: https://github.com/apache/spark/pull/27552#discussion_r385506809
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/HiveResult.scala
##########
@@ -36,27 +37,43 @@ object HiveResult {
* Returns the result as a hive compatible sequence of strings. This is used in tests and
* `SparkSQLDriver` for CLI applications.
*/
- def hiveResultString(executedPlan: SparkPlan): Seq[String] = executedPlan match {
- case ExecutedCommandExec(_: DescribeCommandBase) =>
- // If it is a describe command for a Hive table, we want to have the output format
- // be similar with Hive.
- executedPlan.executeCollectPublic().map {
- case Row(name: String, dataType: String, comment) =>
- Seq(name, dataType,
- Option(comment.asInstanceOf[String]).getOrElse(""))
- .map(s => String.format(s"%-20s", s))
- .mkString("\t")
- }
- // SHOW TABLES in Hive only output table names, while ours output database, table name, isTemp.
- case command @ ExecutedCommandExec(s: ShowTablesCommand) if !s.isExtended =>
- command.executeCollect().map(_.getString(1))
- case other =>
- val result: Seq[Seq[Any]] = other.executeCollectPublic().map(_.toSeq).toSeq
- // We need the types so we can output struct field names
- val types = executedPlan.output.map(_.dataType)
- // Reformat to match hive tab delimited output.
- result.map(_.zip(types).map(e => toHiveString(e)))
- .map(_.mkString("\t"))
+ def hiveResultString(ds: Dataset[_]): Seq[String] = {
+ val executedPlan = ds.queryExecution.executedPlan
+ executedPlan match {
+ case ExecutedCommandExec(_: DescribeCommandBase) =>
+ // If it is a describe command for a Hive table, we want to have the output format
+ // be similar with Hive.
+ executedPlan.executeCollectPublic().map {
+ case Row(name: String, dataType: String, comment) =>
+ Seq(name, dataType,
+ Option(comment.asInstanceOf[String]).getOrElse(""))
+ .map(s => String.format(s"%-20s", s))
+ .mkString("\t")
+ }
+ // SHOW TABLES in Hive only output table names,
+ // while ours output database, table name, isTemp.
+ case command @ ExecutedCommandExec(s: ShowTablesCommand) if !s.isExtended =>
+ command.executeCollect().map(_.getString(1))
+ case _ =>
+ val sessionWithJava8DatetimeEnabled = {
+ val cloned = ds.sparkSession.cloneSession()
+ cloned.conf.set(SQLConf.DATETIME_JAVA8API_ENABLED.key, true)
Review comment:
the old `Date`/`Timestamp` doesn't follow the new calendar and may produce wrong string for some date/timestamp values.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27552: [SPARK-30808][SQL]
Enable Java 8 time API in Thrift server
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server
URL: https://github.com/apache/spark/pull/27552#issuecomment-586860312
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23303/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on issue #27552: [SPARK-30808][SQL]
Enable Java 8 time API in Thrift server
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on issue #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server
URL: https://github.com/apache/spark/pull/27552#issuecomment-586935416
@MaxGekk yea creating the df again may execute the command again. Let's keep the lazy val.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27552: [SPARK-30808][SQL]
Enable Java 8 time API in Thrift server
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server
URL: https://github.com/apache/spark/pull/27552#issuecomment-586980048
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27552:
[SPARK-30808][SQL] Enable Java 8 time API in Thrift server
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server
URL: https://github.com/apache/spark/pull/27552#issuecomment-586853998
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118545/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27552:
[SPARK-30808][SQL] Enable Java 8 time API in Thrift server
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server
URL: https://github.com/apache/spark/pull/27552#issuecomment-586866068
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23306/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27552: [WIP] Enable Java
8 time API in Thrift server
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27552: [WIP] Enable Java 8 time API in Thrift server
URL: https://github.com/apache/spark/pull/27552#issuecomment-585621415
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] yaooqinn commented on a change in pull request #27552:
[SPARK-30808][SQL] Enable Java 8 time API in Thrift server
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on a change in pull request #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server
URL: https://github.com/apache/spark/pull/27552#discussion_r385493404
##########
File path: sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/SparkExecuteStatementOperation.scala
##########
@@ -178,7 +179,14 @@ private[hive] class SparkExecuteStatementOperation(
}
curCol += 1
}
- resultRowSet.addRow(row.toArray.asInstanceOf[Array[Object]])
+ // Convert date-time instances to types that are acceptable by Hive libs
+ // used in conversions to strings.
+ val resultRow = row.map {
+ case i: Instant => Timestamp.from(i)
Review comment:
There seems no java8 datetime values to be add to the `row` buffer here by `SparkExecuteStatementOperation#addNonNullColumnValue `
https://github.com/apache/spark/pull/27552/files#diff-72dcd8f81a51c8a815159fdf0332acdcR84-R116
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] yaooqinn commented on a change in pull request #27552:
[SPARK-30808][SQL] Enable Java 8 time API in Thrift server
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on a change in pull request #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server
URL: https://github.com/apache/spark/pull/27552#discussion_r385533487
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/HiveResult.scala
##########
@@ -36,27 +37,43 @@ object HiveResult {
* Returns the result as a hive compatible sequence of strings. This is used in tests and
* `SparkSQLDriver` for CLI applications.
*/
- def hiveResultString(executedPlan: SparkPlan): Seq[String] = executedPlan match {
- case ExecutedCommandExec(_: DescribeCommandBase) =>
- // If it is a describe command for a Hive table, we want to have the output format
- // be similar with Hive.
- executedPlan.executeCollectPublic().map {
- case Row(name: String, dataType: String, comment) =>
- Seq(name, dataType,
- Option(comment.asInstanceOf[String]).getOrElse(""))
- .map(s => String.format(s"%-20s", s))
- .mkString("\t")
- }
- // SHOW TABLES in Hive only output table names, while ours output database, table name, isTemp.
- case command @ ExecutedCommandExec(s: ShowTablesCommand) if !s.isExtended =>
- command.executeCollect().map(_.getString(1))
- case other =>
- val result: Seq[Seq[Any]] = other.executeCollectPublic().map(_.toSeq).toSeq
- // We need the types so we can output struct field names
- val types = executedPlan.output.map(_.dataType)
- // Reformat to match hive tab delimited output.
- result.map(_.zip(types).map(e => toHiveString(e)))
- .map(_.mkString("\t"))
+ def hiveResultString(ds: Dataset[_]): Seq[String] = {
+ val executedPlan = ds.queryExecution.executedPlan
+ executedPlan match {
+ case ExecutedCommandExec(_: DescribeCommandBase) =>
+ // If it is a describe command for a Hive table, we want to have the output format
+ // be similar with Hive.
+ executedPlan.executeCollectPublic().map {
+ case Row(name: String, dataType: String, comment) =>
+ Seq(name, dataType,
+ Option(comment.asInstanceOf[String]).getOrElse(""))
+ .map(s => String.format(s"%-20s", s))
+ .mkString("\t")
+ }
+ // SHOW TABLES in Hive only output table names,
+ // while ours output database, table name, isTemp.
+ case command @ ExecutedCommandExec(s: ShowTablesCommand) if !s.isExtended =>
+ command.executeCollect().map(_.getString(1))
+ case _ =>
+ val sessionWithJava8DatetimeEnabled = {
+ val cloned = ds.sparkSession.cloneSession()
+ cloned.conf.set(SQLConf.DATETIME_JAVA8API_ENABLED.key, true)
Review comment:
OK
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27552:
[SPARK-30808][SQL] Enable Java 8 time API in Thrift server
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server
URL: https://github.com/apache/spark/pull/27552#issuecomment-586752514
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23265/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27552:
[SPARK-30808][SQL] Enable Java 8 time API in Thrift server
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server
URL: https://github.com/apache/spark/pull/27552#issuecomment-586859947
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27552:
[SPARK-30808][SQL] Enable Java 8 time API in Thrift server
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server
URL: https://github.com/apache/spark/pull/27552#issuecomment-586764229
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118510/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27552:
[SPARK-30808][SQL] Enable Java 8 time API in Thrift server
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server
URL: https://github.com/apache/spark/pull/27552#issuecomment-586767088
**[Test build #118511 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118511/testReport)** for PR 27552 at commit [`80f4c89`](https://github.com/apache/spark/commit/80f4c891d3da7bbefcd5738cb61a8d4dabfa1a96).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk commented on issue #27552: [SPARK-30808][SQL]
Enable Java 8 time API in Thrift server
Posted by GitBox <gi...@apache.org>.
MaxGekk commented on issue #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server
URL: https://github.com/apache/spark/pull/27552#issuecomment-586934685
@cloud-fan Something wrong is going on here. Commands issues from HiveComparisonTest are executed twice, it seems.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27552:
[SPARK-30808][SQL] Enable Java 8 time API in Thrift server
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server
URL: https://github.com/apache/spark/pull/27552#issuecomment-585623968
**[Test build #118347 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118347/testReport)** for PR 27552 at commit [`d98cdbc`](https://github.com/apache/spark/commit/d98cdbcb85ba7df6d24429081da97f5619cfe54b).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27552: [SPARK-30808][SQL]
Enable Java 8 time API in Thrift server
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server
URL: https://github.com/apache/spark/pull/27552#issuecomment-587108057
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] yaooqinn commented on a change in pull request #27552:
[SPARK-30808][SQL] Enable Java 8 time API in Thrift server
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on a change in pull request #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server
URL: https://github.com/apache/spark/pull/27552#discussion_r385503250
##########
File path: sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/SparkExecuteStatementOperation.scala
##########
@@ -178,7 +179,14 @@ private[hive] class SparkExecuteStatementOperation(
}
curCol += 1
}
- resultRowSet.addRow(row.toArray.asInstanceOf[Array[Object]])
+ // Convert date-time instances to types that are acceptable by Hive libs
+ // used in conversions to strings.
+ val resultRow = row.map {
+ case i: Instant => Timestamp.from(i)
Review comment:
We are limited by `hive-jdbc` module, see https://github.com/apache/hive/blob/a7e704c679a00db68db9b9f921d133d79a32cfcc/jdbc/src/java/org/apache/hive/jdbc/HiveBaseResultSet.java#L427-L457, we might need our own jdbc driver implementation to achieve this
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27552: [SPARK-30808][SQL]
Enable Java 8 time API in Thrift server
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server
URL: https://github.com/apache/spark/pull/27552#issuecomment-586846551
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23298/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27552:
[SPARK-30808][SQL] Enable Java 8 time API in Thrift server
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server
URL: https://github.com/apache/spark/pull/27552#issuecomment-585664861
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118352/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27552: [WIP] Enable Java 8 time
API in Thrift server
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27552: [WIP] Enable Java 8 time API in Thrift server
URL: https://github.com/apache/spark/pull/27552#issuecomment-585395017
**[Test build #118319 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118319/testReport)** for PR 27552 at commit [`916838a`](https://github.com/apache/spark/commit/916838a3d43aeac59cdc799fed0de8d279b0ad66).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27552: [WIP]
Enable Java 8 time API in Thrift server
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27552: [WIP] Enable Java 8 time API in Thrift server
URL: https://github.com/apache/spark/pull/27552#issuecomment-585407367
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #27552:
[SPARK-30808][SQL] Enable Java 8 time API in Thrift server
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server
URL: https://github.com/apache/spark/pull/27552#discussion_r385115198
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/SQLQueryTestSuite.scala
##########
@@ -512,7 +511,7 @@ class SQLQueryTestSuite extends QueryTest with SharedSparkSession {
val schema = df.schema.catalogString
// Get answer, but also get rid of the #1234 expression ids that show up in explain plans
val answer = SQLExecution.withNewExecutionId(df.queryExecution, Some(sql)) {
- hiveResultString(df.queryExecution.executedPlan).map(replaceNotIncludedMsg)
+ hiveResultString(df).map(replaceNotIncludedMsg)
Review comment:
in `ThriftServerQueryTestSuite`, we get the result by JDBC, so there is no DataFrame created.
We should follow pgsql and return java 8 datetime when the config is enabled. https://jdbc.postgresql.org/documentation/head/8-date-time.html
cc @wangyum @yaooqinn
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27552: [SPARK-30808][SQL]
Enable Java 8 time API in Thrift server
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server
URL: https://github.com/apache/spark/pull/27552#issuecomment-586904049
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118552/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27552:
[SPARK-30808][SQL] Enable Java 8 time API in Thrift server
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server
URL: https://github.com/apache/spark/pull/27552#issuecomment-586980048
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan closed pull request #27552: [SPARK-30808][SQL]
Enable Java 8 time API in Thrift server
Posted by GitBox <gi...@apache.org>.
cloud-fan closed pull request #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server
URL: https://github.com/apache/spark/pull/27552
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27552:
[SPARK-30808][SQL] Enable Java 8 time API in Thrift server
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server
URL: https://github.com/apache/spark/pull/27552#issuecomment-585644999
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27552:
[SPARK-30808][SQL] Enable Java 8 time API in Thrift server
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server
URL: https://github.com/apache/spark/pull/27552#issuecomment-586927280
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27552: [SPARK-30808][SQL]
Enable Java 8 time API in Thrift server
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server
URL: https://github.com/apache/spark/pull/27552#issuecomment-586853998
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118545/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk commented on issue #27552: [SPARK-30808][SQL]
Enable Java 8 time API in Thrift server
Posted by GitBox <gi...@apache.org>.
MaxGekk commented on issue #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server
URL: https://github.com/apache/spark/pull/27552#issuecomment-586947271
While debugging the `test ambiguousReferences resolved as hive` test from `org.apache.spark.sql.hive.execution.HiveResolutionSuite`, I see CreateDataSourceTableCommand is invoke twice for the same table name **t1** :
1. In `lazy val dataset = Dataset.ofRows(sparkSession, logical)` inside of `TestHiveQueryExecution`
2. In `hiveResultString()`:
```scala
val result: Seq[Seq[Any]] = Dataset.ofRows(ds.sparkSession, ds.queryExecution.logical)
.queryExecution
.executedPlan
.executeCollectPublic().map(_.toSeq).toSeq
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27552:
[SPARK-30808][SQL] Enable Java 8 time API in Thrift server
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server
URL: https://github.com/apache/spark/pull/27552#issuecomment-586860302
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27552: [WIP] Enable Java
8 time API in Thrift server
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27552: [WIP] Enable Java 8 time API in Thrift server
URL: https://github.com/apache/spark/pull/27552#issuecomment-585407367
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27552: [SPARK-30808][SQL]
Enable Java 8 time API in Thrift server
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server
URL: https://github.com/apache/spark/pull/27552#issuecomment-586865476
**[Test build #118552 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118552/testReport)** for PR 27552 at commit [`f9112b7`](https://github.com/apache/spark/commit/f9112b7ab9bb654d880a8bdf0ad28a78c3da0712).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27552:
[SPARK-30808][SQL] Enable Java 8 time API in Thrift server
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server
URL: https://github.com/apache/spark/pull/27552#issuecomment-586748630
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118509/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27552: [SPARK-30808][SQL]
Enable Java 8 time API in Thrift server
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server
URL: https://github.com/apache/spark/pull/27552#issuecomment-586764229
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118510/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27552: [SPARK-30808][SQL]
Enable Java 8 time API in Thrift server
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server
URL: https://github.com/apache/spark/pull/27552#issuecomment-586767088
**[Test build #118511 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118511/testReport)** for PR 27552 at commit [`80f4c89`](https://github.com/apache/spark/commit/80f4c891d3da7bbefcd5738cb61a8d4dabfa1a96).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27552: [SPARK-30808][SQL]
Enable Java 8 time API in Thrift server
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server
URL: https://github.com/apache/spark/pull/27552#issuecomment-586846544
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27552: [SPARK-30808][SQL]
Enable Java 8 time API in Thrift server
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server
URL: https://github.com/apache/spark/pull/27552#issuecomment-586866062
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27552: [WIP] Enable Java
8 time API in Thrift server
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27552: [WIP] Enable Java 8 time API in Thrift server
URL: https://github.com/apache/spark/pull/27552#issuecomment-585395525
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23077/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27552: [WIP] Enable Java
8 time API in Thrift server
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27552: [WIP] Enable Java 8 time API in Thrift server
URL: https://github.com/apache/spark/pull/27552#issuecomment-585407373
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118319/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27552:
[SPARK-30808][SQL] Enable Java 8 time API in Thrift server
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server
URL: https://github.com/apache/spark/pull/27552#issuecomment-586764227
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] yaooqinn commented on a change in pull request #27552:
[SPARK-30808][SQL] Enable Java 8 time API in Thrift server
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on a change in pull request #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server
URL: https://github.com/apache/spark/pull/27552#discussion_r385527140
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/HiveResult.scala
##########
@@ -36,27 +37,43 @@ object HiveResult {
* Returns the result as a hive compatible sequence of strings. This is used in tests and
* `SparkSQLDriver` for CLI applications.
*/
- def hiveResultString(executedPlan: SparkPlan): Seq[String] = executedPlan match {
- case ExecutedCommandExec(_: DescribeCommandBase) =>
- // If it is a describe command for a Hive table, we want to have the output format
- // be similar with Hive.
- executedPlan.executeCollectPublic().map {
- case Row(name: String, dataType: String, comment) =>
- Seq(name, dataType,
- Option(comment.asInstanceOf[String]).getOrElse(""))
- .map(s => String.format(s"%-20s", s))
- .mkString("\t")
- }
- // SHOW TABLES in Hive only output table names, while ours output database, table name, isTemp.
- case command @ ExecutedCommandExec(s: ShowTablesCommand) if !s.isExtended =>
- command.executeCollect().map(_.getString(1))
- case other =>
- val result: Seq[Seq[Any]] = other.executeCollectPublic().map(_.toSeq).toSeq
- // We need the types so we can output struct field names
- val types = executedPlan.output.map(_.dataType)
- // Reformat to match hive tab delimited output.
- result.map(_.zip(types).map(e => toHiveString(e)))
- .map(_.mkString("\t"))
+ def hiveResultString(ds: Dataset[_]): Seq[String] = {
+ val executedPlan = ds.queryExecution.executedPlan
+ executedPlan match {
+ case ExecutedCommandExec(_: DescribeCommandBase) =>
+ // If it is a describe command for a Hive table, we want to have the output format
+ // be similar with Hive.
+ executedPlan.executeCollectPublic().map {
+ case Row(name: String, dataType: String, comment) =>
+ Seq(name, dataType,
+ Option(comment.asInstanceOf[String]).getOrElse(""))
+ .map(s => String.format(s"%-20s", s))
+ .mkString("\t")
+ }
+ // SHOW TABLES in Hive only output table names,
+ // while ours output database, table name, isTemp.
+ case command @ ExecutedCommandExec(s: ShowTablesCommand) if !s.isExtended =>
+ command.executeCollect().map(_.getString(1))
+ case _ =>
+ val sessionWithJava8DatetimeEnabled = {
+ val cloned = ds.sparkSession.cloneSession()
+ cloned.conf.set(SQLConf.DATETIME_JAVA8API_ENABLED.key, true)
Review comment:
```
spark-sql> set spark.sql.session.timeZone=America/Los_Angeles;
spark.sql.session.timeZone America/Los_Angeles
spark-sql> SELECT DATE_TRUNC('MILLENNIUM', DATE '1970-03-20');
1001-01-01 00:00:00
spark-sql> select version();
3.1.0 b3dcb63a682bc31827a86cf381f157a81e9e07ac
```
Also correct with `bin/spark-sql`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #27552:
[SPARK-30808][SQL] Enable Java 8 time API in Thrift server
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server
URL: https://github.com/apache/spark/pull/27552#discussion_r380014884
##########
File path: sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/SparkSQLDriver.scala
##########
@@ -60,9 +60,10 @@ private[hive] class SparkSQLDriver(val context: SQLContext = SparkSQLEnv.sqlCont
// TODO unify the error code
try {
context.sparkContext.setJobDescription(command)
+ val df = context.sql(command)
val execution = context.sessionState.executePlan(context.sql(command).logicalPlan)
hiveResponse = SQLExecution.withNewExecutionId(execution) {
Review comment:
`SQLExecution.withNewExecutionId(df.queryExecution)`?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27552: [SPARK-30808][SQL]
Enable Java 8 time API in Thrift server
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server
URL: https://github.com/apache/spark/pull/27552#issuecomment-586904032
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27552: [SPARK-30808][SQL]
Enable Java 8 time API in Thrift server
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server
URL: https://github.com/apache/spark/pull/27552#issuecomment-586860302
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #27552:
[SPARK-30808][SQL] Enable Java 8 time API in Thrift server
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server
URL: https://github.com/apache/spark/pull/27552#discussion_r380015103
##########
File path: sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/SparkSQLDriver.scala
##########
@@ -60,9 +60,10 @@ private[hive] class SparkSQLDriver(val context: SQLContext = SparkSQLEnv.sqlCont
// TODO unify the error code
try {
context.sparkContext.setJobDescription(command)
+ val df = context.sql(command)
val execution = context.sessionState.executePlan(context.sql(command).logicalPlan)
Review comment:
`val execution = df.queryExecution`?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27552:
[SPARK-30808][SQL] Enable Java 8 time API in Thrift server
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server
URL: https://github.com/apache/spark/pull/27552#issuecomment-586752512
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27552: [SPARK-30808][SQL]
Enable Java 8 time API in Thrift server
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server
URL: https://github.com/apache/spark/pull/27552#issuecomment-585648495
**[Test build #118352 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118352/testReport)** for PR 27552 at commit [`580dd09`](https://github.com/apache/spark/commit/580dd09ca9e3b8df54fcf31806913e581e4da720).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27552: [SPARK-30808][SQL]
Enable Java 8 time API in Thrift server
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server
URL: https://github.com/apache/spark/pull/27552#issuecomment-587108066
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118579/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27552:
[SPARK-30808][SQL] Enable Java 8 time API in Thrift server
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server
URL: https://github.com/apache/spark/pull/27552#issuecomment-586864227
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118549/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #27552:
[SPARK-30808][SQL] Enable Java 8 time API in Thrift server
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server
URL: https://github.com/apache/spark/pull/27552#discussion_r379986061
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/HiveResult.scala
##########
@@ -36,27 +37,40 @@ object HiveResult {
* Returns the result as a hive compatible sequence of strings. This is used in tests and
* `SparkSQLDriver` for CLI applications.
*/
- def hiveResultString(executedPlan: SparkPlan): Seq[String] = executedPlan match {
- case ExecutedCommandExec(_: DescribeCommandBase) =>
- // If it is a describe command for a Hive table, we want to have the output format
- // be similar with Hive.
- executedPlan.executeCollectPublic().map {
- case Row(name: String, dataType: String, comment) =>
- Seq(name, dataType,
- Option(comment.asInstanceOf[String]).getOrElse(""))
- .map(s => String.format(s"%-20s", s))
- .mkString("\t")
- }
- // SHOW TABLES in Hive only output table names, while ours output database, table name, isTemp.
- case command @ ExecutedCommandExec(s: ShowTablesCommand) if !s.isExtended =>
- command.executeCollect().map(_.getString(1))
- case other =>
- val result: Seq[Seq[Any]] = other.executeCollectPublic().map(_.toSeq).toSeq
- // We need the types so we can output struct field names
- val types = executedPlan.output.map(_.dataType)
- // Reformat to match hive tab delimited output.
- result.map(_.zip(types).map(e => toHiveString(e)))
- .map(_.mkString("\t"))
+ def hiveResultString[T](ds: Dataset[T]): Seq[String] = {
+ val executedPlan = ds.queryExecution.executedPlan
+ executedPlan match {
+ case ExecutedCommandExec(_: DescribeCommandBase) =>
+ // If it is a describe command for a Hive table, we want to have the output format
+ // be similar with Hive.
+ executedPlan.executeCollectPublic().map {
+ case Row(name: String, dataType: String, comment) =>
+ Seq(name, dataType,
+ Option(comment.asInstanceOf[String]).getOrElse(""))
+ .map(s => String.format(s"%-20s", s))
+ .mkString("\t")
+ }
+ // SHOW TABLES in Hive only output table names,
+ // while ours output database, table name, isTemp.
+ case command @ ExecutedCommandExec(s: ShowTablesCommand) if !s.isExtended =>
+ command.executeCollect().map(_.getString(1))
+ case _ =>
+ val java8TimeConf = SQLConf.get.getConf(SQLConf.DATETIME_JAVA8API_ENABLED)
Review comment:
shall we follow `QueryExecution.preparations`?
```
val sessionWithJava8DatatimeEnabled = {
val cloned = ds.session.cloneSession()
cloned.... setConf
cloned
}
cloned.withActive {
// the actual work
}
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27552:
[SPARK-30808][SQL] Enable Java 8 time API in Thrift server
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server
URL: https://github.com/apache/spark/pull/27552#issuecomment-586979510
**[Test build #118579 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118579/testReport)** for PR 27552 at commit [`880b1de`](https://github.com/apache/spark/commit/880b1dec95595b58688a3d5f7a2e0637a090cb47).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27552:
[SPARK-30808][SQL] Enable Java 8 time API in Thrift server
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server
URL: https://github.com/apache/spark/pull/27552#issuecomment-586859959
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118543/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27552: [SPARK-30808][SQL]
Enable Java 8 time API in Thrift server
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server
URL: https://github.com/apache/spark/pull/27552#issuecomment-586752514
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23265/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27552: [WIP] Enable Java 8 time
API in Thrift server
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27552: [WIP] Enable Java 8 time API in Thrift server
URL: https://github.com/apache/spark/pull/27552#issuecomment-585623968
**[Test build #118347 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118347/testReport)** for PR 27552 at commit [`d98cdbc`](https://github.com/apache/spark/commit/d98cdbcb85ba7df6d24429081da97f5619cfe54b).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27552: [SPARK-30808][SQL]
Enable Java 8 time API in Thrift server
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server
URL: https://github.com/apache/spark/pull/27552#issuecomment-585644999
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27552: [SPARK-30808][SQL]
Enable Java 8 time API in Thrift server
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server
URL: https://github.com/apache/spark/pull/27552#issuecomment-586753677
**[Test build #118510 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118510/testReport)** for PR 27552 at commit [`80f4c89`](https://github.com/apache/spark/commit/80f4c891d3da7bbefcd5738cb61a8d4dabfa1a96).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27552: [SPARK-30808][SQL]
Enable Java 8 time API in Thrift server
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server
URL: https://github.com/apache/spark/pull/27552#issuecomment-585664688
**[Test build #118352 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118352/testReport)** for PR 27552 at commit [`580dd09`](https://github.com/apache/spark/commit/580dd09ca9e3b8df54fcf31806913e581e4da720).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27552: [SPARK-30808][SQL]
Enable Java 8 time API in Thrift server
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server
URL: https://github.com/apache/spark/pull/27552#issuecomment-586864091
**[Test build #118549 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118549/testReport)** for PR 27552 at commit [`ce77628`](https://github.com/apache/spark/commit/ce77628cad487d7d86cb33577a699ac37bf8f5fd).
* This patch **fails due to an unknown error code, -9**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27552:
[SPARK-30808][SQL] Enable Java 8 time API in Thrift server
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server
URL: https://github.com/apache/spark/pull/27552#issuecomment-585664847
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk commented on a change in pull request #27552:
[SPARK-30808][SQL] Enable Java 8 time API in Thrift server
Posted by GitBox <gi...@apache.org>.
MaxGekk commented on a change in pull request #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server
URL: https://github.com/apache/spark/pull/27552#discussion_r385517974
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/HiveResult.scala
##########
@@ -36,27 +37,43 @@ object HiveResult {
* Returns the result as a hive compatible sequence of strings. This is used in tests and
* `SparkSQLDriver` for CLI applications.
*/
- def hiveResultString(executedPlan: SparkPlan): Seq[String] = executedPlan match {
- case ExecutedCommandExec(_: DescribeCommandBase) =>
- // If it is a describe command for a Hive table, we want to have the output format
- // be similar with Hive.
- executedPlan.executeCollectPublic().map {
- case Row(name: String, dataType: String, comment) =>
- Seq(name, dataType,
- Option(comment.asInstanceOf[String]).getOrElse(""))
- .map(s => String.format(s"%-20s", s))
- .mkString("\t")
- }
- // SHOW TABLES in Hive only output table names, while ours output database, table name, isTemp.
- case command @ ExecutedCommandExec(s: ShowTablesCommand) if !s.isExtended =>
- command.executeCollect().map(_.getString(1))
- case other =>
- val result: Seq[Seq[Any]] = other.executeCollectPublic().map(_.toSeq).toSeq
- // We need the types so we can output struct field names
- val types = executedPlan.output.map(_.dataType)
- // Reformat to match hive tab delimited output.
- result.map(_.zip(types).map(e => toHiveString(e)))
- .map(_.mkString("\t"))
+ def hiveResultString(ds: Dataset[_]): Seq[String] = {
+ val executedPlan = ds.queryExecution.executedPlan
+ executedPlan match {
+ case ExecutedCommandExec(_: DescribeCommandBase) =>
+ // If it is a describe command for a Hive table, we want to have the output format
+ // be similar with Hive.
+ executedPlan.executeCollectPublic().map {
+ case Row(name: String, dataType: String, comment) =>
+ Seq(name, dataType,
+ Option(comment.asInstanceOf[String]).getOrElse(""))
+ .map(s => String.format(s"%-20s", s))
+ .mkString("\t")
+ }
+ // SHOW TABLES in Hive only output table names,
+ // while ours output database, table name, isTemp.
+ case command @ ExecutedCommandExec(s: ShowTablesCommand) if !s.isExtended =>
+ command.executeCollect().map(_.getString(1))
+ case _ =>
+ val sessionWithJava8DatetimeEnabled = {
+ val cloned = ds.sparkSession.cloneSession()
+ cloned.conf.set(SQLConf.DATETIME_JAVA8API_ENABLED.key, true)
Review comment:
> Or does this problem only exists for spark-sql script?
Only when thrift-server is involved in the loop.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] yaooqinn commented on a change in pull request #27552:
[SPARK-30808][SQL] Enable Java 8 time API in Thrift server
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on a change in pull request #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server
URL: https://github.com/apache/spark/pull/27552#discussion_r385493404
##########
File path: sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/SparkExecuteStatementOperation.scala
##########
@@ -178,7 +179,14 @@ private[hive] class SparkExecuteStatementOperation(
}
curCol += 1
}
- resultRowSet.addRow(row.toArray.asInstanceOf[Array[Object]])
+ // Convert date-time instances to types that are acceptable by Hive libs
+ // used in conversions to strings.
+ val resultRow = row.map {
+ case i: Instant => Timestamp.from(i)
Review comment:
There seems no java8 values to be add to the `row` buffer here by `SparkExecuteStatementOperation#addNonNullColumnValue `
https://github.com/apache/spark/pull/27552/files#diff-72dcd8f81a51c8a815159fdf0332acdcR84-R116
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27552: [SPARK-30808][SQL]
Enable Java 8 time API in Thrift server
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server
URL: https://github.com/apache/spark/pull/27552#issuecomment-585664847
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27552: [SPARK-30808][SQL]
Enable Java 8 time API in Thrift server
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server
URL: https://github.com/apache/spark/pull/27552#issuecomment-585649121
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23109/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #27552:
[SPARK-30808][SQL] Enable Java 8 time API in Thrift server
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server
URL: https://github.com/apache/spark/pull/27552#discussion_r385112690
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/HiveResult.scala
##########
@@ -36,27 +37,43 @@ object HiveResult {
* Returns the result as a hive compatible sequence of strings. This is used in tests and
* `SparkSQLDriver` for CLI applications.
*/
- def hiveResultString(executedPlan: SparkPlan): Seq[String] = executedPlan match {
- case ExecutedCommandExec(_: DescribeCommandBase) =>
- // If it is a describe command for a Hive table, we want to have the output format
- // be similar with Hive.
- executedPlan.executeCollectPublic().map {
- case Row(name: String, dataType: String, comment) =>
- Seq(name, dataType,
- Option(comment.asInstanceOf[String]).getOrElse(""))
- .map(s => String.format(s"%-20s", s))
- .mkString("\t")
- }
- // SHOW TABLES in Hive only output table names, while ours output database, table name, isTemp.
- case command @ ExecutedCommandExec(s: ShowTablesCommand) if !s.isExtended =>
- command.executeCollect().map(_.getString(1))
- case other =>
- val result: Seq[Seq[Any]] = other.executeCollectPublic().map(_.toSeq).toSeq
- // We need the types so we can output struct field names
- val types = executedPlan.output.map(_.dataType)
- // Reformat to match hive tab delimited output.
- result.map(_.zip(types).map(e => toHiveString(e)))
- .map(_.mkString("\t"))
+ def hiveResultString(ds: Dataset[_]): Seq[String] = {
+ val executedPlan = ds.queryExecution.executedPlan
+ executedPlan match {
+ case ExecutedCommandExec(_: DescribeCommandBase) =>
+ // If it is a describe command for a Hive table, we want to have the output format
+ // be similar with Hive.
+ executedPlan.executeCollectPublic().map {
+ case Row(name: String, dataType: String, comment) =>
+ Seq(name, dataType,
+ Option(comment.asInstanceOf[String]).getOrElse(""))
+ .map(s => String.format(s"%-20s", s))
+ .mkString("\t")
+ }
+ // SHOW TABLES in Hive only output table names,
+ // while ours output database, table name, isTemp.
+ case command @ ExecutedCommandExec(s: ShowTablesCommand) if !s.isExtended =>
+ command.executeCollect().map(_.getString(1))
+ case _ =>
+ val sessionWithJava8DatetimeEnabled = {
+ val cloned = ds.sparkSession.cloneSession()
+ cloned.conf.set(SQLConf.DATETIME_JAVA8API_ENABLED.key, true)
+ cloned
+ }
+ sessionWithJava8DatetimeEnabled.withActive {
+ // We cannot collect the original dataset because its encoders could be created
+ // with disabled Java 8 date-time API.
+ val result: Seq[Seq[Any]] = Dataset.ofRows(ds.sparkSession, ds.logicalPlan)
Review comment:
found a problem. `Dataset.ofRows` will set the input session as active, so we should write `Dataset.ofRows(sessionWithJava8DatetimeEnabled, ...` and remove the outer `sessionWithJava8DatetimeEnabled.withActive`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #27552:
[SPARK-30808][SQL] Enable Java 8 time API in Thrift server
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server
URL: https://github.com/apache/spark/pull/27552#discussion_r380015431
##########
File path: sql/hive/src/test/scala/org/apache/spark/sql/hive/test/TestHive.scala
##########
@@ -605,6 +605,8 @@ private[hive] class TestHiveQueryExecution(
// Proceed with analysis.
sparkSession.sessionState.analyzer.executeAndCheck(logical, tracker)
}
+
+ lazy val dataset = Dataset.ofRows(sparkSession, logical)
Review comment:
If it's only used once in https://github.com/apache/spark/pull/27552/files#diff-6b519cd0a9dc5b0ee9caab7712d252e9R349 , let's just inline it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27552:
[SPARK-30808][SQL] Enable Java 8 time API in Thrift server
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server
URL: https://github.com/apache/spark/pull/27552#issuecomment-586748629
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #27552:
[SPARK-30808][SQL] Enable Java 8 time API in Thrift server
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server
URL: https://github.com/apache/spark/pull/27552#discussion_r379986254
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/HiveResult.scala
##########
@@ -36,27 +37,40 @@ object HiveResult {
* Returns the result as a hive compatible sequence of strings. This is used in tests and
* `SparkSQLDriver` for CLI applications.
*/
- def hiveResultString(executedPlan: SparkPlan): Seq[String] = executedPlan match {
- case ExecutedCommandExec(_: DescribeCommandBase) =>
- // If it is a describe command for a Hive table, we want to have the output format
- // be similar with Hive.
- executedPlan.executeCollectPublic().map {
- case Row(name: String, dataType: String, comment) =>
- Seq(name, dataType,
- Option(comment.asInstanceOf[String]).getOrElse(""))
- .map(s => String.format(s"%-20s", s))
- .mkString("\t")
- }
- // SHOW TABLES in Hive only output table names, while ours output database, table name, isTemp.
- case command @ ExecutedCommandExec(s: ShowTablesCommand) if !s.isExtended =>
- command.executeCollect().map(_.getString(1))
- case other =>
- val result: Seq[Seq[Any]] = other.executeCollectPublic().map(_.toSeq).toSeq
- // We need the types so we can output struct field names
- val types = executedPlan.output.map(_.dataType)
- // Reformat to match hive tab delimited output.
- result.map(_.zip(types).map(e => toHiveString(e)))
- .map(_.mkString("\t"))
+ def hiveResultString[T](ds: Dataset[T]): Seq[String] = {
+ val executedPlan = ds.queryExecution.executedPlan
+ executedPlan match {
+ case ExecutedCommandExec(_: DescribeCommandBase) =>
+ // If it is a describe command for a Hive table, we want to have the output format
+ // be similar with Hive.
+ executedPlan.executeCollectPublic().map {
+ case Row(name: String, dataType: String, comment) =>
+ Seq(name, dataType,
+ Option(comment.asInstanceOf[String]).getOrElse(""))
+ .map(s => String.format(s"%-20s", s))
+ .mkString("\t")
+ }
+ // SHOW TABLES in Hive only output table names,
+ // while ours output database, table name, isTemp.
+ case command @ ExecutedCommandExec(s: ShowTablesCommand) if !s.isExtended =>
+ command.executeCollect().map(_.getString(1))
+ case _ =>
+ val java8TimeConf = SQLConf.get.getConf(SQLConf.DATETIME_JAVA8API_ENABLED)
+ try {
+ SQLConf.get.setConf(SQLConf.DATETIME_JAVA8API_ENABLED, true)
+ val result: Seq[Seq[Any]] = ds.filter(lit(true))
Review comment:
how about `Dataset.ofRows(session, ds.queryExecution.logicalPlan)`?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27552: [SPARK-30808][SQL]
Enable Java 8 time API in Thrift server
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server
URL: https://github.com/apache/spark/pull/27552#issuecomment-586859902
**[Test build #118543 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118543/testReport)** for PR 27552 at commit [`e491f96`](https://github.com/apache/spark/commit/e491f966b23a734358b222ed8486a547467a22ed).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27552:
[SPARK-30808][SQL] Enable Java 8 time API in Thrift server
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server
URL: https://github.com/apache/spark/pull/27552#issuecomment-585649109
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] yaooqinn commented on a change in pull request #27552:
[SPARK-30808][SQL] Enable Java 8 time API in Thrift server
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on a change in pull request #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server
URL: https://github.com/apache/spark/pull/27552#discussion_r385512544
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/HiveResult.scala
##########
@@ -36,27 +37,43 @@ object HiveResult {
* Returns the result as a hive compatible sequence of strings. This is used in tests and
* `SparkSQLDriver` for CLI applications.
*/
- def hiveResultString(executedPlan: SparkPlan): Seq[String] = executedPlan match {
- case ExecutedCommandExec(_: DescribeCommandBase) =>
- // If it is a describe command for a Hive table, we want to have the output format
- // be similar with Hive.
- executedPlan.executeCollectPublic().map {
- case Row(name: String, dataType: String, comment) =>
- Seq(name, dataType,
- Option(comment.asInstanceOf[String]).getOrElse(""))
- .map(s => String.format(s"%-20s", s))
- .mkString("\t")
- }
- // SHOW TABLES in Hive only output table names, while ours output database, table name, isTemp.
- case command @ ExecutedCommandExec(s: ShowTablesCommand) if !s.isExtended =>
- command.executeCollect().map(_.getString(1))
- case other =>
- val result: Seq[Seq[Any]] = other.executeCollectPublic().map(_.toSeq).toSeq
- // We need the types so we can output struct field names
- val types = executedPlan.output.map(_.dataType)
- // Reformat to match hive tab delimited output.
- result.map(_.zip(types).map(e => toHiveString(e)))
- .map(_.mkString("\t"))
+ def hiveResultString(ds: Dataset[_]): Seq[String] = {
+ val executedPlan = ds.queryExecution.executedPlan
+ executedPlan match {
+ case ExecutedCommandExec(_: DescribeCommandBase) =>
+ // If it is a describe command for a Hive table, we want to have the output format
+ // be similar with Hive.
+ executedPlan.executeCollectPublic().map {
+ case Row(name: String, dataType: String, comment) =>
+ Seq(name, dataType,
+ Option(comment.asInstanceOf[String]).getOrElse(""))
+ .map(s => String.format(s"%-20s", s))
+ .mkString("\t")
+ }
+ // SHOW TABLES in Hive only output table names,
+ // while ours output database, table name, isTemp.
+ case command @ ExecutedCommandExec(s: ShowTablesCommand) if !s.isExtended =>
+ command.executeCollect().map(_.getString(1))
+ case _ =>
+ val sessionWithJava8DatetimeEnabled = {
+ val cloned = ds.sparkSession.cloneSession()
+ cloned.conf.set(SQLConf.DATETIME_JAVA8API_ENABLED.key, true)
Review comment:
```
-- !query
set spark.sql.datetime.java8API.enabled
-- !query schema
struct<key:string,value:string>
-- !query output
spark.sql.datetime.java8API.enabled false
-- !query
set set spark.sql.session.timeZone=America/Los_Angeles
-- !query schema
struct<key:string,value:string>
-- !query output
set spark.sql.session.timeZone America/Los_Angeles
-- !query
SELECT DATE_TRUNC('MILLENNIUM', DATE '1970-03-20')
-- !query schema
struct<date_trunc(MILLENNIUM, CAST(DATE '1970-03-20' AS TIMESTAMP)):timestamp>
-- !query output
1001-01-01 00:00:00
```
I rm this line and run `SQLQueryTestSuite` with cases above, the results are the same. Or does this problem only exists for `spark-sql` script?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27552:
[SPARK-30808][SQL] Enable Java 8 time API in Thrift server
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server
URL: https://github.com/apache/spark/pull/27552#issuecomment-586864214
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27552:
[SPARK-30808][SQL] Enable Java 8 time API in Thrift server
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server
URL: https://github.com/apache/spark/pull/27552#issuecomment-586865476
**[Test build #118552 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118552/testReport)** for PR 27552 at commit [`f9112b7`](https://github.com/apache/spark/commit/f9112b7ab9bb654d880a8bdf0ad28a78c3da0712).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27552: [SPARK-30808][SQL]
Enable Java 8 time API in Thrift server
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server
URL: https://github.com/apache/spark/pull/27552#issuecomment-586864214
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27552:
[SPARK-30808][SQL] Enable Java 8 time API in Thrift server
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server
URL: https://github.com/apache/spark/pull/27552#issuecomment-586846551
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23298/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27552: [SPARK-30808][SQL]
Enable Java 8 time API in Thrift server
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server
URL: https://github.com/apache/spark/pull/27552#issuecomment-586741690
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk commented on issue #27552: [SPARK-30808][SQL]
Enable Java 8 time API in Thrift server
Posted by GitBox <gi...@apache.org>.
MaxGekk commented on issue #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server
URL: https://github.com/apache/spark/pull/27552#issuecomment-586924188
So many HiveComparisonTest related tests failed, I will revert this https://github.com/apache/spark/pull/27552/commits/cdb322d6366e5c92579a8eaa26ccd94626cdf3ee
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27552: [SPARK-30808][SQL]
Enable Java 8 time API in Thrift server
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server
URL: https://github.com/apache/spark/pull/27552#issuecomment-586740962
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23263/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk commented on issue #27552: [WIP] Enable Java 8 time
API in Thrift server
Posted by GitBox <gi...@apache.org>.
MaxGekk commented on issue #27552: [WIP] Enable Java 8 time API in Thrift server
URL: https://github.com/apache/spark/pull/27552#issuecomment-585395953
@cloud-fan I tried to set the SQL config before the collect action in https://github.com/apache/spark/blob/aa0d13683cdf9f38f04cc0e73dc8cf63eed29bf4/sql/core/src/main/scala/org/apache/spark/sql/execution/HiveResult.scala#L54-L59 but it doesn't work - the SQL config didn't change the behavior :-(
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #27552:
[SPARK-30808][SQL] Enable Java 8 time API in Thrift server
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server
URL: https://github.com/apache/spark/pull/27552#discussion_r385496003
##########
File path: sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/SparkExecuteStatementOperation.scala
##########
@@ -178,7 +179,14 @@ private[hive] class SparkExecuteStatementOperation(
}
curCol += 1
}
- resultRowSet.addRow(row.toArray.asInstanceOf[Array[Object]])
+ // Convert date-time instances to types that are acceptable by Hive libs
+ // used in conversions to strings.
+ val resultRow = row.map {
+ case i: Instant => Timestamp.from(i)
Review comment:
can you help fix it? I think we should output java8 datetime values if the config is enabled.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27552:
[SPARK-30808][SQL] Enable Java 8 time API in Thrift server
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server
URL: https://github.com/apache/spark/pull/27552#issuecomment-587108066
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118579/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27552: [SPARK-30808][SQL]
Enable Java 8 time API in Thrift server
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server
URL: https://github.com/apache/spark/pull/27552#issuecomment-586752512
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27552: [SPARK-30808][SQL]
Enable Java 8 time API in Thrift server
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server
URL: https://github.com/apache/spark/pull/27552#issuecomment-586853928
**[Test build #118545 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118545/testReport)** for PR 27552 at commit [`95597a7`](https://github.com/apache/spark/commit/95597a7244e2c99a9ae267e5705748ad8403a169).
* This patch **fails to generate documentation**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27552: [SPARK-30808][SQL]
Enable Java 8 time API in Thrift server
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server
URL: https://github.com/apache/spark/pull/27552#issuecomment-585664861
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118352/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27552: [SPARK-30808][SQL]
Enable Java 8 time API in Thrift server
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server
URL: https://github.com/apache/spark/pull/27552#issuecomment-586748630
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118509/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27552: [SPARK-30808][SQL]
Enable Java 8 time API in Thrift server
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server
URL: https://github.com/apache/spark/pull/27552#issuecomment-586774299
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118511/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #27552:
[SPARK-30808][SQL] Enable Java 8 time API in Thrift server
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server
URL: https://github.com/apache/spark/pull/27552#discussion_r379986401
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/HiveResult.scala
##########
@@ -36,27 +37,40 @@ object HiveResult {
* Returns the result as a hive compatible sequence of strings. This is used in tests and
* `SparkSQLDriver` for CLI applications.
*/
- def hiveResultString(executedPlan: SparkPlan): Seq[String] = executedPlan match {
- case ExecutedCommandExec(_: DescribeCommandBase) =>
- // If it is a describe command for a Hive table, we want to have the output format
- // be similar with Hive.
- executedPlan.executeCollectPublic().map {
- case Row(name: String, dataType: String, comment) =>
- Seq(name, dataType,
- Option(comment.asInstanceOf[String]).getOrElse(""))
- .map(s => String.format(s"%-20s", s))
- .mkString("\t")
- }
- // SHOW TABLES in Hive only output table names, while ours output database, table name, isTemp.
- case command @ ExecutedCommandExec(s: ShowTablesCommand) if !s.isExtended =>
- command.executeCollect().map(_.getString(1))
- case other =>
- val result: Seq[Seq[Any]] = other.executeCollectPublic().map(_.toSeq).toSeq
- // We need the types so we can output struct field names
- val types = executedPlan.output.map(_.dataType)
- // Reformat to match hive tab delimited output.
- result.map(_.zip(types).map(e => toHiveString(e)))
- .map(_.mkString("\t"))
+ def hiveResultString[T](ds: Dataset[T]): Seq[String] = {
Review comment:
how about `def hiveResultString(ds: Dataset[_])`?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27552: [SPARK-30808][SQL]
Enable Java 8 time API in Thrift server
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server
URL: https://github.com/apache/spark/pull/27552#issuecomment-586859947
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27552: [WIP] Enable Java 8 time
API in Thrift server
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27552: [WIP] Enable Java 8 time API in Thrift server
URL: https://github.com/apache/spark/pull/27552#issuecomment-585407325
**[Test build #118319 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118319/testReport)** for PR 27552 at commit [`916838a`](https://github.com/apache/spark/commit/916838a3d43aeac59cdc799fed0de8d279b0ad66).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27552:
[SPARK-30808][SQL] Enable Java 8 time API in Thrift server
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server
URL: https://github.com/apache/spark/pull/27552#issuecomment-586933972
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27552: [SPARK-30808][SQL]
Enable Java 8 time API in Thrift server
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server
URL: https://github.com/apache/spark/pull/27552#issuecomment-586933912
**[Test build #118569 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118569/testReport)** for PR 27552 at commit [`2129f30`](https://github.com/apache/spark/commit/2129f30c8f54c669405db302dbfb52f78631fc05).
* This patch **fails to generate documentation**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27552: [SPARK-30808][SQL]
Enable Java 8 time API in Thrift server
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server
URL: https://github.com/apache/spark/pull/27552#issuecomment-586749159
**[Test build #118508 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118508/testReport)** for PR 27552 at commit [`deaef58`](https://github.com/apache/spark/commit/deaef5837b0d7bbbb1eb0f6b6e90aef4b1eefc7d).
* This patch **fails Spark unit tests**.
* This patch **does not merge cleanly**.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27552:
[SPARK-30808][SQL] Enable Java 8 time API in Thrift server
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server
URL: https://github.com/apache/spark/pull/27552#issuecomment-587108057
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27552: [SPARK-30808][SQL]
Enable Java 8 time API in Thrift server
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server
URL: https://github.com/apache/spark/pull/27552#issuecomment-586933980
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118569/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk edited a comment on issue #27552:
[SPARK-30808][SQL] Enable Java 8 time API in Thrift server
Posted by GitBox <gi...@apache.org>.
MaxGekk edited a comment on issue #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server
URL: https://github.com/apache/spark/pull/27552#issuecomment-586947271
While debugging the `test ambiguousReferences resolved as hive` test from `org.apache.spark.sql.hive.execution.HiveResolutionSuite`, I see CreateDataSourceTableCommand is invoke twice for the same table name **t1** :
1. In `lazy val dataset = Dataset.ofRows(sparkSession, logical)` inside of `TestHiveQueryExecution`
2. In `hiveResultString()`:
```scala
val result: Seq[Seq[Any]] = Dataset.ofRows(ds.sparkSession, ds.queryExecution.logical)
.queryExecution
.executedPlan
.executeCollectPublic().map(_.toSeq).toSeq
```
This causes side effects here:
https://github.com/apache/spark/blob/e2d3983de78f5c80fac066b7ee8bedd0987110dd/sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala#L226
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27552: [WIP]
Enable Java 8 time API in Thrift server
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27552: [WIP] Enable Java 8 time API in Thrift server
URL: https://github.com/apache/spark/pull/27552#issuecomment-585621415
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] yaooqinn commented on a change in pull request #27552:
[SPARK-30808][SQL] Enable Java 8 time API in Thrift server
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on a change in pull request #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server
URL: https://github.com/apache/spark/pull/27552#discussion_r385492731
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/SQLQueryTestSuite.scala
##########
@@ -512,7 +511,7 @@ class SQLQueryTestSuite extends QueryTest with SharedSparkSession {
val schema = df.schema.catalogString
// Get answer, but also get rid of the #1234 expression ids that show up in explain plans
val answer = SQLExecution.withNewExecutionId(df.queryExecution, Some(sql)) {
- hiveResultString(df.queryExecution.executedPlan).map(replaceNotIncludedMsg)
+ hiveResultString(df).map(replaceNotIncludedMsg)
Review comment:
We do not have our own JDBC implementation yet, and the hive JDBC haven't supported this too
https://github.com/apache/hive/blob/ebd41526b9433a196ae0f934bc466cb5c155a202/jdbc/src/java/org/apache/hive/jdbc/HiveQueryResultSet.java#L402-L411
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on issue #27552: [SPARK-30808][SQL]
Enable Java 8 time API in Thrift server
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on issue #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server
URL: https://github.com/apache/spark/pull/27552#issuecomment-586964319
@MaxGekk I see the problem now. We should use `ds.logicalPlan` instead of `ds.queryExecution.logical`. As `ds.logicalPlan` is `LocalRelation` to hold the command result.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27552: [SPARK-30808][SQL]
Enable Java 8 time API in Thrift server
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server
URL: https://github.com/apache/spark/pull/27552#issuecomment-586741695
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23264/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27552:
[SPARK-30808][SQL] Enable Java 8 time API in Thrift server
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server
URL: https://github.com/apache/spark/pull/27552#issuecomment-586767247
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org