You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by navis <gi...@git.apache.org> on 2015/10/14 04:01:34 UTC
[GitHub] spark pull request: [SPARK-11067] Spark SQL thrift server fails to...
GitHub user navis opened a pull request:
https://github.com/apache/spark/pull/9107
[SPARK-11067] Spark SQL thrift server fails to handle decimal value
When executing the following query through beeline connecting to Spark sql thrift server, it errors out for decimal column
Select decimal_column from table
```
WARN 2015-10-09 15:04:00 org.apache.hive.service.cli.thrift.ThriftCLIService: Error fetching results:
java.lang.ClassCastException: java.math.BigDecimal cannot be cast to org.apache.hadoop.hive.common.type.HiveDecimal
at org.apache.hive.service.cli.ColumnValue.toTColumnValue(ColumnValue.java:174) ~[hive-service-0.13.1a.jar:0.13.1a]
at org.apache.hive.service.cli.RowBasedSet.addRow(RowBasedSet.java:60) ~[hive-service-0.13.1a.jar:0.13.1a]
at org.apache.hive.service.cli.RowBasedSet.addRow(RowBasedSet.java:32) ~[hive-service-0.13.1a.jar:0.13.1a]
at org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation.getNextRowSet(Shim13.scala:144) ~[spark-hive-thriftserver_2.10-1.4.1.1.jar:1.4.1.1]
at org.apache.hive.service.cli.operation.OperationManager.getOperationNextRowSet(OperationManager.java:192) ~[hive-service-0.13.1a.jar:0.13.1a]
at org.apache.hive.service.cli.session.HiveSessionImpl.fetchResults(HiveSessionImpl.java:471) ~[hive-service-0.13.1a.jar:0.13.1a]
at org.apache.hive.service.cli.CLIService.fetchResults(CLIService.java:405) ~[hive-service-0.13.1a.jar:0.13.1a]
at org.apache.hive.service.cli.thrift.ThriftCLIService.FetchResults(ThriftCLIService.java:530) ~[hive-service-0.13.1a.jar:0.13.1a]
at org.apache.hive.service.cli.thrift.TCLIService$Processor$FetchResults.getResult(TCLIService.java:1553) [hive-service-0.13.1a.jar:0.13.1a]
at org.apache.hive.service.cli.thrift.TCLIService$Processor$FetchResults.getResult(TCLIService.java:1538) [hive-service-0.13.1a.jar:0.13.1a]
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) [libthrift-0.9.2.jar:0.9.2]
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) [libthrift-0.9.2.jar:0.9.2]
at org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:55) [hive-service-0.13.1a.jar:4.8.1-SNAPSHOT]
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:285) [libthrift-0.9.2.jar:0.9.2]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_55]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_55]
at java.lang.Thread.run(Thread.java:745) [na:1.7.0_55]
```
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/navis/spark SPARK-11067
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/9107.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #9107
----
commit 57c12557cb1280e488a1c6ac7d89a7bc40d557b0
Author: navis.ryu <na...@apache.org>
Date: 2015-10-13T04:34:57Z
[SPARK-11067] Spark SQL thrift server fails to handle decimal value
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11067] Spark SQL thrift server fails to...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9107#issuecomment-147906246
[Test build #43696 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43696/console) for PR 9107 at commit [`57c1255`](https://github.com/apache/spark/commit/57c12557cb1280e488a1c6ac7d89a7bc40d557b0).
* This patch **passes all tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* ` def getPath = path.getOrElse(sys.error("Constructors must start at a class type"))`
* `case class WrapOption(optionType: DataType, child: Expression)`
* `class GenericArrayData(val array: Array[Any]) extends ArrayData `
* `trait QueryExecutionListener `
* `class ExecutionListenerManager extends Logging `
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11067] Spark SQL thrift server fails to...
Posted by davies <gi...@git.apache.org>.
Github user davies commented on the pull request:
https://github.com/apache/spark/pull/9107#issuecomment-148119645
@navis Thanks for sending out the patch, could you update existing tests to cover this case (or add a new one)?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11067] Spark SQL thrift server fails to...
Posted by navis <gi...@git.apache.org>.
Github user navis commented on the pull request:
https://github.com/apache/spark/pull/9107#issuecomment-148281071
@davies It'll be reproduced only when using older version of hive jdbc, which does not support columnar wire format (RowBasedSet is older format). Idea?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11067] Spark SQL thrift server fails to...
Posted by navis <gi...@git.apache.org>.
Github user navis commented on the pull request:
https://github.com/apache/spark/pull/9107#issuecomment-150998671
@chenghao-intel It will not happen with `columnar` format which just calls `String.valueOf(field)` whatever the field is. But in row-based set which was used older version of hive-jdbc, it casts decimal type to HiveDecimal first which makes exception described.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11067] Spark SQL thrift server fails to...
Posted by chenghao-intel <gi...@git.apache.org>.
Github user chenghao-intel commented on the pull request:
https://github.com/apache/spark/pull/9107#issuecomment-150088090
In my understanding it's not related to `columnar` format, probably a general problem for all of the Decimal type.Right?
Add the unit test in the https://github.com/apache/spark/blob/master/sql/hive-thriftserver/src/test/scala/org/apache/spark/sql/hive/thriftserver/HiveThriftServer2Suites.scala should be very easy, you can cast any existed data type to decimal type and then get it thru JDBC.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11067] Spark SQL thrift server fails to...
Posted by navis <gi...@git.apache.org>.
Github user navis closed the pull request at:
https://github.com/apache/spark/pull/9107
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11067] Spark SQL thrift server fails to...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9107#issuecomment-147903403
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11067] Spark SQL thrift server fails to...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9107#issuecomment-147903391
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11067] Spark SQL thrift server fails to...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9107#issuecomment-147906305
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11067] Spark SQL thrift server fails to...
Posted by yhuai <gi...@git.apache.org>.
Github user yhuai commented on the pull request:
https://github.com/apache/spark/pull/9107#issuecomment-171387800
@navis How about we close this for now? Or, if you'd like to continue the work, can you provide more details the case that triggers the error in the description (I feel we do not understand how this failure will be triggered.)?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11067] Spark SQL thrift server fails to...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9107#issuecomment-147903630
[Test build #43696 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43696/consoleFull) for PR 9107 at commit [`57c1255`](https://github.com/apache/spark/commit/57c12557cb1280e488a1c6ac7d89a7bc40d557b0).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11067] Spark SQL thrift server fails to...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9107#issuecomment-147906306
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43696/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11067] Spark SQL thrift server fails to...
Posted by navis <gi...@git.apache.org>.
Github user navis commented on the pull request:
https://github.com/apache/spark/pull/9107#issuecomment-171484501
It's only happened when using older version of hive jdbc. I'm closing.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org