You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by hv...@apache.org on 2016/12/12 22:38:41 UTC
spark git commit: [SPARK-18681][SQL] Fix filtering to compatible with
partition keys of type int
Repository: spark
Updated Branches:
refs/heads/master 476b34c23 -> 90abfd15f
[SPARK-18681][SQL] Fix filtering to compatible with partition keys of type int
## What changes were proposed in this pull request?
Cloudera put `/var/run/cloudera-scm-agent/process/15000-hive-HIVEMETASTORE/hive-site.xml` as the configuration file for the Hive Metastore Server, where `hive.metastore.try.direct.sql=false`. But Spark isn't reading this configuration file and get default value `hive.metastore.try.direct.sql=true`. As mallman said, we should use `getMetaConf` method to obtain the original configuration from Hive Metastore Server. I have tested this method few times and the return value is always consistent with Hive Metastore Server.
## How was this patch tested?
The existing tests.
Author: Yuming Wang <wg...@gmail.com>
Closes #16122 from wangyum/SPARK-18681.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/90abfd15
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/90abfd15
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/90abfd15
Branch: refs/heads/master
Commit: 90abfd15f4b3f612a7b0ff65f03bf319c78a0243
Parents: 476b34c
Author: Yuming Wang <wg...@gmail.com>
Authored: Mon Dec 12 23:38:36 2016 +0100
Committer: Herman van Hovell <hv...@databricks.com>
Committed: Mon Dec 12 23:38:36 2016 +0100
----------------------------------------------------------------------
.../scala/org/apache/spark/sql/hive/client/HiveShim.scala | 7 +++++--
1 file changed, 5 insertions(+), 2 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/spark/blob/90abfd15/sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveShim.scala
----------------------------------------------------------------------
diff --git a/sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveShim.scala b/sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveShim.scala
index 137ec26..b052f1e 100644
--- a/sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveShim.scala
+++ b/sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveShim.scala
@@ -594,8 +594,11 @@ private[client] class Shim_v0_13 extends Shim_v0_12 {
} else {
logDebug(s"Hive metastore filter is '$filter'.")
val tryDirectSqlConfVar = HiveConf.ConfVars.METASTORE_TRY_DIRECT_SQL
- val tryDirectSql =
- hive.getConf.getBoolean(tryDirectSqlConfVar.varname, tryDirectSqlConfVar.defaultBoolVal)
+ // We should get this config value from the metaStore. otherwise hit SPARK-18681.
+ // To be compatible with hive-0.12 and hive-0.13, In the future we can achieve this by:
+ // val tryDirectSql = hive.getMetaConf(tryDirectSqlConfVar.varname).toBoolean
+ val tryDirectSql = hive.getMSC.getConfigValue(tryDirectSqlConfVar.varname,
+ tryDirectSqlConfVar.defaultBoolVal.toString).toBoolean
try {
// Hive may throw an exception when calling this method in some circumstances, such as
// when filtering on a non-string partition column when the hive config key
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org