You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by gu...@apache.org on 2021/11/25 01:08:25 UTC
[spark] branch branch-3.2 updated: [SPARK-37446][SQL] Use reflection for getWithoutRegisterFns to allow different Hive versions for building
This is an automated email from the ASF dual-hosted git repository.
gurwls223 pushed a commit to branch branch-3.2
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.2 by this push:
new a51ad2a [SPARK-37446][SQL] Use reflection for getWithoutRegisterFns to allow different Hive versions for building
a51ad2a is described below
commit a51ad2adaa38de11de5f632d4b8566778860af7a
Author: Angerszhuuuu <an...@gmail.com>
AuthorDate: Thu Nov 25 10:06:42 2021 +0900
[SPARK-37446][SQL] Use reflection for getWithoutRegisterFns to allow different Hive versions for building
### What changes were proposed in this pull request?
Since Hive 2.3.9 start have function `getWithoutRegisterFns`, but user may use hive 2.3.8 or lower version.
Here we should use reflection to let user build with hive 2.3.8 or lower version
### Why are the changes needed?
Support build with hive version lower than 2.3.9 since many user will build spark with it 's own hive code and their own jar (they may do some optimize or. other thing in their own code). This pr make it easier to integrate and won't hurt current logic.
### Does this PR introduce _any_ user-facing change?
User can build spark with hive version lower than 2.3.9
### How was this patch tested?
build with command
```
./dev/make-distribution.sh --tgz -Pyarn -Phive -Phive-thriftserver -Dhive.version=2.3.8
```
Jars under dist
Closes #34690 from AngersZhuuuu/SPARK-37446.
Authored-by: Angerszhuuuu <an...@gmail.com>
Signed-off-by: Hyukjin Kwon <gu...@apache.org>
(cherry picked from commit 04671bd04d3d5688bd0d3fa616365ee32cc1ff41)
Signed-off-by: Hyukjin Kwon <gu...@apache.org>
---
.../main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala | 5 +++--
1 file changed, 3 insertions(+), 2 deletions(-)
diff --git a/sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala b/sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala
index d4362fa..4703f24 100644
--- a/sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala
+++ b/sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala
@@ -202,11 +202,12 @@ private[hive] class HiveClientImpl(
private def getHive(conf: HiveConf): Hive = {
try {
- Hive.getWithoutRegisterFns(conf)
+ classOf[Hive].getMethod("getWithoutRegisterFns", classOf[HiveConf])
+ .invoke(null, conf).asInstanceOf[Hive]
} catch {
// SPARK-37069: not all Hive versions have the above method (e.g., Hive 2.3.9 has it but
// 2.3.8 don't), therefore here we fallback when encountering the exception.
- case _: NoSuchMethodError =>
+ case _: NoSuchMethodException =>
Hive.get(conf)
}
}
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org