You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by gu...@apache.org on 2021/01/08 00:35:08 UTC
[spark] branch master updated: [SPARK-34044][DOCS] Add
spark.sql.hive.metastore.jars.path to sql-data-sources-hive-tables.md
This is an automated email from the ASF dual-hosted git repository.
gurwls223 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 5b16d70 [SPARK-34044][DOCS] Add spark.sql.hive.metastore.jars.path to sql-data-sources-hive-tables.md
5b16d70 is described below
commit 5b16d70d6a51720660e7607c859fae4f28691952
Author: Dongjoon Hyun <dh...@apple.com>
AuthorDate: Fri Jan 8 09:34:40 2021 +0900
[SPARK-34044][DOCS] Add spark.sql.hive.metastore.jars.path to sql-data-sources-hive-tables.md
### What changes were proposed in this pull request?
This PR adds new configuration to `sql-data-sources-hive-tables`.
### Why are the changes needed?
SPARK-32852 added a new configuration, `spark.sql.hive.metastore.jars.path`.
### Does this PR introduce _any_ user-facing change?
Yes, but a document only.
### How was this patch tested?
**BEFORE**
**AFTER**
Closes #31085 from dongjoon-hyun/SPARK-34044.
Authored-by: Dongjoon Hyun <dh...@apple.com>
Signed-off-by: HyukjinKwon <gu...@apache.org>
---
docs/sql-data-sources-hive-tables.md | 27 ++++++++++++++++++++++++++-
1 file changed, 26 insertions(+), 1 deletion(-)
diff --git a/docs/sql-data-sources-hive-tables.md b/docs/sql-data-sources-hive-tables.md
index ae3572c..376c204 100644
--- a/docs/sql-data-sources-hive-tables.md
+++ b/docs/sql-data-sources-hive-tables.md
@@ -139,7 +139,7 @@ The following options can be used to configure the version of Hive that is used
<td><code>builtin</code></td>
<td>
Location of the jars that should be used to instantiate the HiveMetastoreClient. This
- property can be one of three options:
+ property can be one of four options:
<ol>
<li><code>builtin</code></li>
Use Hive 2.3.7, which is bundled with the Spark assembly when <code>-Phive</code> is
@@ -148,6 +148,9 @@ The following options can be used to configure the version of Hive that is used
<li><code>maven</code></li>
Use Hive jars of specified version downloaded from Maven repositories. This configuration
is not generally recommended for production deployments.
+ <li><code>path</code></li>
+ Use Hive jars configured by <code>spark.sql.hive.metastore.jars.path</code>
+ in comma separated format. Support both local or remote paths.
<li>A classpath in the standard format for the JVM. This classpath must include all of Hive
and its dependencies, including the correct version of Hadoop. These jars only need to be
present on the driver, but if you are running in yarn cluster mode then you must ensure
@@ -157,6 +160,28 @@ The following options can be used to configure the version of Hive that is used
<td>1.4.0</td>
</tr>
<tr>
+ <td><code>spark.sql.hive.metastore.jars.path</code></td>
+ <td><code>(empty)</code></td>
+ <td>
+ Comma-separated paths of the jars that used to instantiate the HiveMetastoreClient.
+ This configuration is useful only when <code>spark.sql.hive.metastore.jars</code> is set as <code>path</code>.
+ <br/>
+ The paths can be any of the following format:
+ <ol>
+ <li><code>file://path/to/jar/foo.jar</code></li>
+ <li><code>hdfs://nameservice/path/to/jar/foo.jar</code></li>
+ <li><code>/path/to/jar/</code>(path without URI scheme follow conf <code>fs.defaultFS</code>'s URI schema)</li>
+ <li><code>[http/https/ftp]://path/to/jar/foo.jar</code></li>
+ </ol>
+ Note that 1, 2, and 3 support wildcard. For example:
+ <ol>
+ <li><code>file://path/to/jar/*,file://path2/to/jar/*/*.jar</code></li>
+ <li><code>hdfs://nameservice/path/to/jar/*,hdfs://nameservice2/path/to/jar/*/*.jar</code></li>
+ </ol>
+ </td>
+ <td>3.1.0</td>
+ </tr>
+ <tr>
<td><code>spark.sql.hive.metastore.sharedPrefixes</code></td>
<td><code>com.mysql.jdbc,<br/>org.postgresql,<br/>com.microsoft.sqlserver,<br/>oracle.jdbc</code></td>
<td>
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org