You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@drill.apache.org by br...@apache.org on 2018/07/11 21:40:41 UTC
[drill-site] branch asf-site updated (e875d70 -> 3a06342)
This is an automated email from the ASF dual-hosted git repository.
bridgetb pushed a change to branch asf-site
in repository https://gitbox.apache.org/repos/asf/drill-site.git.
from e875d70 fix broken link
new 08eff5b add doc for hive option to set hive properties at session level
new 3a06342 edits
The 2 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
docs/configuration-options-introduction/index.html | 12 ++++++-
docs/hive-storage-plugin/index.html | 40 +++++++++++++++++-----
feed.xml | 4 +--
3 files changed, 45 insertions(+), 11 deletions(-)
[drill-site] 01/02: add doc for hive option to set hive properties
at session level
Posted by br...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
bridgetb pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/drill-site.git
commit 08eff5b0b7ab3b179ff1c09cca0469767c623f29
Author: Bridget Bevens <bb...@maprtech.com>
AuthorDate: Wed Jul 11 14:32:00 2018 -0700
add doc for hive option to set hive properties at session level
---
docs/configuration-options-introduction/index.html | 12 ++++++-
docs/hive-storage-plugin/index.html | 40 +++++++++++++++++-----
feed.xml | 4 +--
3 files changed, 45 insertions(+), 11 deletions(-)
diff --git a/docs/configuration-options-introduction/index.html b/docs/configuration-options-introduction/index.html
index 7409356..973b17d 100644
--- a/docs/configuration-options-introduction/index.html
+++ b/docs/configuration-options-introduction/index.html
@@ -1238,7 +1238,7 @@
</div>
- Jun 26, 2018
+ Jul 11, 2018
<link href="/css/docpage.css" rel="stylesheet" type="text/css">
@@ -1637,6 +1637,16 @@
<td>By default, Drill reads Hive tables using the native Hive reader. When you enable this option, Drill reads Hive tables using Drill native readers, which enables faster reads and enforces direct memory usage. Starting in Drill 1.14, this option also enables Drill to apply filter push down optimizations. Previously, this was the store.hive.optimize_scan_with_native_readers option, which is scheduled to be deprecated in Drill 1.15.</td>
</tr>
<tr>
+<td></td>
+<td>store.hive.conf.properties</td>
+<td>""</td>
+</tr>
+<tr>
+<td></td>
+<td></td>
+<td></td>
+</tr>
+<tr>
<td>store.json.all_text_mode</td>
<td>FALSE</td>
<td>Drill reads all data from the JSON files as VARCHAR. Prevents schema change errors.</td>
diff --git a/docs/hive-storage-plugin/index.html b/docs/hive-storage-plugin/index.html
index a1e2025..3a075cf 100644
--- a/docs/hive-storage-plugin/index.html
+++ b/docs/hive-storage-plugin/index.html
@@ -1236,7 +1236,7 @@
</div>
- Apr 16, 2018
+ Jul 11, 2018
<link href="/css/docpage.css" rel="stylesheet" type="text/css">
@@ -1245,16 +1245,31 @@
<p>Prior to Drill 1.13, Drill supported Hive 1.0. Drill 1.13 and later includes version 2.3.2 of the Hive client, which adds support for queries on transactional (ACID) and non-transactional Hive bucketed ORC tables. The updated Hive libraries are backward compatible with earlier versions of the Hive server and metastore. </p>
<p>To access Hive tables using custom SerDes or InputFormat/OutputFormat, all nodes running Drillbits must have the SerDes or InputFormat/OutputFormat <code>JAR</code> files in the
-<code><drill_installation_directory>/jars/3rdparty</code> folder.</p>
+<code><drill_installation_directory>/jars/3rdparty</code> folder. </p>
-<p>You can run Hive queries in the following ways by configuring the Hive storage plugin as described in this document:</p>
+<h2 id="setting-hive-properties">Setting Hive Properties</h2>
+
+<p>Prior to Drill 1.14, you could only set <a href="https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties">Hive configuration properties</a> at the system level in <code>hive-site.xml</code> or in the hive storage plugin configuration in the Drill Web UI. Starting in Drill 1.14, Drill supports the <code>store.hive.conf.properties</code> option, which enables you to specify Hive properties at the session level using the <a href="/docs/set/">SET command</a>, as shown:</p>
+<div class="highlight"><pre><code class="language-text" data-lang="text"> alter session set `store.hive.conf.properties` = 'hive.mapred.supports.subdirectories=true\nmapred.input.dir.recursive=true';
+</code></pre></div>
+<p><strong>Note:</strong> The <code>hive.mapred.supports.subdirectories</code> and <code>mapred.input.dir.recursive</code> properties enable Drill to query Hive external tables that have nested folders with data. For example, given the following directory structure: </p>
+<div class="highlight"><pre><code class="language-text" data-lang="text"> /data/my_tbl/sub_dir/data.txt
+</code></pre></div>
+<p>You could query the Hive external table named <code>my_tbl</code>, and Drill would return results that included the data from the <code>data.txt</code> file. </p>
+
+<h3 id="guidelines-for-using-the-store.hive.conf.properties-option">Guidelines for Using the <code>store.hive.conf.properties</code> Option</h3>
+
+<p>When you set Hive properties at the session level, use the following guidelines: </p>
<ul>
-<li><a href="/docs/hive-storage-plugin/#connect-drill-to-the-hive-remote-metastore">Connect Drill to the Hive remote metastore</a><br></li>
-<li><a href="/docs/hive-storage-plugin/#connect-to-the-hive-embedded-metastore">Connect to the Hive embedded metastore</a><br></li>
+<li>Do not set property values in quotes. Setting property values in quotes could result in incorrect parsing.<br></li>
+<li>Separate the property name and value with an equality sign (=).<br></li>
+<li>Use <code>\n</code> as the string delimiter between properties.<br></li>
</ul>
-<p>You update the Hive storage plugin by selecting the <strong>Storage tab</strong> on the <a href="/docs/plugin-configuration-basics/#using-the-drill-web-console">Drill Web Console</a>. From the list of disabled storage plugins in the Drill Web Console, click <strong>Update</strong> next to <code>hive</code>. The default Hive storage plugin configuration appears as follows:</p>
+<h2 id="making-hive-a-consumable-data-source-for-drill">Making Hive a Consumable Data Source for Drill</h2>
+
+<p>You must connect Drill to the Hive metastore before you can run queries on a Hive data source. When you configure Hive as a consumable data source for Drill, you update the Hive storage plugin by selecting the <strong>Storage tab</strong> in the <a href="/docs/plugin-configuration-basics/#using-the-drill-web-console">Drill Web UI</a>. From the list of disabled storage plugins in the Drill Web UI, click <strong>Update</strong> next to <code>hive</code>. The default Hive storage plugin [...]
<div class="highlight"><pre><code class="language-text" data-lang="text"> {
"type": "hive",
"enabled": false,
@@ -1265,9 +1280,18 @@
"fs.default.name": "file:///",
"hive.metastore.sasl.enabled": "false"
}
- }
+ }
</code></pre></div>
-<h2 id="connect-drill-to-the-hive-remote-metastore">Connect Drill to the Hive Remote Metastore</h2>
+<p>Update this storage plugin configuration for Hive in your environment. </p>
+
+<p>The following sections provide information for connecting Drill to a remote or embedded Hive metastore: </p>
+
+<ul>
+<li><a href="/docs/hive-storage-plugin/#connect-drill-to-a-remote-hive-metastore">Connect Drill to the Hive remote metastore</a><br></li>
+<li><a href="/docs/hive-storage-plugin/#connect-to-the-hive-embedded-metastore">Connect to the Hive embedded metastore</a><br></li>
+</ul>
+
+<h2 id="connect-drill-to-a-remote-hive-metastore">Connect Drill to a Remote Hive Metastore</h2>
<p>The Hive metastore runs as a separate service outside
of Hive. Drill can query the Hive metastore through Thrift. The
diff --git a/feed.xml b/feed.xml
index 3f65850..6cbd92b 100644
--- a/feed.xml
+++ b/feed.xml
@@ -6,8 +6,8 @@
</description>
<link>/</link>
<atom:link href="/feed.xml" rel="self" type="application/rss+xml"/>
- <pubDate>Fri, 06 Jul 2018 18:31:09 -0700</pubDate>
- <lastBuildDate>Fri, 06 Jul 2018 18:31:09 -0700</lastBuildDate>
+ <pubDate>Wed, 11 Jul 2018 14:29:11 -0700</pubDate>
+ <lastBuildDate>Wed, 11 Jul 2018 14:29:11 -0700</lastBuildDate>
<generator>Jekyll v2.5.2</generator>
<item>
[drill-site] 02/02: edits
Posted by br...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
bridgetb pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/drill-site.git
commit 3a063429002e253aca846f41c73ec7df617f5657
Author: Bridget Bevens <bb...@maprtech.com>
AuthorDate: Wed Jul 11 14:40:26 2018 -0700
edits
---
docs/hive-storage-plugin/index.html | 2 +-
feed.xml | 4 ++--
2 files changed, 3 insertions(+), 3 deletions(-)
diff --git a/docs/hive-storage-plugin/index.html b/docs/hive-storage-plugin/index.html
index 3a075cf..0a31f90 100644
--- a/docs/hive-storage-plugin/index.html
+++ b/docs/hive-storage-plugin/index.html
@@ -1259,7 +1259,7 @@
<h3 id="guidelines-for-using-the-store.hive.conf.properties-option">Guidelines for Using the <code>store.hive.conf.properties</code> Option</h3>
-<p>When you set Hive properties at the session level, use the following guidelines: </p>
+<p>When you set Hive properties at the session level, follow these guidelines: </p>
<ul>
<li>Do not set property values in quotes. Setting property values in quotes could result in incorrect parsing.<br></li>
diff --git a/feed.xml b/feed.xml
index 6cbd92b..f56c4c2 100644
--- a/feed.xml
+++ b/feed.xml
@@ -6,8 +6,8 @@
</description>
<link>/</link>
<atom:link href="/feed.xml" rel="self" type="application/rss+xml"/>
- <pubDate>Wed, 11 Jul 2018 14:29:11 -0700</pubDate>
- <lastBuildDate>Wed, 11 Jul 2018 14:29:11 -0700</lastBuildDate>
+ <pubDate>Wed, 11 Jul 2018 14:38:49 -0700</pubDate>
+ <lastBuildDate>Wed, 11 Jul 2018 14:38:49 -0700</lastBuildDate>
<generator>Jekyll v2.5.2</generator>
<item>