You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@impala.apache.org by mi...@apache.org on 2018/05/09 21:10:20 UTC
[11/51] [partial] impala git commit: [DOCS] Impala doc site update
for 3.0
http://git-wip-us.apache.org/repos/asf/impala/blob/fae51ec2/docs/build3x/html/topics/impala_proxy.html
----------------------------------------------------------------------
diff --git a/docs/build3x/html/topics/impala_proxy.html b/docs/build3x/html/topics/impala_proxy.html
new file mode 100644
index 0000000..9a2b90d
--- /dev/null
+++ b/docs/build3x/html/topics/impala_proxy.html
@@ -0,0 +1,501 @@
+<!DOCTYPE html
+ SYSTEM "about:legacy-compat">
+<html lang="en"><head><meta http-equiv="Content-Type" content="text/html; charset=UTF-8"><meta charset="UTF-8"><meta name="copyright" content="(C) Copyright 2018"><meta name="DC.rights.owner" content="(C) Copyright 2018"><meta name="DC.Type" content="concept"><meta name="DC.Relation" scheme="URI" content="../topics/impala_admin.html"><meta name="prodname" content="Impala"><meta name="prodname" content="Impala"><meta name="version" content="Impala 3.0.x"><meta name="version" content="Impala 3.0.x"><meta name="DC.Format" content="XHTML"><meta name="DC.Identifier" content="proxy"><link rel="stylesheet" type="text/css" href="../commonltr.css"><title>Using Impala through a Proxy for High Availability</title></head><body id="proxy"><main role="main"><article role="article" aria-labelledby="ariaid-title1">
+
+ <h1 class="title topictitle1" id="ariaid-title1">Using Impala through a Proxy for High Availability</h1>
+
+
+
+ <div class="body conbody">
+
+ <p class="p">
+ For most clusters that have multiple users and production availability requirements, you might set up a proxy
+ server to relay requests to and from Impala.
+ </p>
+
+ <p class="p">
+ Currently, the Impala statestore mechanism does not include such proxying and load-balancing features. Set up
+ a software package of your choice to perform these functions.
+ </p>
+
+ <div class="note note note_note"><span class="note__title notetitle">Note:</span>
+ <p class="p">
+ Most considerations for load balancing and high availability apply to the <span class="keyword cmdname">impalad</span> daemon.
+ The <span class="keyword cmdname">statestored</span> and <span class="keyword cmdname">catalogd</span> daemons do not have special
+ requirements for high availability, because problems with those daemons do not result in data loss.
+ If those daemons become unavailable due to an outage on a particular
+ host, you can stop the Impala service, delete the <span class="ph uicontrol">Impala StateStore</span> and
+ <span class="ph uicontrol">Impala Catalog Server</span> roles, add the roles on a different host, and restart the
+ Impala service.
+ </p>
+ </div>
+
+ <p class="p toc inpage"></p>
+
+ </div>
+
+ <nav role="navigation" class="related-links"><div class="familylinks"><div class="parentlink"><strong>Parent topic:</strong> <a class="link" href="../topics/impala_admin.html">Impala Administration</a></div></div></nav><article class="topic concept nested1" aria-labelledby="ariaid-title2" id="proxy__proxy_overview">
+
+ <h2 class="title topictitle2" id="ariaid-title2">Overview of Proxy Usage and Load Balancing for Impala</h2>
+
+
+ <div class="body conbody">
+
+ <p class="p">
+ Using a load-balancing proxy server for Impala has the following advantages:
+ </p>
+
+ <ul class="ul">
+ <li class="li">
+ Applications connect to a single well-known host and port, rather than keeping track of the hosts where
+ the <span class="keyword cmdname">impalad</span> daemon is running.
+ </li>
+
+ <li class="li">
+ If any host running the <span class="keyword cmdname">impalad</span> daemon becomes unavailable, application connection
+ requests still succeed because you always connect to the proxy server rather than a specific host running
+ the <span class="keyword cmdname">impalad</span> daemon.
+ </li>
+
+ <li class="li">
+ The coordinator node for each Impala query potentially requires more memory and CPU cycles than the other
+ nodes that process the query. The proxy server can issue queries using round-robin scheduling, so that
+ each connection uses a different coordinator node. This load-balancing technique lets the Impala nodes
+ share this additional work, rather than concentrating it on a single machine.
+ </li>
+ </ul>
+
+ <p class="p">
+ The following setup steps are a general outline that apply to any load-balancing proxy software:
+ </p>
+
+ <ol class="ol">
+ <li class="li">
+ Select and download the load-balancing proxy software or other
+ load-balancing hardware appliance. It should only need to be installed
+ and configured on a single host, typically on an edge node. Pick a
+ host other than the DataNodes where <span class="keyword cmdname">impalad</span> is
+ running, because the intention is to protect against the possibility
+ of one or more of these DataNodes becoming unavailable.
+ </li>
+
+ <li class="li">
+ Configure the load balancer (typically by editing a configuration file).
+ In particular:
+ <ul class="ul">
+ <li class="li">
+ Set up a port that the load balancer will listen on to relay
+ Impala requests back and forth. </li>
+ <li class="li">
+ See <a class="xref" href="#proxy_balancing">Choosing the Load-Balancing Algorithm</a> for load
+ balancing algorithm options.
+ </li>
+ <li class="li">
+ For Kerberized clusters, follow the instructions in <a class="xref" href="impala_proxy.html#proxy_kerberos">Special Proxy Considerations for Clusters Using Kerberos</a>.
+ </li>
+ </ul>
+ </li>
+
+ <li class="li">
+ If you are using Hue or JDBC-based applications, you typically set
+ up load balancing for both ports 21000 and 21050, because these client
+ applications connect through port 21050 while the
+ <span class="keyword cmdname">impala-shell</span> command connects through port
+ 21000. See <a class="xref" href="impala_ports.html#ports">Ports Used by Impala</a> for when to use port
+ 21000, 21050, or another value depending on what type of connections
+ you are load balancing.
+ </li>
+
+ <li class="li">
+ Run the load-balancing proxy server, pointing it at the configuration file that you set up.
+ </li>
+
+ <li class="li">
+ For any scripts, jobs, or configuration settings for applications
+ that formerly connected to a specific DataNode to run Impala SQL
+ statements, change the connection information (such as the
+ <code class="ph codeph">-i</code> option in <span class="keyword cmdname">impala-shell</span>) to
+ point to the load balancer instead.
+ </li>
+ </ol>
+
+ <div class="note note note_note"><span class="note__title notetitle">Note:</span>
+ The following sections use the HAProxy software as a representative example of a load balancer
+ that you can use with Impala.
+ </div>
+
+ </div>
+
+ </article>
+
+ <article class="topic concept nested1" aria-labelledby="ariaid-title3" id="proxy__proxy_balancing">
+ <h2 class="title topictitle2" id="ariaid-title3">Choosing the Load-Balancing Algorithm</h2>
+ <div class="body conbody">
+ <p class="p">
+ Load-balancing software offers a number of algorithms to distribute requests.
+ Each algorithm has its own characteristics that make it suitable in some situations
+ but not others.
+ </p>
+
+ <dl class="dl">
+
+ <dt class="dt dlterm">Leastconn</dt>
+ <dd class="dd">
+ Connects sessions to the coordinator with the fewest connections,
+ to balance the load evenly. Typically used for workloads consisting
+ of many independent, short-running queries. In configurations with
+ only a few client machines, this setting can avoid having all
+ requests go to only a small set of coordinators.
+ </dd>
+ <dd class="dd ddexpand">
+ Recommended for Impala with F5.
+ </dd>
+
+
+ <dt class="dt dlterm">Source IP Persistence</dt>
+ <dd class="dd">
+ <p class="p">
+ Sessions from the same IP address always go to the same
+ coordinator. A good choice for Impala workloads containing a mix
+ of queries and DDL statements, such as <code class="ph codeph">CREATE TABLE</code>
+ and <code class="ph codeph">ALTER TABLE</code>. Because the metadata changes from
+ a DDL statement take time to propagate across the cluster, prefer
+ to use the Source IP Persistence in this case. If you are unable
+ to choose Source IP Persistence, run the DDL and subsequent queries
+ that depend on the results of the DDL through the same session,
+ for example by running <code class="ph codeph">impala-shell -f <var class="keyword varname">script_file</var></code>
+ to submit several statements through a single session.
+ </p>
+ </dd>
+ <dd class="dd ddexpand">
+ <p class="p">
+ Required for setting up high availability with Hue.
+ </p>
+ </dd>
+
+
+ <dt class="dt dlterm">Round-robin</dt>
+ <dd class="dd">
+ <p class="p">
+ Distributes connections to all coordinator nodes.
+ Typically not recommended for Impala.
+ </p>
+ </dd>
+
+ </dl>
+
+ <p class="p">
+ You might need to perform benchmarks and load testing to determine
+ which setting is optimal for your use case. Always set up using two
+ load-balancing algorithms: Source IP Persistence for Hue and Leastconn
+ for others.
+ </p>
+
+ </div>
+ </article>
+
+ <article class="topic concept nested1" aria-labelledby="ariaid-title4" id="proxy__proxy_kerberos">
+
+ <h2 class="title topictitle2" id="ariaid-title4">Special Proxy Considerations for Clusters Using Kerberos</h2>
+
+
+ <div class="body conbody">
+
+ <p class="p">
+ In a cluster using Kerberos, applications check host credentials to
+ verify that the host they are connecting to is the same one that is
+ actually processing the request, to prevent man-in-the-middle attacks.
+ </p>
+ <p class="p">
+ In <span class="keyword">Impala 2.11</span> and lower
+ versions, once you enable a proxy server in a Kerberized cluster, users
+ will not be able to connect to individual impala daemons directly from
+ impala-shell.
+ </p>
+
+ <p class="p">
+ In <span class="keyword">Impala 2.12</span> and higher,
+ if you enable a proxy server in a Kerberized cluster, users have an
+ option to connect to Impala daemons directly from
+ <span class="keyword cmdname">impala-shell</span> using the <code class="ph codeph">-b</code> /
+ <code class="ph codeph">--kerberos_host_fqdn</code> option when you start
+ <span class="keyword cmdname">impala-shell</span>. This option can be used for testing or
+ troubleshooting purposes, but not recommended for live production
+ environments as it defeats the purpose of a load balancer/proxy.
+ </p>
+
+ <div class="p">
+ Example:
+<pre class="pre codeblock"><code>
+impala-shell -i impalad-1.mydomain.com -k -b loadbalancer-1.mydomain.com
+</code></pre>
+ </div>
+
+ <div class="p">
+ Alternatively, with the fully qualified
+ configurations:
+<pre class="pre codeblock"><code>impala-shell --impalad=impalad-1.mydomain.com:21000 --kerberos --kerberos_host_fqdn=loadbalancer-1.mydomain.com</code></pre>
+ </div>
+ <p class="p">
+ See <a class="xref" href="impala_shell_options.html#shell_options">impala-shell Configuration Options</a> for
+ information about the option.
+ </p>
+
+ <p class="p">
+ To clarify that the load-balancing proxy server is legitimate, perform
+ these extra Kerberos setup steps:
+ </p>
+
+ <ol class="ol">
+ <li class="li">
+ This section assumes you are starting with a Kerberos-enabled cluster. See
+ <a class="xref" href="impala_kerberos.html#kerberos">Enabling Kerberos Authentication for Impala</a> for instructions for setting up Impala with Kerberos. See
+ <span class="xref">the documentation for your Apache Hadoop distribution</span> for general steps to set up Kerberos.
+ </li>
+
+ <li class="li">
+ Choose the host you will use for the proxy server. Based on the Kerberos setup procedure, it should
+ already have an entry <code class="ph codeph">impala/<var class="keyword varname">proxy_host</var>@<var class="keyword varname">realm</var></code> in
+ its keytab. If not, go back over the initial Kerberos configuration steps for the keytab on each host
+ running the <span class="keyword cmdname">impalad</span> daemon.
+ </li>
+
+ <li class="li">
+ Copy the keytab file from the proxy host to all other hosts in the cluster that run the
+ <span class="keyword cmdname">impalad</span> daemon. (For optimal performance, <span class="keyword cmdname">impalad</span> should be running
+ on all DataNodes in the cluster.) Put the keytab file in a secure location on each of these other hosts.
+ </li>
+
+ <li class="li">
+ Add an entry <code class="ph codeph">impala/<var class="keyword varname">actual_hostname</var>@<var class="keyword varname">realm</var></code> to the keytab on each
+ host running the <span class="keyword cmdname">impalad</span> daemon.
+ </li>
+
+ <li class="li">
+
+ For each impalad node, merge the existing keytab with the proxy’s keytab using
+ <span class="keyword cmdname">ktutil</span>, producing a new keytab file. For example:
+ <pre class="pre codeblock"><code>$ ktutil
+ ktutil: read_kt proxy.keytab
+ ktutil: read_kt impala.keytab
+ ktutil: write_kt proxy_impala.keytab
+ ktutil: quit</code></pre>
+
+ </li>
+
+ <li class="li">
+
+ To verify that the keytabs are merged, run the command:
+<pre class="pre codeblock"><code>
+klist -k <var class="keyword varname">keytabfile</var>
+</code></pre>
+ which lists the credentials for both <code class="ph codeph">principal</code> and <code class="ph codeph">be_principal</code> on
+ all nodes.
+ </li>
+
+
+ <li class="li">
+
+ Make sure that the <code class="ph codeph">impala</code> user has permission to read this merged keytab file.
+
+ </li>
+
+ <li class="li">
+ Change the following configuration settings for each host in the cluster that participates
+ in the load balancing:
+ <ul class="ul">
+ <li class="li">
+ In the <span class="keyword cmdname">impalad</span> option definition, add:
+<pre class="pre codeblock"><code>
+--principal=impala/<em class="ph i">proxy_host@realm</em>
+ --be_principal=impala/<em class="ph i">actual_host@realm</em>
+ --keytab_file=<em class="ph i">path_to_merged_keytab</em>
+</code></pre>
+ <div class="note note note_note"><span class="note__title notetitle">Note:</span>
+ Every host has different <code class="ph codeph">--be_principal</code> because the actual hostname
+ is different on each host.
+
+ Specify the fully qualified domain name (FQDN) for the proxy host, not the IP
+ address. Use the exact FQDN as returned by a reverse DNS lookup for the associated
+ IP address.
+
+ </div>
+ </li>
+
+ <li class="li">
+ Modify the startup options. See <a class="xref" href="impala_config_options.html#config_options">Modifying Impala Startup Options</a> for the procedure to modify the startup
+ options.
+ </li>
+ </ul>
+ </li>
+
+ <li class="li">
+ Restart Impala to make the changes take effect. Restart the <span class="keyword cmdname">impalad</span> daemons on all
+ hosts in the cluster, as well as the <span class="keyword cmdname">statestored</span> and <span class="keyword cmdname">catalogd</span>
+ daemons.
+ </li>
+
+ </ol>
+
+
+
+ </div>
+
+ </article>
+
+ <article class="topic concept nested1" aria-labelledby="ariaid-title5" id="proxy__tut_proxy">
+
+ <h2 class="title topictitle2" id="ariaid-title5">Example of Configuring HAProxy Load Balancer for Impala</h2>
+
+
+ <div class="body conbody">
+
+ <p class="p">
+ If you are not already using a load-balancing proxy, you can experiment with
+ <a class="xref" href="http://haproxy.1wt.eu/" target="_blank">HAProxy</a> a free, open source load
+ balancer. This example shows how you might install and configure that load balancer on a Red Hat Enterprise
+ Linux system.
+ </p>
+
+ <ul class="ul">
+ <li class="li">
+ <p class="p">
+ Install the load balancer: <code class="ph codeph">yum install haproxy</code>
+ </p>
+ </li>
+
+ <li class="li">
+ <p class="p">
+ Set up the configuration file: <span class="ph filepath">/etc/haproxy/haproxy.cfg</span>. See the following section
+ for a sample configuration file.
+ </p>
+ </li>
+
+ <li class="li">
+ <p class="p">
+ Run the load balancer (on a single host, preferably one not running <span class="keyword cmdname">impalad</span>):
+ </p>
+<pre class="pre codeblock"><code>/usr/sbin/haproxy –f /etc/haproxy/haproxy.cfg</code></pre>
+ </li>
+
+ <li class="li">
+ <p class="p">
+ In <span class="keyword cmdname">impala-shell</span>, JDBC applications, or ODBC applications, connect to the listener
+ port of the proxy host, rather than port 21000 or 21050 on a host actually running <span class="keyword cmdname">impalad</span>.
+ The sample configuration file sets haproxy to listen on port 25003, therefore you would send all
+ requests to <code class="ph codeph"><var class="keyword varname">haproxy_host</var>:25003</code>.
+ </p>
+ </li>
+ </ul>
+
+ <p class="p">
+ This is the sample <span class="ph filepath">haproxy.cfg</span> used in this example:
+ </p>
+
+<pre class="pre codeblock"><code>global
+ # To have these messages end up in /var/log/haproxy.log you will
+ # need to:
+ #
+ # 1) configure syslog to accept network log events. This is done
+ # by adding the '-r' option to the SYSLOGD_OPTIONS in
+ # /etc/sysconfig/syslog
+ #
+ # 2) configure local2 events to go to the /var/log/haproxy.log
+ # file. A line like the following can be added to
+ # /etc/sysconfig/syslog
+ #
+ # local2.* /var/log/haproxy.log
+ #
+ log 127.0.0.1 local0
+ log 127.0.0.1 local1 notice
+ chroot /var/lib/haproxy
+ pidfile /var/run/haproxy.pid
+ maxconn 4000
+ user haproxy
+ group haproxy
+ daemon
+
+ # turn on stats unix socket
+ #stats socket /var/lib/haproxy/stats
+
+#---------------------------------------------------------------------
+# common defaults that all the 'listen' and 'backend' sections will
+# use if not designated in their block
+#
+# You might need to adjust timing values to prevent timeouts.
+#
+# The timeout values should be dependant on how you use the cluster
+# and how long your queries run.
+#---------------------------------------------------------------------
+defaults
+ mode http
+ log global
+ option httplog
+ option dontlognull
+ option http-server-close
+ option forwardfor except 127.0.0.0/8
+ option redispatch
+ retries 3
+ maxconn 3000
+ timeout connect 5000
+ timeout client 3600s
+ timeout server 3600s
+
+#
+# This sets up the admin page for HA Proxy at port 25002.
+#
+listen stats :25002
+ balance
+ mode http
+ stats enable
+ stats auth <var class="keyword varname">username</var>:<var class="keyword varname">password</var>
+
+# This is the setup for Impala. Impala client connect to load_balancer_host:25003.
+# HAProxy will balance connections among the list of servers listed below.
+# The list of Impalad is listening at port 21000 for beeswax (impala-shell) or original ODBC driver.
+# For JDBC or ODBC version 2.x driver, use port 21050 instead of 21000.
+listen impala :25003
+ mode tcp
+ option tcplog
+ balance leastconn
+
+ server <var class="keyword varname">symbolic_name_1</var> impala-host-1.example.com:21000
+ server <var class="keyword varname">symbolic_name_2</var> impala-host-2.example.com:21000
+ server <var class="keyword varname">symbolic_name_3</var> impala-host-3.example.com:21000
+ server <var class="keyword varname">symbolic_name_4</var> impala-host-4.example.com:21000
+
+# Setup for Hue or other JDBC-enabled applications.
+# In particular, Hue requires sticky sessions.
+# The application connects to load_balancer_host:21051, and HAProxy balances
+# connections to the associated hosts, where Impala listens for JDBC
+# requests on port 21050.
+listen impalajdbc :21051
+ mode tcp
+ option tcplog
+ balance source
+ server <var class="keyword varname">symbolic_name_5</var> impala-host-1.example.com:21050 check
+ server <var class="keyword varname">symbolic_name_6</var> impala-host-2.example.com:21050 check
+ server <var class="keyword varname">symbolic_name_7</var> impala-host-3.example.com:21050 check
+ server <var class="keyword varname">symbolic_name_8</var> impala-host-4.example.com:21050 check
+</code></pre>
+ <div class="note important note_important"><span class="note__title importanttitle">Important:</span>
+ Hue requires the <code class="ph codeph">check</code> option at end of each line in
+ the above file to ensure HAProxy can detect any unreachable Impalad
+ server, and failover can be successful. Without the TCP check, you may hit
+ an error when the <span class="keyword cmdname">impalad</span> daemon to which Hue tries to
+ connect is down.
+ </div>
+
+ <div class="note note note_note"><span class="note__title notetitle">Note:</span>
+ If your JDBC or ODBC application connects to Impala through a load balancer such as
+ <code class="ph codeph">haproxy</code>, be cautious about reusing the connections. If the load balancer has set up
+ connection timeout values, either check the connection frequently so that it never sits idle longer than
+ the load balancer timeout value, or check the connection validity before using it and create a new one if
+ the connection has been closed.
+ </div>
+
+ </div>
+
+ </article>
+
+</article></main></body></html>
http://git-wip-us.apache.org/repos/asf/impala/blob/fae51ec2/docs/build3x/html/topics/impala_query_options.html
----------------------------------------------------------------------
diff --git a/docs/build3x/html/topics/impala_query_options.html b/docs/build3x/html/topics/impala_query_options.html
new file mode 100644
index 0000000..40b0c8e
--- /dev/null
+++ b/docs/build3x/html/topics/impala_query_options.html
@@ -0,0 +1,55 @@
+<!DOCTYPE html
+ SYSTEM "about:legacy-compat">
+<html lang="en"><head><meta http-equiv="Content-Type" content="text/html; charset=UTF-8"><meta charset="UTF-8"><meta name="copyright" content="(C) Copyright 2018"><meta name="DC.rights.owner" content="(C) Copyright 2018"><meta name="DC.Type" content="concept"><meta name="DC.Relation" scheme="URI" content="../topics/impala_set.html"><meta name="DC.Relation" scheme="URI" content="../topics/impala_abort_on_error.html"><meta name="DC.Relation" scheme="URI" content="../topics/impala_allow_unsupported_formats.html"><meta name="DC.Relation" scheme="URI" content="../topics/impala_appx_count_distinct.html"><meta name="DC.Relation" scheme="URI" content="../topics/impala_batch_size.html"><meta name="DC.Relation" scheme="URI" content="../topics/impala_buffer_pool_limit.html"><meta name="DC.Relation" scheme="URI" content="../topics/impala_compression_codec.html"><meta name="DC.Relation" scheme="URI" content="../topics/impala_compute_stats_min_sample_size.html"><meta name="DC.Relation" scheme="UR
I" content="../topics/impala_debug_action.html"><meta name="DC.Relation" scheme="URI" content="../topics/impala_decimal_v2.html"><meta name="DC.Relation" scheme="URI" content="../topics/impala_default_join_distribution_mode.html"><meta name="DC.Relation" scheme="URI" content="../topics/impala_default_spillable_buffer_size.html"><meta name="DC.Relation" scheme="URI" content="../topics/impala_disable_codegen.html"><meta name="DC.Relation" scheme="URI" content="../topics/impala_disable_row_runtime_filtering.html"><meta name="DC.Relation" scheme="URI" content="../topics/impala_disable_streaming_preaggregations.html"><meta name="DC.Relation" scheme="URI" content="../topics/impala_disable_unsafe_spills.html"><meta name="DC.Relation" scheme="URI" content="../topics/impala_exec_single_node_rows_threshold.html"><meta name="DC.Relation" scheme="URI" content="../topics/impala_exec_time_limit_s.html"><meta name="DC.Relation" scheme="URI" content="../topics/impala_explain_level.html"><meta name=
"DC.Relation" scheme="URI" content="../topics/impala_hbase_cache_blocks.html"><meta name="DC.Relation" scheme="URI" content="../topics/impala_hbase_caching.html"><meta name="DC.Relation" scheme="URI" content="../topics/impala_live_progress.html"><meta name="DC.Relation" scheme="URI" content="../topics/impala_live_summary.html"><meta name="DC.Relation" scheme="URI" content="../topics/impala_max_errors.html"><meta name="DC.Relation" scheme="URI" content="../topics/impala_max_row_size.html"><meta name="DC.Relation" scheme="URI" content="../topics/impala_max_num_runtime_filters.html"><meta name="DC.Relation" scheme="URI" content="../topics/impala_max_scan_range_length.html"><meta name="DC.Relation" scheme="URI" content="../topics/impala_mem_limit.html"><meta name="DC.Relation" scheme="URI" content="../topics/impala_min_spillable_buffer_size.html"><meta name="DC.Relation" scheme="URI" content="../topics/impala_mt_dop.html"><meta name="DC.Relation" scheme="URI" content="../topics/impala_n
um_nodes.html"><meta name="DC.Relation" scheme="URI" content="../topics/impala_num_scanner_threads.html"><meta name="DC.Relation" scheme="URI" content="../topics/impala_optimize_partition_key_scans.html"><meta name="DC.Relation" scheme="URI" content="../topics/impala_parquet_compression_codec.html"><meta name="DC.Relation" scheme="URI" content="../topics/impala_parquet_annotate_strings_utf8.html"><meta name="DC.Relation" scheme="URI" content="../topics/impala_parquet_array_resolution.html"><meta name="DC.Relation" scheme="URI" content="../topics/impala_parquet_fallback_schema_resolution.html"><meta name="DC.Relation" scheme="URI" content="../topics/impala_parquet_file_size.html"><meta name="DC.Relation" scheme="URI" content="../topics/impala_prefetch_mode.html"><meta name="DC.Relation" scheme="URI" content="../topics/impala_query_timeout_s.html"><meta name="DC.Relation" scheme="URI" content="../topics/impala_request_pool.html"><meta name="DC.Relation" scheme="URI" content="../topics
/impala_replica_preference.html"><meta name="DC.Relation" scheme="URI" content="../topics/impala_runtime_bloom_filter_size.html"><meta name="DC.Relation" scheme="URI" content="../topics/impala_runtime_filter_max_size.html"><meta name="DC.Relation" scheme="URI" content="../topics/impala_runtime_filter_min_size.html"><meta name="DC.Relation" scheme="URI" content="../topics/impala_runtime_filter_mode.html"><meta name="DC.Relation" scheme="URI" content="../topics/impala_runtime_filter_wait_time_ms.html"><meta name="DC.Relation" scheme="URI" content="../topics/impala_s3_skip_insert_staging.html"><meta name="DC.Relation" scheme="URI" content="../topics/impala_schedule_random_replica.html"><meta name="DC.Relation" scheme="URI" content="../topics/impala_scratch_limit.html"><meta name="DC.Relation" scheme="URI" content="../topics/impala_shuffle_distinct_exprs.html"><meta name="DC.Relation" scheme="URI" content="../topics/impala_support_start_over.html"><meta name="DC.Relation" scheme="URI" c
ontent="../topics/impala_sync_ddl.html"><meta name="prodname" content="Impala"><meta name="prodname" content="Impala"><meta name="version" content="Impala 3.0.x"><meta name="version" content="Impala 3.0.x"><meta name="DC.Format" content="XHTML"><meta name="DC.Identifier" content="query_options"><link rel="stylesheet" type="text/css" href="../commonltr.css"><title>Query Options for the SET Statement</title></head><body id="query_options"><main role="main"><article role="article" aria-labelledby="ariaid-title1">
+
+ <h1 class="title topictitle1" id="ariaid-title1">Query Options for the SET Statement</h1>
+
+
+ <div class="body conbody">
+
+ <p class="p">
+ You can specify the following options using the <code class="ph codeph">SET</code> statement, and those settings affect all
+ queries issued from that session.
+ </p>
+
+ <p class="p">
+ Some query options are useful in day-to-day operations for improving usability, performance, or flexibility.
+ </p>
+
+ <p class="p">
+ Other query options control special-purpose aspects of Impala operation and are intended primarily for
+ advanced debugging or troubleshooting.
+ </p>
+
+ <p class="p">
+ Options with Boolean parameters can be set to 1 or <code class="ph codeph">true</code> to enable, or 0 or <code class="ph codeph">false</code>
+ to turn off.
+ </p>
+
+ <div class="note note note_note"><span class="note__title notetitle">Note:</span>
+ <p class="p">
+ In Impala 2.0 and later, you can set query options directly through the JDBC and ODBC interfaces by using the
+ <code class="ph codeph">SET</code> statement. Formerly, <code class="ph codeph">SET</code> was only available as a command within the
+ <span class="keyword cmdname">impala-shell</span> interpreter.
+ </p>
+ <p class="p">
+ In <span class="keyword">Impala 2.11</span> and later, you can set query options for an <span class="keyword cmdname">impala-shell</span> session
+ by specifying one or more command-line arguments of the form
+ <code class="ph codeph">--query_option=<var class="keyword varname">option</var>=<var class="keyword varname">value</var></code>.
+ See <a class="xref" href="impala_shell_options.html">impala-shell Configuration Options</a> for details.
+ </p>
+ </div>
+
+
+
+ <p class="p toc"></p>
+
+ <p class="p">
+ <strong class="ph b">Related information:</strong>
+ </p>
+
+ <p class="p">
+ <a class="xref" href="impala_set.html#set">SET Statement</a>
+ </p>
+ </div>
+<nav role="navigation" class="related-links"><ul class="ullinks"><li class="link ulchildlink"><strong><a href="../topics/impala_abort_on_error.html">ABORT_ON_ERROR Query Option</a></strong><br></li><li class="link ulchildlink"><strong><a href="../topics/impala_allow_unsupported_formats.html">ALLOW_UNSUPPORTED_FORMATS Query Option</a></strong><br></li><li class="link ulchildlink"><strong><a href="../topics/impala_appx_count_distinct.html">APPX_COUNT_DISTINCT Query Option (Impala 2.0 or higher only)</a></strong><br></li><li class="link ulchildlink"><strong><a href="../topics/impala_batch_size.html">BATCH_SIZE Query Option</a></strong><br></li><li class="link ulchildlink"><strong><a href="../topics/impala_buffer_pool_limit.html">BUFFER_POOL_LIMIT Query Option</a></strong><br></li><li class="link ulchildlink"><strong><a href="../topics/impala_compression_codec.html">COMPRESSION_CODEC Query Option (Impala 2.0 or higher only)</a></strong><br></li><li class="link ulchildlink"><strong><a hr
ef="../topics/impala_compute_stats_min_sample_size.html">COMPUTE_STATS_MIN_SAMPLE_SIZE Query Option</a></strong><br></li><li class="link ulchildlink"><strong><a href="../topics/impala_debug_action.html">DEBUG_ACTION Query Option</a></strong><br></li><li class="link ulchildlink"><strong><a href="../topics/impala_decimal_v2.html">DECIMAL_V2 Query Option</a></strong><br></li><li class="link ulchildlink"><strong><a href="../topics/impala_default_join_distribution_mode.html">DEFAULT_JOIN_DISTRIBUTION_MODE Query Option</a></strong><br></li><li class="link ulchildlink"><strong><a href="../topics/impala_default_spillable_buffer_size.html">DEFAULT_SPILLABLE_BUFFER_SIZE Query Option</a></strong><br></li><li class="link ulchildlink"><strong><a href="../topics/impala_disable_codegen.html">DISABLE_CODEGEN Query Option</a></strong><br></li><li class="link ulchildlink"><strong><a href="../topics/impala_disable_row_runtime_filtering.html">DISABLE_ROW_RUNTIME_FILTERING Query Option (Impala 2.5 or hi
gher only)</a></strong><br></li><li class="link ulchildlink"><strong><a href="../topics/impala_disable_streaming_preaggregations.html">DISABLE_STREAMING_PREAGGREGATIONS Query Option (Impala 2.5 or higher only)</a></strong><br></li><li class="link ulchildlink"><strong><a href="../topics/impala_disable_unsafe_spills.html">DISABLE_UNSAFE_SPILLS Query Option (Impala 2.0 or higher only)</a></strong><br></li><li class="link ulchildlink"><strong><a href="../topics/impala_exec_single_node_rows_threshold.html">EXEC_SINGLE_NODE_ROWS_THRESHOLD Query Option (Impala 2.1 or higher only)</a></strong><br></li><li class="link ulchildlink"><strong><a href="../topics/impala_exec_time_limit_s.html">EXEC_TIME_LIMIT_S Query Option (Impala 2.12 or higher only)</a></strong><br></li><li class="link ulchildlink"><strong><a href="../topics/impala_explain_level.html">EXPLAIN_LEVEL Query Option</a></strong><br></li><li class="link ulchildlink"><strong><a href="../topics/impala_hbase_cache_blocks.html">HBASE_CAC
HE_BLOCKS Query Option</a></strong><br></li><li class="link ulchildlink"><strong><a href="../topics/impala_hbase_caching.html">HBASE_CACHING Query Option</a></strong><br></li><li class="link ulchildlink"><strong><a href="../topics/impala_live_progress.html">LIVE_PROGRESS Query Option (Impala 2.3 or higher only)</a></strong><br></li><li class="link ulchildlink"><strong><a href="../topics/impala_live_summary.html">LIVE_SUMMARY Query Option (Impala 2.3 or higher only)</a></strong><br></li><li class="link ulchildlink"><strong><a href="../topics/impala_max_errors.html">MAX_ERRORS Query Option</a></strong><br></li><li class="link ulchildlink"><strong><a href="../topics/impala_max_row_size.html">MAX_ROW_SIZE Query Option</a></strong><br></li><li class="link ulchildlink"><strong><a href="../topics/impala_max_num_runtime_filters.html">MAX_NUM_RUNTIME_FILTERS Query Option (Impala 2.5 or higher only)</a></strong><br></li><li class="link ulchildlink"><strong><a href="../topics/impala_max_scan_r
ange_length.html">MAX_SCAN_RANGE_LENGTH Query Option</a></strong><br></li><li class="link ulchildlink"><strong><a href="../topics/impala_mem_limit.html">MEM_LIMIT Query Option</a></strong><br></li><li class="link ulchildlink"><strong><a href="../topics/impala_min_spillable_buffer_size.html">MIN_SPILLABLE_BUFFER_SIZE Query Option</a></strong><br></li><li class="link ulchildlink"><strong><a href="../topics/impala_mt_dop.html">MT_DOP Query Option</a></strong><br></li><li class="link ulchildlink"><strong><a href="../topics/impala_num_nodes.html">NUM_NODES Query Option</a></strong><br></li><li class="link ulchildlink"><strong><a href="../topics/impala_num_scanner_threads.html">NUM_SCANNER_THREADS Query Option</a></strong><br></li><li class="link ulchildlink"><strong><a href="../topics/impala_optimize_partition_key_scans.html">OPTIMIZE_PARTITION_KEY_SCANS Query Option (Impala 2.5 or higher only)</a></strong><br></li><li class="link ulchildlink"><strong><a href="../topics/impala_parquet_co
mpression_codec.html">PARQUET_COMPRESSION_CODEC Query Option</a></strong><br></li><li class="link ulchildlink"><strong><a href="../topics/impala_parquet_annotate_strings_utf8.html">PARQUET_ANNOTATE_STRINGS_UTF8 Query Option (Impala 2.6 or higher only)</a></strong><br></li><li class="link ulchildlink"><strong><a href="../topics/impala_parquet_array_resolution.html">PARQUET_ARRAY_RESOLUTION Query Option (Impala 2.9 or higher only)</a></strong><br></li><li class="link ulchildlink"><strong><a href="../topics/impala_parquet_fallback_schema_resolution.html">PARQUET_FALLBACK_SCHEMA_RESOLUTION Query Option (Impala 2.6 or higher only)</a></strong><br></li><li class="link ulchildlink"><strong><a href="../topics/impala_parquet_file_size.html">PARQUET_FILE_SIZE Query Option</a></strong><br></li><li class="link ulchildlink"><strong><a href="../topics/impala_prefetch_mode.html">PREFETCH_MODE Query Option (Impala 2.6 or higher only)</a></strong><br></li><li class="link ulchildlink"><strong><a href
="../topics/impala_query_timeout_s.html">QUERY_TIMEOUT_S Query Option (Impala 2.0 or higher only)</a></strong><br></li><li class="link ulchildlink"><strong><a href="../topics/impala_request_pool.html">REQUEST_POOL Query Option</a></strong><br></li><li class="link ulchildlink"><strong><a href="../topics/impala_replica_preference.html">REPLICA_PREFERENCE Query Option (Impala 2.7 or higher only)</a></strong><br></li><li class="link ulchildlink"><strong><a href="../topics/impala_runtime_bloom_filter_size.html">RUNTIME_BLOOM_FILTER_SIZE Query Option (Impala 2.5 or higher only)</a></strong><br></li><li class="link ulchildlink"><strong><a href="../topics/impala_runtime_filter_max_size.html">RUNTIME_FILTER_MAX_SIZE Query Option (Impala 2.6 or higher only)</a></strong><br></li><li class="link ulchildlink"><strong><a href="../topics/impala_runtime_filter_min_size.html">RUNTIME_FILTER_MIN_SIZE Query Option (Impala 2.6 or higher only)</a></strong><br></li><li class="link ulchildlink"><strong><a
href="../topics/impala_runtime_filter_mode.html">RUNTIME_FILTER_MODE Query Option (Impala 2.5 or higher only)</a></strong><br></li><li class="link ulchildlink"><strong><a href="../topics/impala_runtime_filter_wait_time_ms.html">RUNTIME_FILTER_WAIT_TIME_MS Query Option (Impala 2.5 or higher only)</a></strong><br></li><li class="link ulchildlink"><strong><a href="../topics/impala_s3_skip_insert_staging.html">S3_SKIP_INSERT_STAGING Query Option (Impala 2.6 or higher only)</a></strong><br></li><li class="link ulchildlink"><strong><a href="../topics/impala_schedule_random_replica.html">SCHEDULE_RANDOM_REPLICA Query Option (Impala 2.5 or higher only)</a></strong><br></li><li class="link ulchildlink"><strong><a href="../topics/impala_scratch_limit.html">SCRATCH_LIMIT Query Option</a></strong><br></li><li class="link ulchildlink"><strong><a href="../topics/impala_shuffle_distinct_exprs.html">SHUFFLE_DISTINCT_EXPRS Query Option</a></strong><br></li><li class="link ulchildlink"><strong><a hr
ef="../topics/impala_support_start_over.html">SUPPORT_START_OVER Query Option</a></strong><br></li><li class="link ulchildlink"><strong><a href="../topics/impala_sync_ddl.html">SYNC_DDL Query Option</a></strong><br></li></ul><div class="familylinks"><div class="parentlink"><strong>Parent topic:</strong> <a class="link" href="../topics/impala_set.html">SET Statement</a></div></div></nav></article></main></body></html>
http://git-wip-us.apache.org/repos/asf/impala/blob/fae51ec2/docs/build3x/html/topics/impala_query_timeout_s.html
----------------------------------------------------------------------
diff --git a/docs/build3x/html/topics/impala_query_timeout_s.html b/docs/build3x/html/topics/impala_query_timeout_s.html
new file mode 100644
index 0000000..d6c11e6
--- /dev/null
+++ b/docs/build3x/html/topics/impala_query_timeout_s.html
@@ -0,0 +1,62 @@
+<!DOCTYPE html
+ SYSTEM "about:legacy-compat">
+<html lang="en"><head><meta http-equiv="Content-Type" content="text/html; charset=UTF-8"><meta charset="UTF-8"><meta name="copyright" content="(C) Copyright 2018"><meta name="DC.rights.owner" content="(C) Copyright 2018"><meta name="DC.Type" content="concept"><meta name="DC.Relation" scheme="URI" content="../topics/impala_query_options.html"><meta name="prodname" content="Impala"><meta name="prodname" content="Impala"><meta name="version" content="Impala 3.0.x"><meta name="version" content="Impala 3.0.x"><meta name="DC.Format" content="XHTML"><meta name="DC.Identifier" content="query_timeout_s"><link rel="stylesheet" type="text/css" href="../commonltr.css"><title>QUERY_TIMEOUT_S Query Option (Impala 2.0 or higher only)</title></head><body id="query_timeout_s"><main role="main"><article role="article" aria-labelledby="ariaid-title1">
+
+ <h1 class="title topictitle1" id="ariaid-title1">QUERY_TIMEOUT_S Query Option (<span class="keyword">Impala 2.0</span> or higher only)</h1>
+
+
+
+ <div class="body conbody">
+
+ <p class="p">
+
+ Sets the idle query timeout value for the session, in seconds. Queries that sit idle for longer than the
+ timeout value are automatically cancelled. If the system administrator specified the
+ <code class="ph codeph">--idle_query_timeout</code> startup option, <code class="ph codeph">QUERY_TIMEOUT_S</code> must be smaller than
+ or equal to the <code class="ph codeph">--idle_query_timeout</code> value.
+ </p>
+
+ <div class="note note note_note"><span class="note__title notetitle">Note:</span>
+ <p class="p">
+ The timeout clock for queries and sessions only starts ticking when the query or session is idle.
+ For queries, this means the query has results ready but is waiting for a client to fetch the data. A
+ query can run for an arbitrary time without triggering a timeout, because the query is computing results
+ rather than sitting idle waiting for the results to be fetched. The timeout period is intended to prevent
+ unclosed queries from consuming resources and taking up slots in the admission count of running queries,
+ potentially preventing other queries from starting.
+ </p>
+ <p class="p">
+ For sessions, this means that no query has been submitted for some period of time.
+ </p>
+ </div>
+
+ <p class="p">
+ <strong class="ph b">Syntax:</strong>
+ </p>
+
+<pre class="pre codeblock"><code>SET QUERY_TIMEOUT_S=<var class="keyword varname">seconds</var>;</code></pre>
+
+
+
+ <p class="p">
+ <strong class="ph b">Type:</strong> numeric
+ </p>
+
+ <p class="p">
+ <strong class="ph b">Default:</strong> 0 (no timeout if <code class="ph codeph">--idle_query_timeout</code> not in effect; otherwise, use
+ <code class="ph codeph">--idle_query_timeout</code> value)
+ </p>
+
+ <p class="p">
+ <strong class="ph b">Added in:</strong> <span class="keyword">Impala 2.0.0</span>
+ </p>
+
+ <p class="p">
+ <strong class="ph b">Related information:</strong>
+ </p>
+
+ <p class="p">
+ <a class="xref" href="impala_timeouts.html#timeouts">Setting Timeout Periods for Daemons, Queries, and Sessions</a>
+ </p>
+ </div>
+<nav role="navigation" class="related-links"><div class="familylinks"><div class="parentlink"><strong>Parent topic:</strong> <a class="link" href="../topics/impala_query_options.html">Query Options for the SET Statement</a></div></div></nav></article></main></body></html>
http://git-wip-us.apache.org/repos/asf/impala/blob/fae51ec2/docs/build3x/html/topics/impala_rcfile.html
----------------------------------------------------------------------
diff --git a/docs/build3x/html/topics/impala_rcfile.html b/docs/build3x/html/topics/impala_rcfile.html
new file mode 100644
index 0000000..72b1bd8
--- /dev/null
+++ b/docs/build3x/html/topics/impala_rcfile.html
@@ -0,0 +1,246 @@
+<!DOCTYPE html
+ SYSTEM "about:legacy-compat">
+<html lang="en"><head><meta http-equiv="Content-Type" content="text/html; charset=UTF-8"><meta charset="UTF-8"><meta name="copyright" content="(C) Copyright 2018"><meta name="DC.rights.owner" content="(C) Copyright 2018"><meta name="DC.Type" content="concept"><meta name="DC.Relation" scheme="URI" content="../topics/impala_file_formats.html"><meta name="prodname" content="Impala"><meta name="prodname" content="Impala"><meta name="version" content="Impala 3.0.x"><meta name="version" content="Impala 3.0.x"><meta name="DC.Format" content="XHTML"><meta name="DC.Identifier" content="rcfile"><link rel="stylesheet" type="text/css" href="../commonltr.css"><title>Using the RCFile File Format with Impala Tables</title></head><body id="rcfile"><main role="main"><article role="article" aria-labelledby="ariaid-title1">
+
+ <h1 class="title topictitle1" id="ariaid-title1">Using the RCFile File Format with Impala Tables</h1>
+
+
+
+ <div class="body conbody">
+
+ <p class="p">
+
+ Impala supports using RCFile data files.
+ </p>
+
+ <table class="table"><caption><span class="table--title-label">Table 1. </span><span class="title">RCFile Format Support in Impala</span></caption><colgroup><col style="width:10%"><col style="width:10%"><col style="width:20%"><col style="width:30%"><col style="width:30%"></colgroup><thead class="thead">
+ <tr class="row">
+ <th class="entry nocellnorowborder" id="rcfile__entry__1">
+ File Type
+ </th>
+ <th class="entry nocellnorowborder" id="rcfile__entry__2">
+ Format
+ </th>
+ <th class="entry nocellnorowborder" id="rcfile__entry__3">
+ Compression Codecs
+ </th>
+ <th class="entry nocellnorowborder" id="rcfile__entry__4">
+ Impala Can CREATE?
+ </th>
+ <th class="entry nocellnorowborder" id="rcfile__entry__5">
+ Impala Can INSERT?
+ </th>
+ </tr>
+ </thead><tbody class="tbody">
+ <tr class="row">
+ <td class="entry nocellnorowborder" headers="rcfile__entry__1 ">
+ <a class="xref" href="impala_rcfile.html#rcfile">RCFile</a>
+ </td>
+ <td class="entry nocellnorowborder" headers="rcfile__entry__2 ">
+ Structured
+ </td>
+ <td class="entry nocellnorowborder" headers="rcfile__entry__3 ">
+ Snappy, gzip, deflate, bzip2
+ </td>
+ <td class="entry nocellnorowborder" headers="rcfile__entry__4 ">
+ Yes.
+ </td>
+ <td class="entry nocellnorowborder" headers="rcfile__entry__5 ">
+ No. Import data by using <code class="ph codeph">LOAD DATA</code> on data files already in the right format, or use
+ <code class="ph codeph">INSERT</code> in Hive followed by <code class="ph codeph">REFRESH <var class="keyword varname">table_name</var></code> in Impala.
+ </td>
+
+ </tr>
+ </tbody></table>
+
+ <p class="p toc inpage"></p>
+ </div>
+
+ <nav role="navigation" class="related-links"><div class="familylinks"><div class="parentlink"><strong>Parent topic:</strong> <a class="link" href="../topics/impala_file_formats.html">How Impala Works with Hadoop File Formats</a></div></div></nav><article class="topic concept nested1" aria-labelledby="ariaid-title2" id="rcfile__rcfile_create">
+
+ <h2 class="title topictitle2" id="ariaid-title2">Creating RCFile Tables and Loading Data</h2>
+
+
+ <div class="body conbody">
+
+ <p class="p">
+ If you do not have an existing data file to use, begin by creating one in the appropriate format.
+ </p>
+
+ <p class="p">
+ <strong class="ph b">To create an RCFile table:</strong>
+ </p>
+
+ <p class="p">
+ In the <code class="ph codeph">impala-shell</code> interpreter, issue a command similar to:
+ </p>
+
+<pre class="pre codeblock"><code>create table rcfile_table (<var class="keyword varname">column_specs</var>) stored as rcfile;</code></pre>
+
+ <p class="p">
+ Because Impala can query some kinds of tables that it cannot currently write to, after creating tables of
+ certain file formats, you might use the Hive shell to load the data. See
+ <a class="xref" href="impala_file_formats.html#file_formats">How Impala Works with Hadoop File Formats</a> for details. After loading data into a table through
+ Hive or other mechanism outside of Impala, issue a <code class="ph codeph">REFRESH <var class="keyword varname">table_name</var></code>
+ statement the next time you connect to the Impala node, before querying the table, to make Impala recognize
+ the new data.
+ </p>
+
+ <div class="note important note_important"><span class="note__title importanttitle">Important:</span>
+ See <a class="xref" href="impala_known_issues.html#known_issues">Known Issues and Workarounds in Impala</a> for potential compatibility issues with
+ RCFile tables created in Hive 0.12, due to a change in the default RCFile SerDe for Hive.
+ </div>
+
+ <p class="p">
+ For example, here is how you might create some RCFile tables in Impala (by specifying the columns
+ explicitly, or cloning the structure of another table), load data through Hive, and query them through
+ Impala:
+ </p>
+
+<pre class="pre codeblock"><code>$ impala-shell -i localhost
+[localhost:21000] > create table rcfile_table (x int) stored as rcfile;
+[localhost:21000] > create table rcfile_clone like some_other_table stored as rcfile;
+[localhost:21000] > quit;
+
+$ hive
+hive> insert into table rcfile_table select x from some_other_table;
+3 Rows loaded to rcfile_table
+Time taken: 19.015 seconds
+hive> quit;
+
+$ impala-shell -i localhost
+[localhost:21000] > select * from rcfile_table;
+Returned 0 row(s) in 0.23s
+[localhost:21000] > -- Make Impala recognize the data loaded through Hive;
+[localhost:21000] > refresh rcfile_table;
+[localhost:21000] > select * from rcfile_table;
++---+
+| x |
++---+
+| 1 |
+| 2 |
+| 3 |
++---+
+Returned 3 row(s) in 0.23s</code></pre>
+
+ <p class="p">
+ <strong class="ph b">Complex type considerations:</strong>
+ Although you can create tables in this file format using
+ the complex types (<code class="ph codeph">ARRAY</code>, <code class="ph codeph">STRUCT</code>,
+ and <code class="ph codeph">MAP</code>) available in <span class="keyword">Impala 2.3</span> and higher,
+ currently, Impala can query these types only in Parquet tables.
+ <span class="ph">
+ The one exception to the preceding rule is <code class="ph codeph">COUNT(*)</code> queries on RCFile tables that include complex types.
+ Such queries are allowed in <span class="keyword">Impala 2.6</span> and higher.
+ </span>
+ </p>
+
+ </div>
+ </article>
+
+ <article class="topic concept nested1" aria-labelledby="ariaid-title3" id="rcfile__rcfile_compression">
+
+ <h2 class="title topictitle2" id="ariaid-title3">Enabling Compression for RCFile Tables</h2>
+
+
+ <div class="body conbody">
+
+ <p class="p">
+
+ You may want to enable compression on existing tables. Enabling compression provides performance gains in
+ most cases and is supported for RCFile tables. For example, to enable Snappy compression, you would specify
+ the following additional settings when loading data through the Hive shell:
+ </p>
+
+<pre class="pre codeblock"><code>hive> SET hive.exec.compress.output=true;
+hive> SET mapred.max.split.size=256000000;
+hive> SET mapred.output.compression.type=BLOCK;
+hive> SET mapred.output.compression.codec=org.apache.hadoop.io.compress.SnappyCodec;
+hive> INSERT OVERWRITE TABLE <var class="keyword varname">new_table</var> SELECT * FROM <var class="keyword varname">old_table</var>;</code></pre>
+
+ <p class="p">
+ If you are converting partitioned tables, you must complete additional steps. In such a case, specify
+ additional settings similar to the following:
+ </p>
+
+<pre class="pre codeblock"><code>hive> CREATE TABLE <var class="keyword varname">new_table</var> (<var class="keyword varname">your_cols</var>) PARTITIONED BY (<var class="keyword varname">partition_cols</var>) STORED AS <var class="keyword varname">new_format</var>;
+hive> SET hive.exec.dynamic.partition.mode=nonstrict;
+hive> SET hive.exec.dynamic.partition=true;
+hive> INSERT OVERWRITE TABLE <var class="keyword varname">new_table</var> PARTITION(<var class="keyword varname">comma_separated_partition_cols</var>) SELECT * FROM <var class="keyword varname">old_table</var>;</code></pre>
+
+ <p class="p">
+ Remember that Hive does not require that you specify a source format for it. Consider the case of
+ converting a table with two partition columns called <code class="ph codeph">year</code> and <code class="ph codeph">month</code> to a
+ Snappy compressed RCFile. Combining the components outlined previously to complete this table conversion,
+ you would specify settings similar to the following:
+ </p>
+
+<pre class="pre codeblock"><code>hive> CREATE TABLE tbl_rc (int_col INT, string_col STRING) STORED AS RCFILE;
+hive> SET hive.exec.compress.output=true;
+hive> SET mapred.max.split.size=256000000;
+hive> SET mapred.output.compression.type=BLOCK;
+hive> SET mapred.output.compression.codec=org.apache.hadoop.io.compress.SnappyCodec;
+hive> SET hive.exec.dynamic.partition.mode=nonstrict;
+hive> SET hive.exec.dynamic.partition=true;
+hive> INSERT OVERWRITE TABLE tbl_rc SELECT * FROM tbl;</code></pre>
+
+ <p class="p">
+ To complete a similar process for a table that includes partitions, you would specify settings similar to
+ the following:
+ </p>
+
+<pre class="pre codeblock"><code>hive> CREATE TABLE tbl_rc (int_col INT, string_col STRING) PARTITIONED BY (year INT) STORED AS RCFILE;
+hive> SET hive.exec.compress.output=true;
+hive> SET mapred.max.split.size=256000000;
+hive> SET mapred.output.compression.type=BLOCK;
+hive> SET mapred.output.compression.codec=org.apache.hadoop.io.compress.SnappyCodec;
+hive> SET hive.exec.dynamic.partition.mode=nonstrict;
+hive> SET hive.exec.dynamic.partition=true;
+hive> INSERT OVERWRITE TABLE tbl_rc PARTITION(year) SELECT * FROM tbl;</code></pre>
+
+ <div class="note note note_note"><span class="note__title notetitle">Note:</span>
+ <p class="p">
+ The compression type is specified in the following command:
+ </p>
+<pre class="pre codeblock"><code>SET mapred.output.compression.codec=org.apache.hadoop.io.compress.SnappyCodec;</code></pre>
+ <p class="p">
+ You could elect to specify alternative codecs such as <code class="ph codeph">GzipCodec</code> here.
+ </p>
+ </div>
+ </div>
+ </article>
+
+ <article class="topic concept nested1" aria-labelledby="ariaid-title4" id="rcfile__rcfile_performance">
+
+ <h2 class="title topictitle2" id="ariaid-title4">Query Performance for Impala RCFile Tables</h2>
+
+ <div class="body conbody">
+
+ <p class="p">
+ In general, expect query performance with RCFile tables to be
+ faster than with tables using text data, but slower than with
+ Parquet tables. See <a class="xref" href="impala_parquet.html#parquet">Using the Parquet File Format with Impala Tables</a>
+ for information about using the Parquet file format for
+ high-performance analytic queries.
+ </p>
+
+ <p class="p">
+ In <span class="keyword">Impala 2.6</span> and higher, Impala queries are optimized for files stored in Amazon S3.
+ For Impala tables that use the file formats Parquet, RCFile, SequenceFile,
+ Avro, and uncompressed text, the setting <code class="ph codeph">fs.s3a.block.size</code>
+ in the <span class="ph filepath">core-site.xml</span> configuration file determines
+ how Impala divides the I/O work of reading the data files. This configuration
+ setting is specified in bytes. By default, this
+ value is 33554432 (32 MB), meaning that Impala parallelizes S3 read operations on the files
+ as if they were made up of 32 MB blocks. For example, if your S3 queries primarily access
+ Parquet files written by MapReduce or Hive, increase <code class="ph codeph">fs.s3a.block.size</code>
+ to 134217728 (128 MB) to match the row group size of those files. If most S3 queries involve
+ Parquet files written by Impala, increase <code class="ph codeph">fs.s3a.block.size</code>
+ to 268435456 (256 MB) to match the row group size produced by Impala.
+ </p>
+
+ </div>
+ </article>
+
+
+</article></main></body></html>
http://git-wip-us.apache.org/repos/asf/impala/blob/fae51ec2/docs/build3x/html/topics/impala_real.html
----------------------------------------------------------------------
diff --git a/docs/build3x/html/topics/impala_real.html b/docs/build3x/html/topics/impala_real.html
new file mode 100644
index 0000000..5e772c2
--- /dev/null
+++ b/docs/build3x/html/topics/impala_real.html
@@ -0,0 +1,39 @@
+<!DOCTYPE html
+ SYSTEM "about:legacy-compat">
+<html lang="en"><head><meta http-equiv="Content-Type" content="text/html; charset=UTF-8"><meta charset="UTF-8"><meta name="copyright" content="(C) Copyright 2018"><meta name="DC.rights.owner" content="(C) Copyright 2018"><meta name="DC.Type" content="concept"><meta name="DC.Relation" scheme="URI" content="../topics/impala_datatypes.html"><meta name="prodname" content="Impala"><meta name="prodname" content="Impala"><meta name="version" content="Impala 3.0.x"><meta name="version" content="Impala 3.0.x"><meta name="DC.Format" content="XHTML"><meta name="DC.Identifier" content="real"><link rel="stylesheet" type="text/css" href="../commonltr.css"><title>REAL Data Type</title></head><body id="real"><main role="main"><article role="article" aria-labelledby="ariaid-title1">
+
+ <h1 class="title topictitle1" id="ariaid-title1">REAL Data Type</h1>
+
+
+
+ <div class="body conbody">
+
+ <p class="p">
+ An alias for the <code class="ph codeph">DOUBLE</code> data type. See <a class="xref" href="impala_double.html#double">DOUBLE Data Type</a> for details.
+ </p>
+
+ <p class="p">
+ <strong class="ph b">Examples:</strong>
+ </p>
+
+ <p class="p">
+ These examples show how you can use the type names <code class="ph codeph">REAL</code> and <code class="ph codeph">DOUBLE</code>
+ interchangeably, and behind the scenes Impala treats them always as <code class="ph codeph">DOUBLE</code>.
+ </p>
+
+<pre class="pre codeblock"><code>[localhost:21000] > create table r1 (x real);
+[localhost:21000] > describe r1;
++------+--------+---------+
+| name | type | comment |
++------+--------+---------+
+| x | double | |
++------+--------+---------+
+[localhost:21000] > insert into r1 values (1.5), (cast (2.2 as double));
+[localhost:21000] > select cast (1e6 as real);
++---------------------------+
+| cast(1000000.0 as double) |
++---------------------------+
+| 1000000 |
++---------------------------+</code></pre>
+ </div>
+<nav role="navigation" class="related-links"><div class="familylinks"><div class="parentlink"><strong>Parent topic:</strong> <a class="link" href="../topics/impala_datatypes.html">Data Types</a></div></div></nav></article></main></body></html>
http://git-wip-us.apache.org/repos/asf/impala/blob/fae51ec2/docs/build3x/html/topics/impala_refresh.html
----------------------------------------------------------------------
diff --git a/docs/build3x/html/topics/impala_refresh.html b/docs/build3x/html/topics/impala_refresh.html
new file mode 100644
index 0000000..5359668
--- /dev/null
+++ b/docs/build3x/html/topics/impala_refresh.html
@@ -0,0 +1,408 @@
+<!DOCTYPE html
+ SYSTEM "about:legacy-compat">
+<html lang="en"><head><meta http-equiv="Content-Type" content="text/html; charset=UTF-8"><meta charset="UTF-8"><meta name="copyright" content="(C) Copyright 2018"><meta name="DC.rights.owner" content="(C) Copyright 2018"><meta name="DC.Type" content="concept"><meta name="DC.Relation" scheme="URI" content="../topics/impala_langref_sql.html"><meta name="prodname" content="Impala"><meta name="prodname" content="Impala"><meta name="version" content="Impala 3.0.x"><meta name="version" content="Impala 3.0.x"><meta name="DC.Format" content="XHTML"><meta name="DC.Identifier" content="refresh"><link rel="stylesheet" type="text/css" href="../commonltr.css"><title>REFRESH Statement</title></head><body id="refresh"><main role="main"><article role="article" aria-labelledby="ariaid-title1">
+
+ <h1 class="title topictitle1" id="ariaid-title1">REFRESH Statement</h1>
+
+
+
+ <div class="body conbody">
+
+ <p class="p">
+
+ To accurately respond to queries, the Impala node that acts as the coordinator (the node to which you are
+ connected through <span class="keyword cmdname">impala-shell</span>, JDBC, or ODBC) must have current metadata about those
+ databases and tables that are referenced in Impala queries. If you are not familiar with the way Impala uses
+ metadata and how it shares the same metastore database as Hive, see
+ <a class="xref" href="impala_hadoop.html#intro_metastore">Overview of Impala Metadata and the Metastore</a> for background information.
+ </p>
+
+ <p class="p">
+ <strong class="ph b">Syntax:</strong>
+ </p>
+
+<pre class="pre codeblock"><code>REFRESH [<var class="keyword varname">db_name</var>.]<var class="keyword varname">table_name</var> [PARTITION (<var class="keyword varname">key_col1</var>=<var class="keyword varname">val1</var> [, <var class="keyword varname">key_col2</var>=<var class="keyword varname">val2</var>...])]
+<span class="ph">REFRESH FUNCTIONS <var class="keyword varname">db_name</var></span>
+</code></pre>
+
+ <p class="p">
+ <strong class="ph b">Usage notes:</strong>
+ </p>
+
+ <p class="p">
+ Use the <code class="ph codeph">REFRESH</code> statement to load the latest metastore metadata and block location data for
+ a particular table in these scenarios:
+ </p>
+
+ <ul class="ul">
+ <li class="li">
+ After loading new data files into the HDFS data directory for the table. (Once you have set up an ETL
+ pipeline to bring data into Impala on a regular basis, this is typically the most frequent reason why
+ metadata needs to be refreshed.)
+ </li>
+
+ <li class="li">
+ After issuing <code class="ph codeph">ALTER TABLE</code>, <code class="ph codeph">INSERT</code>, <code class="ph codeph">LOAD DATA</code>, or other
+ table-modifying SQL statement in Hive.
+ </li>
+ </ul>
+
+ <div class="note note note_note"><span class="note__title notetitle">Note:</span>
+ <p class="p">
+ In <span class="keyword">Impala 2.3</span> and higher, the syntax <code class="ph codeph">ALTER TABLE <var class="keyword varname">table_name</var> RECOVER PARTITIONS</code>
+ is a faster alternative to <code class="ph codeph">REFRESH</code> when the only change to the table data is the addition of
+ new partition directories through Hive or manual HDFS operations.
+ See <a class="xref" href="impala_alter_table.html#alter_table">ALTER TABLE Statement</a> for details.
+ </p>
+ </div>
+
+ <p class="p">
+ You only need to issue the <code class="ph codeph">REFRESH</code> statement on the node to which you connect to issue
+ queries. The coordinator node divides the work among all the Impala nodes in a cluster, and sends read
+ requests for the correct HDFS blocks without relying on the metadata on the other nodes.
+ </p>
+
+ <p class="p">
+ <code class="ph codeph">REFRESH</code> reloads the metadata for the table from the metastore database, and does an
+ incremental reload of the low-level block location data to account for any new data files added to the HDFS
+ data directory for the table. It is a low-overhead, single-table operation, specifically tuned for the common
+ scenario where new data files are added to HDFS.
+ </p>
+
+ <p class="p">
+ Only the metadata for the specified table is flushed. The table must already exist and be known to Impala,
+ either because the <code class="ph codeph">CREATE TABLE</code> statement was run in Impala rather than Hive, or because a
+ previous <code class="ph codeph">INVALIDATE METADATA</code> statement caused Impala to reload its entire metadata catalog.
+ </p>
+
+ <div class="note note note_note"><span class="note__title notetitle">Note:</span>
+ <p class="p">
+ The catalog service broadcasts any changed metadata as a result of Impala
+ <code class="ph codeph">ALTER TABLE</code>, <code class="ph codeph">INSERT</code> and <code class="ph codeph">LOAD DATA</code> statements to all
+ Impala nodes. Thus, the <code class="ph codeph">REFRESH</code> statement is only required if you load data through Hive
+ or by manipulating data files in HDFS directly. See <a class="xref" href="impala_components.html#intro_catalogd">The Impala Catalog Service</a> for
+ more information on the catalog service.
+ </p>
+ <p class="p">
+ Another way to avoid inconsistency across nodes is to enable the
+ <code class="ph codeph">SYNC_DDL</code> query option before performing a DDL statement or an <code class="ph codeph">INSERT</code> or
+ <code class="ph codeph">LOAD DATA</code>.
+ </p>
+ <p class="p">
+ The table name is a required parameter. To flush the metadata for all tables, use the
+ <code class="ph codeph"><a class="xref" href="impala_invalidate_metadata.html#invalidate_metadata">INVALIDATE METADATA</a></code>
+ command.
+ </p>
+ <p class="p">
+ Because <code class="ph codeph">REFRESH <var class="keyword varname">table_name</var></code> only works for tables that the current
+ Impala node is already aware of, when you create a new table in the Hive shell, enter
+ <code class="ph codeph">INVALIDATE METADATA <var class="keyword varname">new_table</var></code> before you can see the new table in
+ <span class="keyword cmdname">impala-shell</span>. Once the table is known by Impala, you can issue <code class="ph codeph">REFRESH
+ <var class="keyword varname">table_name</var></code> after you add data files for that table.
+ </p>
+ </div>
+
+ <p class="p">
+ <code class="ph codeph">INVALIDATE METADATA</code> and <code class="ph codeph">REFRESH</code> are counterparts: <code class="ph codeph">INVALIDATE
+ METADATA</code> waits to reload the metadata when needed for a subsequent query, but reloads all the
+ metadata for the table, which can be an expensive operation, especially for large tables with many
+ partitions. <code class="ph codeph">REFRESH</code> reloads the metadata immediately, but only loads the block location
+ data for newly added data files, making it a less expensive operation overall. If data was altered in some
+ more extensive way, such as being reorganized by the HDFS balancer, use <code class="ph codeph">INVALIDATE
+ METADATA</code> to avoid a performance penalty from reduced local reads. If you used Impala version 1.0,
+ the <code class="ph codeph">INVALIDATE METADATA</code> statement works just like the Impala 1.0 <code class="ph codeph">REFRESH</code>
+ statement did, while the Impala 1.1 <code class="ph codeph">REFRESH</code> is optimized for the common use case of adding
+ new data files to an existing table, thus the table name argument is now required.
+ </p>
+
+ <p class="p">
+ A metadata update for an <code class="ph codeph">impalad</code> instance <strong class="ph b">is</strong> required if:
+ </p>
+
+ <ul class="ul">
+ <li class="li">
+ A metadata change occurs.
+ </li>
+
+ <li class="li">
+ <strong class="ph b">and</strong> the change is made through Hive.
+ </li>
+
+ <li class="li">
+ <strong class="ph b">and</strong> the change is made to a metastore database to which clients such as the Impala shell or ODBC directly
+ connect.
+ </li>
+ </ul>
+
+ <p class="p">
+ A metadata update for an Impala node is <strong class="ph b">not</strong> required after you run <code class="ph codeph">ALTER TABLE</code>,
+ <code class="ph codeph">INSERT</code>, or other table-modifying statement in Impala rather than Hive. Impala handles the
+ metadata synchronization automatically through the catalog service.
+ </p>
+
+ <p class="p">
+ Database and table metadata is typically modified by:
+ </p>
+
+ <ul class="ul">
+ <li class="li">
+ Hive - through <code class="ph codeph">ALTER</code>, <code class="ph codeph">CREATE</code>, <code class="ph codeph">DROP</code> or
+ <code class="ph codeph">INSERT</code> operations.
+ </li>
+
+ <li class="li">
+ Impalad - through <code class="ph codeph">CREATE TABLE</code>, <code class="ph codeph">ALTER TABLE</code>, and <code class="ph codeph">INSERT</code>
+ operations. <span class="ph">Such changes are propagated to all Impala nodes by the
+ Impala catalog service.</span>
+ </li>
+ </ul>
+
+ <p class="p">
+ <code class="ph codeph">REFRESH</code> causes the metadata for that table to be immediately reloaded. For a huge table,
+ that process could take a noticeable amount of time; but doing the refresh up front avoids an unpredictable
+ delay later, for example if the next reference to the table is during a benchmark test.
+ </p>
+
+ <p class="p">
+ <strong class="ph b">Refreshing a single partition:</strong>
+ </p>
+
+ <p class="p">
+ In <span class="keyword">Impala 2.7</span> and higher, the <code class="ph codeph">REFRESH</code> statement can apply to a single partition at a time,
+ rather than the whole table. Include the optional <code class="ph codeph">PARTITION (<var class="keyword varname">partition_spec</var>)</code>
+ clause and specify values for each of the partition key columns.
+ </p>
+
+ <p class="p">
+ The following examples show how to make Impala aware of data added to a single partition, after data is loaded into
+ a partition's data directory using some mechanism outside Impala, such as Hive or Spark. The partition can be one that
+ Impala created and is already aware of, or a new partition created through Hive.
+ </p>
+
+<pre class="pre codeblock"><code>
+impala> create table p (x int) partitioned by (y int);
+impala> insert into p (x,y) values (1,2), (2,2), (2,1);
+impala> show partitions p;
++-------+-------+--------+------+...
+| y | #Rows | #Files | Size |...
++-------+-------+--------+------+...
+| 1 | -1 | 1 | 2B |...
+| 2 | -1 | 1 | 4B |...
+| Total | -1 | 2 | 6B |...
++-------+-------+--------+------+...
+
+-- ... Data is inserted into one of the partitions by some external mechanism ...
+beeline> insert into p partition (y = 1) values(1000);
+
+impala> refresh p partition (y=1);
+impala> select x from p where y=1;
++------+
+| x |
++------+
+| 2 | <- Original data created by Impala
+| 1000 | <- Additional data inserted through Beeline
++------+
+
+</code></pre>
+
+ <p class="p">
+ The same applies for tables with more than one partition key column.
+ The <code class="ph codeph">PARTITION</code> clause of the <code class="ph codeph">REFRESH</code>
+ statement must include all the partition key columns.
+ </p>
+
+<pre class="pre codeblock"><code>
+impala> create table p2 (x int) partitioned by (y int, z int);
+impala> insert into p2 (x,y,z) values (0,0,0), (1,2,3), (2,2,3);
+impala> show partitions p2;
++-------+---+-------+--------+------+...
+| y | z | #Rows | #Files | Size |...
++-------+---+-------+--------+------+...
+| 0 | 0 | -1 | 1 | 2B |...
+| 2 | 3 | -1 | 1 | 4B |...
+| Total | | -1 | 2 | 6B |...
++-------+---+-------+--------+------+...
+
+-- ... Data is inserted into one of the partitions by some external mechanism ...
+beeline> insert into p2 partition (y = 2, z = 3) values(1000);
+
+impala> refresh p2 partition (y=2, z=3);
+impala> select x from p where y=2 and z = 3;
++------+
+| x |
++------+
+| 1 | <- Original data created by Impala
+| 2 | <- Original data created by Impala
+| 1000 | <- Additional data inserted through Beeline
++------+
+
+</code></pre>
+
+ <p class="p">
+ The following examples show how specifying a nonexistent partition does not cause any error,
+ and the order of the partition key columns does not have to match the column order in the table.
+ The partition spec must include all the partition key columns; specifying an incomplete set of
+ columns does cause an error.
+ </p>
+
+<pre class="pre codeblock"><code>
+-- Partition doesn't exist.
+refresh p2 partition (y=0, z=3);
+refresh p2 partition (y=0, z=-1)
+-- Key columns specified in a different order than the table definition.
+refresh p2 partition (z=1, y=0)
+-- Incomplete partition spec causes an error.
+refresh p2 partition (y=0)
+ERROR: AnalysisException: Items in partition spec must exactly match the partition columns in the table definition: default.p2 (1 vs 2)
+
+</code></pre>
+
+ <p class="p">
+ If you connect to different Impala nodes within an <span class="keyword cmdname">impala-shell</span> session for
+ load-balancing purposes, you can enable the <code class="ph codeph">SYNC_DDL</code> query option to make each DDL
+ statement wait before returning, until the new or changed metadata has been received by all the Impala
+ nodes. See <a class="xref" href="../shared/../topics/impala_sync_ddl.html#sync_ddl">SYNC_DDL Query Option</a> for details.
+ </p>
+
+ <p class="p">
+ <strong class="ph b">Examples:</strong>
+ </p>
+
+ <p class="p">
+ The following example shows how you might use the <code class="ph codeph">REFRESH</code> statement after manually adding
+ new HDFS data files to the Impala data directory for a table:
+ </p>
+
+<pre class="pre codeblock"><code>[impalad-host:21000] > refresh t1;
+[impalad-host:21000] > refresh t2;
+[impalad-host:21000] > select * from t1;
+...
+[impalad-host:21000] > select * from t2;
+... </code></pre>
+
+ <p class="p">
+ For more examples of using <code class="ph codeph">REFRESH</code> and <code class="ph codeph">INVALIDATE METADATA</code> with a
+ combination of Impala and Hive operations, see <a class="xref" href="impala_tutorial.html#tutorial_impala_hive">Switching Back and Forth Between Impala and Hive</a>.
+ </p>
+
+ <p class="p">
+ <strong class="ph b">Related impala-shell options:</strong>
+ </p>
+
+ <p class="p">
+ The <span class="keyword cmdname">impala-shell</span> option <code class="ph codeph">-r</code> issues an <code class="ph codeph">INVALIDATE METADATA</code> statement
+ when starting up the shell, effectively performing a <code class="ph codeph">REFRESH</code> of all tables.
+ Due to the expense of reloading the metadata for all tables, the <span class="keyword cmdname">impala-shell</span> <code class="ph codeph">-r</code>
+ option is not recommended for day-to-day use in a production environment. (This option was mainly intended as a workaround
+ for synchronization issues in very old Impala versions.)
+ </p>
+
+ <p class="p">
+ <strong class="ph b">HDFS permissions:</strong>
+ </p>
+ <p class="p">
+ The user ID that the <span class="keyword cmdname">impalad</span> daemon runs under,
+ typically the <code class="ph codeph">impala</code> user, must have execute
+ permissions for all the relevant directories holding table data.
+ (A table could have data spread across multiple directories,
+ or in unexpected paths, if it uses partitioning or
+ specifies a <code class="ph codeph">LOCATION</code> attribute for
+ individual partitions or the entire table.)
+ Issues with permissions might not cause an immediate error for this statement,
+ but subsequent statements such as <code class="ph codeph">SELECT</code>
+ or <code class="ph codeph">SHOW TABLE STATS</code> could fail.
+ </p>
+ <p class="p">
+ All HDFS and Sentry permissions and privileges are the same whether you refresh the entire table
+ or a single partition.
+ </p>
+
+ <p class="p">
+ <strong class="ph b">HDFS considerations:</strong>
+ </p>
+
+ <p class="p">
+ The <code class="ph codeph">REFRESH</code> command checks HDFS permissions of the underlying data files and directories,
+ caching this information so that a statement can be cancelled immediately if for example the
+ <code class="ph codeph">impala</code> user does not have permission to write to the data directory for the table. Impala
+ reports any lack of write permissions as an <code class="ph codeph">INFO</code> message in the log file, in case that
+ represents an oversight. If you change HDFS permissions to make data readable or writeable by the Impala
+ user, issue another <code class="ph codeph">REFRESH</code> to make Impala aware of the change.
+ </p>
+
+ <div class="note important note_important"><span class="note__title importanttitle">Important:</span>
+ After adding or replacing data in a table used in performance-critical queries, issue a <code class="ph codeph">COMPUTE
+ STATS</code> statement to make sure all statistics are up-to-date. Consider updating statistics for a
+ table after any <code class="ph codeph">INSERT</code>, <code class="ph codeph">LOAD DATA</code>, or <code class="ph codeph">CREATE TABLE AS
+ SELECT</code> statement in Impala, or after loading data through Hive and doing a <code class="ph codeph">REFRESH
+ <var class="keyword varname">table_name</var></code> in Impala. This technique is especially important for tables that
+ are very large, used in join queries, or both.
+ </div>
+
+ <p class="p">
+ <strong class="ph b">Amazon S3 considerations:</strong>
+ </p>
+ <p class="p">
+ The <code class="ph codeph">REFRESH</code> and <code class="ph codeph">INVALIDATE METADATA</code> statements also cache metadata
+ for tables where the data resides in the Amazon Simple Storage Service (S3).
+ In particular, issue a <code class="ph codeph">REFRESH</code> for a table after adding or removing files
+ in the associated S3 data directory.
+ See <a class="xref" href="../shared/../topics/impala_s3.html#s3">Using Impala with the Amazon S3 Filesystem</a> for details about working with S3 tables.
+ </p>
+
+ <p class="p">
+ <strong class="ph b">Cancellation:</strong> Cannot be cancelled.
+ </p>
+
+ <p class="p">
+ <strong class="ph b">Kudu considerations:</strong>
+ </p>
+ <p class="p">
+ Much of the metadata for Kudu tables is handled by the underlying
+ storage layer. Kudu tables have less reliance on the metastore
+ database, and require less metadata caching on the Impala side.
+ For example, information about partitions in Kudu tables is managed
+ by Kudu, and Impala does not cache any block locality metadata
+ for Kudu tables.
+ </p>
+ <p class="p">
+ The <code class="ph codeph">REFRESH</code> and <code class="ph codeph">INVALIDATE METADATA</code>
+ statements are needed less frequently for Kudu tables than for
+ HDFS-backed tables. Neither statement is needed when data is
+ added to, removed, or updated in a Kudu table, even if the changes
+ are made directly to Kudu through a client program using the Kudu API.
+ Run <code class="ph codeph">REFRESH <var class="keyword varname">table_name</var></code> or
+ <code class="ph codeph">INVALIDATE METADATA <var class="keyword varname">table_name</var></code>
+ for a Kudu table only after making a change to the Kudu table schema,
+ such as adding or dropping a column, by a mechanism other than
+ Impala.
+ </p>
+
+ <p class="p">
+ <strong class="ph b">UDF considerations:</strong>
+ </p>
+ <div class="p">
+ In <span class="keyword">Impala 2.9</span> and higher, you can refresh the user-defined functions (UDFs)
+ that Impala recognizes, at the database level, by running the <code class="ph codeph">REFRESH FUNCTIONS</code>
+ statement with the database name as an argument. Java-based UDFs can be added to the metastore
+ database through Hive <code class="ph codeph">CREATE FUNCTION</code> statements, and made visible to Impala
+ by subsequently running <code class="ph codeph">REFRESH FUNCTIONS</code>. For example:
+
+<pre class="pre codeblock"><code>CREATE DATABASE shared_udfs;
+USE shared_udfs;
+...use CREATE FUNCTION statements in Hive to create some Java-based UDFs
+ that Impala is not initially aware of...
+REFRESH FUNCTIONS shared_udfs;
+SELECT udf_created_by_hive(c1) FROM ...
+</code></pre>
+ </div>
+
+ <p class="p">
+ <strong class="ph b">Related information:</strong>
+ </p>
+ <p class="p">
+ <a class="xref" href="impala_hadoop.html#intro_metastore">Overview of Impala Metadata and the Metastore</a>,
+ <a class="xref" href="impala_invalidate_metadata.html#invalidate_metadata">INVALIDATE METADATA Statement</a>
+ </p>
+ </div>
+<nav role="navigation" class="related-links"><div class="familylinks"><div class="parentlink"><strong>Parent topic:</strong> <a class="link" href="../topics/impala_langref_sql.html">Impala SQL Statements</a></div></div></nav></article></main></body></html>
http://git-wip-us.apache.org/repos/asf/impala/blob/fae51ec2/docs/build3x/html/topics/impala_release_notes.html
----------------------------------------------------------------------
diff --git a/docs/build3x/html/topics/impala_release_notes.html b/docs/build3x/html/topics/impala_release_notes.html
new file mode 100644
index 0000000..86359c9
--- /dev/null
+++ b/docs/build3x/html/topics/impala_release_notes.html
@@ -0,0 +1,26 @@
+<!DOCTYPE html
+ SYSTEM "about:legacy-compat">
+<html lang="en"><head><meta http-equiv="Content-Type" content="text/html; charset=UTF-8"><meta charset="UTF-8"><meta name="copyright" content="(C) Copyright 2018"><meta name="DC.rights.owner" content="(C) Copyright 2018"><meta name="DC.Type" content="concept"><meta name="DC.Relation" scheme="URI" content="../topics/impala_new_features.html"><meta name="DC.Relation" scheme="URI" content="../topics/impala_incompatible_changes.html"><meta name="DC.Relation" scheme="URI" content="../topics/impala_known_issues.html"><meta name="DC.Relation" scheme="URI" content="../topics/impala_fixed_issues.html"><meta name="prodname" content="Impala"><meta name="version" content="Impala 3.0.x"><meta name="DC.Format" content="XHTML"><meta name="DC.Identifier" content="impala_release_notes"><link rel="stylesheet" type="text/css" href="../commonltr.css"><title>Impala Release Notes</title></head><body id="impala_release_notes"><main role="main"><article role="article" aria-labelledby="ariaid-title1">
+
+ <h1 class="title topictitle1" id="ariaid-title1">Impala Release Notes</h1>
+
+
+ <div class="body conbody">
+
+ <p class="p">
+ These release notes provide information on the <a class="xref" href="impala_new_features.html#new_features">new
+ features</a> and <a class="xref" href="impala_known_issues.html#known_issues">known issues and limitations</a> for
+ Impala versions up to <span class="ph">Impala 3.0.x</span>. For users
+ upgrading from earlier Impala releases, or using Impala in combination with specific versions of other
+ software, <a class="xref" href="impala_incompatible_changes.html#incompatible_changes">Incompatible Changes and Limitations in Apache Impala</a> lists any changes to
+ file formats, SQL syntax, or software dependencies to take into account.
+ </p>
+
+ <p class="p">
+ Once you are finished reviewing these release notes, for more information about using Impala, see
+ <a class="xref" href="impala_concepts.html">Impala Concepts and Architecture</a>.
+ </p>
+
+ <p class="p toc"></p>
+ </div>
+<nav role="navigation" class="related-links"><ul class="ullinks"><li class="link ulchildlink"><strong><a href="../topics/impala_new_features.html">New Features in Apache Impala</a></strong><br></li><li class="link ulchildlink"><strong><a href="../topics/impala_incompatible_changes.html">Incompatible Changes and Limitations in Apache Impala</a></strong><br></li><li class="link ulchildlink"><strong><a href="../topics/impala_known_issues.html">Known Issues and Workarounds in Impala</a></strong><br></li><li class="link ulchildlink"><strong><a href="../topics/impala_fixed_issues.html">Fixed Issues in Apache Impala</a></strong><br></li></ul></nav></article></main></body></html>
http://git-wip-us.apache.org/repos/asf/impala/blob/fae51ec2/docs/build3x/html/topics/impala_relnotes.html
----------------------------------------------------------------------
diff --git a/docs/build3x/html/topics/impala_relnotes.html b/docs/build3x/html/topics/impala_relnotes.html
new file mode 100644
index 0000000..f9b8d62
--- /dev/null
+++ b/docs/build3x/html/topics/impala_relnotes.html
@@ -0,0 +1,26 @@
+<!DOCTYPE html
+ SYSTEM "about:legacy-compat">
+<html lang="en"><head><meta http-equiv="Content-Type" content="text/html; charset=UTF-8"><meta charset="UTF-8"><meta name="copyright" content="(C) Copyright 2018"><meta name="DC.rights.owner" content="(C) Copyright 2018"><meta name="DC.Type" content="concept"><meta name="DC.Format" content="XHTML"><meta name="DC.Identifier" content="relnotes"><link rel="stylesheet" type="text/css" href="../commonltr.css"><title>Impala Release Notes</title></head><body id="relnotes"><main role="main"><article role="article" aria-labelledby="ariaid-title1">
+
+ <h1 class="title topictitle1" id="ariaid-title1">Impala Release Notes</h1>
+
+
+ <div class="body conbody" id="relnotes__relnotes_intro">
+
+ <p class="p">
+ These release notes provide information on the <a class="xref" href="impala_new_features.html#new_features">new
+ features</a> and <a class="xref" href="impala_known_issues.html#known_issues">known issues and limitations</a> for
+ Impala versions up to <span class="ph">Impala 3.0.x</span>. For users
+ upgrading from earlier Impala releases, or using Impala in combination with specific versions of other
+ software, <a class="xref" href="impala_incompatible_changes.html#incompatible_changes">Incompatible Changes and Limitations in Apache Impala</a> lists any changes to
+ file formats, SQL syntax, or software dependencies to take into account.
+ </p>
+
+ <p class="p">
+ Once you are finished reviewing these release notes, for more information about using Impala, see
+ <a class="xref" href="impala_concepts.html">Impala Concepts and Architecture</a>.
+ </p>
+
+ <p class="p toc"></p>
+ </div>
+</article></main></body></html>