You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@impala.apache.org by jb...@apache.org on 2017/04/12 18:25:34 UTC
[30/51] [partial] incubator-impala git commit: IMPALA-4181 [DOCS]
Publish rendered Impala documentation to ASF site
http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/75c46918/docs/build/html/topics/impala_incompatible_changes.html
----------------------------------------------------------------------
diff --git a/docs/build/html/topics/impala_incompatible_changes.html b/docs/build/html/topics/impala_incompatible_changes.html
new file mode 100644
index 0000000..642a334
--- /dev/null
+++ b/docs/build/html/topics/impala_incompatible_changes.html
@@ -0,0 +1,1443 @@
+<!DOCTYPE html
+ SYSTEM "about:legacy-compat">
+<html lang="en"><head><meta http-equiv="Content-Type" content="text/html; charset=UTF-8"><meta charset="UTF-8"><meta name="copyright" content="(C) Copyright 2017"><meta name="DC.rights.owner" content="(C) Copyright 2017"><meta name="DC.Type" content="concept"><meta name="DC.Relation" scheme="URI" content="../topics/impala_release_notes.html"><meta name="prodname" content="Impala"><meta name="prodname" content="Impala"><meta name="version" content="Impala 2.8.x"><meta name="version" content="Impala 2.8.x"><meta name="DC.Format" content="XHTML"><meta name="DC.Identifier" content="incompatible_changes"><link rel="stylesheet" type="text/css" href="../commonltr.css"><title>Incompatible Changes and Limitations in Apache Impala (incubating)</title></head><body id="incompatible_changes"><main role="main"><article role="article" aria-labelledby="ariaid-title1">
+
+ <h1 class="title topictitle1" id="ariaid-title1"><span class="ph">Incompatible Changes and Limitations in Apache Impala (incubating)</span></h1>
+
+
+ <div class="body conbody">
+
+ <p class="p">
+ The Impala version covered by this documentation library contains the following incompatible changes. These
+ are things such as file format changes, removed features, or changes to implementation, default
+ configuration, dependencies, or prerequisites that could cause issues during or after an Impala upgrade.
+ </p>
+
+ <p class="p">
+ Even added SQL statements or clauses can produce incompatibilities, if you have databases, tables, or columns
+ whose names conflict with the new keywords. <span class="ph">See
+ <a class="xref" href="impala_reserved_words.html#reserved_words">Impala Reserved Words</a> for the set of reserved words for the current
+ release, and the quoting techniques to avoid name conflicts.</span>
+ </p>
+
+ <p class="p toc inpage"></p>
+ </div>
+
+ <nav role="navigation" class="related-links"><div class="familylinks"><div class="parentlink"><strong>Parent topic:</strong> <a class="link" href="../topics/impala_release_notes.html">Impala Release Notes</a></div></div></nav><article class="topic concept nested1" aria-labelledby="ariaid-title2" id="incompatible_changes__incompatible_changes_28x">
+
+ <h2 class="title topictitle2" id="ariaid-title2">Incompatible Changes Introduced in Impala 2.8.x</h2>
+
+ <div class="body conbody">
+ <ul class="ul">
+ <li class="li">
+ <p class="p">
+ Llama support is removed completely from Impala. Related flags (<code class="ph codeph">--enable_rm</code>)
+ and query options (such as <code class="ph codeph">V_CPU_CORES</code>) remain but do not have any effect.
+ </p>
+ <p class="p">
+ If <code class="ph codeph">--enable_rm</code> is passed to Impala, a warning is printed to the log on startup.
+ </p>
+ </li>
+ <li class="li">
+ <p class="p">
+ The syntax related to Kudu tables includes a number of new reserved words,
+ such as <code class="ph codeph">COMPRESSION</code>, <code class="ph codeph">DEFAULT</code>, and <code class="ph codeph">ENCODING</code>, that
+ might conflict with names of existing tables, columns, or other identifiers from older Impala versions.
+ See <a class="xref" href="impala_reserved_words.html#reserved_words">Impala Reserved Words</a> for the full list of reserved words.
+ </p>
+ </li>
+ <li class="li">
+ <p class="p">
+ The DDL syntax for Kudu tables, particularly in the <code class="ph codeph">CREATE TABLE</code> statement, is different
+ from the special <code class="ph codeph">impala_next</code> fork that was previously used for accessing Kudu tables
+ from Impala:
+ </p>
+ <ul class="ul">
+ <li class="li">
+ <p class="p">
+ The <code class="ph codeph">DISTRIBUTE BY</code> clause is now <code class="ph codeph">PARTITIONED BY</code>.
+ </p>
+ </li>
+ <li class="li">
+ <p class="p">
+ The <code class="ph codeph">INTO <var class="keyword varname">N</var> BUCKETS</code>
+ clause is now <code class="ph codeph">PARTITIONS <var class="keyword varname">N</var></code>.
+ </p>
+ </li>
+ <li class="li">
+ <p class="p">
+ The <code class="ph codeph">SPLIT ROWS</code> clause is replaced by different syntax for specifying
+ the ranges covered by each partition.
+ </p>
+ </li>
+ </ul>
+ </li>
+ <li class="li">
+ <p class="p">
+ The <code class="ph codeph">DESCRIBE</code> output for Kudu tables includes several extra columns.
+ </p>
+ </li>
+ <li class="li">
+ <p class="p">
+ Non-primary-key columns can contain <code class="ph codeph">NULL</code> values by default. The
+ <code class="ph codeph">SHOW CREATE TABLE</code> output for these columns displays the <code class="ph codeph">NULL</code>
+ attribute. There was a period during early experimental versions of Impala + Kudu where
+ non-primary-key columns had the <code class="ph codeph">NOT NULL</code> attribute by default.
+ </p>
+ </li>
+ <li class="li">
+ <p class="p">
+ The <code class="ph codeph">IGNORE</code> keyword that was present in early experimental versions of Impala + Kudu
+ is no longer present. The behavior of the <code class="ph codeph">IGNORE</code> keyword is now the default:
+ DML statements continue with warnings, instead of failing with errors, if they encounter conditions
+ such as <span class="q">"primary key already exists"</span> for an <code class="ph codeph">INSERT</code> statement or
+ <span class="q">"primary key already deleted"</span> for a <code class="ph codeph">DELETE</code> statement.
+ </p>
+ </li>
+ <li class="li">
+ <p class="p">
+ The replication factor for Kudu tables must be an odd number.
+ </p>
+ </li>
+ </ul>
+ </div>
+
+ </article>
+
+ <article class="topic concept nested1" aria-labelledby="ariaid-title3" id="incompatible_changes__incompatible_changes_27x">
+
+ <h2 class="title topictitle2" id="ariaid-title3">Incompatible Changes Introduced in Impala 2.7.x</h2>
+
+ <div class="body conbody">
+ <ul class="ul">
+ <li class="li">
+ <p class="p">
+ Bug fixes related to parsing of floating-point values (IMPALA-1731 and IMPALA-3868) can change
+ the results of casting strings that represent invalid floating-point values.
+ For example, formerly a string value beginning or ending with <code class="ph codeph">inf</code>,
+ such as <code class="ph codeph">1.23inf</code> or <code class="ph codeph">infinite</code>, now are converted to <code class="ph codeph">NULL</code>
+ when interpreted as a floating-point value.
+ Formerly, they were interpreted as the special <span class="q">"infinity"</span> value when converting from string to floating-point.
+ Similarly, now only the string <code class="ph codeph">NaN</code> (case-sensitive) is interpreted as the special <span class="q">"not a number"</span>
+ value. String values containing multiple dots, such as <code class="ph codeph">3..141</code> or <code class="ph codeph">3.1.4.1</code>,
+ are now interpreted as <code class="ph codeph">NULL</code> rather than being converted to valid floating-point values.
+ </p>
+ </li>
+ </ul>
+ </div>
+
+ </article>
+
+ <article class="topic concept nested1" aria-labelledby="ariaid-title4" id="incompatible_changes__incompatible_changes_26x">
+
+ <h2 class="title topictitle2" id="ariaid-title4">Incompatible Changes Introduced in Impala 2.6.x</h2>
+
+ <div class="body conbody">
+ <ul class="ul">
+ <li class="li">
+ <p class="p">
+ The default for the <code class="ph codeph">RUNTIME_FILTER_MODE</code>
+ query option is changed to <code class="ph codeph">GLOBAL</code> (the highest setting).
+ </p>
+ </li>
+ <li class="li">
+ <p class="p">
+ The <code class="ph codeph">RUNTIME_BLOOM_FILTER_SIZE</code> setting is now only used
+ as a fallback if statistics are not available; otherwise, Impala
+ uses the statistics to estimate the appropriate size to use for each filter.
+ </p>
+ </li>
+ <li class="li">
+ <p class="p">
+ Admission control and dynamic resource pools are enabled by default.
+ When upgrading from an earlier release, you must turn on these settings yourself
+ if they are not already enabled.
+ See <a class="xref" href="impala_admission.html#admission_control">Admission Control and Query Queuing</a> for details
+ about admission control.
+ </p>
+ </li>
+ <li class="li">
+ <p class="p">
+ Impala reserves some new keywords, in preparation for support for Kudu syntax:
+ <code class="ph codeph">buckets</code>, <code class="ph codeph">delete</code>, <code class="ph codeph">distribute</code>,
+ <code class="ph codeph">hash</code>, <code class="ph codeph">ignore</code>, <code class="ph codeph">split</code>, and <code class="ph codeph">update</code>.
+ </p>
+ </li>
+ <li class="li">
+ <p class="p">
+ For Kerberized clusters, the Catalog service now uses
+ the Kerberos principal instead of the operating sytem user that runs
+ the <span class="keyword cmdname">catalogd</span> daemon.
+ This eliminates the requirement to configure a <code class="ph codeph">hadoop.user.group.static.mapping.overrides</code>
+ setting to put the OS user into the Sentry administrative group, on clusters where the principal
+ and the OS user name for this user are different.
+ </p>
+ </li>
+ <li class="li">
+ <p class="p">
+ The mechanism for interpreting <code class="ph codeph">DECIMAL</code> literals is
+ improved, no longer going through an intermediate conversion step
+ to <code class="ph codeph">DOUBLE</code>:
+ </p>
+ <ul class="ul">
+ <li class="li">
+ <p class="p">
+ Casting a <code class="ph codeph">DECIMAL</code> value to <code class="ph codeph">TIMESTAMP</code>
+ <code class="ph codeph">DOUBLE</code> produces a more precise
+ value for the <code class="ph codeph">TIMESTAMP</code> than formerly.
+ </p>
+ </li>
+ <li class="li">
+ <p class="p">
+ Certain function calls involving <code class="ph codeph">DECIMAL</code> literals
+ now succeed, when formerly they failed due to lack of a function
+ signature with a <code class="ph codeph">DOUBLE</code> argument.
+ </p>
+ </li>
+ </ul>
+ </li>
+ <li class="li">
+ <p class="p">
+ Improved type accuracy for <code class="ph codeph">CASE</code> return values.
+ If all <code class="ph codeph">WHEN</code> clauses of the <code class="ph codeph">CASE</code>
+ expression are of <code class="ph codeph">CHAR</code> type, the final result
+ is also <code class="ph codeph">CHAR</code> instead of being converted to
+ <code class="ph codeph">STRING</code>.
+ </p>
+ </li>
+ <li class="li">
+ <div class="p">
+ The initial release of <span class="keyword">Impala 2.5</span> sometimes has a higher peak memory usage than in previous releases
+ while reading Parquet files.
+ The following query options might help to reduce memory consumption in the Parquet scanner:
+ <ul class="ul">
+ <li class="li">
+ Reduce the number of scanner threads, for example: <code class="ph codeph">set num_scanner_threads=30</code>
+ </li>
+ <li class="li">
+ Reduce the batch size, for example: <code class="ph codeph">set batch_size=512</code>
+ </li>
+ <li class="li">
+ Increase the memory limit, for example: <code class="ph codeph">set mem_limit=64g</code>
+ </li>
+ </ul>
+ You can track the status of the fix for this issue at
+ <a class="xref" href="https://issues.apache.org/jira/browse/IMPALA-3662" target="_blank">IMPALA-3662</a>.
+ </div>
+ </li>
+ <li class="li">
+ <p class="p">
+ The <code class="ph codeph">S3_SKIP_INSERT_STAGING</code> query option, which is enabled by
+ default, increases the speed of <code class="ph codeph">INSERT</code> operations for S3 tables.
+ The speedup applies to regular <code class="ph codeph">INSERT</code>, but not <code class="ph codeph">INSERT OVERWRITE</code>.
+ The tradeoff is the possibility of inconsistent output files left behind if a
+ node fails during <code class="ph codeph">INSERT</code> execution.
+ See <a class="xref" href="impala_s3_skip_insert_staging.html#s3_skip_insert_staging">S3_SKIP_INSERT_STAGING Query Option (Impala 2.6 or higher only)</a> for details.
+ </p>
+ </li>
+ </ul>
+ <p class="p">
+ Certain features are turned off by default, to avoid regressions or unexpected
+ behavior following an upgrade. Consider turning on these features after suitable testing:
+ </p>
+ <ul class="ul">
+ <li class="li">
+ <p class="p">
+ Impala now recognizes the <code class="ph codeph">auth_to_local</code> setting,
+ specified through the HDFS configuration setting
+ <code class="ph codeph">hadoop.security.auth_to_local</code>.
+ This feature is disabled by default; to enable it,
+ specify <code class="ph codeph">--load_auth_to_local_rules=true</code>
+ in the <span class="keyword cmdname">impalad</span> configuration settings.
+ </p>
+ </li>
+ <li class="li">
+ <p class="p">
+ A new query option, <code class="ph codeph">PARQUET_ANNOTATE_STRINGS_UTF8</code>,
+ makes Impala include the <code class="ph codeph">UTF-8</code> annotation
+ metadata for <code class="ph codeph">STRING</code>, <code class="ph codeph">CHAR</code>,
+ and <code class="ph codeph">VARCHAR</code> columns in Parquet files created
+ by <code class="ph codeph">INSERT</code> or <code class="ph codeph">CREATE TABLE AS SELECT</code>
+ statements.
+ </p>
+ </li>
+ <li class="li">
+ <p class="p">
+ A new query option,
+ <code class="ph codeph">PARQUET_FALLBACK_SCHEMA_RESOLUTION</code>,
+ lets Impala locate columns within Parquet files based on
+ column name rather than ordinal position.
+ This enhancement improves interoperability with applications
+ that write Parquet files with a different order or subset of
+ columns than are used in the Impala table.
+ </p>
+ </li>
+ </ul>
+ </div>
+
+ </article>
+
+ <article class="topic concept nested1" aria-labelledby="ariaid-title5" id="incompatible_changes__incompatible_changes_25x">
+
+ <h2 class="title topictitle2" id="ariaid-title5">Incompatible Changes Introduced in Impala 2.5.x</h2>
+
+ <div class="body conbody">
+ <ul class="ul">
+ <li class="li">
+ <p class="p">
+ The admission control default limit for concurrent queries (the <span class="ph uicontrol">max requests</span>
+ setting) is now unlimited instead of 200.
+ </p>
+ </li>
+
+ <li class="li">
+ <p class="p">
+ Multiplying a mixture of <code class="ph codeph">DECIMAL</code> and <code class="ph codeph">FLOAT</code> or
+ <code class="ph codeph">DOUBLE</code> values now returns
+ <code class="ph codeph">DOUBLE</code> rather than <code class="ph codeph">DECIMAL</code>. This
+ change avoids some cases where an intermediate value would underflow or overflow
+ and become <code class="ph codeph">NULL</code> unexpectedly. The results of
+ multiplying <code class="ph codeph">DECIMAL</code> and <code class="ph codeph">FLOAT</code> or
+ <code class="ph codeph">DOUBLE</code> might now be slightly less precise than
+ before. Previously, the intermediate types and thus the final result
+ depended on the exact order of the values of different types being
+ multiplied, which made the final result values difficult to
+ reason about.
+ </p>
+ </li>
+ <li class="li">
+ <p class="p">
+ Previously, the <code class="ph codeph">_</code> and <code class="ph codeph">%</code> wildcard
+ characters for the <code class="ph codeph">LIKE</code> operator would not match
+ characters on the second or subsequent lines of multi-line string values. The fix for issue
+ <a class="xref" href="https://issues.apache.org/jira/browse/IMPALA-2204" target="_blank">IMPALA-2204</a> causes
+ the wildcard matching to apply to the entire string for values
+ containing embedded <code class="ph codeph">\n</code> characters. This could cause
+ different results than in previous Impala releases for identical
+ queries on identical data.
+ </p>
+ </li>
+ <li class="li">
+ <p class="p">
+ Formerly, all Impala UDFs and UDAs required running the
+ <code class="ph codeph">CREATE FUNCTION</code> statements to
+ re-create them after each <span class="keyword cmdname">catalogd</span> restart.
+ In <span class="keyword">Impala 2.5</span> and higher, functions written in C++ are persisted across
+ restarts, and the requirement to
+ re-create functions only applies to functions written in Java. Adapt any
+ function-reloading logic that you have added to your Impala environment.
+ </p>
+ </li>
+ <li class="li">
+ <p class="p">
+ <code class="ph codeph">CREATE TABLE LIKE</code> no longer inherits HDFS caching settings from the source table.
+ </p>
+ </li>
+ <li class="li">
+ <p class="p">
+ The <code class="ph codeph">SHOW DATABASES</code> statement now returns two columns rather than one.
+ The second column includes the associated comment string, if any, for each database.
+ Adjust any application code that examines the list of databases and assumes the
+ result set contains only a single column.
+ </p>
+ </li>
+ <li class="li">
+ <p class="p">
+ The output of the <code class="ph codeph">SHOW FUNCTIONS</code> statement includes
+ two new columns, showing the kind of the function (for example,
+ <code class="ph codeph">BUILTIN</code>) and whether or not the function persists
+ across catalog server restarts. For example, the <code class="ph codeph">SHOW
+ FUNCTIONS</code> output for the
+ <code class="ph codeph">_impala_builtins</code> database starts with:
+ </p>
+<pre class="pre codeblock"><code>
++--------------+-------------------------------------------------+-------------+---------------+
+| return type | signature | binary type | is persistent |
++--------------+-------------------------------------------------+-------------+---------------+
+| BIGINT | abs(BIGINT) | BUILTIN | true |
+| DECIMAL(*,*) | abs(DECIMAL(*,*)) | BUILTIN | true |
+| DOUBLE | abs(DOUBLE) | BUILTIN | true |
+...
+</code></pre>
+ </li>
+ </ul>
+ </div>
+
+ </article>
+
+ <article class="topic concept nested1" aria-labelledby="ariaid-title6" id="incompatible_changes__incompatible_changes_24x">
+
+ <h2 class="title topictitle2" id="ariaid-title6">Incompatible Changes Introduced in Impala 2.4.x</h2>
+
+ <div class="body conbody">
+ <p class="p">
+ Other than support for DSSD storage, the Impala feature set for <span class="keyword">Impala 2.4</span> is the same as for <span class="keyword">Impala 2.3</span>.
+ Therefore, there are no incompatible changes for Impala introduced in <span class="keyword">Impala 2.4</span>.
+ </p>
+ </div>
+
+ </article>
+
+
+
+
+
+ <article class="topic concept nested1" aria-labelledby="ariaid-title7" id="incompatible_changes__incompatible_changes_23x">
+
+ <h2 class="title topictitle2" id="ariaid-title7">Incompatible Changes Introduced in Impala 2.3.x</h2>
+
+ <div class="body conbody">
+
+ <div class="note note note_note"><span class="note__title notetitle">Note:</span>
+ <p class="p">
+ The use of the Llama component for integrated resource management within YARN
+ is no longer supported with <span class="keyword">Impala 2.3</span> and higher.
+ The Llama support code is removed entirely in <span class="keyword">Impala 2.8</span> and higher.
+ </p>
+ <p class="p">
+ For clusters running Impala alongside
+ other data management components, you define static service pools to define the resources
+ available to Impala and other components. Then within the area allocated for Impala,
+ you can create dynamic service pools, each with its own settings for the Impala admission control feature.
+ </p>
+ </div>
+
+ <ul class="ul">
+
+ <li class="li">
+ <p class="p">
+ If Impala encounters a Parquet file that is invalid because of an incorrect magic number,
+ the query skips the file. This change is caused by the fix for issue <a class="xref" href="https://issues.apache.org/jira/browse/IMPALA-2130" target="_blank">IMPALA-2130</a>.
+ Previously, Impala would attempt to read the file despite the possibility that the file was corrupted.
+ </p>
+ </li>
+ <li class="li">
+ <p class="p">
+ Previously, calls to overloaded built-in functions could treat parameters as <code class="ph codeph">DOUBLE</code>
+ or <code class="ph codeph">FLOAT</code> when no overload had a signature that matched the exact argument types.
+ Now Impala prefers the function signature with <code class="ph codeph">DECIMAL</code> parameters in this case.
+ This change avoids a possible loss of precision in function calls such as <code class="ph codeph">greatest(0, 99999.8888)</code>;
+ now both parameters are treated as <code class="ph codeph">DECIMAL</code> rather than <code class="ph codeph">DOUBLE</code>, avoiding
+ any loss of precision in the fractional value.
+ This could cause slightly different results than in previous Impala releases for certain function calls.
+ </p>
+ </li>
+ <li class="li">
+ <p class="p">
+ Formerly, adding or subtracting a large interval value to a <code class="ph codeph">TIMESTAMP</code> could produce
+ a nonsensical result. Now when the result goes outside the range of <code class="ph codeph">TIMESTAMP</code> values,
+ Impala returns <code class="ph codeph">NULL</code>.
+ </p>
+ </li>
+ <li class="li">
+ <p class="p">
+ Formerly, it was possible to accidentally create a table with identical row and column delimiters.
+ This could happen unintentionally, when specifying one of the delimiters and using the
+ default value for the other. Now an attempt to use identical delimiters still succeeds,
+ but displays a warning message.
+ </p>
+ </li>
+ <li class="li">
+ <p class="p">
+ Formerly, Impala could include snippets of table data in log files by default, for example
+ when reporting conversion errors for data values. Now any such log messages are only produced
+ at higher logging levels that you would enable only during debugging.
+ </p>
+ </li>
+
+ </ul>
+ </div>
+
+ </article>
+
+
+
+ <article class="topic concept nested1" aria-labelledby="ariaid-title8" id="incompatible_changes__incompatible_changes_22x">
+
+ <h2 class="title topictitle2" id="ariaid-title8">Incompatible Changes Introduced in Impala 2.2.x</h2>
+
+ <div class="body conbody">
+
+ <section class="section" id="incompatible_changes_22x__files_220"><h3 class="title sectiontitle">
+ Changes to File Handling
+ </h3>
+
+ <p class="p">
+ Impala queries ignore files with extensions commonly used for temporary work files by Hadoop tools. Any
+ files with extensions <code class="ph codeph">.tmp</code> or <code class="ph codeph">.copying</code> are not considered part of the
+ Impala table. The suffix matching is case-insensitive, so for example Impala ignores both
+ <code class="ph codeph">.copying</code> and <code class="ph codeph">.COPYING</code> suffixes.
+ </p>
+ <p class="p">
+ The log rotation feature in Impala 2.2.0 and higher
+ means that older log files are now removed by default.
+ The default is to preserve the latest 10 log files for each
+ severity level, for each Impala-related daemon. If you have
+ set up your own log rotation processes that expect older
+ files to be present, either adjust your procedures or
+ change the Impala <code class="ph codeph">-max_log_files</code> setting.
+ <span class="ph">See <a class="xref" href="impala_logging.html#logs_rotate">Rotating Impala Logs</a> for details.</span>
+ </p>
+ </section>
+
+ <section class="section" id="incompatible_changes_22x__prereqs_210"><h3 class="title sectiontitle">
+ Changes to Prerequisites
+ </h3>
+
+ <p class="p">
+ The prerequisite for CPU architecture has been relaxed in Impala 2.2.0 and higher. From this release
+ onward, Impala works on CPUs that have the SSSE3 instruction set. The SSE4 instruction set is no longer
+ required. This relaxed requirement simplifies the upgrade planning from Impala 1.x releases, which also
+ worked on SSSE3-enabled processors.
+ </p>
+ </section>
+
+ </div>
+ </article>
+
+
+
+ <article class="topic concept nested1" aria-labelledby="ariaid-title9" id="incompatible_changes__incompatible_changes_21x">
+
+ <h2 class="title topictitle2" id="ariaid-title9">Incompatible Changes Introduced in Impala 2.1.x</h2>
+
+ <div class="body conbody">
+
+ <section class="section" id="incompatible_changes_21x__prereqs_210"><h3 class="title sectiontitle">
+ Changes to Prerequisites
+ </h3>
+
+ <p class="p">
+ Currently, Impala 2.1.x does not function on CPUs without the SSE4.1 instruction set. This minimum CPU
+ requirement is higher than in previous versions, which relied on the older SSSE3 instruction set. Check
+ the CPU level of the hosts in your cluster before upgrading to <span class="keyword">Impala 2.1</span>.
+ </p>
+ </section>
+
+ <section class="section" id="incompatible_changes_21x__output_format_210"><h3 class="title sectiontitle">
+ Changes to Output Format
+ </h3>
+
+ <p class="p">
+ The <span class="q">"small query"</span> optimization feature introduces some new information in the
+ <code class="ph codeph">EXPLAIN</code> plan, which you might need to account for if you parse the text of the plan
+ output.
+ </p>
+ </section>
+
+ <section class="section" id="incompatible_changes_21x__reserved_words_210"><h3 class="title sectiontitle">
+ New Reserved Words
+ </h3>
+
+ <p class="p">
+ New SQL syntax introduces additional reserved words:
+ <code class="ph codeph">FOR</code>, <code class="ph codeph">GRANT</code>, <code class="ph codeph">REVOKE</code>, <code class="ph codeph">ROLE</code>, <code class="ph codeph">ROLES</code>,
+ <code class="ph codeph">INCREMENTAL</code>.
+ <span class="ph">As always, see <a class="xref" href="impala_reserved_words.html#reserved_words">Impala Reserved Words</a>
+ for the set of reserved words for the current release, and the quoting techniques to avoid name conflicts.</span>
+ </p>
+ </section>
+ </div>
+ </article>
+
+
+
+ <article class="topic concept nested1" aria-labelledby="ariaid-title10" id="incompatible_changes__incompatible_changes_205">
+
+ <h2 class="title topictitle2" id="ariaid-title10">Incompatible Changes Introduced in Impala 2.0.5</h2>
+
+ <div class="body conbody">
+
+ <p class="p">
+ No incompatible changes.
+ </p>
+
+ </div>
+ </article>
+
+ <article class="topic concept nested1" aria-labelledby="ariaid-title11" id="incompatible_changes__incompatible_changes_204">
+
+ <h2 class="title topictitle2" id="ariaid-title11">Incompatible Changes Introduced in Impala 2.0.4</h2>
+
+ <div class="body conbody">
+
+ <p class="p">
+ No incompatible changes.
+ </p>
+
+ </div>
+ </article>
+
+ <article class="topic concept nested1" aria-labelledby="ariaid-title12" id="incompatible_changes__incompatible_changes_203">
+
+ <h2 class="title topictitle2" id="ariaid-title12">Incompatible Changes Introduced in Impala 2.0.3</h2>
+
+ <div class="body conbody">
+
+ </div>
+ </article>
+
+ <article class="topic concept nested1" aria-labelledby="ariaid-title13" id="incompatible_changes__incompatible_changes_202">
+
+ <h2 class="title topictitle2" id="ariaid-title13">Incompatible Changes Introduced in Impala 2.0.2</h2>
+
+ <div class="body conbody">
+
+ <p class="p">
+ No incompatible changes.
+ </p>
+
+ </div>
+ </article>
+
+ <article class="topic concept nested1" aria-labelledby="ariaid-title14" id="incompatible_changes__incompatible_changes_201">
+
+ <h2 class="title topictitle2" id="ariaid-title14">Incompatible Changes Introduced in Impala 2.0.1</h2>
+
+ <div class="body conbody">
+
+ <ul class="ul">
+ <li class="li">
+ <p class="p">
+ The <code class="ph codeph">INSERT</code> statement has always left behind a hidden work directory inside the data
+ directory of the table. Formerly, this hidden work directory was named
+ <span class="ph filepath">.impala_insert_staging</span> . In Impala 2.0.1 and later, this directory name is changed to
+ <span class="ph filepath">_impala_insert_staging</span> . (While HDFS tools are expected to treat names beginning
+ either with underscore and dot as hidden, in practice names beginning with an underscore are more widely
+ supported.) If you have any scripts, cleanup jobs, and so on that rely on the name of this work directory,
+ adjust them to use the new name.
+ </p>
+ </li>
+
+ <li class="li">
+ <p class="p">
+ The <code class="ph codeph">abs()</code> function now takes a broader range of numeric types as arguments, and the
+ return type is the same as the argument type.
+ </p>
+ </li>
+
+ <li class="li">
+ <p class="p">
+ Shorthand notation for character classes in regular expressions, such as <code class="ph codeph">\d</code> for digit,
+ are now available again in regular expression operators and functions such as
+ <code class="ph codeph">regexp_extract()</code> and <code class="ph codeph">regexp_replace()</code>. Some other differences in
+ regular expression behavior remain between Impala 1.x and Impala 2.x releases. See
+ <a class="xref" href="impala_incompatible_changes.html#incompatible_changes_200">Incompatible Changes Introduced in Impala 2.0.0</a> for details.
+ </p>
+ </li>
+ </ul>
+ </div>
+ </article>
+
+ <article class="topic concept nested1" aria-labelledby="ariaid-title15" id="incompatible_changes__incompatible_changes_200">
+
+ <h2 class="title topictitle2" id="ariaid-title15">Incompatible Changes Introduced in Impala 2.0.0</h2>
+
+ <div class="body conbody">
+
+ <section class="section" id="incompatible_changes_200__prereqs_200"><h3 class="title sectiontitle">
+ Changes to Prerequisites
+ </h3>
+
+ <p class="p">
+ Currently, Impala 2.0.x does not function on CPUs without the SSE4.1 instruction set. This minimum CPU
+ requirement is higher than in previous versions, which relied on the older SSSE3 instruction set. Check
+ the CPU level of the hosts in your cluster before upgrading to <span class="keyword">Impala 2.0</span>.
+ </p>
+ </section>
+
+ <section class="section" id="incompatible_changes_200__queries_200"><h3 class="title sectiontitle">
+ Changes to Query Syntax
+ </h3>
+
+
+ <p class="p">
+ The new syntax where query hints are allowed in comments causes some changes in the way comments are
+ parsed in the <span class="keyword cmdname">impala-shell</span> interpreter. Previously, you could end a
+ <code class="ph codeph">--</code> comment line with a semicolon and <span class="keyword cmdname">impala-shell</span> would treat that
+ as a no-op statement. Now, a comment line ending with a semicolon is passed as an empty statement to
+ the Impala daemon, where it is flagged as an error.
+ </p>
+
+ <p class="p">
+ Impala 2.0 and later uses a different support library for regular expression parsing than in earlier
+ Impala versions. Now, Impala uses the
+ <a class="xref" href="https://code.google.com/p/re2/" target="_blank">Google RE2 library</a>
+ rather than Boost for evaluating regular expressions. This implementation change causes some
+ differences in the allowed regular expression syntax, and in the way certain regex operators are
+ interpreted. The following are some of the major differences (not necessarily a complete list):
+ </p>
+ <ul class="ul">
+ <li class="li">
+ <p class="p">
+ <code class="ph codeph">.*?</code> notation for non-greedy matches is now supported, where it was not in earlier
+ Impala releases.
+ </p>
+ </li>
+
+ <li class="li">
+ <p class="p">
+ By default, <code class="ph codeph">^</code> and <code class="ph codeph">$</code> now match only begin/end of buffer, not
+ begin/end of each line. This behavior can be overridden in the regex itself using the
+ <code class="ph codeph">m</code> flag.
+ </p>
+ </li>
+
+ <li class="li">
+ <p class="p">
+ By default, <code class="ph codeph">.</code> does not match newline. This behavior can be overridden in the regex
+ itself using the <code class="ph codeph">s</code> flag.
+ </p>
+ </li>
+
+ <li class="li">
+ <p class="p">
+ <code class="ph codeph">\Z</code> is not supported.
+ </p>
+ </li>
+
+ <li class="li">
+ <p class="p">
+ <code class="ph codeph"><</code> and <code class="ph codeph">></code> for start of word and end of word are not
+ supported.
+ </p>
+ </li>
+
+ <li class="li">
+ <p class="p">
+ Lookahead and lookbehind are not supported.
+ </p>
+ </li>
+
+ <li class="li">
+ <p class="p">
+ Shorthand notation for character classes, such as <code class="ph codeph">\d</code> for digit, is not recognized.
+ (This restriction is lifted in Impala 2.0.1, which restores the shorthand notation.)
+ </p>
+ </li>
+ </ul>
+ </section>
+
+ <section class="section" id="incompatible_changes_200__output_format_210"><h3 class="title sectiontitle">
+ Changes to Output Format
+ </h3>
+
+
+ <p class="p">
+ In Impala 2.0 and later, <code class="ph codeph">user()</code> returns the full Kerberos principal string, such as
+ <code class="ph codeph">user@example.com</code>, in a Kerberized environment.
+ </p>
+
+ <p class="p">
+ The changed format for the user name in secure environments is also reflected where the user name is
+ displayed in the output of the <code class="ph codeph">PROFILE</code> command.
+ </p>
+
+ <p class="p">
+ In the output from <code class="ph codeph">SHOW FUNCTIONS</code>, <code class="ph codeph">SHOW AGGREGATE FUNCTIONS</code>, and
+ <code class="ph codeph">SHOW ANALYTIC FUNCTIONS</code>, arguments and return types of arbitrary
+ <code class="ph codeph">DECIMAL</code> scale and precision are represented as <code class="ph codeph">DECIMAL(*,*)</code>.
+ Formerly, these items were displayed as <code class="ph codeph">DECIMAL(-1,-1)</code>.
+ </p>
+
+ </section>
+
+ <section class="section" id="incompatible_changes_200__query_options_200"><h3 class="title sectiontitle">
+ Changes to Query Options
+ </h3>
+
+ <p class="p">
+ The <code class="ph codeph">PARQUET_COMPRESSION_CODEC</code> query option has been replaced by the
+ <code class="ph codeph">COMPRESSION_CODEC</code> query option.
+ <span class="ph">See <a class="xref" href="impala_compression_codec.html#compression_codec">COMPRESSION_CODEC Query Option (Impala 2.0 or higher only)</a> for details.</span>
+ </p>
+ </section>
+
+ <section class="section" id="incompatible_changes_200__config_options_200"><h3 class="title sectiontitle">
+ Changes to Configuration Options
+ </h3>
+
+
+ <p class="p">
+ The meaning of the <code class="ph codeph">--idle_query_timeout</code> configuration option is changed, to
+ accommodate the new <code class="ph codeph">QUERY_TIMEOUT_S</code> query option. Rather than setting an absolute
+ timeout period that applies to all queries, it now sets a maximum timeout period, which can be adjusted
+ downward for individual queries by specifying a value for the <code class="ph codeph">QUERY_TIMEOUT_S</code> query
+ option. In sessions where no <code class="ph codeph">QUERY_TIMEOUT_S</code> query option is specified, the
+ <code class="ph codeph">--idle_query_timeout</code> timeout period applies the same as in earlier versions.
+ </p>
+
+ <p class="p">
+ The <code class="ph codeph">--strict_unicode</code> option of <span class="keyword cmdname">impala-shell</span> was removed. To avoid
+ problems with Unicode values in <span class="keyword cmdname">impala-shell</span>, define the following locale setting
+ before running <span class="keyword cmdname">impala-shell</span>:
+ </p>
+<pre class="pre codeblock"><code>export LC_CTYPE=en_US.UTF-8
+</code></pre>
+
+ </section>
+
+ <section class="section" id="incompatible_changes_200__reserved_words_210"><h3 class="title sectiontitle">
+ New Reserved Words
+ </h3>
+
+ <p class="p">
+ Some new SQL syntax requires the addition of new reserved words: <code class="ph codeph">ANTI</code>,
+ <code class="ph codeph">ANALYTIC</code>, <code class="ph codeph">OVER</code>, <code class="ph codeph">PRECEDING</code>,
+ <code class="ph codeph">UNBOUNDED</code>, <code class="ph codeph">FOLLOWING</code>, <code class="ph codeph">CURRENT</code>,
+ <code class="ph codeph">ROWS</code>, <code class="ph codeph">RANGE</code>, <code class="ph codeph">CHAR</code>, <code class="ph codeph">VARCHAR</code>.
+ <span class="ph">As always, see <a class="xref" href="impala_reserved_words.html#reserved_words">Impala Reserved Words</a>
+ for the set of reserved words for the current release, and the quoting techniques to avoid name conflicts.</span>
+ </p>
+ </section>
+
+ <section class="section" id="incompatible_changes_200__output_files_200"><h3 class="title sectiontitle">
+ Changes to Data Files
+ </h3>
+
+
+ <p class="p" id="incompatible_changes_200__parquet_block_size">
+ The default Parquet block size for Impala is changed from 1 GB to 256 MB. This change could have
+ implications for the sizes of Parquet files produced by <code class="ph codeph">INSERT</code> and <code class="ph codeph">CREATE
+ TABLE AS SELECT</code> statements.
+ </p>
+ <p class="p">
+ Although older Impala releases typically produced files that were smaller than the old default size of
+ 1 GB, now the file size matches more closely whatever value is specified for the
+ <code class="ph codeph">PARQUET_FILE_SIZE</code> query option. Thus, if you use a non-default value for this setting,
+ the output files could be larger than before. They still might be somewhat smaller than the specified
+ value, because Impala makes conservative estimates about the space needed to represent each column as
+ it encodes the data.
+ </p>
+ <p class="p">
+ When you do not specify an explicit value for the <code class="ph codeph">PARQUET_FILE_SIZE</code> query option,
+ Impala tries to keep the file size within the 256 MB default size, but Impala might adjust the file
+ size to be somewhat larger if needed to accommodate the layout for <dfn class="term">wide</dfn> tables, that is,
+ tables with hundreds or thousands of columns.
+ </p>
+ <p class="p">
+ This change is unlikely to affect memory usage while writing Parquet files, because Impala does not
+ pre-allocate the memory needed to hold the entire Parquet block.
+ </p>
+
+ </section>
+
+ </div>
+ </article>
+
+ <article class="topic concept nested1" aria-labelledby="ariaid-title16" id="incompatible_changes__incompatible_changes_144">
+ <h2 class="title topictitle2" id="ariaid-title16">Incompatible Changes Introduced in Impala 1.4.4</h2>
+ <div class="body conbody">
+ <p class="p">
+ No incompatible changes.
+ </p>
+
+ </div>
+ </article>
+
+ <article class="topic concept nested1" aria-labelledby="ariaid-title17" id="incompatible_changes__incompatible_changes_143">
+
+ <h2 class="title topictitle2" id="ariaid-title17">Incompatible Changes Introduced in Impala 1.4.3</h2>
+
+ <div class="body conbody">
+
+ <p class="p">
+ No incompatible changes. The TLS/SSL security fix does not require any change in the way you interact with
+ Impala.
+ </p>
+
+ </div>
+ </article>
+
+ <article class="topic concept nested1" aria-labelledby="ariaid-title18" id="incompatible_changes__incompatible_changes_142">
+
+ <h2 class="title topictitle2" id="ariaid-title18">Incompatible Changes Introduced in Impala 1.4.2</h2>
+
+ <div class="body conbody">
+
+ <p class="p">
+ None. Impala 1.4.2 is purely a bug-fix release. It does not include any incompatible changes.
+ </p>
+
+ </div>
+ </article>
+
+ <article class="topic concept nested1" aria-labelledby="ariaid-title19" id="incompatible_changes__incompatible_changes_141">
+
+ <h2 class="title topictitle2" id="ariaid-title19">Incompatible Changes Introduced in Impala 1.4.1</h2>
+
+ <div class="body conbody">
+
+ <p class="p">
+ None. Impala 1.4.1 is purely a bug-fix release. It does not include any incompatible changes.
+ </p>
+ </div>
+ </article>
+
+ <article class="topic concept nested1" aria-labelledby="ariaid-title20" id="incompatible_changes__incompatible_changes_140">
+
+ <h2 class="title topictitle2" id="ariaid-title20">Incompatible Changes Introduced in Impala 1.4.0</h2>
+
+
+ <div class="body conbody">
+
+ <ul class="ul">
+ <li class="li">
+ <p class="p">
+ There is a slight change to required security privileges in the Sentry framework. To create a new
+ object, now you need the <code class="ph codeph">ALL</code> privilege on the parent object. For example, to create a
+ new table, view, or function requires having the <code class="ph codeph">ALL</code> privilege on the database
+ containing the new object. See <a class="xref" href="impala_authorization.html">Enabling Sentry Authorization for Impala</a> for a full list of operations and
+ associated privileges.
+ </p>
+ </li>
+
+ <li class="li">
+ <p class="p">
+ With the ability of <code class="ph codeph">ORDER BY</code> queries to process unlimited amounts of data with no
+ <code class="ph codeph">LIMIT</code> clause, the query options <code class="ph codeph">DEFAULT_ORDER_BY_LIMIT</code> and
+ <code class="ph codeph">ABORT_ON_DEFAULT_LIMIT_EXCEEDED</code> are now deprecated and have no effect.
+ <span class="ph">See <a class="xref" href="impala_order_by.html#order_by">ORDER BY Clause</a> for details about improvements to
+ the <code class="ph codeph">ORDER BY</code> clause.</span>
+ </p>
+ </li>
+
+ <li class="li">
+ <p class="p">
+ There are some changes to the list of reserved words. <span class="ph">See
+ <a class="xref" href="impala_reserved_words.html#reserved_words">Impala Reserved Words</a> for the most current list.</span> The following
+ keywords are new:
+ </p>
+ <ul class="ul">
+ <li class="li">
+ <code class="ph codeph">API_VERSION</code>
+ </li>
+
+ <li class="li">
+ <code class="ph codeph">BINARY</code>
+ </li>
+
+ <li class="li">
+ <code class="ph codeph">CACHED</code>
+ </li>
+
+ <li class="li">
+ <code class="ph codeph">CLASS</code>
+ </li>
+
+ <li class="li">
+ <code class="ph codeph">PARTITIONS</code>
+ </li>
+
+ <li class="li">
+ <code class="ph codeph">PRODUCED</code>
+ </li>
+
+ <li class="li">
+ <code class="ph codeph">UNCACHED</code>
+ </li>
+ </ul>
+ <p class="p">
+ The following were formerly reserved keywords, but are no longer reserved:
+ </p>
+ <ul class="ul">
+ <li class="li">
+ <code class="ph codeph">COUNT</code>
+ </li>
+
+ <li class="li">
+ <code class="ph codeph">GROUP_CONCAT</code>
+ </li>
+
+ <li class="li">
+ <code class="ph codeph">NDV</code>
+ </li>
+
+ <li class="li">
+ <code class="ph codeph">SUM</code>
+ </li>
+ </ul>
+ </li>
+
+ <li class="li">
+ <p class="p">
+ The fix for issue
+ <a class="xref" href="https://issues.apache.org/jira/browse/IMPALA-973" target="_blank">IMPALA-973</a>
+ changes the behavior of the <code class="ph codeph">INVALIDATE METADATA</code> statement regarding nonexistent
+ tables. In Impala 1.4.0 and higher, the statement returns an error if the specified table is not in the
+ metastore database at all. It completes successfully if the specified table is in the metastore
+ database but not yet recognized by Impala, for example if the table was created through Hive. Formerly,
+ you could issue this statement for a completely nonexistent table, with no error.
+ </p>
+ </li>
+ </ul>
+ </div>
+ </article>
+
+ <article class="topic concept nested1" aria-labelledby="ariaid-title21" id="incompatible_changes__incompatible_changes_133">
+
+ <h2 class="title topictitle2" id="ariaid-title21">Incompatible Changes Introduced in Impala 1.3.3</h2>
+
+ <div class="body conbody">
+
+ <p class="p">
+ No incompatible changes. The TLS/SSL security fix does not require any change in the way you interact with
+ Impala.
+ </p>
+
+ </div>
+ </article>
+
+ <article class="topic concept nested1" aria-labelledby="ariaid-title22" id="incompatible_changes__incompatible_changes_132">
+
+ <h2 class="title topictitle2" id="ariaid-title22">Incompatible Changes Introduced in Impala 1.3.2</h2>
+
+ <div class="body conbody">
+
+ <p class="p">
+ With the fix for IMPALA-1019, you can use HDFS caching for files that are accessed by Impala.
+ </p>
+
+ </div>
+ </article>
+
+ <article class="topic concept nested1" aria-labelledby="ariaid-title23" id="incompatible_changes__incompatible_changes_131">
+
+ <h2 class="title topictitle2" id="ariaid-title23">Incompatible Changes Introduced in Impala 1.3.1</h2>
+
+ <div class="body conbody">
+
+ <ul class="ul">
+ <li class="li">
+ <p class="p">
+ In Impala 1.3.1 and higher, the <code class="ph codeph">REGEXP</code> and <code class="ph codeph">RLIKE</code> operators now match a
+ regular expression string that occurs anywhere inside the target string, the same as if the regular
+ expression was enclosed on each side by <code class="ph codeph">.*</code>. See
+ <a class="xref" href="../shared/../topics/impala_operators.html#regexp">REGEXP Operator</a> for examples. Previously, these operators only
+ succeeded when the regular expression matched the entire target string. This change improves compatibility
+ with the regular expression support for popular database systems. There is no change to the behavior of the
+ <code class="ph codeph">regexp_extract()</code> and <code class="ph codeph">regexp_replace()</code> built-in functions.
+ </p>
+ </li>
+
+ <li class="li">
+ <p class="p">
+ The result set for the <code class="ph codeph">SHOW FUNCTIONS</code> statement includes a new first column, with the
+ data type of the return value. <span class="ph">See <a class="xref" href="impala_show.html#show">SHOW Statement</a> for
+ examples.</span>
+ </p>
+ </li>
+ </ul>
+ </div>
+ </article>
+
+ <article class="topic concept nested1" aria-labelledby="ariaid-title24" id="incompatible_changes__incompatible_changes_130">
+
+ <h2 class="title topictitle2" id="ariaid-title24">Incompatible Changes Introduced in Impala 1.3.0</h2>
+
+ <div class="body conbody">
+
+ <ul class="ul">
+ <li class="li">
+ <p class="p">
+ The <code class="ph codeph">EXPLAIN_LEVEL</code> query option now accepts numeric options from 0 (most concise) to 3
+ (most verbose), rather than only 0 or 1. If you formerly used <code class="ph codeph">SET EXPLAIN_LEVEL=1</code> to
+ get detailed explain plans, switch to <code class="ph codeph">SET EXPLAIN_LEVEL=3</code>. If you used the mnemonic
+ keyword (<code class="ph codeph">SET EXPLAIN_LEVEL=verbose</code>), you do not need to change your code because now
+ level 3 corresponds to <code class="ph codeph">verbose</code>. <span class="ph">See
+ <a class="xref" href="impala_explain_level.html#explain_level">EXPLAIN_LEVEL Query Option</a> for details about the allowed explain levels, and
+ <a class="xref" href="impala_explain_plan.html#explain_plan">Understanding Impala Query Performance - EXPLAIN Plans and Query Profiles</a> for usage information.</span>
+ </p>
+ </li>
+
+ <li class="li">
+ <div class="p">
+ The keyword <code class="ph codeph">DECIMAL</code> is now a reserved word. If you have any databases, tables,
+ columns, or other objects already named <code class="ph codeph">DECIMAL</code>, quote any references to them using
+ backticks (<code class="ph codeph">``</code>) to avoid name conflicts with the keyword.
+ <div class="note note note_note"><span class="note__title notetitle">Note:</span>
+ Although the <code class="ph codeph">DECIMAL</code> keyword is a reserved word, currently Impala does not support
+ <code class="ph codeph">DECIMAL</code> as a data type for columns.
+ </div>
+ </div>
+ </li>
+
+ <li class="li">
+ <p class="p">
+ The query option formerly named <code class="ph codeph">YARN_POOL</code> is now named
+ <code class="ph codeph">REQUEST_POOL</code> to reflect its broader use with the Impala admission control feature.
+ <span class="ph">See <a class="xref" href="impala_request_pool.html#request_pool">REQUEST_POOL Query Option</a> for information about the
+ option, and <a class="xref" href="impala_admission.html#admission_control">Admission Control and Query Queuing</a> for details about its use with the
+ admission control feature.</span>
+ </p>
+ </li>
+
+ <li class="li">
+ <p class="p">
+ There are some changes to the list of reserved words. <span class="ph">See
+ <a class="xref" href="impala_reserved_words.html#reserved_words">Impala Reserved Words</a> for the most current list.</span>
+ </p>
+ <ul class="ul">
+ <li class="li">
+ <p class="p">
+ The names of aggregate functions are no longer reserved words, so you can have databases, tables,
+ columns, or other objects named <code class="ph codeph">AVG</code>, <code class="ph codeph">MIN</code>, and so on without any
+ name conflicts.
+ </p>
+ </li>
+
+ <li class="li">
+ <p class="p">
+ The internal function names <code class="ph codeph">DISTINCTPC</code> and <code class="ph codeph">DISTINCTPCSA</code> are no
+ longer reserved words, although <code class="ph codeph">DISTINCT</code> is still a reserved word.
+ </p>
+ </li>
+
+ <li class="li">
+ <p class="p">
+ The keywords <code class="ph codeph">CLOSE_FN</code> and <code class="ph codeph">PREPARE_FN</code> are now reserved words.
+ <span class="ph">See <a class="xref" href="impala_create_function.html#create_function">CREATE FUNCTION Statement</a> for their role in
+ the <code class="ph codeph">CREATE FUNCTION</code> statement, and <a class="xref" href="impala_udf.html#udf_threads">Thread-Safe Work Area for UDFs</a> for
+ usage information.</span>
+ </p>
+ </li>
+ </ul>
+ </li>
+
+ <li class="li">
+ <p class="p">
+ The HDFS property <code class="ph codeph">dfs.client.file-block-storage-locations.timeout</code> was renamed to
+ <code class="ph codeph">dfs.client.file-block-storage-locations.timeout.millis</code>, to emphasize that the unit of
+ measure is milliseconds, not seconds. Impala requires a timeout of at least 10 seconds, making the
+ minimum value for this setting 10000. If you are not using cluster management software, you might need to
+ edit the <span class="ph filepath">hdfs-site.xml</span> file in the Impala configuration directory for the new name
+ and minimum value.
+ </p>
+ </li>
+ </ul>
+ </div>
+ </article>
+
+ <article class="topic concept nested1" aria-labelledby="ariaid-title25" id="incompatible_changes__incompatible_changes_124">
+
+ <h2 class="title topictitle2" id="ariaid-title25">Incompatible Changes Introduced in Impala 1.2.4</h2>
+
+ <div class="body conbody">
+
+ <p class="p">
+ There are no incompatible changes introduced in Impala 1.2.4.
+ </p>
+
+ <p class="p">
+ Previously, after creating a table in Hive, you had to issue the <code class="ph codeph">INVALIDATE METADATA</code>
+ statement with no table name, a potentially expensive operation on clusters with many databases, tables,
+ and partitions. Starting in Impala 1.2.4, you can issue the statement <code class="ph codeph">INVALIDATE METADATA
+ <var class="keyword varname">table_name</var></code> for a table newly created through Hive. Loading the metadata for
+ only this one table is faster and involves less network overhead. Therefore, you might revisit your setup
+ DDL scripts to add the table name to <code class="ph codeph">INVALIDATE METADATA</code> statements, in cases where you
+ create and populate the tables through Hive before querying them through Impala.
+ </p>
+ </div>
+ </article>
+
+ <article class="topic concept nested1" aria-labelledby="ariaid-title26" id="incompatible_changes__incompatible_changes_123">
+
+ <h2 class="title topictitle2" id="ariaid-title26">Incompatible Changes Introduced in Impala 1.2.3</h2>
+
+ <div class="body conbody">
+
+ <p class="p">
+ Because the feature set of Impala 1.2.3 is identical to Impala 1.2.2, there are no new incompatible
+ changes. See <a class="xref" href="impala_incompatible_changes.html#incompatible_changes_122">Incompatible Changes Introduced in Impala 1.2.2</a> if you are upgrading
+ from Impala 1.2.1 or 1.1.x.
+ </p>
+ </div>
+ </article>
+
+ <article class="topic concept nested1" aria-labelledby="ariaid-title27" id="incompatible_changes__incompatible_changes_122">
+
+ <h2 class="title topictitle2" id="ariaid-title27">Incompatible Changes Introduced in Impala 1.2.2</h2>
+
+ <div class="body conbody">
+
+ <p class="p">
+ The following changes to SQL syntax and semantics in Impala 1.2.2 could require updates to your SQL code,
+ or schema objects such as tables or views:
+ </p>
+
+ <ul class="ul">
+ <li class="li">
+ <p class="p">
+ With the addition of the <code class="ph codeph">CROSS JOIN</code> keyword, you might need to rewrite any queries
+ that refer to a table named <code class="ph codeph">CROSS</code> or use the name <code class="ph codeph">CROSS</code> as a table
+ alias:
+ </p>
+<pre class="pre codeblock"><code>-- Formerly, 'cross' in this query was an alias for t1
+-- and it was a normal join query.
+-- In 1.2.2 and higher, CROSS JOIN is a keyword, so 'cross'
+-- is not interpreted as a table alias, and the query
+-- uses the special CROSS JOIN processing rather than a
+-- regular join.
+select * from t1 cross join t2...
+
+-- Now if CROSS is used in other context such as a table or column name,
+-- use backticks to escape it.
+create table `cross` (x int);
+select * from `cross`;</code></pre>
+ </li>
+
+ <li class="li">
+ <p class="p">
+ Formerly, a <code class="ph codeph">DROP DATABASE</code> statement in Impala would not remove the top-level HDFS
+ directory for that database. The <code class="ph codeph">DROP DATABASE</code> has been enhanced to remove that
+ directory. (You still need to drop all the tables inside the database first; this change only applies
+ to the top-level directory for the entire database.)
+ </p>
+ </li>
+
+ <li class="li">
+ The keyword <code class="ph codeph">PARQUET</code> is introduced as a synonym for <code class="ph codeph">PARQUETFILE</code> in the
+ <code class="ph codeph">CREATE TABLE</code> and <code class="ph codeph">ALTER TABLE</code> statements, because that is the common
+ name for the file format. (As opposed to SequenceFile and RCFile where the <span class="q">"File"</span> suffix is part of
+ the name.) Documentation examples have been changed to prefer the new shorter keyword. The
+ <code class="ph codeph">PARQUETFILE</code> keyword is still available for backward compatibility with older Impala
+ versions.
+ </li>
+
+ <li class="li">
+ New overloads are available for several operators and built-in functions, allowing you to insert their
+ result values into smaller numeric columns such as <code class="ph codeph">INT</code>, <code class="ph codeph">SMALLINT</code>,
+ <code class="ph codeph">TINYINT</code>, and <code class="ph codeph">FLOAT</code> without using a <code class="ph codeph">CAST()</code> call. If you
+ remove the <code class="ph codeph">CAST()</code> calls from <code class="ph codeph">INSERT</code> statements, those statements might
+ not work with earlier versions of Impala.
+ </li>
+ </ul>
+
+ <p class="p">
+ Because many users are likely to upgrade straight from Impala 1.x to Impala 1.2.2, also read
+ <a class="xref" href="impala_incompatible_changes.html#incompatible_changes_121">Incompatible Changes Introduced in Impala 1.2.1</a> for things to note about upgrading
+ to Impala 1.2.x in general.
+ </p>
+
+ </div>
+ </article>
+
+ <article class="topic concept nested1" aria-labelledby="ariaid-title28" id="incompatible_changes__incompatible_changes_121">
+
+ <h2 class="title topictitle2" id="ariaid-title28">Incompatible Changes Introduced in Impala 1.2.1</h2>
+
+ <div class="body conbody">
+
+ <p class="p">
+ The following changes to SQL syntax and semantics in Impala 1.2.1 could require updates to your SQL code,
+ or schema objects such as tables or views:
+ </p>
+
+ <ul class="ul">
+ <li class="li">
+ <p class="p">
+ In Impala 1.2.1 and higher, all <code class="ph codeph">NULL</code> values come at the end of the result set for
+ <code class="ph codeph">ORDER BY ... ASC</code> queries, and at the beginning of the result set for <code class="ph codeph">ORDER BY ...
+ DESC</code> queries. In effect, <code class="ph codeph">NULL</code> is considered greater than all other values for
+ sorting purposes. The original Impala behavior always put <code class="ph codeph">NULL</code> values at the end, even for
+ <code class="ph codeph">ORDER BY ... DESC</code> queries. The new behavior in Impala 1.2.1 makes Impala more compatible
+ with other popular database systems. In Impala 1.2.1 and higher, you can override or specify the sorting
+ behavior for <code class="ph codeph">NULL</code> by adding the clause <code class="ph codeph">NULLS FIRST</code> or <code class="ph codeph">NULLS
+ LAST</code> at the end of the <code class="ph codeph">ORDER BY</code> clause.
+ </p>
+ <p class="p">
+ See <a class="xref" href="impala_literals.html#null">NULL</a> for more information.
+ </p>
+ </li>
+ </ul>
+
+ <p class="p">
+ The new <span class="keyword cmdname">catalogd</span> service might require changes to any user-written scripts that stop,
+ start, or restart Impala services, install or upgrade Impala packages, or issue <code class="ph codeph">REFRESH</code> or
+ <code class="ph codeph">INVALIDATE METADATA</code> statements:
+ </p>
+
+ <ul class="ul">
+ <li class="li">
+ <p class="p">
+ See <a class="xref" href="../shared/../topics/impala_install.html#install">Installing Impala</a>,
+ <a class="xref" href="../shared/../topics/impala_upgrading.html#upgrading">Upgrading Impala</a> and
+ <a class="xref" href="../shared/../topics/impala_processes.html#processes">Starting Impala</a>, for usage information for the
+ <span class="keyword cmdname">catalogd</span> daemon.
+ </p>
+ </li>
+
+ <li class="li">
+ <p class="p">
+ The <code class="ph codeph">REFRESH</code> and <code class="ph codeph">INVALIDATE METADATA</code> statements are no longer needed
+ when the <code class="ph codeph">CREATE TABLE</code>, <code class="ph codeph">INSERT</code>, or other table-changing or
+ data-changing operation is performed through Impala. These statements are still needed if such
+ operations are done through Hive or by manipulating data files directly in HDFS, but in those cases the
+ statements only need to be issued on one Impala node rather than on all nodes. See
+ <a class="xref" href="../shared/../topics/impala_refresh.html#refresh">REFRESH Statement</a> and
+ <a class="xref" href="../shared/../topics/impala_invalidate_metadata.html#invalidate_metadata">INVALIDATE METADATA Statement</a> for the latest usage
+ information for those statements.
+ </p>
+ </li>
+
+ <li class="li">
+ <p class="p">
+ See <a class="xref" href="../shared/../topics/impala_components.html#intro_catalogd">The Impala Catalog Service</a> for background information on the
+ <span class="keyword cmdname">catalogd</span> service.
+ </p>
+ </li>
+ </ul>
+
+ </div>
+ </article>
+
+ <article class="topic concept nested1" aria-labelledby="ariaid-title29" id="incompatible_changes__incompatible_changes_120">
+
+ <h2 class="title topictitle2" id="ariaid-title29">Incompatible Changes Introduced in Impala 1.2.0 (Beta)</h2>
+
+ <div class="body conbody">
+
+ <p class="p">
+ There are no incompatible changes to SQL syntax in Impala 1.2.0 (beta).
+ </p>
+
+ <p class="p">
+ The new <span class="keyword cmdname">catalogd</span> service might require changes to any user-written scripts that stop,
+ start, or restart Impala services, install or upgrade Impala packages, or issue <code class="ph codeph">REFRESH</code> or
+ <code class="ph codeph">INVALIDATE METADATA</code> statements:
+ </p>
+
+ <ul class="ul">
+ <li class="li">
+ <p class="p">
+ See <a class="xref" href="../shared/../topics/impala_install.html#install">Installing Impala</a>,
+ <a class="xref" href="../shared/../topics/impala_upgrading.html#upgrading">Upgrading Impala</a> and
+ <a class="xref" href="../shared/../topics/impala_processes.html#processes">Starting Impala</a>, for usage information for the
+ <span class="keyword cmdname">catalogd</span> daemon.
+ </p>
+ </li>
+
+ <li class="li">
+ <p class="p">
+ The <code class="ph codeph">REFRESH</code> and <code class="ph codeph">INVALIDATE METADATA</code> statements are no longer needed
+ when the <code class="ph codeph">CREATE TABLE</code>, <code class="ph codeph">INSERT</code>, or other table-changing or
+ data-changing operation is performed through Impala. These statements are still needed if such
+ operations are done through Hive or by manipulating data files directly in HDFS, but in those cases the
+ statements only need to be issued on one Impala node rather than on all nodes. See
+ <a class="xref" href="../shared/../topics/impala_refresh.html#refresh">REFRESH Statement</a> and
+ <a class="xref" href="../shared/../topics/impala_invalidate_metadata.html#invalidate_metadata">INVALIDATE METADATA Statement</a> for the latest usage
+ information for those statements.
+ </p>
+ </li>
+
+ <li class="li">
+ <p class="p">
+ See <a class="xref" href="../shared/../topics/impala_components.html#intro_catalogd">The Impala Catalog Service</a> for background information on the
+ <span class="keyword cmdname">catalogd</span> service.
+ </p>
+ </li>
+ </ul>
+
+ <p class="p">
+ The new resource management feature interacts with both YARN and Llama services.
+ <span class="ph">See
+ <a class="xref" href="impala_resource_management.html#resource_management">Resource Management for Impala</a> for usage information for Impala resource
+ management.</span>
+ </p>
+ </div>
+ </article>
+
+ <article class="topic concept nested1" aria-labelledby="ariaid-title30" id="incompatible_changes__incompatible_changes_111">
+
+ <h2 class="title topictitle2" id="ariaid-title30">Incompatible Changes Introduced in Impala 1.1.1</h2>
+
+ <div class="body conbody">
+
+ <p class="p">
+ There are no incompatible changes in Impala 1.1.1.
+ </p>
+
+
+
+
+
+
+
+
+
+
+
+ <p class="p">
+ Previously, it was not possible to create Parquet data through Impala and reuse that table within Hive. Now
+ that Parquet support is available for Hive 10, reusing existing Impala Parquet data files in Hive requires
+ updating the table metadata. Use the following command if you are already running Impala 1.1.1:
+ </p>
+
+<pre class="pre codeblock"><code>ALTER TABLE <var class="keyword varname">table_name</var> SET FILEFORMAT PARQUETFILE;
+</code></pre>
+
+ <p class="p">
+ If you are running a level of Impala that is older than 1.1.1, do the metadata update through Hive:
+ </p>
+
+<pre class="pre codeblock"><code>ALTER TABLE <var class="keyword varname">table_name</var> SET SERDE 'parquet.hive.serde.ParquetHiveSerDe';
+ALTER TABLE <var class="keyword varname">table_name</var> SET FILEFORMAT
+ INPUTFORMAT "parquet.hive.DeprecatedParquetInputFormat"
+ OUTPUTFORMAT "parquet.hive.DeprecatedParquetOutputFormat";
+</code></pre>
+
+ <p class="p">
+ Impala 1.1.1 and higher can reuse Parquet data files created by Hive, without any action required.
+ </p>
+
+ <p class="p">
+ As usual, make sure to upgrade the Impala LZO package to the latest level at the same
+ time as you upgrade the Impala server.
+ </p>
+ </div>
+ </article>
+
+ <article class="topic concept nested1" aria-labelledby="ariaid-title31" id="incompatible_changes__incompatible_changes_11">
+
+ <h2 class="title topictitle2" id="ariaid-title31">Incompatible Change Introduced in Impala 1.1</h2>
+
+ <div class="body conbody">
+
+ <ul class="ul">
+ <li class="li">
+ <p class="p">
+ The <code class="ph codeph">REFRESH</code> statement now requires a table name; in Impala 1.0, the table name was
+ optional. This syntax change is part of the internal rework to make <code class="ph codeph">REFRESH</code> a true
+ Impala SQL statement so that it can be called through the JDBC and ODBC APIs. <code class="ph codeph">REFRESH</code>
+ now reloads the metadata immediately, rather than marking it for update the next time any affected
+ table is accessed. The previous behavior, where omitting the table name caused a refresh of the entire
+ Impala metadata catalog, is available through the new <code class="ph codeph">INVALIDATE METADATA</code> statement.
+ <code class="ph codeph">INVALIDATE METADATA</code> can be specified with a table name to affect a single table, or
+ without a table name to affect the entire metadata catalog; the relevant metadata is reloaded the next
+ time it is requested during the processing for a SQL statement. See
+ <a class="xref" href="impala_refresh.html#refresh">REFRESH Statement</a> and
+ <a class="xref" href="impala_invalidate_metadata.html#invalidate_metadata">INVALIDATE METADATA Statement</a> for the latest details about these
+ statements.
+ </p>
+ </li>
+ </ul>
+ </div>
+ </article>
+
+ <article class="topic concept nested1" aria-labelledby="ariaid-title32" id="incompatible_changes__incompatible_changes_10">
+
+ <h2 class="title topictitle2" id="ariaid-title32">Incompatible Changes Introduced in Impala 1.0</h2>
+
+ <div class="body conbody">
+
+ <ul class="ul">
+ <li class="li">
+ If you use LZO-compressed text files, when you upgrade Impala to version 1.0, also update the
+ Impala LZO package to the latest level. See <a class="xref" href="impala_txtfile.html#lzo">Using LZO-Compressed Text Files</a> for
+ details.
+ </li>
+ </ul>
+ </div>
+ </article>
+
+</article></main></body></html>
\ No newline at end of file