You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@impala.apache.org by jb...@apache.org on 2016/11/17 23:12:03 UTC
[25/51] [partial] incubator-impala git commit: IMPALA-3398: Add docs
to main Impala branch.
http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_identifiers.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_identifiers.xml b/docs/topics/impala_identifiers.xml
new file mode 100644
index 0000000..7a947b3
--- /dev/null
+++ b/docs/topics/impala_identifiers.xml
@@ -0,0 +1,114 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept id="identifiers">
+
+ <title>Overview of Impala Identifiers</title>
+ <titlealts audience="PDF"><navtitle>Identifiers</navtitle></titlealts>
+ <prolog>
+ <metadata>
+ <data name="Category" value="Impala"/>
+ <data name="Category" value="SQL"/>
+ <data name="Category" value="Data Analysts"/>
+ <data name="Category" value="Developers"/>
+ <data name="Category" value="Querying"/>
+ <data name="Category" value="Databases"/>
+ <data name="Category" value="Tables"/>
+ </metadata>
+ </prolog>
+
+ <conbody>
+
+ <p>
+ Identifiers are the names of databases, tables, or columns that you specify in a SQL statement. The rules for
+ identifiers govern what names you can give to things you create, the notation for referring to names
+ containing unusual characters, and other aspects such as case sensitivity.
+ </p>
+
+ <ul>
+ <li>
+ <p>
+ The minimum length of an identifier is 1 character.
+ </p>
+ </li>
+
+ <li>
+ <p>
+ The maximum length of an identifier is currently 128 characters, enforced by the metastore database.
+ </p>
+ </li>
+
+ <li>
+ <p>
+ An identifier must start with an alphabetic character. The remainder can contain any combination of
+ alphanumeric characters and underscores. Quoting the identifier with backticks has no effect on the allowed
+ characters in the name.
+ </p>
+ </li>
+
+ <li>
+ <p>
+ An identifier can contain only ASCII characters.
+ </p>
+ </li>
+
+ <li>
+ <p>
+ To use an identifier name that matches one of the Impala reserved keywords (listed in
+ <xref href="impala_reserved_words.xml#reserved_words"/>), surround the identifier with <codeph>``</codeph>
+ characters (backticks). Quote the reserved word even if it is part of a fully qualified name.
+ The following example shows how a reserved word can be used as a column name if it is quoted
+ with backticks in the <codeph>CREATE TABLE</codeph> statement, and how the column name
+ must also be quoted with backticks in a query:
+ </p>
+<codeblock>[localhost:21000] > create table reserved (`data` string);
+
+[localhost:21000] > select data from reserved;
+ERROR: AnalysisException: Syntax error in line 1:
+select data from reserved
+ ^
+Encountered: DATA
+Expected: ALL, CASE, CAST, DISTINCT, EXISTS, FALSE, IF, INTERVAL, NOT, NULL, STRAIGHT_JOIN, TRUE, IDENTIFIER
+CAUSED BY: Exception: Syntax error
+
+[localhost:21000] > select reserved.data from reserved;
+ERROR: AnalysisException: Syntax error in line 1:
+select reserved.data from reserved
+ ^
+Encountered: DATA
+Expected: IDENTIFIER
+CAUSED BY: Exception: Syntax error
+
+[localhost:21000] > select reserved.`data` from reserved;
+
+[localhost:21000] >
+</codeblock>
+
+ <note type="important">
+ Because the list of reserved words grows over time as new SQL syntax is added,
+ consider adopting coding conventions (especially for any automated scripts
+ or in packaged applications) to always quote all identifiers with backticks.
+ Quoting all identifiers protects your SQL from compatibility issues if
+ new reserved words are added in later releases.
+ </note>
+
+ </li>
+
+ <li>
+ <p>
+ Impala identifiers are always case-insensitive. That is, tables named <codeph>t1</codeph> and
+ <codeph>T1</codeph> always refer to the same table, regardless of quote characters. Internally, Impala
+ always folds all specified table and column names to lowercase. This is why the column headers in query
+ output are always displayed in lowercase.
+ </p>
+ </li>
+ </ul>
+
+ <p>
+ See <xref href="impala_aliases.xml#aliases"/> for how to define shorter or easier-to-remember aliases if the
+ original names are long or cryptic identifiers.
+ <ph conref="../shared/impala_common.xml#common/aliases_vs_identifiers"/>
+ </p>
+
+ <p conref="../shared/impala_common.xml#common/views_vs_identifiers"/>
+ </conbody>
+</concept>
http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_impala_shell.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_impala_shell.xml b/docs/topics/impala_impala_shell.xml
new file mode 100644
index 0000000..afbfcaa
--- /dev/null
+++ b/docs/topics/impala_impala_shell.xml
@@ -0,0 +1,101 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept id="impala_shell">
+
+ <title>Using the Impala Shell (impala-shell Command)</title>
+ <titlealts audience="PDF"><navtitle>The Impala Shell</navtitle></titlealts>
+ <prolog>
+ <metadata>
+ <data name="Category" value="Impala"/>
+ <data name="Category" value="impala-shell"/>
+ <data name="Category" value="SQL"/>
+ <data name="Category" value="Querying"/>
+ <data name="Category" value="Data Analysts"/>
+ <data name="Category" value="Developers"/>
+ <data name="Category" value="Stub Pages"/>
+ </metadata>
+ </prolog>
+
+ <conbody>
+
+ <p>
+ <indexterm audience="Cloudera">impala-shell</indexterm>
+ You can use the Impala shell tool (<codeph>impala-shell</codeph>) to set up databases and tables, insert
+ data, and issue queries. For ad hoc queries and exploration, you can submit SQL statements in an interactive
+ session. To automate your work, you can specify command-line options to process a single statement or a
+ script file. The <cmdname>impala-shell</cmdname> interpreter accepts all the same SQL statements listed in
+ <xref href="impala_langref_sql.xml#langref_sql"/>, plus some shell-only commands that you can use for tuning
+ performance and diagnosing problems.
+ </p>
+
+ <p>
+ The <codeph>impala-shell</codeph> command fits into the familiar Unix toolchain:
+ </p>
+
+ <ul>
+ <li>
+ The <codeph>-q</codeph> option lets you issue a single query from the command line, without starting the
+ interactive interpreter. You could use this option to run <codeph>impala-shell</codeph> from inside a shell
+ script or with the command invocation syntax from a Python, Perl, or other kind of script.
+ </li>
+
+ <li>
+ The <codeph>-f</codeph> option lets you process a file containing multiple SQL statements,
+ such as a set of reports or DDL statements to create a group of tables and views.
+ </li>
+
+ <li rev="2.5.0 IMPALA-2179">
+ The <codeph>--var</codeph> option lets you pass substitution variables to the statements that
+ are executed by that <cmdname>impala-shell</cmdname> session, for example the statements
+ in a script file processed by the <codeph>-f</codeph> option. You encode the substitution variable
+ on the command line using the notation
+ <codeph>--var=<varname>variable_name</varname>=<varname>value</varname></codeph>.
+ Within a SQL statement, you substitute the value by using the notation <codeph>${var:<varname>variable_name</varname>}</codeph>.
+ This feature is available in <keyword keyref="impala25_full"/> and higher.
+ </li>
+
+ <li>
+ The <codeph>-o</codeph> option lets you save query output to a file.
+ </li>
+
+ <li>
+ The <codeph>-B</codeph> option turns off pretty-printing, so that you can produce comma-separated,
+ tab-separated, or other delimited text files as output. (Use the <codeph>--output_delimiter</codeph> option
+ to choose the delimiter character; the default is the tab character.)
+ </li>
+
+ <li>
+ In non-interactive mode, query output is printed to <codeph>stdout</codeph> or to the file specified by the
+ <codeph>-o</codeph> option, while incidental output is printed to <codeph>stderr</codeph>, so that you can
+ process just the query output as part of a Unix pipeline.
+ </li>
+
+ <li>
+ In interactive mode, <codeph>impala-shell</codeph> uses the <codeph>readline</codeph> facility to recall
+ and edit previous commands.
+ </li>
+ </ul>
+
+ <p>
+ For information on installing the Impala shell, see <xref href="impala_install.xml#install"/>. In Cloudera
+ Manager 4.1 and higher, Cloudera Manager installs <cmdname>impala-shell</cmdname> automatically. You might
+ install <cmdname>impala-shell</cmdname> manually on other systems not managed by Cloudera Manager, so that
+ you can issue queries from client systems that are not also running the Impala daemon or other Apache Hadoop
+ components.
+ </p>
+
+ <p>
+ For information about establishing a connection to a DataNode running the <codeph>impalad</codeph> daemon
+ through the <codeph>impala-shell</codeph> command, see <xref href="impala_connecting.xml#connecting"/>.
+ </p>
+
+ <p>
+ For a list of the <codeph>impala-shell</codeph> command-line options, see
+ <xref href="impala_shell_options.xml#shell_options"/>. For reference information about the
+ <codeph>impala-shell</codeph> interactive commands, see
+ <xref href="impala_shell_commands.xml#shell_commands"/>.
+ </p>
+
+ <p outputclass="toc"/>
+ </conbody>
+</concept>
http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_incompatible_changes.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_incompatible_changes.xml b/docs/topics/impala_incompatible_changes.xml
new file mode 100644
index 0000000..88a306f
--- /dev/null
+++ b/docs/topics/impala_incompatible_changes.xml
@@ -0,0 +1,1519 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept rev="ver" id="incompatible_changes">
+
+ <title><ph audience="standalone">Incompatible Changes and Limitations in Apache Impala (incubating)</ph><ph audience="integrated">Apache Impala (incubating) Incompatible Changes and Limitations</ph></title>
+ <prolog>
+ <metadata>
+ <data name="Category" value="Impala"/>
+ <data name="Category" value="Release Notes"/>
+ <data name="Category" value="Incompatible Changes"/>
+ <data name="Category" value="Limitations"/>
+ <data name="Category" value="Upgrading"/>
+ <data name="Category" value="Troubleshooting"/>
+ <data name="Category" value="Administrators"/>
+ <data name="Category" value="Developers"/>
+ <data name="Category" value="Data Analysts"/>
+ </metadata>
+ </prolog>
+
+ <conbody>
+
+ <p>
+ The Impala version covered by this documentation library contains the following incompatible changes. These
+ are things such as file format changes, removed features, or changes to implementation, default
+ configuration, dependencies, or prerequisites that could cause issues during or after an Impala upgrade.
+ </p>
+
+ <p>
+ Even added SQL statements or clauses can produce incompatibilities, if you have databases, tables, or columns
+ whose names conflict with the new keywords. <ph audience="PDF">See
+ <xref href="impala_reserved_words.xml#reserved_words"/> for the set of reserved words for the current
+ release, and the quoting techniques to avoid name conflicts.</ph>
+ </p>
+
+ <p outputclass="toc inpage"/>
+ </conbody>
+
+ <concept rev="2.7.0" id="incompatible_changes_27x">
+
+ <title>Incompatible Changes Introduced in Impala for CDH 5.9.x / Impala 2.7.x</title>
+
+ <conbody>
+ <ul>
+ <li>
+ <p rev="IMPALA-1731 IMPALA-3868 CDH-43734">
+ Bug fixes related to parsing of floating-point values (IMPALA-1731 and IMPALA-3868) can change
+ the results of casting strings that represent invalid floating-point values.
+ For example, formerly a string value beginning or ending with <codeph>inf</codeph>,
+ such as <codeph>1.23inf</codeph> or <codeph>infinite</codeph>, now are converted to <codeph>NULL</codeph>
+ when interpreted as a floating-point value.
+ Formerly, they were interpreted as the special <q>infinity</q> value when converting from string to floating-point.
+ Similarly, now only the string <codeph>NaN</codeph> (case-sensitive) is interpreted as the special <q>not a number</q>
+ value. String values containing multiple dots, such as <codeph>3..141</codeph> or <codeph>3.1.4.1</codeph>,
+ are now interpreted as <codeph>NULL</codeph> rather than being converted to valid floating-point values.
+ </p>
+ </li>
+ <li>
+ <p rev="IMPALA-4372">
+ The column types shown in the <codeph>DESCRIBE FORMATTED</codeph> output are in uppercase, where formerly they
+ were in lowercase. This is not an intended change and could be reverted in the future, when IMPALA-4372
+ is resolved.
+ </p>
+ </li>
+ </ul>
+ </conbody>
+
+ </concept>
+
+ <concept rev="2.6.0" id="incompatible_changes_26x">
+
+ <title>Incompatible Changes Introduced in Impala for CDH 5.8.x / Impala 2.6.x</title>
+
+ <conbody>
+ <ul>
+ <li>
+ <p rev="CDH-41184">
+ The default for the <codeph>RUNTIME_FILTER_MODE</codeph>
+ query option is changed to <codeph>GLOBAL</codeph> (the highest setting).
+ </p>
+ </li>
+ <li rev="CDH-41184 IMPALA-3007">
+ <p>
+ The <codeph>RUNTIME_BLOOM_FILTER_SIZE</codeph> setting is now only used
+ as a fallback if statistics are not available; otherwise, Impala
+ uses the statistics to estimate the appropriate size to use for each filter.
+ </p>
+ </li>
+ <li>
+ <p rev="IMPALA-3199">
+ Admission control and dynamic resource pools are enabled by default.
+ When upgrading from an earlier release, you must turn on these settings yourself
+ if they are not already enabled.
+ See <xref href="impala_admission.xml#admission_control"/> for details
+ about admission control.
+ </p>
+ </li>
+ <li>
+ <p>
+ Impala reserves some new keywords, in preparation for support for Kudu syntax:
+ <codeph>buckets</codeph>, <codeph>delete</codeph>, <codeph>distribute</codeph>,
+ <codeph>hash</codeph>, <codeph>ignore</codeph>, <codeph>split</codeph>, and <codeph>update</codeph>.
+ </p>
+ </li>
+ <li>
+ <p rev="IMPALA-3554">
+ For Kerberized clusters, the Catalog service now uses
+ the Kerberos principal instead of the operating sytem user that runs
+ the <cmdname>catalogd</cmdname> daemon.
+ This eliminates the requirement to configure a <codeph>hadoop.user.group.static.mapping.overrides</codeph>
+ setting to put the OS user into the Sentry administrative group, on clusters where the principal
+ and the OS user name for this user are different.
+ </p>
+ </li>
+ <li>
+ <p>
+ The mechanism for interpreting <codeph>DECIMAL</codeph> literals is
+ improved, no longer going through an intermediate conversion step
+ to <codeph>DOUBLE</codeph>:
+ </p>
+ <ul>
+ <li>
+ <p rev="IMPALA-3163">
+ Casting a <codeph>DECIMAL</codeph> value to <codeph>TIMESTAMP</codeph>
+ <codeph>DOUBLE</codeph> produces a more precise
+ value for the <codeph>TIMESTAMP</codeph> than formerly.
+ </p>
+ </li>
+ <li>
+ <p rev="IMPALA-3439">
+ Certain function calls involving <codeph>DECIMAL</codeph> literals
+ now succeed, when formerly they failed due to lack of a function
+ signature with a <codeph>DOUBLE</codeph> argument.
+ </p>
+ </li>
+ </ul>
+ </li>
+ <li>
+ <p rev="IMPALA-3155">
+ Improved type accuracy for <codeph>CASE</codeph> return values.
+ If all <codeph>WHEN</codeph> clauses of the <codeph>CASE</codeph>
+ expression are of <codeph>CHAR</codeph> type, the final result
+ is also <codeph>CHAR</codeph> instead of being converted to
+ <codeph>STRING</codeph>.
+ </p>
+ </li>
+ <li>
+ <p conref="../shared/impala_common.xml#common/IMPALA-3662"/>
+ </li>
+ <li rev="IMPALA-3452 CDH-39913">
+ <p>
+ The <codeph>S3_SKIP_INSERT_STAGING</codeph> query option, which is enabled by
+ default, increases the speed of <codeph>INSERT</codeph> operations for S3 tables.
+ The speedup applies to regular <codeph>INSERT</codeph>, but not <codeph>INSERT OVERWRITE</codeph>.
+ The tradeoff is the possibility of inconsistent output files left behind if a
+ node fails during <codeph>INSERT</codeph> execution.
+ See <xref href="impala_s3_skip_insert_staging.xml#s3_skip_insert_staging"/> for details.
+ </p>
+ </li>
+ </ul>
+ <p>
+ Certain features are turned off by default, to avoid regressions or unexpected
+ behavior following an upgrade. Consider turning on these features after suitable testing:
+ </p>
+ <ul>
+ <li>
+ <p rev="IMPALA-2660 CDH-40241">
+ Impala now recognizes the <codeph>auth_to_local</codeph> setting,
+ specified through the HDFS configuration setting
+ <codeph>hadoop.security.auth_to_local</codeph>.
+ This feature is disabled by default; to enable it,
+ specify <codeph>--load_auth_to_local_rules=true</codeph>
+ in the <cmdname>impalad</cmdname> configuration settings.
+ </p>
+ </li>
+ <li>
+ <p rev="IMPALA-2069">
+ A new query option, <codeph>PARQUET_ANNOTATE_STRINGS_UTF8</codeph>,
+ makes Impala include the <codeph>UTF-8</codeph> annotation
+ metadata for <codeph>STRING</codeph>, <codeph>CHAR</codeph>,
+ and <codeph>VARCHAR</codeph> columns in Parquet files created
+ by <codeph>INSERT</codeph> or <codeph>CREATE TABLE AS SELECT</codeph>
+ statements.
+ </p>
+ </li>
+ <li>
+ <p rev="IMPALA-2835 CDH-33330">
+ A new query option,
+ <codeph>PARQUET_FALLBACK_SCHEMA_RESOLUTION</codeph>,
+ lets Impala locate columns within Parquet files based on
+ column name rather than ordinal position.
+ This enhancement improves interoperability with applications
+ that write Parquet files with a different order or subset of
+ columns than are used in the Impala table.
+ </p>
+ </li>
+ </ul>
+ </conbody>
+
+ </concept>
+
+ <concept rev="2.5.x" id="incompatible_changes_25x">
+
+ <title>Incompatible Changes Introduced in Impala for CDH 5.7.x / Impala 2.5.x</title>
+
+ <conbody>
+ <ul>
+ <li rev="IMPALA-3044">
+ <p>
+ The admission control default limit for concurrent queries (the <uicontrol>max requests</uicontrol>
+ setting) is now unlimited instead of 200.
+ </p>
+ </li>
+
+ <li>
+ <p rev="IMPALA-2749">
+ Multiplying a mixture of <codeph>DECIMAL</codeph> and <codeph>FLOAT</codeph> or
+ <codeph>DOUBLE</codeph> values now returns
+ <codeph>DOUBLE</codeph> rather than <codeph>DECIMAL</codeph>. This
+ change avoids some cases where an intermediate value would underflow or overflow
+ and become <codeph>NULL</codeph> unexpectedly. The results of
+ multiplying <codeph>DECIMAL</codeph> and <codeph>FLOAT</codeph> or
+ <codeph>DOUBLE</codeph> might now be slightly less precise than
+ before. Previously, the intermediate types and thus the final result
+ depended on the exact order of the values of different types being
+ multiplied, which made the final result values difficult to
+ reason about.
+ </p>
+ </li>
+ <li rev="IMPALA-2204 CDH-33139">
+ <p>
+ Previously, the <codeph>_</codeph> and <codeph>%</codeph> wildcard
+ characters for the <codeph>LIKE</codeph> operator would not match
+ characters on the second or subsequent lines of multi-line string values. The fix for issue
+ <xref href="https://issues.cloudera.org/browse/IMPALA-2204" scope="external" format="html">IMPALA-2204</xref> causes
+ the wildcard matching to apply to the entire string for values
+ containing embedded <codeph>\n</codeph> characters. This could cause
+ different results than in previous Impala releases for identical
+ queries on identical data.
+ </p>
+ </li>
+ <li rev="IMPALA-1748 CDH-38369">
+ <p>
+ Formerly, all Impala UDFs and UDAs required running the
+ <codeph>CREATE FUNCTION</codeph> statements to
+ re-create them after each <cmdname>catalogd</cmdname> restart.
+ In <keyword keyref="impala25_full"/> and higher, functions written in C++ are persisted across
+ restarts, and the requirement to
+ re-create functions only applies to functions written in Java. Adapt any
+ function-reloading logic that you have added to your Impala environment.
+ </p>
+ </li>
+ <li>
+ <p rev="IMPALA-1651">
+ <codeph>CREATE TABLE LIKE</codeph> no longer inherits HDFS caching settings from the source table.
+ </p>
+ </li>
+ <li>
+ <p rev="IMPALA-2070">
+ The <codeph>SHOW DATABASES</codeph> statement now returns two columns rather than one.
+ The second column includes the associated comment string, if any, for each database.
+ Adjust any application code that examines the list of databases and assumes the
+ result set contains only a single column.
+ </p>
+ </li>
+ <li>
+ <p>
+ The output of the <codeph>SHOW FUNCTIONS</codeph> statement includes
+ two new columns, showing the kind of the function (for example,
+ <codeph>BUILTIN</codeph>) and whether or not the function persists
+ across catalog server restarts. For example, the <codeph>SHOW
+ FUNCTIONS</codeph> output for the
+ <codeph>_impala_builtins</codeph> database starts with:
+ </p>
+<codeblock>
++--------------+-------------------------------------------------+-------------+---------------+
+| return type | signature | binary type | is persistent |
++--------------+-------------------------------------------------+-------------+---------------+
+| BIGINT | abs(BIGINT) | BUILTIN | true |
+| DECIMAL(*,*) | abs(DECIMAL(*,*)) | BUILTIN | true |
+| DOUBLE | abs(DOUBLE) | BUILTIN | true |
+...
+</codeblock>
+ </li>
+ </ul>
+ </conbody>
+
+ </concept>
+
+ <concept rev="2.4.x" id="incompatible_changes_24x">
+
+ <title>Incompatible Changes Introduced in Impala for CDH 5.6.x / Impala 2.4.x</title>
+
+ <conbody>
+ <p>
+ Other than support for DSSD storage, the Impala feature set for CDH 5.6 is the same as for CDH 5.5.
+ Therefore, there are no incompatible changes for Impala introduced in CDH 5.6.
+ </p>
+ </conbody>
+
+ </concept>
+
+<!-- All 2.3.x subsections go under here -->
+
+<!-- Actually for 2.3 / 5.5, let's get away from doing a separate subhead for each maintenance release,
+ because in the normal course of events there will be nothing to add here until 5.6. If something new
+ needs to get noted, just add a new bullet with wording to indicate which 5.5.x release it applies to. -->
+
+ <concept rev="2.3.x" id="incompatible_changes_23x">
+
+ <title>Incompatible Changes Introduced in Impala for CDH 5.5.x / Impala 2.3.x</title>
+
+ <conbody>
+
+ <note conref="../shared/impala_common.xml#common/impala_llama_obsolete"/>
+
+ <ul>
+ <li rev="IMPALA-2005" audience="Cloudera">
+ <p>
+ If a <codeph>CREATE TABLE AS SELECT</codeph> operation fails while data is being inserted,
+ the table is automatically removed. Previously, the table was left behind with no data.
+ </p>
+ </li>
+ <li rev="IMPALA-2130">
+ <p>
+ If Impala encounters a Parquet file that is invalid because of an incorrect magic number,
+ the query skips the file. This change is caused by the fix for issue <xref href="https://issues.cloudera.org/browse/IMPALA-2130" scope="external" format="html">IMPALA-2130</xref>.
+ Previously, Impala would attempt to read the file despite the possibility that the file was corrupted.
+ </p>
+ </li>
+ <li rev="IMPALA-2233 CDH-33145">
+ <p>
+ Previously, calls to overloaded built-in functions could treat parameters as <codeph>DOUBLE</codeph>
+ or <codeph>FLOAT</codeph> when no overload had a signature that matched the exact argument types.
+ Now Impala prefers the function signature with <codeph>DECIMAL</codeph> parameters in this case.
+ This change avoids a possible loss of precision in function calls such as <codeph>greatest(0, 99999.8888)</codeph>;
+ now both parameters are treated as <codeph>DECIMAL</codeph> rather than <codeph>DOUBLE</codeph>, avoiding
+ any loss of precision in the fractional value.
+ This could cause slightly different results than in previous Impala releases for certain function calls.
+ </p>
+ </li>
+ <li rev="IMPALA-1675">
+ <p>
+ Formerly, adding or subtracting a large interval value to a <codeph>TIMESTAMP</codeph> could produce
+ a nonsensical result. Now when the result goes outside the range of <codeph>TIMESTAMP</codeph> values,
+ Impala returns <codeph>NULL</codeph>.
+ </p>
+ </li>
+ <li rev="IMPALA-2251 IMPALA-2257">
+ <p>
+ Formerly, it was possible to accidentally create a table with identical row and column delimiters.
+ This could happen unintentionally, when specifying one of the delimiters and using the
+ default value for the other. Now an attempt to use identical delimiters still succeeds,
+ but displays a warning message.
+ </p>
+ </li>
+ <li rev="CDH-28071">
+ <p>
+ Formerly, Impala could include snippets of table data in log files by default, for example
+ when reporting conversion errors for data values. Now any such log messages are only produced
+ at higher logging levels that you would enable only during debugging.
+ </p>
+ </li>
+<!-- placeholder -->
+ </ul>
+ </conbody>
+
+ </concept>
+
+<!-- All 2.2.x subsections go under here -->
+
+ <concept rev="2.2.x" id="incompatible_changes_22x">
+
+ <title>Incompatible Changes Introduced in Impala for CDH 5.4.x</title>
+
+ <conbody>
+
+<!--
+ <p>
+ No incompatible changes. CDH maintenance releases such as 5.4.1, 5.4.2, and so on are exclusively bug fix releases.
+ See <xref href="impala_incompatible_changes.xml#incompatible_changes_220"/> for the initial Impala feature release,
+ which are the releases that typically include incompatible changes.
+ </p>
+
+ <note conref="../shared/impala_common.xml#common/only_cdh5_22x"/>
+
+ </conbody>
+ </concept>
+
+ <concept rev="2.2.0" id="incompatible_changes_220">
+
+ <title>Incompatible Changes Introduced in Impala 2.2.0 / CDH 5.4.0</title>
+
+ <conbody>
+-->
+
+ <note conref="../shared/impala_common.xml#common/only_cdh5_220"/>
+
+ <section id="files_220">
+ <title>
+ Changes to File Handling
+ </title>
+ <p conref="../shared/impala_common.xml#common/ignore_file_extensions"/>
+ <p>
+ The log rotation feature in Impala 2.2.0 and higher
+ means that older log files are now removed by default.
+ The default is to preserve the latest 10 log files for each
+ severity level, for each Impala-related daemon. If you have
+ set up your own log rotation processes that expect older
+ files to be present, either adjust your procedures or
+ change the Impala <codeph>-max_log_files</codeph> setting.
+ <ph audience="PDF">See <xref href="impala_logging.xml#logs_rotate"/> for details.</ph>
+ </p>
+ </section>
+
+
+ <section id="prereqs_210">
+ <title>
+ Changes to Prerequisites
+ </title>
+ <p conref="../shared/impala_common.xml#common/cpu_prereq"/>
+ </section>
+
+ </conbody>
+ </concept>
+
+<!-- All 2.1.x subsections go under here -->
+
+ <concept rev="2.1.x" id="incompatible_changes_21x">
+
+ <title>Incompatible Changes Introduced in Impala for CDH 5.3.x</title>
+
+ <conbody>
+
+<!--
+ <p>
+ No incompatible changes. CDH maintenance releases such as 5.3.1, 5.3.2, and so on are exclusively bug fix releases.
+ See <xref href="impala_incompatible_changes.xml#incompatible_changes_210"/> for the initial Impala feature release,
+ which are the releases that typically include incompatible changes.
+ </p>
+
+ <note conref="../shared/impala_common.xml#common/only_cdh5_21x"/>
+
+ </conbody>
+ </concept>
+
+ <concept rev="2.1.7" id="incompatible_changes_217">
+ <title>Incompatible Changes Introduced in Cloudera Impala 2.1.7 / CDH 5.3.9</title>
+ <conbody>
+ <p>
+ No incompatible changes.
+ </p>
+
+ <note conref="../shared/impala_common.xml#common/only_cdh5_21x"/>
+
+ </conbody>
+ </concept>
+
+ <concept rev="2.1.6" id="incompatible_changes_216">
+ <title>Incompatible Changes Introduced in Impala 2.1.6 / CDH 5.3.8</title>
+ <conbody>
+ <p>
+ No incompatible changes.
+ </p>
+
+ <note conref="../shared/impala_common.xml#common/only_cdh5_21x"/>
+
+ </conbody>
+ </concept>
+
+ <concept rev="2.1.5" id="incompatible_changes_215">
+ <title>Incompatible Changes Introduced in Impala 2.1.5 / CDH 5.3.6</title>
+ <conbody>
+ <p>
+ No incompatible changes.
+ </p>
+
+ <note conref="../shared/impala_common.xml#common/only_cdh5_21x"/>
+
+ </conbody>
+ </concept>
+
+ <concept rev="2.1.4" id="incompatible_changes_214">
+ <title>Incompatible Changes Introduced in Impala 2.1.4 / CDH 5.3.4</title>
+ <conbody>
+ <p>
+ No incompatible changes.
+ <ph conref="../shared/impala_common.xml#common/impala_214_redux"/>
+ </p>
+
+ <note conref="../shared/impala_common.xml#common/only_cdh5_21x"/>
+
+ </conbody>
+ </concept>
+
+ <concept rev="2.1.3" id="incompatible_changes_213">
+ <title>Incompatible Changes Introduced in Impala 2.1.3 / CDH 5.3.3</title>
+ <conbody>
+ <p>
+ No incompatible changes.
+ </p>
+
+ <note conref="../shared/impala_common.xml#common/only_cdh5_213"/>
+
+ </conbody>
+ </concept>
+
+ <concept rev="2.1.2" id="incompatible_changes_212">
+ <title>Incompatible Changes Introduced in Impala 2.1.2 / CDH 5.3.2</title>
+ <conbody>
+ <p>
+ No incompatible changes.
+ </p>
+
+ <note conref="../shared/impala_common.xml#common/only_cdh5_212"/>
+
+ </conbody>
+ </concept>
+
+ <concept rev="2.1.1" id="incompatible_changes_211">
+
+ <title>Incompatible Changes Introduced in Impala 2.1.1 / CDH 5.3.1</title>
+
+ <conbody>
+
+ <p>
+ No incompatible changes.
+ </p>
+ </conbody>
+ </concept>
+
+ <concept rev="2.1.0" id="incompatible_changes_210">
+
+ <title>Incompatible Changes Introduced in Impala 2.1.0 / CDH 5.3.0</title>
+
+ <conbody>
+-->
+
+ <section id="prereqs_210">
+ <title>
+ Changes to Prerequisites
+ </title>
+ <p rev="CDH-24874">
+ Currently, Impala 2.1.x does not function on CPUs without the SSE4.1 instruction set. This minimum CPU
+ requirement is higher than in previous versions, which relied on the older SSSE3 instruction set. Check
+ the CPU level of the hosts in your cluster before upgrading to Impala 2.1.x or CDH 5.3.x.
+ </p>
+ </section>
+
+ <section id="output_format_210">
+ <title>
+ Changes to Output Format
+ </title>
+ <p>
+ The <q>small query</q> optimization feature introduces some new information in the
+ <codeph>EXPLAIN</codeph> plan, which you might need to account for if you parse the text of the plan
+ output.
+ </p>
+ </section>
+
+ <section id="reserved_words_210">
+ <title>
+ New Reserved Words
+ </title>
+ <p>
+ New SQL syntax introduces additional reserved words:
+ <codeph>FOR</codeph>, <codeph>GRANT</codeph>, <codeph>REVOKE</codeph>, <codeph>ROLE</codeph>, <codeph>ROLES</codeph>,
+ <codeph>INCREMENTAL</codeph>.
+ <ph audience="PDF">As always, see <xref href="impala_reserved_words.xml#reserved_words"/>
+ for the set of reserved words for the current release, and the quoting techniques to avoid name conflicts.</ph>
+ </p>
+ </section>
+ </conbody>
+ </concept>
+
+<!-- All 2.0.x subsections go under here -->
+
+ <concept rev="2.0.5" id="incompatible_changes_205">
+
+ <title>Incompatible Changes Introduced in Impala 2.0.5 / CDH 5.2.6</title>
+
+ <conbody>
+
+ <p>
+ No incompatible changes.
+ </p>
+
+ <note conref="../shared/impala_common.xml#common/only_cdh5_205"/>
+
+ </conbody>
+ </concept>
+
+ <concept rev="2.0.4" id="incompatible_changes_204">
+
+ <title>Incompatible Changes Introduced in Impala 2.0.4 / CDH 5.2.5</title>
+
+ <conbody>
+
+ <p>
+ No incompatible changes.
+ </p>
+
+ <note conref="../shared/impala_common.xml#common/only_cdh5_204"/>
+
+ </conbody>
+ </concept>
+
+ <concept rev="2.0.3" id="incompatible_changes_203">
+
+ <title>Incompatible Changes Introduced in Impala 2.0.3 / CDH 5.2.4</title>
+
+ <conbody>
+
+ <note conref="../shared/impala_common.xml#common/only_cdh5_203"/>
+
+ </conbody>
+ </concept>
+
+ <concept rev="2.0.2" id="incompatible_changes_202">
+
+ <title>Incompatible Changes Introduced in Impala 2.0.2 / CDH 5.2.3</title>
+
+ <conbody>
+
+ <p>
+ No incompatible changes.
+ </p>
+
+ <note conref="../shared/impala_common.xml#common/only_cdh5_202"/>
+
+ </conbody>
+ </concept>
+
+ <concept rev="2.0.1" id="incompatible_changes_201">
+
+ <title>Incompatible Changes Introduced in Impala 2.0.1 / CDH 5.2.1</title>
+
+ <conbody>
+
+ <ul>
+ <li>
+ <p conref="../shared/impala_common.xml#common/insert_hidden_work_directory"/>
+ </li>
+
+ <li>
+ <p>
+ The <codeph>abs()</codeph> function now takes a broader range of numeric types as arguments, and the
+ return type is the same as the argument type.
+ </p>
+ </li>
+
+ <li>
+ <p>
+ Shorthand notation for character classes in regular expressions, such as <codeph>\d</codeph> for digit,
+ are now available again in regular expression operators and functions such as
+ <codeph>regexp_extract()</codeph> and <codeph>regexp_replace()</codeph>. Some other differences in
+ regular expression behavior remain between Impala 1.x and Impala 2.x releases. See
+ <xref href="impala_incompatible_changes.xml#incompatible_changes_200"/> for details.
+ </p>
+ </li>
+ </ul>
+ </conbody>
+ </concept>
+
+ <concept rev="2.0.0" id="incompatible_changes_200">
+
+ <title>Incompatible Changes Introduced in Impala 2.0.0 / CDH 5.2.0</title>
+
+ <conbody>
+
+ <section id="prereqs_200">
+ <title>
+ Changes to Prerequisites
+ </title>
+ <p rev="CDH-24874">
+ Currently, Impala 2.0.x does not function on CPUs without the SSE4.1 instruction set. This minimum CPU
+ requirement is higher than in previous versions, which relied on the older SSSE3 instruction set. Check
+ the CPU level of the hosts in your cluster before upgrading to Impala 2.0.x or CDH 5.2.x.
+ </p>
+ </section>
+
+ <section id="queries_200">
+ <title>
+ Changes to Query Syntax
+ </title>
+
+ <p>
+ The new syntax where query hints are allowed in comments causes some changes in the way comments are
+ parsed in the <cmdname>impala-shell</cmdname> interpreter. Previously, you could end a
+ <codeph>--</codeph> comment line with a semicolon and <cmdname>impala-shell</cmdname> would treat that
+ as a no-op statement. Now, a comment line ending with a semicolon is passed as an empty statement to
+ the Impala daemon, where it is flagged as an error.
+ </p>
+
+ <p>
+ Impala 2.0 and later uses a different support library for regular expression parsing than in earlier
+ Impala versions. Now, Impala uses the
+ <xref href="https://code.google.com/p/re2/" scope="external" format="html">Google RE2 library</xref>
+ rather than Boost for evaluating regular expressions. This implementation change causes some
+ differences in the allowed regular expression syntax, and in the way certain regex operators are
+ interpreted. The following are some of the major differences (not necessarily a complete list):
+ </p>
+ <ul>
+ <li>
+ <p>
+ <codeph>.*?</codeph> notation for non-greedy matches is now supported, where it was not in earlier
+ Impala releases.
+ </p>
+ </li>
+
+ <li>
+ <p>
+ By default, <codeph>^</codeph> and <codeph>$</codeph> now match only begin/end of buffer, not
+ begin/end of each line. This behavior can be overridden in the regex itself using the
+ <codeph>m</codeph> flag.
+ </p>
+ </li>
+
+ <li>
+ <p>
+ By default, <codeph>.</codeph> does not match newline. This behavior can be overridden in the regex
+ itself using the <codeph>s</codeph> flag.
+ </p>
+ </li>
+
+ <li>
+ <p>
+ <codeph>\Z</codeph> is not supported.
+ </p>
+ </li>
+
+ <li>
+ <p>
+ <codeph><</codeph> and <codeph>></codeph> for start of word and end of word are not
+ supported.
+ </p>
+ </li>
+
+ <li>
+ <p>
+ Lookahead and lookbehind are not supported.
+ </p>
+ </li>
+
+ <li>
+ <p>
+ Shorthand notation for character classes, such as <codeph>\d</codeph> for digit, is not recognized.
+ (This restriction is lifted in Impala 2.0.1, which restores the shorthand notation.)
+ </p>
+ </li>
+ </ul>
+ </section>
+
+ <section id="output_format_210">
+ <title>
+ Changes to Output Format
+ </title>
+
+ <p conref="../shared/impala_common.xml#common/user_kerberized"/>
+
+ <p>
+ The changed format for the user name in secure environments is also reflected where the user name is
+ displayed in the output of the <codeph>PROFILE</codeph> command.
+ </p>
+
+ <p>
+ In the output from <codeph>SHOW FUNCTIONS</codeph>, <codeph>SHOW AGGREGATE FUNCTIONS</codeph>, and
+ <codeph>SHOW ANALYTIC FUNCTIONS</codeph>, arguments and return types of arbitrary
+ <codeph>DECIMAL</codeph> scale and precision are represented as <codeph>DECIMAL(*,*)</codeph>.
+ Formerly, these items were displayed as <codeph>DECIMAL(-1,-1)</codeph>.
+ </p>
+
+ </section>
+
+ <section id="query_options_200">
+ <title>
+ Changes to Query Options
+ </title>
+ <p>
+ The <codeph>PARQUET_COMPRESSION_CODEC</codeph> query option has been replaced by the
+ <codeph>COMPRESSION_CODEC</codeph> query option.
+ <ph audience="PDF">See <xref href="impala_compression_codec.xml#compression_codec"/> for details.</ph>
+ </p>
+ </section>
+
+ <section id="config_options_200">
+ <title>
+ Changes to Configuration Options
+ </title>
+
+ <p>
+ The meaning of the <codeph>--idle_query_timeout</codeph> configuration option is changed, to
+ accommodate the new <codeph>QUERY_TIMEOUT_S</codeph> query option. Rather than setting an absolute
+ timeout period that applies to all queries, it now sets a maximum timeout period, which can be adjusted
+ downward for individual queries by specifying a value for the <codeph>QUERY_TIMEOUT_S</codeph> query
+ option. In sessions where no <codeph>QUERY_TIMEOUT_S</codeph> query option is specified, the
+ <codeph>--idle_query_timeout</codeph> timeout period applies the same as in earlier versions.
+ </p>
+
+ <p>
+ The <codeph>--strict_unicode</codeph> option of <cmdname>impala-shell</cmdname> was removed. To avoid
+ problems with Unicode values in <cmdname>impala-shell</cmdname>, define the following locale setting
+ before running <cmdname>impala-shell</cmdname>:
+ </p>
+<codeblock>export LC_CTYPE=en_US.UTF-8
+</codeblock>
+
+ </section>
+
+ <section id="reserved_words_210">
+ <title>
+ New Reserved Words
+ </title>
+ <p>
+ Some new SQL syntax requires the addition of new reserved words: <codeph>ANTI</codeph>,
+ <codeph>ANALYTIC</codeph>, <codeph>OVER</codeph>, <codeph>PRECEDING</codeph>,
+ <codeph>UNBOUNDED</codeph>, <codeph>FOLLOWING</codeph>, <codeph>CURRENT</codeph>,
+ <codeph>ROWS</codeph>, <codeph>RANGE</codeph>, <codeph>CHAR</codeph>, <codeph>VARCHAR</codeph>.
+ <ph audience="PDF">As always, see <xref href="impala_reserved_words.xml#reserved_words"/>
+ for the set of reserved words for the current release, and the quoting techniques to avoid name conflicts.</ph>
+ </p>
+ </section>
+
+ <section id="output_files_200">
+ <title>
+ Changes to Data Files
+ </title>
+
+ <p id="parquet_block_size">
+ The default Parquet block size for Impala is changed from 1 GB to 256 MB. This change could have
+ implications for the sizes of Parquet files produced by <codeph>INSERT</codeph> and <codeph>CREATE
+ TABLE AS SELECT</codeph> statements.
+ </p>
+ <p>
+ Although older Impala releases typically produced files that were smaller than the old default size of
+ 1 GB, now the file size matches more closely whatever value is specified for the
+ <codeph>PARQUET_FILE_SIZE</codeph> query option. Thus, if you use a non-default value for this setting,
+ the output files could be larger than before. They still might be somewhat smaller than the specified
+ value, because Impala makes conservative estimates about the space needed to represent each column as
+ it encodes the data.
+ </p>
+ <p>
+ When you do not specify an explicit value for the <codeph>PARQUET_FILE_SIZE</codeph> query option,
+ Impala tries to keep the file size within the 256 MB default size, but Impala might adjust the file
+ size to be somewhat larger if needed to accommodate the layout for <term>wide</term> tables, that is,
+ tables with hundreds or thousands of columns.
+ </p>
+ <p>
+ This change is unlikely to affect memory usage while writing Parquet files, because Impala does not
+ pre-allocate the memory needed to hold the entire Parquet block.
+ </p>
+
+ </section>
+
+ </conbody>
+ </concept>
+
+ <concept rev="1.4.4" id="incompatible_changes_144">
+ <title>Incompatible Changes Introduced in Impala 1.4.4 / CDH 5.1.5</title>
+ <conbody>
+ <p>
+ No incompatible changes.
+ </p>
+
+ <note conref="../shared/impala_common.xml#common/only_cdh5_144"/>
+
+ </conbody>
+ </concept>
+
+ <concept rev="1.4.3" id="incompatible_changes_143">
+
+ <title>Incompatible Changes Introduced in Impala 1.4.3 / CDH 5.1.4</title>
+
+ <conbody>
+
+ <p>
+ No incompatible changes. The TLS/SSL security fix does not require any change in the way you interact with
+ Impala.
+ </p>
+
+ <note conref="../shared/impala_common.xml#common/only_cdh5_143"/>
+
+ </conbody>
+ </concept>
+
+ <concept rev="1.4.2" id="incompatible_changes_142">
+
+ <title>Incompatible Changes Introduced in Impala 1.4.2 / CDH 5.1.3</title>
+
+ <conbody>
+
+ <p>
+ None. Impala 1.4.2 is purely a bug-fix release. It does not include any incompatible changes.
+ </p>
+
+ <note conref="../shared/impala_common.xml#common/only_cdh5_142"/>
+
+ </conbody>
+ </concept>
+
+ <concept rev="1.4.1" id="incompatible_changes_141">
+
+ <title>Incompatible Changes Introduced in Impala 1.4.1 / CDH 5.1.2</title>
+
+ <conbody>
+
+ <p>
+ None. Impala 1.4.1 is purely a bug-fix release. It does not include any incompatible changes.
+ </p>
+ </conbody>
+ </concept>
+
+ <concept rev="1.4.0" id="incompatible_changes_140">
+
+ <title>Incompatible Changes Introduced in Impala 1.4.0 / CDH 5.1.0</title>
+ <prolog>
+ <metadata>
+ <data name="Category" value="Deprecated Features"/>
+ </metadata>
+ </prolog>
+
+ <conbody>
+
+ <ul>
+ <li>
+ <p>
+ There is a slight change to required security privileges in the Sentry framework. To create a new
+ object, now you need the <codeph>ALL</codeph> privilege on the parent object. For example, to create a
+ new table, view, or function requires having the <codeph>ALL</codeph> privilege on the database
+ containing the new object. <ph audience="PDF">See
+ <xref
+ href="http://www.cloudera.com/documentation/enterprise/latest/topics/impala_authorization.html"
+ scope="external" format="html">Privilege Model and Object Hierarchy</xref>
+ for a full list of operations and
+ associated privileges.</ph>
+ </p>
+ </li>
+
+ <li>
+ <p>
+ With the ability of <codeph>ORDER BY</codeph> queries to process unlimited amounts of data with no
+ <codeph>LIMIT</codeph> clause, the query options <codeph>DEFAULT_ORDER_BY_LIMIT</codeph> and
+ <codeph>ABORT_ON_DEFAULT_LIMIT_EXCEEDED</codeph> are now deprecated and have no effect.
+ <ph audience="PDF">See <xref href="impala_order_by.xml#order_by"/> for details about improvements to
+ the <codeph>ORDER BY</codeph> clause.</ph>
+ </p>
+ </li>
+
+ <li>
+ <p>
+ There are some changes to the list of reserved words. <ph audience="PDF">See
+ <xref href="impala_reserved_words.xml#reserved_words"/> for the most current list.</ph> The following
+ keywords are new:
+ </p>
+ <ul>
+ <li>
+ <codeph>API_VERSION</codeph>
+ </li>
+
+ <li>
+ <codeph>BINARY</codeph>
+ </li>
+
+ <li>
+ <codeph>CACHED</codeph>
+ </li>
+
+ <li>
+ <codeph>CLASS</codeph>
+ </li>
+
+ <li>
+ <codeph>PARTITIONS</codeph>
+ </li>
+
+ <li>
+ <codeph>PRODUCED</codeph>
+ </li>
+
+ <li>
+ <codeph>UNCACHED</codeph>
+ </li>
+ </ul>
+ <p>
+ The following were formerly reserved keywords, but are no longer reserved:
+ </p>
+ <ul>
+ <li>
+ <codeph>COUNT</codeph>
+ </li>
+
+ <li>
+ <codeph>GROUP_CONCAT</codeph>
+ </li>
+
+ <li>
+ <codeph>NDV</codeph>
+ </li>
+
+ <li>
+ <codeph>SUM</codeph>
+ </li>
+ </ul>
+ </li>
+
+ <li>
+ <p>
+ The fix for issue
+ <xref href="https://issues.cloudera.org/browse/IMPALA-973" scope="external" format="html">IMPALA-973</xref>
+ changes the behavior of the <codeph>INVALIDATE METADATA</codeph> statement regarding nonexistent
+ tables. In Impala 1.4.0 and higher, the statement returns an error if the specified table is not in the
+ metastore database at all. It completes successfully if the specified table is in the metastore
+ database but not yet recognized by Impala, for example if the table was created through Hive. Formerly,
+ you could issue this statement for a completely nonexistent table, with no error.
+ </p>
+ </li>
+ </ul>
+ </conbody>
+ </concept>
+
+ <concept rev="1.3.3" id="incompatible_changes_133">
+
+ <title>Incompatible Changes Introduced in Impala 1.3.3 / CDH 5.0.5</title>
+
+ <conbody>
+
+ <p>
+ No incompatible changes. The TLS/SSL security fix does not require any change in the way you interact with
+ Impala.
+ </p>
+
+ <note conref="../shared/impala_common.xml#common/only_cdh5_133"/>
+
+ </conbody>
+ </concept>
+
+ <concept rev="1.3.2" id="incompatible_changes_132">
+
+ <title>Incompatible Changes Introduced in Impala 1.3.2 / CDH 5.0.4</title>
+
+ <conbody>
+
+ <p>
+ With the fix for IMPALA-1019, you can use HDFS caching for files that are accessed by Impala.
+ </p>
+
+ <note conref="../shared/impala_common.xml#common/only_cdh5_132"/>
+
+ </conbody>
+ </concept>
+
+ <concept rev="1.3.1" id="incompatible_changes_131">
+
+ <title>Incompatible Changes Introduced in Impala 1.3.1 / CDH 5.0.3</title>
+
+ <conbody>
+
+ <ul>
+ <li>
+ <p conref="../shared/impala_common.xml#common/regexp_matching"/>
+ </li>
+
+ <li>
+ <p>
+ The result set for the <codeph>SHOW FUNCTIONS</codeph> statement includes a new first column, with the
+ data type of the return value. <ph audience="PDF">See <xref href="impala_show.xml#show"/> for
+ examples.</ph>
+ </p>
+ </li>
+ </ul>
+ </conbody>
+ </concept>
+
+ <concept rev="1.3.0" id="incompatible_changes_130">
+
+ <title>Incompatible Changes Introduced in Impala 1.3.0 / CDH 5.0.0</title>
+
+ <conbody>
+
+ <ul>
+ <li>
+ <p>
+ The <codeph>EXPLAIN_LEVEL</codeph> query option now accepts numeric options from 0 (most concise) to 3
+ (most verbose), rather than only 0 or 1. If you formerly used <codeph>SET EXPLAIN_LEVEL=1</codeph> to
+ get detailed explain plans, switch to <codeph>SET EXPLAIN_LEVEL=3</codeph>. If you used the mnemonic
+ keyword (<codeph>SET EXPLAIN_LEVEL=verbose</codeph>), you do not need to change your code because now
+ level 3 corresponds to <codeph>verbose</codeph>. <ph audience="PDF">See
+ <xref href="impala_explain_level.xml#explain_level"/> for details about the allowed explain levels, and
+ <xref href="impala_explain_plan.xml#explain_plan"/> for usage information.</ph>
+ </p>
+ </li>
+
+ <li>
+ <p>
+ The keyword <codeph>DECIMAL</codeph> is now a reserved word. If you have any databases, tables,
+ columns, or other objects already named <codeph>DECIMAL</codeph>, quote any references to them using
+ backticks (<codeph>``</codeph>) to avoid name conflicts with the keyword.
+ <note>
+ Although the <codeph>DECIMAL</codeph> keyword is a reserved word, currently Impala does not support
+ <codeph>DECIMAL</codeph> as a data type for columns.
+ </note>
+ </p>
+ </li>
+
+ <li>
+ <p>
+ The query option named <codeph>YARN_POOL</codeph> during the CDH 5 beta period is now named
+ <codeph>REQUEST_POOL</codeph> to reflect its broader use with the Impala admission control feature.
+ <ph audience="PDF">See <xref href="impala_request_pool.xml#request_pool"/> for information about the
+ option, and <xref href="impala_admission.xml#admission_control"/> for details about its use with the
+ admission control feature.</ph>
+ </p>
+ </li>
+
+ <li>
+ <p>
+ There are some changes to the list of reserved words. <ph audience="PDF">See
+ <xref href="impala_reserved_words.xml#reserved_words"/> for the most current list.</ph>
+ </p>
+ <ul>
+ <li>
+ <p>
+ The names of aggregate functions are no longer reserved words, so you can have databases, tables,
+ columns, or other objects named <codeph>AVG</codeph>, <codeph>MIN</codeph>, and so on without any
+ name conflicts.
+ </p>
+ </li>
+
+ <li>
+ <p>
+ The internal function names <codeph>DISTINCTPC</codeph> and <codeph>DISTINCTPCSA</codeph> are no
+ longer reserved words, although <codeph>DISTINCT</codeph> is still a reserved word.
+ </p>
+ </li>
+
+ <li>
+ <p>
+ The keywords <codeph>CLOSE_FN</codeph> and <codeph>PREPARE_FN</codeph> are now reserved words.
+ <ph audience="PDF">See <xref href="impala_create_function.xml#create_function"/> for their role in
+ the <codeph>CREATE FUNCTION</codeph> statement, and <xref href="impala_udf.xml#udf_threads"/> for
+ usage information.</ph>
+ </p>
+ </li>
+ </ul>
+ </li>
+
+ <li>
+ <p>
+ The HDFS property <codeph>dfs.client.file-block-storage-locations.timeout</codeph> was renamed to
+ <codeph>dfs.client.file-block-storage-locations.timeout.millis</codeph>, to emphasize that the unit of
+ measure is milliseconds, not seconds. Impala requires a timeout of at least 10 seconds, making the
+ minimum value for this setting 10000. On systems not managed by Cloudera Manager, you might need to
+ edit the <filepath>hdfs-site.xml</filepath> file in the Impala configuration directory for the new name
+ and minimum value.
+ </p>
+ </li>
+ </ul>
+ </conbody>
+ </concept>
+
+ <concept rev="1.2.4" id="incompatible_changes_124">
+
+ <title>Incompatible Changes Introduced in Impala 1.2.4</title>
+
+ <conbody>
+
+ <p>
+ There are no incompatible changes introduced in Impala 1.2.4.
+ </p>
+
+ <p>
+ Previously, after creating a table in Hive, you had to issue the <codeph>INVALIDATE METADATA</codeph>
+ statement with no table name, a potentially expensive operation on clusters with many databases, tables,
+ and partitions. Starting in Impala 1.2.4, you can issue the statement <codeph>INVALIDATE METADATA
+ <varname>table_name</varname></codeph> for a table newly created through Hive. Loading the metadata for
+ only this one table is faster and involves less network overhead. Therefore, you might revisit your setup
+ DDL scripts to add the table name to <codeph>INVALIDATE METADATA</codeph> statements, in cases where you
+ create and populate the tables through Hive before querying them through Impala.
+ </p>
+ </conbody>
+ </concept>
+
+ <concept rev="1.2.3" id="incompatible_changes_123">
+
+ <title>Incompatible Changes Introduced in Impala 1.2.3</title>
+
+ <conbody>
+
+ <p>
+ Because the feature set of Impala 1.2.3 is identical to Impala 1.2.2, there are no new incompatible
+ changes. See <xref href="impala_incompatible_changes.xml#incompatible_changes_122"/> if you are upgrading
+ from Impala 1.2.1 or 1.1.x.
+ </p>
+ </conbody>
+ </concept>
+
+ <concept rev="1.2.2" id="incompatible_changes_122">
+
+ <title>Incompatible Changes Introduced in Impala 1.2.2</title>
+
+ <conbody>
+
+ <p>
+ The following changes to SQL syntax and semantics in Impala 1.2.2 could require updates to your SQL code,
+ or schema objects such as tables or views:
+ </p>
+
+ <ul>
+ <li>
+ <p>
+ With the addition of the <codeph>CROSS JOIN</codeph> keyword, you might need to rewrite any queries
+ that refer to a table named <codeph>CROSS</codeph> or use the name <codeph>CROSS</codeph> as a table
+ alias:
+ </p>
+<codeblock>-- Formerly, 'cross' in this query was an alias for t1
+-- and it was a normal join query.
+-- In 1.2.2 and higher, CROSS JOIN is a keyword, so 'cross'
+-- is not interpreted as a table alias, and the query
+-- uses the special CROSS JOIN processing rather than a
+-- regular join.
+select * from t1 cross join t2...
+
+-- Now if CROSS is used in other context such as a table or column name,
+-- use backticks to escape it.
+create table `cross` (x int);
+select * from `cross`;</codeblock>
+ </li>
+
+ <li>
+ <p>
+ Formerly, a <codeph>DROP DATABASE</codeph> statement in Impala would not remove the top-level HDFS
+ directory for that database. The <codeph>DROP DATABASE</codeph> has been enhanced to remove that
+ directory. (You still need to drop all the tables inside the database first; this change only applies
+ to the top-level directory for the entire database.)
+ </p>
+ </li>
+
+ <li>
+ The keyword <codeph>PARQUET</codeph> is introduced as a synonym for <codeph>PARQUETFILE</codeph> in the
+ <codeph>CREATE TABLE</codeph> and <codeph>ALTER TABLE</codeph> statements, because that is the common
+ name for the file format. (As opposed to SequenceFile and RCFile where the <q>File</q> suffix is part of
+ the name.) Documentation examples have been changed to prefer the new shorter keyword. The
+ <codeph>PARQUETFILE</codeph> keyword is still available for backward compatibility with older Impala
+ versions.
+ </li>
+
+ <li>
+ New overloads are available for several operators and built-in functions, allowing you to insert their
+ result values into smaller numeric columns such as <codeph>INT</codeph>, <codeph>SMALLINT</codeph>,
+ <codeph>TINYINT</codeph>, and <codeph>FLOAT</codeph> without using a <codeph>CAST()</codeph> call. If you
+ remove the <codeph>CAST()</codeph> calls from <codeph>INSERT</codeph> statements, those statements might
+ not work with earlier versions of Impala.
+ </li>
+ </ul>
+
+ <p>
+ Because many users are likely to upgrade straight from Impala 1.x to Impala 1.2.2, also read
+ <xref href="impala_incompatible_changes.xml#incompatible_changes_121"/> for things to note about upgrading
+ to Impala 1.2.x in general.
+ </p>
+
+ <p conref="../shared/impala_common.xml#common/cm48_upgrade"/>
+
+<!-- <note conref="common.xml#common/cdh4_cdh5_upgrade"/> -->
+ </conbody>
+ </concept>
+
+ <concept rev="1.2.1" id="incompatible_changes_121">
+
+ <title>Incompatible Changes Introduced in Impala 1.2.1</title>
+
+ <conbody>
+
+ <p>
+ The following changes to SQL syntax and semantics in Impala 1.2.1 could require updates to your SQL code,
+ or schema objects such as tables or views:
+ </p>
+
+ <ul>
+ <li>
+ <p conref="../shared/impala_common.xml#common/null_sorting_change"/>
+ <p audience="PDF">
+ See <xref href="impala_literals.xml#null"/> for more information.
+ </p>
+ </li>
+ </ul>
+
+ <p>
+ Impala 1.2.1 goes along with CDH 4.5 and Cloudera Manager 4.8. If you used the beta version Impala 1.2.0
+ that came with the beta of CDH 5, Impala 1.2.1 includes all the features of Impala 1.2.0 except for
+ resource management, which relies on the YARN framework from CDH 5.
+ </p>
+
+ <p>
+ The new <cmdname>catalogd</cmdname> service might require changes to any user-written scripts that stop,
+ start, or restart Impala services, install or upgrade Impala packages, or issue <codeph>REFRESH</codeph> or
+ <codeph>INVALIDATE METADATA</codeph> statements:
+ </p>
+
+ <ul conref="../shared/impala_common.xml#common/catalogd_xrefs">
+ <li/>
+ </ul>
+
+ <p conref="../shared/impala_common.xml#common/cm48_upgrade"/>
+
+<!-- <note conref="common.xml#common/cdh4_cdh5_upgrade"/> -->
+ </conbody>
+ </concept>
+
+ <concept rev="1.2" id="incompatible_changes_120">
+
+ <title>Incompatible Changes Introduced in Impala 1.2.0 (Beta)</title>
+
+ <conbody>
+
+ <p>
+ There are no incompatible changes to SQL syntax in Impala 1.2.0 (beta).
+ </p>
+
+ <p>
+ Because Impala 1.2.0 is bundled with the CDH 5 beta download and depends on specific levels of Apache
+ Hadoop components supplied with CDH 5, you can only install it in combination with the CDH 5 beta.
+ </p>
+
+ <p>
+ The new <cmdname>catalogd</cmdname> service might require changes to any user-written scripts that stop,
+ start, or restart Impala services, install or upgrade Impala packages, or issue <codeph>REFRESH</codeph> or
+ <codeph>INVALIDATE METADATA</codeph> statements:
+ </p>
+
+ <ul conref="../shared/impala_common.xml#common/catalogd_xrefs">
+ <li/>
+ </ul>
+
+ <p>
+ The new resource management feature interacts with both YARN and Llama services, which are available in CDH
+ 5. These services are set up for you automatically in a Cloudera Manager (CM) environment. For information
+ about setting up the YARN and Llama services, see the instructions for
+<!-- Original URL: http://www.cloudera.com/content/cloudera-content/cloudera-docs/CDH5/latest/CDH5-Installation-Guide/cdh_ig_yarn_cluster_deploy.html -->
+ <xref href="http://www.cloudera.com/documentation/enterprise/latest/topics/cdh_ig_yarn_cluster_deploy.html" scope="external" format="html">YARN</xref>
+ and
+<!-- Original URL: http://www.cloudera.com/content/cloudera-content/cloudera-docs/CDH5/latest/CDH5-Installation-Guide/cdh_ig_llama_installation.html -->
+<!-- Then this, which was removed: http://www.cloudera.com/documentation/enterprise/latest/topics/cdh_ig_llama_installation.html -->
+ <xref href="http://www.cloudera.com/documentation/enterprise/latest/topics/admin_llama.html" scope="external" format="html">Llama</xref>
+ in the <cite>CDH 5 Documentation</cite>. <ph audience="PDF">See
+ <xref href="impala_resource_management.xml#resource_management"/> for usage information for Impala resource
+ management.</ph>
+ </p>
+ </conbody>
+ </concept>
+
+ <concept id="incompatible_changes_111">
+
+ <title>Incompatible Changes Introduced in Impala 1.1.1</title>
+
+ <conbody>
+
+ <p>
+ There are no incompatible changes in Impala 1.1.1.
+ </p>
+
+<!-- These couple of paragraphs were originally intended to be conref'ed from the Parquet section of Installing/Using. -->
+
+<!-- But conbodydiv tag too restrictive, can't have just paragraphs and codeblocks inside. -->
+
+<!-- So I will physically copy the info for the time being. -->
+
+<!-- Also copying it under the Upgrading topic. -->
+
+<!-- <conbodydiv conref="impala_parquet.xml#upgrade_parquet_metadata"/> -->
+
+ <p>
+ Previously, it was not possible to create Parquet data through Impala and reuse that table within Hive. Now
+ that Parquet support is available for Hive 10, reusing existing Impala Parquet data files in Hive requires
+ updating the table metadata. Use the following command if you are already running Impala 1.1.1:
+ </p>
+
+<codeblock>ALTER TABLE <varname>table_name</varname> SET FILEFORMAT PARQUETFILE;
+</codeblock>
+
+ <p>
+ If you are running a level of Impala that is older than 1.1.1, do the metadata update through Hive:
+ </p>
+
+<codeblock>ALTER TABLE <varname>table_name</varname> SET SERDE 'parquet.hive.serde.ParquetHiveSerDe';
+ALTER TABLE <varname>table_name</varname> SET FILEFORMAT
+ INPUTFORMAT "parquet.hive.DeprecatedParquetInputFormat"
+ OUTPUTFORMAT "parquet.hive.DeprecatedParquetOutputFormat";
+</codeblock>
+
+ <p>
+ Impala 1.1.1 and higher can reuse Parquet data files created by Hive, without any action required.
+ </p>
+
+ <p>
+ As usual, make sure to upgrade the <codeph>impala-lzo-cdh4</codeph> package to the latest level at the same
+ time as you upgrade the Impala server.
+ </p>
+ </conbody>
+ </concept>
+
+ <concept id="incompatible_changes_11">
+
+ <title>Incompatible Change Introduced in Impala 1.1</title>
+
+ <conbody>
+
+ <ul>
+ <li>
+ <p>
+ The <codeph>REFRESH</codeph> statement now requires a table name; in Impala 1.0, the table name was
+ optional. This syntax change is part of the internal rework to make <codeph>REFRESH</codeph> a true
+ Impala SQL statement so that it can be called through the JDBC and ODBC APIs. <codeph>REFRESH</codeph>
+ now reloads the metadata immediately, rather than marking it for update the next time any affected
+ table is accessed. The previous behavior, where omitting the table name caused a refresh of the entire
+ Impala metadata catalog, is available through the new <codeph>INVALIDATE METADATA</codeph> statement.
+ <codeph>INVALIDATE METADATA</codeph> can be specified with a table name to affect a single table, or
+ without a table name to affect the entire metadata catalog; the relevant metadata is reloaded the next
+ time it is requested during the processing for a SQL statement. See
+ <xref href="impala_refresh.xml#refresh"/> and
+ <xref href="impala_invalidate_metadata.xml#invalidate_metadata"/> for the latest details about these
+ statements.
+ </p>
+ </li>
+ </ul>
+ </conbody>
+ </concept>
+
+ <concept id="incompatible_changes_10">
+
+ <title>Incompatible Changes Introduced in Impala 1.0</title>
+
+ <conbody>
+
+ <ul>
+ <li>
+ If you use LZO-compressed text files, when you upgrade Impala to version 1.0, also update the
+ <codeph>impala-lzo-cdh4</codeph> to the latest level. See <xref href="impala_txtfile.xml#lzo"/> for
+ details.
+ </li>
+
+ <li>
+ Cloudera Manager 4.5.2 and higher only supports Impala 1.0 and higher, and vice versa. If you upgrade to
+ Impala 1.0 or higher managed by Cloudera Manager, you must also upgrade Cloudera Manager to version 4.5.2
+ or higher. If you upgrade from an earlier version of Cloudera Manager, and were using Impala, you must
+ also upgrade Impala to version 1.0 or higher. The beta versions of Impala are no longer supported as of
+ the release of Impala 1.0.
+ </li>
+ </ul>
+ </conbody>
+ </concept>
+
+ <concept id="incompatible_changes_07">
+
+ <title>Incompatible Change Introduced in Version 0.7 of the Impala Beta Release</title>
+
+ <conbody>
+
+ <ul>
+ <li>
+ The defaults for the <codeph>-nn</codeph> and <codeph>-nn_port</codeph> flags have changed and are now
+ read from <codeph>core-site.xml</codeph>. Impala prints the values of <codeph>-nn</codeph> and
+ <codeph>-nn_port</codeph> to the log when it starts. The ability to set <codeph>-nn</codeph> and
+ <codeph>-nn_port</codeph> on the command line is deprecated in 0.7 and may be removed in Impala 0.8.
+ </li>
+ </ul>
+ </conbody>
+ </concept>
+
+ <concept id="incompatible_changes_06">
+
+ <title>Incompatible Change Introduced in Version 0.6 of the Impala Beta Release</title>
+
+ <conbody>
+
+ <ul>
+ <li>
+ Cloudera Manager 4.5 supports only version 0.6 of the Impala Beta Release. It does not support
+ the earlier beta versions. If you upgrade your Cloudera Manager installation, you must also upgrade
+ Impala to beta version 0.6. If you upgrade Impala to beta version 0.6, you must upgrade Cloudera Manager
+ to 4.5.
+ </li>
+ </ul>
+ </conbody>
+ </concept>
+
+ <concept id="incompatible_changes_04">
+
+ <title>Incompatible Change Introduced in Version 0.4 of the Impala Beta Release</title>
+
+ <conbody>
+
+ <ul>
+ <li>
+ Cloudera Manager 4.1.3 supports only version 0.4 of the Impala Beta Release. It does not support
+ the earlier beta versions. If you upgrade your Cloudera Manager installation, you must also upgrade
+ Impala to beta version 0.4. If you upgrade Impala to beta version 0.4, you must upgrade Cloudera Manager
+ to 4.1.3.
+ </li>
+ </ul>
+ </conbody>
+ </concept>
+
+ <concept id="incompatible_changes_03">
+
+ <title>Incompatible Change Introduced in Version 0.3 of the Impala Beta Release</title>
+
+ <conbody>
+
+ <ul>
+ <li>
+ Cloudera Manager 4.1.2 supports only version 0.3 of the Impala Beta Release. It does not support
+ the earlier beta versions. If you upgrade your Cloudera Manager installation, you must also upgrade
+ Impala to beta version 0.3. If you upgrade Impala to beta version 0.3, you must upgrade Cloudera Manager
+ to 4.1.2.
+ </li>
+ </ul>
+ </conbody>
+ </concept>
+</concept>