You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@impala.apache.org by jr...@apache.org on 2016/11/01 23:13:13 UTC
[15/23] incubator-impala git commit: Update all impala* files to the
latest CDH 5.9/5.10 versions.
http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3c2c8f12/docs/topics/impala_incompatible_changes.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_incompatible_changes.xml b/docs/topics/impala_incompatible_changes.xml
index 1814553..a7bafcd 100644
--- a/docs/topics/impala_incompatible_changes.xml
+++ b/docs/topics/impala_incompatible_changes.xml
@@ -3,7 +3,19 @@
<concept rev="ver" id="incompatible_changes">
<title><ph audience="standalone">Incompatible Changes and Limitations in Apache Impala (incubating)</ph><ph audience="integrated">Apache Impala (incubating) Incompatible Changes and Limitations</ph></title>
-
+ <prolog>
+ <metadata>
+ <data name="Category" value="Impala"/>
+ <data name="Category" value="Release Notes"/>
+ <data name="Category" value="Incompatible Changes"/>
+ <data name="Category" value="Limitations"/>
+ <data name="Category" value="Upgrading"/>
+ <data name="Category" value="Troubleshooting"/>
+ <data name="Category" value="Administrators"/>
+ <data name="Category" value="Developers"/>
+ <data name="Category" value="Data Analysts"/>
+ </metadata>
+ </prolog>
<conbody>
@@ -13,7 +25,1495 @@
configuration, dependencies, or prerequisites that could cause issues during or after an Impala upgrade.
</p>
-
+ <p>
+ Even added SQL statements or clauses can produce incompatibilities, if you have databases, tables, or columns
+ whose names conflict with the new keywords. <ph audience="PDF">See
+ <xref href="impala_reserved_words.xml#reserved_words"/> for the set of reserved words for the current
+ release, and the quoting techniques to avoid name conflicts.</ph>
+ </p>
+
+ <p outputclass="toc inpage"/>
+ </conbody>
+
+ <concept rev="2.7.0" id="incompatible_changes_27x">
+
+ <title>Incompatible Changes Introduced in Impala for CDH 5.9.x / Impala 2.7.x</title>
+
+ <conbody>
+ <ul>
+ <li>
+ <p rev="IMPALA-1731 IMPALA-3868 CDH-43734">
+ Bug fixes related to parsing of floating-point values (IMPALA-1731 and IMPALA-3868) can change
+ the results of casting strings that represent invalid floating-point values.
+ For example, formerly a string value beginning or ending with <codeph>inf</codeph>,
+ such as <codeph>1.23inf</codeph> or <codeph>infinite</codeph>, now are converted to <codeph>NULL</codeph>
+ when interpreted as a floating-point value.
+ Formerly, they were interpreted as the special <q>infinity</q> value when converting from string to floating-point.
+ Similarly, now only the string <codeph>NaN</codeph> (case-sensitive) is interpreted as the special <q>not a number</q>
+ value. String values containing multiple dots, such as <codeph>3..141</codeph> or <codeph>3.1.4.1</codeph>,
+ are now interpreted as <codeph>NULL</codeph> rather than being converted to valid floating-point values.
+ </p>
+ </li>
+ <li>
+ <p rev="IMPALA-4372">
+ The column types shown in the <codeph>DESCRIBE FORMATTED</codeph> output are in uppercase, where formerly they
+ were in lowercase. This is not an intended change and could be reverted in the future, when IMPALA-4372
+ is resolved.
+ </p>
+ </li>
+ </ul>
+ </conbody>
+
+ </concept>
+
+ <concept rev="2.6.0" id="incompatible_changes_26x">
+
+ <title>Incompatible Changes Introduced in Impala for CDH 5.8.x / Impala 2.6.x</title>
+
+ <conbody>
+ <ul>
+ <li>
+ <p rev="CDH-41184">
+ The default for the <codeph>RUNTIME_FILTER_MODE</codeph>
+ query option is changed to <codeph>GLOBAL</codeph> (the highest setting).
+ </p>
+ </li>
+ <li rev="CDH-41184 IMPALA-3007">
+ <p>
+ The <codeph>RUNTIME_BLOOM_FILTER_SIZE</codeph> setting is now only used
+ as a fallback if statistics are not available; otherwise, Impala
+ uses the statistics to estimate the appropriate size to use for each filter.
+ </p>
+ </li>
+ <li>
+ <p rev="IMPALA-3199">
+ Admission control and dynamic resource pools are enabled by default.
+ When upgrading from an earlier release, you must turn on these settings yourself
+ if they are not already enabled.
+ See <xref href="impala_admission.xml#admission_control"/> for details
+ about admission control.
+ </p>
+ </li>
+ <li>
+ <p>
+ Impala reserves some new keywords, in preparation for support for Kudu syntax:
+ <codeph>buckets</codeph>, <codeph>delete</codeph>, <codeph>distribute</codeph>,
+ <codeph>hash</codeph>, <codeph>ignore</codeph>, <codeph>split</codeph>, and <codeph>update</codeph>.
+ </p>
+ </li>
+ <li>
+ <p rev="IMPALA-3554">
+ For Kerberized clusters, the Catalog service now uses
+ the Kerberos principal instead of the operating sytem user that runs
+ the <cmdname>catalogd</cmdname> daemon.
+ This eliminates the requirement to configure a <codeph>hadoop.user.group.static.mapping.overrides</codeph>
+ setting to put the OS user into the Sentry administrative group, on clusters where the principal
+ and the OS user name for this user are different.
+ </p>
+ </li>
+ <li>
+ <p>
+ The mechanism for interpreting <codeph>DECIMAL</codeph> literals is
+ improved, no longer going through an intermediate conversion step
+ to <codeph>DOUBLE</codeph>:
+ </p>
+ <ul>
+ <li>
+ <p rev="IMPALA-3163">
+ Casting a <codeph>DECIMAL</codeph> value to <codeph>TIMESTAMP</codeph>
+ <codeph>DOUBLE</codeph> produces a more precise
+ value for the <codeph>TIMESTAMP</codeph> than formerly.
+ </p>
+ </li>
+ <li>
+ <p rev="IMPALA-3439">
+ Certain function calls involving <codeph>DECIMAL</codeph> literals
+ now succeed, when formerly they failed due to lack of a function
+ signature with a <codeph>DOUBLE</codeph> argument.
+ </p>
+ </li>
+ </ul>
+ </li>
+ <li>
+ <p rev="IMPALA-3155">
+ Improved type accuracy for <codeph>CASE</codeph> return values.
+ If all <codeph>WHEN</codeph> clauses of the <codeph>CASE</codeph>
+ expression are of <codeph>CHAR</codeph> type, the final result
+ is also <codeph>CHAR</codeph> instead of being converted to
+ <codeph>STRING</codeph>.
+ </p>
+ </li>
+ <li>
+ <p conref="../shared/impala_common.xml#common/IMPALA-3662"/>
+ </li>
+ <li rev="IMPALA-3452 CDH-39913">
+ <p>
+ The <codeph>S3_SKIP_INSERT_STAGING</codeph> query option, which is enabled by
+ default, increases the speed of <codeph>INSERT</codeph> operations for S3 tables.
+ The speedup applies to regular <codeph>INSERT</codeph>, but not <codeph>INSERT OVERWRITE</codeph>.
+ The tradeoff is the possibility of inconsistent output files left behind if a
+ node fails during <codeph>INSERT</codeph> execution.
+ See <xref href="impala_s3_skip_insert_staging.xml#s3_skip_insert_staging"/> for details.
+ </p>
+ </li>
+ </ul>
+ <p>
+ Certain features are turned off by default, to avoid regressions or unexpected
+ behavior following an upgrade. Consider turning on these features after suitable testing:
+ </p>
+ <ul>
+ <li>
+ <p rev="IMPALA-2660 CDH-40241">
+ Impala now recognizes the <codeph>auth_to_local</codeph> setting,
+ specified through the HDFS configuration setting
+ <codeph>hadoop.security.auth_to_local</codeph>.
+ This feature is disabled by default; to enable it,
+ specify <codeph>--load_auth_to_local_rules=true</codeph>
+ in the <cmdname>impalad</cmdname> configuration settings.
+ </p>
+ </li>
+ <li>
+ <p rev="IMPALA-2069">
+ A new query option, <codeph>PARQUET_ANNOTATE_STRINGS_UTF8</codeph>,
+ makes Impala include the <codeph>UTF-8</codeph> annotation
+ metadata for <codeph>STRING</codeph>, <codeph>CHAR</codeph>,
+ and <codeph>VARCHAR</codeph> columns in Parquet files created
+ by <codeph>INSERT</codeph> or <codeph>CREATE TABLE AS SELECT</codeph>
+ statements.
+ </p>
+ </li>
+ <li>
+ <p rev="IMPALA-2835 CDH-33330">
+ A new query option,
+ <codeph>PARQUET_FALLBACK_SCHEMA_RESOLUTION</codeph>,
+ lets Impala locate columns within Parquet files based on
+ column name rather than ordinal position.
+ This enhancement improves interoperability with applications
+ that write Parquet files with a different order or subset of
+ columns than are used in the Impala table.
+ </p>
+ </li>
+ </ul>
+ </conbody>
+
+ </concept>
+
+ <concept rev="2.5.x" id="incompatible_changes_25x">
+
+ <title>Incompatible Changes Introduced in Impala for CDH 5.7.x / Impala 2.5.x</title>
+
+ <conbody>
+ <ul>
+ <li rev="IMPALA-3044">
+ <p>
+ The admission control default limit for concurrent queries (the <uicontrol>max requests</uicontrol>
+ setting) is now unlimited instead of 200.
+ </p>
+ </li>
+
+ <li>
+ <p rev="IMPALA-2749">
+ Multiplying a mixture of <codeph>DECIMAL</codeph> and <codeph>FLOAT</codeph> or
+ <codeph>DOUBLE</codeph> values now returns
+ <codeph>DOUBLE</codeph> rather than <codeph>DECIMAL</codeph>. This
+ change avoids some cases where an intermediate value would underflow or overflow
+ and become <codeph>NULL</codeph> unexpectedly. The results of
+ multiplying <codeph>DECIMAL</codeph> and <codeph>FLOAT</codeph> or
+ <codeph>DOUBLE</codeph> might now be slightly less precise than
+ before. Previously, the intermediate types and thus the final result
+ depended on the exact order of the values of different types being
+ multiplied, which made the final result values difficult to
+ reason about.
+ </p>
+ </li>
+ <li rev="IMPALA-2204 CDH-33139">
+ <p>
+ Previously, the <codeph>_</codeph> and <codeph>%</codeph> wildcard
+ characters for the <codeph>LIKE</codeph> operator would not match
+ characters on the second or subsequent lines of multi-line string values. The fix for issue
+ <xref href="https://issues.cloudera.org/browse/IMPALA-2204" scope="external" format="html">IMPALA-2204</xref> causes
+ the wildcard matching to apply to the entire string for values
+ containing embedded <codeph>\n</codeph> characters. This could cause
+ different results than in previous Impala releases for identical
+ queries on identical data.
+ </p>
+ </li>
+ <li rev="IMPALA-1748 CDH-38369">
+ <p>
+ Formerly, all Impala UDFs and UDAs required running the
+ <codeph>CREATE FUNCTION</codeph> statements to
+ re-create them after each <cmdname>catalogd</cmdname> restart.
+ In CDH 5.7 / Impala 2.5 and higher, functions written in C++ are persisted across
+ restarts, and the requirement to
+ re-create functions only applies to functions written in Java. Adapt any
+ function-reloading logic that you have added to your Impala environment.
+ </p>
+ </li>
+ <li>
+ <p rev="IMPALA-1651">
+ <codeph>CREATE TABLE LIKE</codeph> no longer inherits HDFS caching settings from the source table.
+ </p>
+ </li>
+ <li>
+ <p rev="IMPALA-2070">
+ The <codeph>SHOW DATABASES</codeph> statement now returns two columns rather than one.
+ The second column includes the associated comment string, if any, for each database.
+ Adjust any application code that examines the list of databases and assumes the
+ result set contains only a single column.
+ </p>
+ </li>
+ <li>
+ <p>
+ The output of the <codeph>SHOW FUNCTIONS</codeph> statement includes
+ two new columns, showing the kind of the function (for example,
+ <codeph>BUILTIN</codeph>) and whether or not the function persists
+ across catalog server restarts. For example, the <codeph>SHOW
+ FUNCTIONS</codeph> output for the
+ <codeph>_impala_builtins</codeph> database starts with:
+ </p>
+<codeblock>
++--------------+-------------------------------------------------+-------------+---------------+
+| return type | signature | binary type | is persistent |
++--------------+-------------------------------------------------+-------------+---------------+
+| BIGINT | abs(BIGINT) | BUILTIN | true |
+| DECIMAL(*,*) | abs(DECIMAL(*,*)) | BUILTIN | true |
+| DOUBLE | abs(DOUBLE) | BUILTIN | true |
+...
+</codeblock>
+ </li>
+ </ul>
+ </conbody>
+
+ </concept>
+
+ <concept rev="2.4.x" id="incompatible_changes_24x">
+
+ <title>Incompatible Changes Introduced in Impala for CDH 5.6.x / Impala 2.4.x</title>
+
+ <conbody>
+ <p>
+ Other than support for DSSD storage, the Impala feature set for CDH 5.6 is the same as for CDH 5.5.
+ Therefore, there are no incompatible changes for Impala introduced in CDH 5.6.
+ </p>
+ </conbody>
+
+ </concept>
+
+<!-- All 2.3.x subsections go under here -->
+
+<!-- Actually for 2.3 / 5.5, let's get away from doing a separate subhead for each maintenance release,
+ because in the normal course of events there will be nothing to add here until 5.6. If something new
+ needs to get noted, just add a new bullet with wording to indicate which 5.5.x release it applies to. -->
+
+ <concept rev="2.3.x" id="incompatible_changes_23x">
+
+ <title>Incompatible Changes Introduced in Impala for CDH 5.5.x / Impala 2.3.x</title>
+
+ <conbody>
+
+ <note conref="../shared/impala_common.xml#common/impala_llama_obsolete"/>
+
+ <ul>
+ <li rev="IMPALA-2005" audience="Cloudera">
+ <p>
+ If a <codeph>CREATE TABLE AS SELECT</codeph> operation fails while data is being inserted,
+ the table is automatically removed. Previously, the table was left behind with no data.
+ </p>
+ </li>
+ <li rev="IMPALA-2130">
+ <p>
+ If Impala encounters a Parquet file that is invalid because of an incorrect magic number,
+ the query skips the file. This change is caused by the fix for issue <xref href="https://issues.cloudera.org/browse/IMPALA-2130" scope="external" format="html">IMPALA-2130</xref>.
+ Previously, Impala would attempt to read the file despite the possibility that the file was corrupted.
+ </p>
+ </li>
+ <li rev="IMPALA-2233 CDH-33145">
+ <p>
+ Previously, calls to overloaded built-in functions could treat parameters as <codeph>DOUBLE</codeph>
+ or <codeph>FLOAT</codeph> when no overload had a signature that matched the exact argument types.
+ Now Impala prefers the function signature with <codeph>DECIMAL</codeph> parameters in this case.
+ This change avoids a possible loss of precision in function calls such as <codeph>greatest(0, 99999.8888)</codeph>;
+ now both parameters are treated as <codeph>DECIMAL</codeph> rather than <codeph>DOUBLE</codeph>, avoiding
+ any loss of precision in the fractional value.
+ This could cause slightly different results than in previous Impala releases for certain function calls.
+ </p>
+ </li>
+ <li rev="IMPALA-1675">
+ <p>
+ Formerly, adding or subtracting a large interval value to a <codeph>TIMESTAMP</codeph> could produce
+ a nonsensical result. Now when the result goes outside the range of <codeph>TIMESTAMP</codeph> values,
+ Impala returns <codeph>NULL</codeph>.
+ </p>
+ </li>
+ <li rev="IMPALA-2251 IMPALA-2257">
+ <p>
+ Formerly, it was possible to accidentally create a table with identical row and column delimiters.
+ This could happen unintentionally, when specifying one of the delimiters and using the
+ default value for the other. Now an attempt to use identical delimiters still succeeds,
+ but displays a warning message.
+ </p>
+ </li>
+ <li rev="CDH-28071">
+ <p>
+ Formerly, Impala could include snippets of table data in log files by default, for example
+ when reporting conversion errors for data values. Now any such log messages are only produced
+ at higher logging levels that you would enable only during debugging.
+ </p>
+ </li>
+<!-- placeholder -->
+ </ul>
+ </conbody>
+
+ </concept>
+
+<!-- All 2.2.x subsections go under here -->
+
+ <concept rev="2.2.x" id="incompatible_changes_22x">
+
+ <title>Incompatible Changes Introduced in Impala for CDH 5.4.x</title>
+
+ <conbody>
+
+<!--
+ <p>
+ No incompatible changes. CDH maintenance releases such as 5.4.1, 5.4.2, and so on are exclusively bug fix releases.
+ See <xref href="impala_incompatible_changes.xml#incompatible_changes_220"/> for the initial Impala feature release,
+ which are the releases that typically include incompatible changes.
+ </p>
+
+ <note conref="../shared/impala_common.xml#common/only_cdh5_22x"/>
+
+ </conbody>
+ </concept>
+
+ <concept rev="2.2.0" id="incompatible_changes_220">
+
+ <title>Incompatible Changes Introduced in Impala 2.2.0 / CDH 5.4.0</title>
+
+ <conbody>
+-->
+
+ <note conref="../shared/impala_common.xml#common/only_cdh5_220"/>
+
+ <section id="files_220">
+ <title>
+ Changes to File Handling
+ </title>
+ <p conref="../shared/impala_common.xml#common/ignore_file_extensions"/>
+ <p>
+ The log rotation feature in Impala 2.2.0 and higher
+ means that older log files are now removed by default.
+ The default is to preserve the latest 10 log files for each
+ severity level, for each Impala-related daemon. If you have
+ set up your own log rotation processes that expect older
+ files to be present, either adjust your procedures or
+ change the Impala <codeph>-max_log_files</codeph> setting.
+ <ph audience="PDF">See <xref href="impala_logging.xml#logs_rotate"/> for details.</ph>
+ </p>
+ </section>
+
+
+ <section id="prereqs_210">
+ <title>
+ Changes to Prerequisites
+ </title>
+ <p conref="../shared/impala_common.xml#common/cpu_prereq"/>
+ </section>
+
+ </conbody>
+ </concept>
+
+<!-- All 2.1.x subsections go under here -->
+
+ <concept rev="2.1.x" id="incompatible_changes_21x">
+
+ <title>Incompatible Changes Introduced in Impala for CDH 5.3.x</title>
+
+ <conbody>
+
+<!--
+ <p>
+ No incompatible changes. CDH maintenance releases such as 5.3.1, 5.3.2, and so on are exclusively bug fix releases.
+ See <xref href="impala_incompatible_changes.xml#incompatible_changes_210"/> for the initial Impala feature release,
+ which are the releases that typically include incompatible changes.
+ </p>
+
+ <note conref="../shared/impala_common.xml#common/only_cdh5_21x"/>
+
+ </conbody>
+ </concept>
+
+ <concept rev="2.1.7" id="incompatible_changes_217">
+ <title>Incompatible Changes Introduced in Cloudera Impala 2.1.7 / CDH 5.3.9</title>
+ <conbody>
+ <p>
+ No incompatible changes.
+ </p>
+
+ <note conref="../shared/impala_common.xml#common/only_cdh5_21x"/>
+
+ </conbody>
+ </concept>
+
+ <concept rev="2.1.6" id="incompatible_changes_216">
+ <title>Incompatible Changes Introduced in Impala 2.1.6 / CDH 5.3.8</title>
+ <conbody>
+ <p>
+ No incompatible changes.
+ </p>
+
+ <note conref="../shared/impala_common.xml#common/only_cdh5_21x"/>
+
+ </conbody>
+ </concept>
+
+ <concept rev="2.1.5" id="incompatible_changes_215">
+ <title>Incompatible Changes Introduced in Impala 2.1.5 / CDH 5.3.6</title>
+ <conbody>
+ <p>
+ No incompatible changes.
+ </p>
+
+ <note conref="../shared/impala_common.xml#common/only_cdh5_21x"/>
+
+ </conbody>
+ </concept>
+
+ <concept rev="2.1.4" id="incompatible_changes_214">
+ <title>Incompatible Changes Introduced in Impala 2.1.4 / CDH 5.3.4</title>
+ <conbody>
+ <p>
+ No incompatible changes.
+ <ph conref="../shared/impala_common.xml#common/impala_214_redux"/>
+ </p>
+
+ <note conref="../shared/impala_common.xml#common/only_cdh5_21x"/>
+
+ </conbody>
+ </concept>
+
+ <concept rev="2.1.3" id="incompatible_changes_213">
+ <title>Incompatible Changes Introduced in Impala 2.1.3 / CDH 5.3.3</title>
+ <conbody>
+ <p>
+ No incompatible changes.
+ </p>
+
+ <note conref="../shared/impala_common.xml#common/only_cdh5_213"/>
+
+ </conbody>
+ </concept>
+
+ <concept rev="2.1.2" id="incompatible_changes_212">
+ <title>Incompatible Changes Introduced in Impala 2.1.2 / CDH 5.3.2</title>
+ <conbody>
+ <p>
+ No incompatible changes.
+ </p>
+
+ <note conref="../shared/impala_common.xml#common/only_cdh5_212"/>
+
+ </conbody>
+ </concept>
+
+ <concept rev="2.1.1" id="incompatible_changes_211">
+
+ <title>Incompatible Changes Introduced in Impala 2.1.1 / CDH 5.3.1</title>
+
+ <conbody>
+
+ <p>
+ No incompatible changes.
+ </p>
+ </conbody>
+ </concept>
+
+ <concept rev="2.1.0" id="incompatible_changes_210">
+
+ <title>Incompatible Changes Introduced in Impala 2.1.0 / CDH 5.3.0</title>
+
+ <conbody>
+-->
+
+ <section id="prereqs_210">
+ <title>
+ Changes to Prerequisites
+ </title>
+ <p rev="CDH-24874">
+ Currently, Impala 2.1.x does not function on CPUs without the SSE4.1 instruction set. This minimum CPU
+ requirement is higher than in previous versions, which relied on the older SSSE3 instruction set. Check
+ the CPU level of the hosts in your cluster before upgrading to Impala 2.1.x or CDH 5.3.x.
+ </p>
+ </section>
+
+ <section id="output_format_210">
+ <title>
+ Changes to Output Format
+ </title>
+ <p>
+ The <q>small query</q> optimization feature introduces some new information in the
+ <codeph>EXPLAIN</codeph> plan, which you might need to account for if you parse the text of the plan
+ output.
+ </p>
+ </section>
+
+ <section id="reserved_words_210">
+ <title>
+ New Reserved Words
+ </title>
+ <p>
+ New SQL syntax introduces additional reserved words:
+ <codeph>FOR</codeph>, <codeph>GRANT</codeph>, <codeph>REVOKE</codeph>, <codeph>ROLE</codeph>, <codeph>ROLES</codeph>,
+ <codeph>INCREMENTAL</codeph>.
+ <ph audience="PDF">As always, see <xref href="impala_reserved_words.xml#reserved_words"/>
+ for the set of reserved words for the current release, and the quoting techniques to avoid name conflicts.</ph>
+ </p>
+ </section>
+ </conbody>
+ </concept>
+
+<!-- All 2.0.x subsections go under here -->
+
+ <concept rev="2.0.5" id="incompatible_changes_205">
+
+ <title>Incompatible Changes Introduced in Impala 2.0.5 / CDH 5.2.6</title>
+
+ <conbody>
+
+ <p>
+ No incompatible changes.
+ </p>
+
+ <note conref="../shared/impala_common.xml#common/only_cdh5_205"/>
+
+ </conbody>
+ </concept>
+
+ <concept rev="2.0.4" id="incompatible_changes_204">
+
+ <title>Incompatible Changes Introduced in Impala 2.0.4 / CDH 5.2.5</title>
+
+ <conbody>
+
+ <p>
+ No incompatible changes.
+ </p>
+
+ <note conref="../shared/impala_common.xml#common/only_cdh5_204"/>
+
+ </conbody>
+ </concept>
+
+ <concept rev="2.0.3" id="incompatible_changes_203">
+
+ <title>Incompatible Changes Introduced in Impala 2.0.3 / CDH 5.2.4</title>
+
+ <conbody>
+
+ <note conref="../shared/impala_common.xml#common/only_cdh5_203"/>
+
+ </conbody>
+ </concept>
+
+ <concept rev="2.0.2" id="incompatible_changes_202">
+
+ <title>Incompatible Changes Introduced in Impala 2.0.2 / CDH 5.2.3</title>
+
+ <conbody>
+
+ <p>
+ No incompatible changes.
+ </p>
+
+ <note conref="../shared/impala_common.xml#common/only_cdh5_202"/>
+
+ </conbody>
+ </concept>
+
+ <concept rev="2.0.1" id="incompatible_changes_201">
+
+ <title>Incompatible Changes Introduced in Impala 2.0.1 / CDH 5.2.1</title>
+
+ <conbody>
+
+ <ul>
+ <li>
+ <p conref="../shared/impala_common.xml#common/insert_hidden_work_directory"/>
+ </li>
+
+ <li>
+ <p>
+ The <codeph>abs()</codeph> function now takes a broader range of numeric types as arguments, and the
+ return type is the same as the argument type.
+ </p>
+ </li>
+
+ <li>
+ <p>
+ Shorthand notation for character classes in regular expressions, such as <codeph>\d</codeph> for digit,
+ are now available again in regular expression operators and functions such as
+ <codeph>regexp_extract()</codeph> and <codeph>regexp_replace()</codeph>. Some other differences in
+ regular expression behavior remain between Impala 1.x and Impala 2.x releases. See
+ <xref href="impala_incompatible_changes.xml#incompatible_changes_200"/> for details.
+ </p>
+ </li>
+ </ul>
+ </conbody>
+ </concept>
+
+ <concept rev="2.0.0" id="incompatible_changes_200">
+
+ <title>Incompatible Changes Introduced in Impala 2.0.0 / CDH 5.2.0</title>
+
+ <conbody>
+
+ <section id="prereqs_200">
+ <title>
+ Changes to Prerequisites
+ </title>
+ <p rev="CDH-24874">
+ Currently, Impala 2.0.x does not function on CPUs without the SSE4.1 instruction set. This minimum CPU
+ requirement is higher than in previous versions, which relied on the older SSSE3 instruction set. Check
+ the CPU level of the hosts in your cluster before upgrading to Impala 2.0.x or CDH 5.2.x.
+ </p>
+ </section>
+
+ <section id="queries_200">
+ <title>
+ Changes to Query Syntax
+ </title>
+
+ <p>
+ The new syntax where query hints are allowed in comments causes some changes in the way comments are
+ parsed in the <cmdname>impala-shell</cmdname> interpreter. Previously, you could end a
+ <codeph>--</codeph> comment line with a semicolon and <cmdname>impala-shell</cmdname> would treat that
+ as a no-op statement. Now, a comment line ending with a semicolon is passed as an empty statement to
+ the Impala daemon, where it is flagged as an error.
+ </p>
+
+ <p>
+ Impala 2.0 and later uses a different support library for regular expression parsing than in earlier
+ Impala versions. Now, Impala uses the
+ <xref href="https://code.google.com/p/re2/" scope="external" format="html">Google RE2 library</xref>
+ rather than Boost for evaluating regular expressions. This implementation change causes some
+ differences in the allowed regular expression syntax, and in the way certain regex operators are
+ interpreted. The following are some of the major differences (not necessarily a complete list):
+ </p>
+ <ul>
+ <li>
+ <p>
+ <codeph>.*?</codeph> notation for non-greedy matches is now supported, where it was not in earlier
+ Impala releases.
+ </p>
+ </li>
+
+ <li>
+ <p>
+ By default, <codeph>^</codeph> and <codeph>$</codeph> now match only begin/end of buffer, not
+ begin/end of each line. This behavior can be overridden in the regex itself using the
+ <codeph>m</codeph> flag.
+ </p>
+ </li>
+
+ <li>
+ <p>
+ By default, <codeph>.</codeph> does not match newline. This behavior can be overridden in the regex
+ itself using the <codeph>s</codeph> flag.
+ </p>
+ </li>
+
+ <li>
+ <p>
+ <codeph>\Z</codeph> is not supported.
+ </p>
+ </li>
+
+ <li>
+ <p>
+ <codeph><</codeph> and <codeph>></codeph> for start of word and end of word are not
+ supported.
+ </p>
+ </li>
+
+ <li>
+ <p>
+ Lookahead and lookbehind are not supported.
+ </p>
+ </li>
+
+ <li>
+ <p>
+ Shorthand notation for character classes, such as <codeph>\d</codeph> for digit, is not recognized.
+ (This restriction is lifted in Impala 2.0.1, which restores the shorthand notation.)
+ </p>
+ </li>
+ </ul>
+ </section>
+
+ <section id="output_format_210">
+ <title>
+ Changes to Output Format
+ </title>
+
+ <p conref="../shared/impala_common.xml#common/user_kerberized"/>
+
+ <p>
+ The changed format for the user name in secure environments is also reflected where the user name is
+ displayed in the output of the <codeph>PROFILE</codeph> command.
+ </p>
+
+ <p>
+ In the output from <codeph>SHOW FUNCTIONS</codeph>, <codeph>SHOW AGGREGATE FUNCTIONS</codeph>, and
+ <codeph>SHOW ANALYTIC FUNCTIONS</codeph>, arguments and return types of arbitrary
+ <codeph>DECIMAL</codeph> scale and precision are represented as <codeph>DECIMAL(*,*)</codeph>.
+ Formerly, these items were displayed as <codeph>DECIMAL(-1,-1)</codeph>.
+ </p>
+
+ </section>
+
+ <section id="query_options_200">
+ <title>
+ Changes to Query Options
+ </title>
+ <p>
+ The <codeph>PARQUET_COMPRESSION_CODEC</codeph> query option has been replaced by the
+ <codeph>COMPRESSION_CODEC</codeph> query option.
+ <ph audience="PDF">See <xref href="impala_compression_codec.xml#compression_codec"/> for details.</ph>
+ </p>
+ </section>
+
+ <section id="config_options_200">
+ <title>
+ Changes to Configuration Options
+ </title>
+
+ <p>
+ The meaning of the <codeph>--idle_query_timeout</codeph> configuration option is changed, to
+ accommodate the new <codeph>QUERY_TIMEOUT_S</codeph> query option. Rather than setting an absolute
+ timeout period that applies to all queries, it now sets a maximum timeout period, which can be adjusted
+ downward for individual queries by specifying a value for the <codeph>QUERY_TIMEOUT_S</codeph> query
+ option. In sessions where no <codeph>QUERY_TIMEOUT_S</codeph> query option is specified, the
+ <codeph>--idle_query_timeout</codeph> timeout period applies the same as in earlier versions.
+ </p>
+
+ <p>
+ The <codeph>--strict_unicode</codeph> option of <cmdname>impala-shell</cmdname> was removed. To avoid
+ problems with Unicode values in <cmdname>impala-shell</cmdname>, define the following locale setting
+ before running <cmdname>impala-shell</cmdname>:
+ </p>
+<codeblock>export LC_CTYPE=en_US.UTF-8
+</codeblock>
+
+ </section>
+
+ <section id="reserved_words_210">
+ <title>
+ New Reserved Words
+ </title>
+ <p>
+ Some new SQL syntax requires the addition of new reserved words: <codeph>ANTI</codeph>,
+ <codeph>ANALYTIC</codeph>, <codeph>OVER</codeph>, <codeph>PRECEDING</codeph>,
+ <codeph>UNBOUNDED</codeph>, <codeph>FOLLOWING</codeph>, <codeph>CURRENT</codeph>,
+ <codeph>ROWS</codeph>, <codeph>RANGE</codeph>, <codeph>CHAR</codeph>, <codeph>VARCHAR</codeph>.
+ <ph audience="PDF">As always, see <xref href="impala_reserved_words.xml#reserved_words"/>
+ for the set of reserved words for the current release, and the quoting techniques to avoid name conflicts.</ph>
+ </p>
+ </section>
+
+ <section id="output_files_200">
+ <title>
+ Changes to Data Files
+ </title>
+
+ <p id="parquet_block_size">
+ The default Parquet block size for Impala is changed from 1 GB to 256 MB. This change could have
+ implications for the sizes of Parquet files produced by <codeph>INSERT</codeph> and <codeph>CREATE
+ TABLE AS SELECT</codeph> statements.
+ </p>
+ <p>
+ Although older Impala releases typically produced files that were smaller than the old default size of
+ 1 GB, now the file size matches more closely whatever value is specified for the
+ <codeph>PARQUET_FILE_SIZE</codeph> query option. Thus, if you use a non-default value for this setting,
+ the output files could be larger than before. They still might be somewhat smaller than the specified
+ value, because Impala makes conservative estimates about the space needed to represent each column as
+ it encodes the data.
+ </p>
+ <p>
+ When you do not specify an explicit value for the <codeph>PARQUET_FILE_SIZE</codeph> query option,
+ Impala tries to keep the file size within the 256 MB default size, but Impala might adjust the file
+ size to be somewhat larger if needed to accommodate the layout for <term>wide</term> tables, that is,
+ tables with hundreds or thousands of columns.
+ </p>
+ <p>
+ This change is unlikely to affect memory usage while writing Parquet files, because Impala does not
+ pre-allocate the memory needed to hold the entire Parquet block.
+ </p>
+
+ </section>
+
+ </conbody>
+ </concept>
+
+ <concept rev="1.4.4" id="incompatible_changes_144">
+ <title>Incompatible Changes Introduced in Impala 1.4.4 / CDH 5.1.5</title>
+ <conbody>
+ <p>
+ No incompatible changes.
+ </p>
+
+ <note conref="../shared/impala_common.xml#common/only_cdh5_144"/>
+
+ </conbody>
+ </concept>
+
+ <concept rev="1.4.3" id="incompatible_changes_143">
+
+ <title>Incompatible Changes Introduced in Impala 1.4.3 / CDH 5.1.4</title>
+
+ <conbody>
+
+ <p>
+ No incompatible changes. The TLS/SSL security fix does not require any change in the way you interact with
+ Impala.
+ </p>
+
+ <note conref="../shared/impala_common.xml#common/only_cdh5_143"/>
+
+ </conbody>
+ </concept>
+
+ <concept rev="1.4.2" id="incompatible_changes_142">
+
+ <title>Incompatible Changes Introduced in Impala 1.4.2 / CDH 5.1.3</title>
+
+ <conbody>
+
+ <p>
+ None. Impala 1.4.2 is purely a bug-fix release. It does not include any incompatible changes.
+ </p>
+
+ <note conref="../shared/impala_common.xml#common/only_cdh5_142"/>
+
+ </conbody>
+ </concept>
+
+ <concept rev="1.4.1" id="incompatible_changes_141">
+
+ <title>Incompatible Changes Introduced in Impala 1.4.1 / CDH 5.1.2</title>
+
+ <conbody>
+
+ <p>
+ None. Impala 1.4.1 is purely a bug-fix release. It does not include any incompatible changes.
+ </p>
+ </conbody>
+ </concept>
+
+ <concept rev="1.4.0" id="incompatible_changes_140">
+
+ <title>Incompatible Changes Introduced in Impala 1.4.0 / CDH 5.1.0</title>
+ <prolog>
+ <metadata>
+ <data name="Category" value="Deprecated Features"/>
+ </metadata>
+ </prolog>
+
+ <conbody>
+
+ <ul>
+ <li>
+ <p>
+ There is a slight change to required security privileges in the Sentry framework. To create a new
+ object, now you need the <codeph>ALL</codeph> privilege on the parent object. For example, to create a
+ new table, view, or function requires having the <codeph>ALL</codeph> privilege on the database
+ containing the new object. <ph audience="PDF">See
+ <xref
+ href="http://www.cloudera.com/documentation/enterprise/latest/topics/impala_authorization.html"
+ scope="external" format="html">Privilege Model and Object Hierarchy</xref>
+ for a full list of operations and
+ associated privileges.</ph>
+ </p>
+ </li>
+
+ <li>
+ <p>
+ With the ability of <codeph>ORDER BY</codeph> queries to process unlimited amounts of data with no
+ <codeph>LIMIT</codeph> clause, the query options <codeph>DEFAULT_ORDER_BY_LIMIT</codeph> and
+ <codeph>ABORT_ON_DEFAULT_LIMIT_EXCEEDED</codeph> are now deprecated and have no effect.
+ <ph audience="PDF">See <xref href="impala_order_by.xml#order_by"/> for details about improvements to
+ the <codeph>ORDER BY</codeph> clause.</ph>
+ </p>
+ </li>
+
+ <li>
+ <p>
+ There are some changes to the list of reserved words. <ph audience="PDF">See
+ <xref href="impala_reserved_words.xml#reserved_words"/> for the most current list.</ph> The following
+ keywords are new:
+ </p>
+ <ul>
+ <li>
+ <codeph>API_VERSION</codeph>
+ </li>
+
+ <li>
+ <codeph>BINARY</codeph>
+ </li>
+
+ <li>
+ <codeph>CACHED</codeph>
+ </li>
+
+ <li>
+ <codeph>CLASS</codeph>
+ </li>
+
+ <li>
+ <codeph>PARTITIONS</codeph>
+ </li>
+
+ <li>
+ <codeph>PRODUCED</codeph>
+ </li>
+
+ <li>
+ <codeph>UNCACHED</codeph>
+ </li>
+ </ul>
+ <p>
+ The following were formerly reserved keywords, but are no longer reserved:
+ </p>
+ <ul>
+ <li>
+ <codeph>COUNT</codeph>
+ </li>
+
+ <li>
+ <codeph>GROUP_CONCAT</codeph>
+ </li>
+
+ <li>
+ <codeph>NDV</codeph>
+ </li>
+
+ <li>
+ <codeph>SUM</codeph>
+ </li>
+ </ul>
+ </li>
+
+ <li>
+ <p>
+ The fix for issue
+ <xref href="https://issues.cloudera.org/browse/IMPALA-973" scope="external" format="html">IMPALA-973</xref>
+ changes the behavior of the <codeph>INVALIDATE METADATA</codeph> statement regarding nonexistent
+ tables. In Impala 1.4.0 and higher, the statement returns an error if the specified table is not in the
+ metastore database at all. It completes successfully if the specified table is in the metastore
+ database but not yet recognized by Impala, for example if the table was created through Hive. Formerly,
+ you could issue this statement for a completely nonexistent table, with no error.
+ </p>
+ </li>
+ </ul>
+ </conbody>
+ </concept>
+
+ <concept rev="1.3.3" id="incompatible_changes_133">
+
+ <title>Incompatible Changes Introduced in Impala 1.3.3 / CDH 5.0.5</title>
+
+ <conbody>
+
+ <p>
+ No incompatible changes. The TLS/SSL security fix does not require any change in the way you interact with
+ Impala.
+ </p>
+
+ <note conref="../shared/impala_common.xml#common/only_cdh5_133"/>
+
+ </conbody>
+ </concept>
+
+ <concept rev="1.3.2" id="incompatible_changes_132">
+
+ <title>Incompatible Changes Introduced in Impala 1.3.2 / CDH 5.0.4</title>
+
+ <conbody>
+
+ <p>
+ With the fix for IMPALA-1019, you can use HDFS caching for files that are accessed by Impala.
+ </p>
+
+ <note conref="../shared/impala_common.xml#common/only_cdh5_132"/>
+
+ </conbody>
+ </concept>
+
+ <concept rev="1.3.1" id="incompatible_changes_131">
+
+ <title>Incompatible Changes Introduced in Impala 1.3.1 / CDH 5.0.3</title>
+
+ <conbody>
+
+ <ul>
+ <li>
+ <p conref="../shared/impala_common.xml#common/regexp_matching"/>
+ </li>
+
+ <li>
+ <p>
+ The result set for the <codeph>SHOW FUNCTIONS</codeph> statement includes a new first column, with the
+ data type of the return value. <ph audience="PDF">See <xref href="impala_show.xml#show"/> for
+ examples.</ph>
+ </p>
+ </li>
+ </ul>
+ </conbody>
+ </concept>
+
+ <concept rev="1.3.0" id="incompatible_changes_130">
+
+ <title>Incompatible Changes Introduced in Impala 1.3.0 / CDH 5.0.0</title>
+
+ <conbody>
+
+ <ul>
+ <li>
+ <p>
+ The <codeph>EXPLAIN_LEVEL</codeph> query option now accepts numeric options from 0 (most concise) to 3
+ (most verbose), rather than only 0 or 1. If you formerly used <codeph>SET EXPLAIN_LEVEL=1</codeph> to
+ get detailed explain plans, switch to <codeph>SET EXPLAIN_LEVEL=3</codeph>. If you used the mnemonic
+ keyword (<codeph>SET EXPLAIN_LEVEL=verbose</codeph>), you do not need to change your code because now
+ level 3 corresponds to <codeph>verbose</codeph>. <ph audience="PDF">See
+ <xref href="impala_explain_level.xml#explain_level"/> for details about the allowed explain levels, and
+ <xref href="impala_explain_plan.xml#explain_plan"/> for usage information.</ph>
+ </p>
+ </li>
+
+ <li>
+ <p>
+ The keyword <codeph>DECIMAL</codeph> is now a reserved word. If you have any databases, tables,
+ columns, or other objects already named <codeph>DECIMAL</codeph>, quote any references to them using
+ backticks (<codeph>``</codeph>) to avoid name conflicts with the keyword.
+ <note>
+ Although the <codeph>DECIMAL</codeph> keyword is a reserved word, currently Impala does not support
+ <codeph>DECIMAL</codeph> as a data type for columns.
+ </note>
+ </p>
+ </li>
+
+ <li>
+ <p>
+ The query option named <codeph>YARN_POOL</codeph> during the CDH 5 beta period is now named
+ <codeph>REQUEST_POOL</codeph> to reflect its broader use with the Impala admission control feature.
+ <ph audience="PDF">See <xref href="impala_request_pool.xml#request_pool"/> for information about the
+ option, and <xref href="impala_admission.xml#admission_control"/> for details about its use with the
+ admission control feature.</ph>
+ </p>
+ </li>
+
+ <li>
+ <p>
+ There are some changes to the list of reserved words. <ph audience="PDF">See
+ <xref href="impala_reserved_words.xml#reserved_words"/> for the most current list.</ph>
+ </p>
+ <ul>
+ <li>
+ <p>
+ The names of aggregate functions are no longer reserved words, so you can have databases, tables,
+ columns, or other objects named <codeph>AVG</codeph>, <codeph>MIN</codeph>, and so on without any
+ name conflicts.
+ </p>
+ </li>
+
+ <li>
+ <p>
+ The internal function names <codeph>DISTINCTPC</codeph> and <codeph>DISTINCTPCSA</codeph> are no
+ longer reserved words, although <codeph>DISTINCT</codeph> is still a reserved word.
+ </p>
+ </li>
+
+ <li>
+ <p>
+ The keywords <codeph>CLOSE_FN</codeph> and <codeph>PREPARE_FN</codeph> are now reserved words.
+ <ph audience="PDF">See <xref href="impala_create_function.xml#create_function"/> for their role in
+ the <codeph>CREATE FUNCTION</codeph> statement, and <xref href="impala_udf.xml#udf_threads"/> for
+ usage information.</ph>
+ </p>
+ </li>
+ </ul>
+ </li>
+
+ <li>
+ <p>
+ The HDFS property <codeph>dfs.client.file-block-storage-locations.timeout</codeph> was renamed to
+ <codeph>dfs.client.file-block-storage-locations.timeout.millis</codeph>, to emphasize that the unit of
+ measure is milliseconds, not seconds. Impala requires a timeout of at least 10 seconds, making the
+ minimum value for this setting 10000. On systems not managed by Cloudera Manager, you might need to
+ edit the <filepath>hdfs-site.xml</filepath> file in the Impala configuration directory for the new name
+ and minimum value.
+ </p>
+ </li>
+ </ul>
+ </conbody>
+ </concept>
+
+ <concept rev="1.2.4" id="incompatible_changes_124">
+
+ <title>Incompatible Changes Introduced in Impala 1.2.4</title>
+
+ <conbody>
+
+ <p>
+ There are no incompatible changes introduced in Impala 1.2.4.
+ </p>
+
+ <p>
+ Previously, after creating a table in Hive, you had to issue the <codeph>INVALIDATE METADATA</codeph>
+ statement with no table name, a potentially expensive operation on clusters with many databases, tables,
+ and partitions. Starting in Impala 1.2.4, you can issue the statement <codeph>INVALIDATE METADATA
+ <varname>table_name</varname></codeph> for a table newly created through Hive. Loading the metadata for
+ only this one table is faster and involves less network overhead. Therefore, you might revisit your setup
+ DDL scripts to add the table name to <codeph>INVALIDATE METADATA</codeph> statements, in cases where you
+ create and populate the tables through Hive before querying them through Impala.
+ </p>
+ </conbody>
+ </concept>
+
+ <concept rev="1.2.3" id="incompatible_changes_123">
+
+ <title>Incompatible Changes Introduced in Impala 1.2.3</title>
+
+ <conbody>
+
+ <p>
+ Because the feature set of Impala 1.2.3 is identical to Impala 1.2.2, there are no new incompatible
+ changes. See <xref href="impala_incompatible_changes.xml#incompatible_changes_122"/> if you are upgrading
+ from Impala 1.2.1 or 1.1.x.
+ </p>
+ </conbody>
+ </concept>
+
+ <concept rev="1.2.2" id="incompatible_changes_122">
+
+ <title>Incompatible Changes Introduced in Impala 1.2.2</title>
+
+ <conbody>
+
+ <p>
+ The following changes to SQL syntax and semantics in Impala 1.2.2 could require updates to your SQL code,
+ or schema objects such as tables or views:
+ </p>
+
+ <ul>
+ <li>
+ <p>
+ With the addition of the <codeph>CROSS JOIN</codeph> keyword, you might need to rewrite any queries
+ that refer to a table named <codeph>CROSS</codeph> or use the name <codeph>CROSS</codeph> as a table
+ alias:
+ </p>
+<codeblock>-- Formerly, 'cross' in this query was an alias for t1
+-- and it was a normal join query.
+-- In 1.2.2 and higher, CROSS JOIN is a keyword, so 'cross'
+-- is not interpreted as a table alias, and the query
+-- uses the special CROSS JOIN processing rather than a
+-- regular join.
+select * from t1 cross join t2...
+
+-- Now if CROSS is used in other context such as a table or column name,
+-- use backticks to escape it.
+create table `cross` (x int);
+select * from `cross`;</codeblock>
+ </li>
+
+ <li>
+ <p>
+ Formerly, a <codeph>DROP DATABASE</codeph> statement in Impala would not remove the top-level HDFS
+ directory for that database. The <codeph>DROP DATABASE</codeph> has been enhanced to remove that
+ directory. (You still need to drop all the tables inside the database first; this change only applies
+ to the top-level directory for the entire database.)
+ </p>
+ </li>
+
+ <li>
+ The keyword <codeph>PARQUET</codeph> is introduced as a synonym for <codeph>PARQUETFILE</codeph> in the
+ <codeph>CREATE TABLE</codeph> and <codeph>ALTER TABLE</codeph> statements, because that is the common
+ name for the file format. (As opposed to SequenceFile and RCFile where the <q>File</q> suffix is part of
+ the name.) Documentation examples have been changed to prefer the new shorter keyword. The
+ <codeph>PARQUETFILE</codeph> keyword is still available for backward compatibility with older Impala
+ versions.
+ </li>
+
+ <li>
+ New overloads are available for several operators and built-in functions, allowing you to insert their
+ result values into smaller numeric columns such as <codeph>INT</codeph>, <codeph>SMALLINT</codeph>,
+ <codeph>TINYINT</codeph>, and <codeph>FLOAT</codeph> without using a <codeph>CAST()</codeph> call. If you
+ remove the <codeph>CAST()</codeph> calls from <codeph>INSERT</codeph> statements, those statements might
+ not work with earlier versions of Impala.
+ </li>
+ </ul>
+
+ <p>
+ Because many users are likely to upgrade straight from Impala 1.x to Impala 1.2.2, also read
+ <xref href="impala_incompatible_changes.xml#incompatible_changes_121"/> for things to note about upgrading
+ to Impala 1.2.x in general.
+ </p>
+
+ <p conref="../shared/impala_common.xml#common/cm48_upgrade"/>
+
+<!-- <note conref="common.xml#common/cdh4_cdh5_upgrade"/> -->
+ </conbody>
+ </concept>
+
+ <concept rev="1.2.1" id="incompatible_changes_121">
+
+ <title>Incompatible Changes Introduced in Impala 1.2.1</title>
+
+ <conbody>
+
+ <p>
+ The following changes to SQL syntax and semantics in Impala 1.2.1 could require updates to your SQL code,
+ or schema objects such as tables or views:
+ </p>
+
+ <ul>
+ <li>
+ <p conref="../shared/impala_common.xml#common/null_sorting_change"/>
+ <p audience="PDF">
+ See <xref href="impala_literals.xml#null"/> for more information.
+ </p>
+ </li>
+ </ul>
+
+ <p>
+ Impala 1.2.1 goes along with CDH 4.5 and Cloudera Manager 4.8. If you used the beta version Impala 1.2.0
+ that came with the beta of CDH 5, Impala 1.2.1 includes all the features of Impala 1.2.0 except for
+ resource management, which relies on the YARN framework from CDH 5.
+ </p>
+
+ <p>
+ The new <cmdname>catalogd</cmdname> service might require changes to any user-written scripts that stop,
+ start, or restart Impala services, install or upgrade Impala packages, or issue <codeph>REFRESH</codeph> or
+ <codeph>INVALIDATE METADATA</codeph> statements:
+ </p>
+
+ <ul conref="../shared/impala_common.xml#common/catalogd_xrefs">
+ <li/>
+ </ul>
+
+ <p conref="../shared/impala_common.xml#common/cm48_upgrade"/>
+
+<!-- <note conref="common.xml#common/cdh4_cdh5_upgrade"/> -->
+ </conbody>
+ </concept>
+
+ <concept rev="1.2" id="incompatible_changes_120">
+
+ <title>Incompatible Changes Introduced in Impala 1.2.0 (Beta)</title>
+
+ <conbody>
+
+ <p>
+ There are no incompatible changes to SQL syntax in Impala 1.2.0 (beta).
+ </p>
+
+ <p>
+ Because Impala 1.2.0 is bundled with the CDH 5 beta download and depends on specific levels of Apache
+ Hadoop components supplied with CDH 5, you can only install it in combination with the CDH 5 beta.
+ </p>
+
+ <p>
+ The new <cmdname>catalogd</cmdname> service might require changes to any user-written scripts that stop,
+ start, or restart Impala services, install or upgrade Impala packages, or issue <codeph>REFRESH</codeph> or
+ <codeph>INVALIDATE METADATA</codeph> statements:
+ </p>
+
+ <ul conref="../shared/impala_common.xml#common/catalogd_xrefs">
+ <li/>
+ </ul>
+
+ <p>
+ The new resource management feature interacts with both YARN and Llama services, which are available in CDH
+ 5. These services are set up for you automatically in a Cloudera Manager (CM) environment. For information
+ about setting up the YARN and Llama services, see the instructions for
+<!-- Original URL: http://www.cloudera.com/content/cloudera-content/cloudera-docs/CDH5/latest/CDH5-Installation-Guide/cdh_ig_yarn_cluster_deploy.html -->
+ <xref href="http://www.cloudera.com/documentation/enterprise/latest/topics/cdh_ig_yarn_cluster_deploy.html" scope="external" format="html">YARN</xref>
+ and
+<!-- Original URL: http://www.cloudera.com/content/cloudera-content/cloudera-docs/CDH5/latest/CDH5-Installation-Guide/cdh_ig_llama_installation.html -->
+<!-- Then this, which was removed: http://www.cloudera.com/documentation/enterprise/latest/topics/cdh_ig_llama_installation.html -->
+ <xref href="http://www.cloudera.com/documentation/enterprise/latest/topics/admin_llama.html" scope="external" format="html">Llama</xref>
+ in the <cite>CDH 5 Documentation</cite>. <ph audience="PDF">See
+ <xref href="impala_resource_management.xml#resource_management"/> for usage information for Impala resource
+ management.</ph>
+ </p>
+ </conbody>
+ </concept>
+
+ <concept id="incompatible_changes_111">
+
+ <title>Incompatible Changes Introduced in Impala 1.1.1</title>
+
+ <conbody>
+
+ <p>
+ There are no incompatible changes in Impala 1.1.1.
+ </p>
+
+<!-- These couple of paragraphs were originally intended to be conref'ed from the Parquet section of Installing/Using. -->
+
+<!-- But conbodydiv tag too restrictive, can't have just paragraphs and codeblocks inside. -->
+
+<!-- So I will physically copy the info for the time being. -->
+
+<!-- Also copying it under the Upgrading topic. -->
+
+<!-- <conbodydiv conref="impala_parquet.xml#upgrade_parquet_metadata"/> -->
+
+ <p>
+ Previously, it was not possible to create Parquet data through Impala and reuse that table within Hive. Now
+ that Parquet support is available for Hive 10, reusing existing Impala Parquet data files in Hive requires
+ updating the table metadata. Use the following command if you are already running Impala 1.1.1:
+ </p>
+
+<codeblock>ALTER TABLE <varname>table_name</varname> SET FILEFORMAT PARQUETFILE;
+</codeblock>
+
+ <p>
+ If you are running a level of Impala that is older than 1.1.1, do the metadata update through Hive:
+ </p>
+
+<codeblock>ALTER TABLE <varname>table_name</varname> SET SERDE 'parquet.hive.serde.ParquetHiveSerDe';
+ALTER TABLE <varname>table_name</varname> SET FILEFORMAT
+ INPUTFORMAT "parquet.hive.DeprecatedParquetInputFormat"
+ OUTPUTFORMAT "parquet.hive.DeprecatedParquetOutputFormat";
+</codeblock>
+
+ <p>
+ Impala 1.1.1 and higher can reuse Parquet data files created by Hive, without any action required.
+ </p>
+
+ <p>
+ As usual, make sure to upgrade the <codeph>impala-lzo-cdh4</codeph> package to the latest level at the same
+ time as you upgrade the Impala server.
+ </p>
</conbody>
</concept>
+ <concept id="incompatible_changes_11">
+
+ <title>Incompatible Change Introduced in Impala 1.1</title>
+
+ <conbody>
+
+ <ul>
+ <li>
+ <p>
+ The <codeph>REFRESH</codeph> statement now requires a table name; in Impala 1.0, the table name was
+ optional. This syntax change is part of the internal rework to make <codeph>REFRESH</codeph> a true
+ Impala SQL statement so that it can be called through the JDBC and ODBC APIs. <codeph>REFRESH</codeph>
+ now reloads the metadata immediately, rather than marking it for update the next time any affected
+ table is accessed. The previous behavior, where omitting the table name caused a refresh of the entire
+ Impala metadata catalog, is available through the new <codeph>INVALIDATE METADATA</codeph> statement.
+ <codeph>INVALIDATE METADATA</codeph> can be specified with a table name to affect a single table, or
+ without a table name to affect the entire metadata catalog; the relevant metadata is reloaded the next
+ time it is requested during the processing for a SQL statement. See
+ <xref href="impala_refresh.xml#refresh"/> and
+ <xref href="impala_invalidate_metadata.xml#invalidate_metadata"/> for the latest details about these
+ statements.
+ </p>
+ </li>
+ </ul>
+ </conbody>
+ </concept>
+
+ <concept id="incompatible_changes_10">
+
+ <title>Incompatible Changes Introduced in Impala 1.0</title>
+
+ <conbody>
+
+ <ul>
+ <li>
+ If you use LZO-compressed text files, when you upgrade Impala to version 1.0, also update the
+ <codeph>impala-lzo-cdh4</codeph> to the latest level. See <xref href="impala_txtfile.xml#lzo"/> for
+ details.
+ </li>
+
+ <li>
+ Cloudera Manager 4.5.2 and higher only supports Impala 1.0 and higher, and vice versa. If you upgrade to
+ Impala 1.0 or higher managed by Cloudera Manager, you must also upgrade Cloudera Manager to version 4.5.2
+ or higher. If you upgrade from an earlier version of Cloudera Manager, and were using Impala, you must
+ also upgrade Impala to version 1.0 or higher. The beta versions of Impala are no longer supported as of
+ the release of Impala 1.0.
+ </li>
+ </ul>
+ </conbody>
+ </concept>
+
+ <concept id="incompatible_changes_07">
+
+ <title>Incompatible Change Introduced in Version 0.7 of the Impala Beta Release</title>
+
+ <conbody>
+
+ <ul>
+ <li>
+ The defaults for the <codeph>-nn</codeph> and <codeph>-nn_port</codeph> flags have changed and are now
+ read from <codeph>core-site.xml</codeph>. Impala prints the values of <codeph>-nn</codeph> and
+ <codeph>-nn_port</codeph> to the log when it starts. The ability to set <codeph>-nn</codeph> and
+ <codeph>-nn_port</codeph> on the command line is deprecated in 0.7 and may be removed in Impala 0.8.
+ </li>
+ </ul>
+ </conbody>
+ </concept>
+
+ <concept id="incompatible_changes_06">
+
+ <title>Incompatible Change Introduced in Version 0.6 of the Impala Beta Release</title>
+
+ <conbody>
+
+ <ul>
+ <li>
+ Cloudera Manager 4.5 supports only version 0.6 of the Impala Beta Release. It does not support
+ the earlier beta versions. If you upgrade your Cloudera Manager installation, you must also upgrade
+ Impala to beta version 0.6. If you upgrade Impala to beta version 0.6, you must upgrade Cloudera Manager
+ to 4.5.
+ </li>
+ </ul>
+ </conbody>
+ </concept>
+
+ <concept id="incompatible_changes_04">
+
+ <title>Incompatible Change Introduced in Version 0.4 of the Impala Beta Release</title>
+
+ <conbody>
+
+ <ul>
+ <li>
+ Cloudera Manager 4.1.3 supports only version 0.4 of the Impala Beta Release. It does not support
+ the earlier beta versions. If you upgrade your Cloudera Manager installation, you must also upgrade
+ Impala to beta version 0.4. If you upgrade Impala to beta version 0.4, you must upgrade Cloudera Manager
+ to 4.1.3.
+ </li>
+ </ul>
+ </conbody>
+ </concept>
+
+ <concept id="incompatible_changes_03">
+
+ <title>Incompatible Change Introduced in Version 0.3 of the Impala Beta Release</title>
+
+ <conbody>
+
+ <ul>
+ <li>
+ Cloudera Manager 4.1.2 supports only version 0.3 of the Impala Beta Release. It does not support
+ the earlier beta versions. If you upgrade your Cloudera Manager installation, you must also upgrade
+ Impala to beta version 0.3. If you upgrade Impala to beta version 0.3, you must upgrade Cloudera Manager
+ to 4.1.2.
+ </li>
+ </ul>
+ </conbody>
+ </concept>
+</concept>
http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3c2c8f12/docs/topics/impala_insert.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_insert.xml b/docs/topics/impala_insert.xml
index 6d0f68b..02ad8c6 100644
--- a/docs/topics/impala_insert.xml
+++ b/docs/topics/impala_insert.xml
@@ -3,7 +3,7 @@
<concept id="insert">
<title>INSERT Statement</title>
- <titlealts><navtitle>INSERT</navtitle></titlealts>
+ <titlealts audience="PDF"><navtitle>INSERT</navtitle></titlealts>
<prolog>
<metadata>
<data name="Category" value="Impala"/>
@@ -14,7 +14,8 @@
<data name="Category" value="Data Analysts"/>
<data name="Category" value="Developers"/>
<data name="Category" value="Tables"/>
- <data audience="impala_next" name="Category" value="Kudu"/>
+ <data name="Category" value="S3"/>
+ <!-- <data name="Category" value="Kudu"/> -->
<!-- This is such an important statement, think if there are more applicable categories. -->
</metadata>
</prolog>
@@ -71,11 +72,11 @@ hint_clause ::= [SHUFFLE] | [NOSHUFFLE] (Note: the square brackets are part o
See <xref href="impala_complex_types.xml#complex_types"/> for details about working with complex types.
</p>
- <p rev="kudu" audience="impala_next">
+ <p rev="kudu">
<b>Ignoring duplicate partition keys for Kudu tables (IGNORE clause)</b>
</p>
- <p rev="kudu" audience="impala_next">
+ <p rev="kudu">
Normally, an <codeph>INSERT</codeph> operation into a Kudu table fails if
it would result in duplicate partition key columns for any rows.
Specify <codeph>INSERT IGNORE <varname>rest_of_statement</varname></codeph> to
@@ -165,7 +166,7 @@ hint_clause ::= [SHUFFLE] | [NOSHUFFLE] (Note: the square brackets are part o
<li>
Insert commands that partition or add files result in changes to Hive metadata. Because Impala uses Hive
metadata, such changes may necessitate a metadata refresh. For more information, see the
- <xref href="impala_refresh.xml#refresh" format="dita">REFRESH</xref> function.
+ <xref href="impala_refresh.xml#refresh">REFRESH</xref> function.
</li>
<li>
@@ -194,6 +195,16 @@ hint_clause ::= [SHUFFLE] | [NOSHUFFLE] (Note: the square brackets are part o
in the <codeph>INSERT</codeph> statement to make the conversion explicit.
</p>
+ <p conref="../shared/impala_common.xml#common/file_format_blurb"/>
+
+ <p rev="DOCS-1523">
+ Because Impala can read certain file formats that it cannot write,
+ the <codeph>INSERT</codeph> statement does not work for all kinds of
+ Impala tables. See <xref href="impala_file_formats.xml#file_formats"/>
+ for details about what file formats are supported by the
+ <codeph>INSERT</codeph> statement.
+ </p>
+
<p conref="../shared/impala_common.xml#common/insert_parquet_blocksize"/>
<p conref="../shared/impala_common.xml#common/sync_ddl_blurb"/>
@@ -204,7 +215,7 @@ hint_clause ::= [SHUFFLE] | [NOSHUFFLE] (Note: the square brackets are part o
<p>
The following example sets up new tables with the same definition as the <codeph>TAB1</codeph> table from the
- <xref href="impala_tutorial.xml#tutorial" format="dita">Tutorial</xref> section, using different file
+ <xref href="impala_tutorial.xml#tutorial">Tutorial</xref> section, using different file
formats, and demonstrates inserting data into the tables created with the <codeph>STORED AS TEXTFILE</codeph>
and <codeph>STORED AS PARQUET</codeph> clauses:
</p>
@@ -641,6 +652,8 @@ Inserted 2 rows in 0.16s
<p conref="../shared/impala_common.xml#common/s3_blurb"/>
<p conref="../shared/impala_common.xml#common/s3_dml"/>
+ <p conref="../shared/impala_common.xml#common/s3_dml_performance"/>
+ <p>See <xref href="../topics/impala_s3.xml#s3"/> for details about reading and writing S3 data with Impala.</p>
<p conref="../shared/impala_common.xml#common/security_blurb"/>
<p conref="../shared/impala_common.xml#common/redaction_yes"/>
http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3c2c8f12/docs/topics/impala_install.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_install.xml b/docs/topics/impala_install.xml
index aab0a63..452ed03 100644
--- a/docs/topics/impala_install.xml
+++ b/docs/topics/impala_install.xml
@@ -3,17 +3,132 @@
<concept id="install">
<title><ph audience="standalone">Installing Impala</ph><ph audience="integrated">Impala Installation</ph></title>
-
+ <prolog>
+ <metadata>
+ <data name="Category" value="Impala"/>
+ <data name="Category" value="Installing"/>
+ <data name="Category" value="Administrators"/>
+ </metadata>
+ </prolog>
<conbody>
<p>
-
+ <indexterm audience="Cloudera">installation</indexterm>
+ <indexterm audience="Cloudera">pseudo-distributed cluster</indexterm>
+ <indexterm audience="Cloudera">cluster</indexterm>
+ <indexterm audience="Cloudera">DataNodes</indexterm>
+ <indexterm audience="Cloudera">NameNode</indexterm>
+ <indexterm audience="Cloudera">Cloudera Manager</indexterm>
+ <indexterm audience="Cloudera">impalad</indexterm>
+ <indexterm audience="Cloudera">impala-shell</indexterm>
+ <indexterm audience="Cloudera">statestored</indexterm>
Impala is an open-source add-on to the Cloudera Enterprise Core that returns rapid responses to
queries.
</p>
-
+ <note>
+ <p>
+ Under CDH 5, Impala is included as part of the CDH installation and no separate steps are needed.
+ <ph audience="standalone">Therefore, the instruction steps in this section apply to CDH 4 only.</ph>
+ </p>
+ </note>
+
+ <p outputclass="toc inpage"/>
+ </conbody>
+
+ <concept id="install_details">
+
+ <title>What is Included in an Impala Installation</title>
+
+ <conbody>
+
+ <p>
+ Impala is made up of a set of components that can be installed on multiple nodes throughout your cluster.
+ The key installation step for performance is to install the <cmdname>impalad</cmdname> daemon (which does
+ most of the query processing work) on <i>all</i> DataNodes in the cluster.
+ </p>
+
+ <p>
+ The Impala package installs these binaries:
+ </p>
+
+ <ul>
+ <li>
+ <p>
+ <cmdname>impalad</cmdname> - The Impala daemon. Plans and executes queries against HDFS, HBase, <ph rev="2.2.0">and Amazon S3 data</ph>.
+ <xref href="impala_processes.xml#processes">Run one impalad process</xref> on each node in the cluster
+ that has a DataNode.
+ </p>
+ </li>
+
+ <li>
+ <p>
+ <cmdname>statestored</cmdname> - Name service that tracks location and status of all
+ <codeph>impalad</codeph> instances in the cluster. <xref href="impala_processes.xml#processes">Run one
+ instance of this daemon</xref> on a node in your cluster. Most production deployments run this daemon
+ on the namenode.
+ </p>
+ </li>
+
+ <li rev="1.2">
+ <p>
+ <cmdname>catalogd</cmdname> - Metadata coordination service that broadcasts changes from Impala DDL and
+ DML statements to all affected Impala nodes, so that new tables, newly loaded data, and so on are
+ immediately visible to queries submitted through any Impala node.
+<!-- Consider removing this when 1.2 gets far in the past. -->
+ (Prior to Impala 1.2, you had to run the <codeph>REFRESH</codeph> or <codeph>INVALIDATE
+ METADATA</codeph> statement on each node to synchronize changed metadata. Now those statements are only
+ required if you perform the DDL or DML through an external mechanism such as Hive <ph rev="2.2.0">or by uploading
+ data to the Amazon S3 filesystem</ph>.)
+ <xref href="impala_processes.xml#processes">Run one instance of this daemon</xref> on a node in your cluster,
+ preferably on the same host as the <codeph>statestored</codeph> daemon.
+ </p>
+ </li>
+
+ <li>
+ <p>
+ <cmdname>impala-shell</cmdname> - <xref href="impala_impala_shell.xml#impala_shell">Command-line
+ interface</xref> for issuing queries to the Impala daemon. You install this on one or more hosts
+ anywhere on your network, not necessarily DataNodes or even within the same cluster as Impala. It can
+ connect remotely to any instance of the Impala daemon.
+ </p>
+ </li>
+ </ul>
+
+ <p>
+ Before doing the installation, ensure that you have all necessary prerequisites. See
+ <xref href="impala_prereqs.xml#prereqs"/> for details.
+ </p>
</conbody>
</concept>
+ <concept audience="standalone" id="install_cdh4">
+
+ <title>Impala Installation Procedure for CDH 4 Users</title>
+
+ <conbody>
+
+ <p>
+ You can install Impala under CDH 4 in one of two ways:
+ </p>
+
+ <ul>
+ <li>
+ Using the Cloudera Manager installer. This is the recommended technique for doing a reliable and verified
+ Impala installation. Cloudera Manager 4.8 or higher can automatically install, configure, manage, and
+ monitor Impala 1.2.1 and higher. The latest Cloudera Manager is always preferable, because newer Cloudera
+ Manager releases have configuration settings for the most recent Impala features.
+ </li>
+
+ <li>
+ Using a manual process for systems not managed by Cloudera Manager. You must do additional verification
+ steps in this case, to check that Impala can interact with other Hadoop components correctly, and that
+ your cluster is configured for efficient Impala execution.
+ </li>
+ </ul>
+
+ <p outputclass="toc"/>
+ </conbody>
+ </concept>
+</concept>
http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3c2c8f12/docs/topics/impala_int.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_int.xml b/docs/topics/impala_int.xml
index 514d377..aeead5b 100644
--- a/docs/topics/impala_int.xml
+++ b/docs/topics/impala_int.xml
@@ -3,7 +3,7 @@
<concept id="int">
<title>INT Data Type</title>
- <titlealts><navtitle>INT</navtitle></titlealts>
+ <titlealts audience="PDF"><navtitle>INT</navtitle></titlealts>
<prolog>
<metadata>
<data name="Category" value="Impala"/>
@@ -73,7 +73,7 @@ SELECT CAST(1000 AS INT);
<p conref="../shared/impala_common.xml#common/text_bulky"/>
-<!-- <p conref="/Content/impala_common_xi44078.xml#common/compatibility_blurb"/> -->
+<!-- <p conref="../shared/impala_common.xml#common/compatibility_blurb"/> -->
<p conref="../shared/impala_common.xml#common/internals_4_bytes"/>
@@ -81,7 +81,7 @@ SELECT CAST(1000 AS INT);
<p conref="../shared/impala_common.xml#common/column_stats_constant"/>
-<!-- <p conref="/Content/impala_common_xi44078.xml#common/restrictions_blurb"/> -->
+<!-- <p conref="../shared/impala_common.xml#common/restrictions_blurb"/> -->
<p conref="../shared/impala_common.xml#common/related_info"/>
http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3c2c8f12/docs/topics/impala_invalidate_metadata.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_invalidate_metadata.xml b/docs/topics/impala_invalidate_metadata.xml
index 96fca7d..c41a996 100644
--- a/docs/topics/impala_invalidate_metadata.xml
+++ b/docs/topics/impala_invalidate_metadata.xml
@@ -3,15 +3,19 @@
<concept rev="1.1" id="invalidate_metadata">
<title>INVALIDATE METADATA Statement</title>
- <titlealts><navtitle>INVALIDATE METADATA</navtitle></titlealts>
+ <titlealts audience="PDF"><navtitle>INVALIDATE METADATA</navtitle></titlealts>
<prolog>
<metadata>
<data name="Category" value="Impala"/>
<data name="Category" value="SQL"/>
<data name="Category" value="DDL"/>
+ <data name="Category" value="ETL"/>
+ <data name="Category" value="Ingest"/>
<data name="Category" value="Metastore"/>
<data name="Category" value="Schemas"/>
<data name="Category" value="Tables"/>
+ <data name="Category" value="Developers"/>
+ <data name="Category" value="Data Analysts"/>
</metadata>
</prolog>
@@ -66,14 +70,7 @@
requires a table name parameter, to flush the metadata for all tables at once, use the <codeph>INVALIDATE
METADATA</codeph> statement.
</p>
- <draft-comment translate="no"> Almost-identical wording here, under INVALIDATE METADATA, and in Release Notes :: New Features. Makes sense to conref. </draft-comment>
- <p>
- Because <codeph>REFRESH <varname>table_name</varname></codeph> only works for tables that the current
- Impala node is already aware of, when you create a new table in the Hive shell, you must enter
- <codeph>INVALIDATE METADATA</codeph> with no table parameter before you can see the new table in
- <cmdname>impala-shell</cmdname>. Once the table is known by the Impala node, you can issue <codeph>REFRESH
- <varname>table_name</varname></codeph> after you add data files for that table.
- </p>
+ <p conref="../shared/impala_common.xml#common/invalidate_then_refresh"/>
</note>
<p conref="../shared/impala_common.xml#common/refresh_vs_invalidate"/>
@@ -95,7 +92,7 @@
</li>
<li>
- <b>and</b> the change is made to a database to which clients such as the Impala shell or ODBC directly
+ <b>and</b> the change is made to a metastore database to which clients such as the Impala shell or ODBC directly
connect.
</li>
</ul>
http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3c2c8f12/docs/topics/impala_isilon.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_isilon.xml b/docs/topics/impala_isilon.xml
index fe6a5de..f631268 100644
--- a/docs/topics/impala_isilon.xml
+++ b/docs/topics/impala_isilon.xml
@@ -4,7 +4,17 @@
<title>Using Impala with Isilon Storage</title>
<titlealts audience="PDF"><navtitle>Isilon Storage</navtitle></titlealts>
-
+ <prolog>
+ <metadata>
+ <data name="Category" value="CDH"/>
+ <data name="Category" value="Impala"/>
+ <data name="Category" value="Isilon"/>
+ <data name="Category" value="Disk Storage"/>
+ <data name="Category" value="Administrators"/>
+ <data name="Category" value="Developers"/>
+ <data name="Category" value="Data Analysts"/>
+ </metadata>
+ </prolog>
<conbody>
@@ -16,8 +26,89 @@
certified on CDH 5.4.4 or higher.
</p>
-
+ <p conref="../shared/impala_common.xml#common/isilon_block_size_caveat"/>
+
+ <p>
+ The typical use case for Impala and Isilon together is to use Isilon for the
+ default filesystem, replacing HDFS entirely. In this configuration,
+ when you create a database, table, or partition, the data always resides on
+ Isilon storage and you do not need to specify any special <codeph>LOCATION</codeph>
+ attribute. If you do specify a <codeph>LOCATION</codeph> attribute, its value refers
+ to a path within the Isilon filesystem.
+ For example:
+ </p>
+<codeblock>-- If the default filesystem is Isilon, all Impala data resides there
+-- and all Impala databases and tables are located there.
+CREATE TABLE t1 (x INT, s STRING);
+
+-- You can specify LOCATION for database, table, or partition,
+-- using values from the Isilon filesystem.
+CREATE DATABASE d1 LOCATION '/some/path/on/isilon/server/d1.db';
+CREATE TABLE d1.t2 (a TINYINT, b BOOLEAN);
+</codeblock>
+
+ <p>
+ Impala can write to, delete, and rename data files and database, table,
+ and partition directories on Isilon storage. Therefore, Impala statements such
+ as
+ <codeph>CREATE TABLE</codeph>, <codeph>DROP TABLE</codeph>,
+ <codeph>CREATE DATABASE</codeph>, <codeph>DROP DATABASE</codeph>,
+ <codeph>ALTER TABLE</codeph>,
+ and
+ <codeph>INSERT</codeph> work the same with Isilon storage as with HDFS.
+ </p>
+
+ <p>
+ When the Impala spill-to-disk feature is activated by a query that approaches
+ the memory limit, Impala writes all the temporary data to a local (not Isilon)
+ storage device. Because the I/O bandwidth for the temporary data depends on
+ the number of local disks, and clusters using Isilon storage might not have
+ as many local disks attached, pay special attention on Isilon-enabled clusters
+ to any queries that use the spill-to-disk feature. Where practical, tune the
+ queries or allocate extra memory for Impala to avoid spilling.
+ Although you can specify an Isilon storage device as the destination for
+ the temporary data for the spill-to-disk feature, that configuration is
+ not recommended due to the need to transfer the data both ways using remote I/O.
+ </p>
+
+ <p>
+ When tuning Impala queries on HDFS, you typically try to avoid any remote reads.
+ When the data resides on Isilon storage, all the I/O consists of remote reads.
+ Do not be alarmed when you see non-zero numbers for remote read measurements
+ in query profile output. The benefit of the Impala and Isilon integration is
+ primarily convenience of not having to move or copy large volumes of data to HDFS,
+ rather than raw query performance. You can increase the performance of Impala
+ I/O for Isilon systems by increasing the value for the
+ <codeph>num_remote_hdfs_io_threads</codeph> configuration parameter,
+ in the Cloudera Manager user interface for clusters using Cloudera Manager,
+ or through the <codeph>--num_remote_hdfs_io_threads</codeph> startup option
+ for the <cmdname>impalad</cmdname> daemon on clusters not using Cloudera Manager.
+ </p>
+
+ <p>
+<!--
+ For information about tasks performed on
+ Isilon OneFS, see the information hub for Cloudera on the EMC Community Network:
+ <xref href="https://community.emc.com/docs/DOC-39522" format="html" scope="external">https://community.emc.com/docs/DOC-39522</xref>.
+-->
+ <!-- This is a little bit of a circular loop when this topic is conrefed into the main Isilon page,
+ consider if there's a way to conditionalize it out in that case. -->
+ For information about managing Isilon storage devices through Cloudera Manager, see
+ <xref audience="integrated" href="cm_mc_isilon_service.xml"/><xref audience="standalone" href="http://www.cloudera.com/documentation/enterprise/latest/topics/cm_mc_isilon_service.html" scope="external" format="html"/>.
+ </p>
+
+ <!-- <p outputclass="toc inpage"/> -->
+ </conbody>
+<concept id="isilon_cm_configs">
+<title>Required Configurations</title>
+<conbody>
+<p>Specify the following configurations in Cloudera Manager on the <menucascade><uicontrol>Clusters</uicontrol><uicontrol><varname>Isilon Service</varname></uicontrol><uicontrol>Configuration</uicontrol></menucascade> tab:<ul id="ul_vpx_bw5_vv">
+<li>In <uicontrol>HDFS Client Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml</uicontrol> <codeph>hdfs-site.xml</codeph> and the <uicontrol>Cluster-wide Advanced Configuration Snippet (Safety Valve) for core-site.xml</uicontrol> properties for the Isilon service, set the value of the <codeph>dfs.client.file-block-storage-locations.timeout.millis</codeph> property to <codeph>10000</codeph>.</li>
+<li>In the Isilon <uicontrol>Cluster-wide Advanced Configuration Snippet (Safety Valve) for core-site.xml</uicontrol> property for the Isilon service, set the value of the <codeph>hadoop.security.token.service.use_ip</codeph> property to <codeph>FALSE</codeph>. </li>
+<li>If you see errors that reference the <codeph>.Trash</codeph> directory, make sure that the <uicontrol>Use Trash</uicontrol> property is selected.</li>
+</ul></p>
+
</conbody>
</concept>
-
+</concept>