You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@impala.apache.org by ta...@apache.org on 2018/03/15 22:58:49 UTC
[1/8] impala git commit: IMPALA-3916: [DOCS] Reserved keywords
updated for Impala 3.0
Repository: impala
Updated Branches:
refs/heads/master 03794389f -> 241031d33
http://git-wip-us.apache.org/repos/asf/impala/blob/6a07076d/docs/topics/impala_reserved_words.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_reserved_words.xml b/docs/topics/impala_reserved_words.xml
index 0a44dba..abd5c6a 100644
--- a/docs/topics/impala_reserved_words.xml
+++ b/docs/topics/impala_reserved_words.xml
@@ -34,32 +34,78 @@ under the License.
<conbody>
<p>
- <indexterm audience="hidden">reserved words</indexterm>
- The following are the reserved words for the current release of Impala. A reserved word is one that
- cannot be used directly as an identifier; you must quote it with backticks. For example, a statement
- <codeph>CREATE TABLE select (x INT)</codeph> fails, while <codeph>CREATE TABLE `select` (x INT)</codeph>
- succeeds. Impala does not reserve the names of aggregate or scalar built-in functions. (Formerly, Impala did
- reserve the names of some aggregate functions.)
+ <indexterm audience="hidden">reserved words</indexterm> This topic lists
+ the reserved words in Impala.
+ </p>
+ <p>
+ A reserved word is one that cannot be used directly as an identifier. If
+ you need to use it as an identifier, you must quote it with backticks.
+ For example:
+ <ul>
+ <li>
+ <codeph>CREATE TABLE select (x INT)</codeph>: fails
+ </li>
+ <li>
+ <codeph>CREATE TABLE `select` (x INT)</codeph>: succeeds
+ </li>
+ </ul>
</p>
<p>
- Because different database systems have different sets of reserved words, and the reserved words change from
- release to release, carefully consider database, table, and column names to ensure maximum compatibility
- between products and versions.
+ Because different database systems have different sets of reserved words,
+ and the reserved words change from release to release, carefully consider
+ database, table, and column names to ensure maximum compatibility between
+ products and versions.
</p>
<p>
- Because you might switch between Impala and Hive when doing analytics and ETL, also consider whether
- your object names are the same as any Hive keywords, and rename or quote any that conflict. Consult the
- <xref href="https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL#LanguageManualDDL-Keywords,Non-reservedKeywordsandReservedKeywords" scope="external" format="html">list of Hive keywords</xref>.
+ Also consider whether your object names are the same as any Hive
+ keywords, and rename or quote any that conflict as you might switch
+ between Impala and Hive when doing analytics and ETL. Consult the
+ <xref keyref="hive_reserved_words">list of Hive keywords</xref>.
+ </p>
+ <p>
+ To future-proof your code, you should avoid additional words in case they
+ become reserved words if Impala adds features in later releases. This kind
+ of planning can also help to avoid name conflicts in case you port SQL
+ from other systems that have different sets of reserved words. The Future
+ Keyword column in the table below indicates those additional words that
+ you should avoid for table, column, or other object names, even though
+ they are not currently reserved by Impala.
</p>
+ <p>
+ The following is a summary of the process for deciding whether a
+ particular SQL 2016 word is to be reserved in Impala.
+ </p>
+ <ul>
+ <li>
+ By default, Impala targets to have the same list of reserved words as
+ SQL 2016.
+ </li>
+ <li>
+ At the same time, to be compatible with earlier versions of Impala
+ and to avoid breaking existing tables/workloads, Impala built-in
+ function names are removed from the reserved words list, e.g. COUNT,
+ AVG, as Impala generally does not need to reserve the names of built-in
+ functions for parsing to work.
+ </li>
+ <li>
+ For those remaining SQL 2016 reserved words, if a word is likely to be
+ in-use by users of older Impala versions and if there is a low chance of
+ Impala needing to reserve that word in the future, then the word is not
+ reserved.
+ </li>
+ <li>
+ Otherwise, the word is reserved in Impala.
+ </li>
+ </ul>
<p outputclass="toc inpage"/>
</conbody>
<concept id="reserved_words_current">
-<title>List of Current Reserved Words</title>
+<title>List of Reserved Words</title>
<conbody>
<!-- This list is derived from the source code at:
https://github.com/apache/impala/blob/master/fe/src/main/jflex/sql-scanner.flex
@@ -67,322 +113,3781 @@ under the License.
See the history, any recent changes, here:
https://github.com/apache/impala/commits/master/fe/src/main/jflex/sql-scanner.flex
-->
-
-<codeblock rev="ver">add
-aggregate
-all
-alter
-<ph rev="2.0.0">analytic</ph>
-and
-<ph rev="2.0.0">anti</ph>
-<ph rev="1.4.0">api_version</ph>
-as
-asc
-avro
-between
-bigint
-<ph rev="1.4.0">binary</ph>
-<ph rev="kudu">blocksize</ph>
-boolean
-<!-- <ph rev="kudu">buckets</ph> -->
-by
-<ph rev="1.4.0">cached</ph>
-<ph rev="2.3.0">cascade</ph>
-case
-cast
-change
-<ph rev="2.0.0">char</ph>
-<ph rev="1.4.0">class</ph>
-<ph rev="1.2.1">close_fn</ph>
-column
-columns
-comment
-<ph rev="kudu">compression</ph>
-compute
-create
-cross
-<ph rev="2.0.0">current</ph>
-data
-database
-databases
-date
-datetime
-decimal
-<ph rev="kudu">default</ph>
-<ph rev="kudu">delete</ph>
-delimited
-desc
-describe
-distinct
-<!-- <ph rev="kudu">distribute</ph> -->
-div
-double
-drop
-else
-<ph rev="kudu">encoding</ph>
-end
-escaped
-exists
-explain
-<ph rev="2.5.0">extended</ph>
-external
-false
-fields
-fileformat
-<ph rev="1.2.1">finalize_fn</ph>
-first
-float
-<ph rev="2.0.0">following</ph>
-<ph rev="2.1.0">for</ph>
-format
-formatted
-from
-full
-function
-functions
-<ph rev="2.1.0">grant</ph>
-group
-<ph rev="kudu">hash</ph>
-having
-if
-<!-- <ph rev="kudu">ignore</ph> -->
-<ph rev="2.5.0">ilike</ph>
-in
-<ph rev="2.1.0">incremental</ph>
-<ph rev="1.2.1">init_fn</ph>
-inner
-inpath
-insert
-int
-integer
-intermediate
-interval
-into
-invalidate
-<ph rev="2.5.0">iregexp</ph>
-is
-join
-last
-left
-like
-limit
-lines
-load
-location
-<ph rev="1.2.1">merge_fn</ph>
-metadata
-not
-null
-nulls
-offset
-on
-or
-order
-outer
-<ph rev="2.0.0">over</ph>
-overwrite
-parquet
-parquetfile
-partition
-partitioned
-<ph rev="1.4.0">partitions</ph>
-<ph rev="2.0.0">preceding</ph>
-<ph rev="1.2.1">prepare_fn</ph>
-<ph rev="1.4.0">produced</ph>
-<ph rev="2.3.0">purge</ph>
-<ph rev="2.0.0">range</ph>
-rcfile
-real
-refresh
-regexp
-rename
-<ph rev="IMPALA-5309">repeatable</ph>
-replace
-<ph rev="2.3.0">restrict</ph>
-returns
-<ph rev="2.1.0">revoke</ph>
-right
-rlike
-<ph rev="2.1.0">role</ph>
-<ph rev="2.1.0">roles</ph>
-row
-<ph rev="2.0.0">rows</ph>
-schema
-schemas
-select
-semi
-sequencefile
-serdeproperties
-<ph rev="2.0.0">serialize_fn</ph>
-set
-show
-smallint
-<!-- <ph rev="kudu">split</ph> -->
-stats
-stored
-straight_join
-string
-symbol
-table
-tables
-<ph rev="2.9.0">tablesample</ph>
-tblproperties
-terminated
-textfile
-then
-timestamp
-tinyint
-to
-true
-<ph rev="2.0.0">truncate</ph>
-<ph rev="2.0.0">unbounded</ph>
-<ph rev="1.4.0">uncached</ph>
-union
-<ph rev="kudu">update</ph>
-<ph rev="1.2.1">update_fn</ph>
-<ph rev="kudu">upsert</ph>
-use
-using
-values
-<ph rev="2.0.0">varchar</ph>
-view
-when
-where
-with</codeblock>
-</conbody>
-</concept>
-
-<concept id="reserved_words_planning">
-<title>Planning for Future Reserved Words</title>
-<conbody>
-<p>
-The previous list of reserved words includes all the keywords
-used in the current level of Impala SQL syntax.
-To future-proof your code,
-you should avoid additional words in case they
-become reserved words if
-Impala adds features in later releases.
-This kind of planning can also help to avoid
-name conflicts in case you port SQL from other systems that
-have different sets of reserved words.
-</p>
-
-<p>
-The following list contains additional words that you should
-avoid for table, column, or other object names,
-even though they are not currently reserved by Impala.
-</p>
-
-<codeblock>any
-authorization
-backup
-begin
-break
-browse
-bulk
-cascade
-check
-checkpoint
-close
-clustered
-coalesce
-collate
-commit
-constraint
-contains
-continue
-convert
-current
-current_date
-current_time
-current_timestamp
-current_user
-cursor
-dbcc
-deallocate
-declare
-default
-deny
-disk
-distributed
-dump
-errlvl
-escape
-except
-exec
-execute
-exit
-fetch
-file
-fillfactor
-for
-foreign
-freetext
-goto
-holdlock
-identity
-index
-intersect
-key
-kill
-lineno
-merge
-national
-nocheck
-nonclustered
-nullif
-of
-off
-offsets
-open
-option
-percent
-pivot
-plan
-precision
-primary
-print
-proc
-procedure
-public
-raiserror
-read
-readtext
-reconfigure
-references
-replication
-restore
-restrict
-return
-revert
-rollback
-rowcount
-rule
-save
-securityaudit
-session_user
-setuser
-shutdown
-some
-statistics
-system_user
-textsize
-then
-top
-tran
-transaction
-trigger
-try_convert
-unique
-unpivot
-updatetext
-user
-varying
-waitfor
-while
-within
-writetext
-</codeblock>
+ <p>
+ <table dir="ltr" id="table_lfw_pjs_cdb" colsep="1" rowsep="1"
+ frame="all">
+ <tgroup cols="5" align="left">
+ <colspec colnum="1" colname="col1"/>
+ <colspec colnum="2" colname="col2"/>
+ <colspec colnum="3" colname="col3"/>
+ <colspec colnum="4" colname="col4"/>
+ <colspec colname="newCol6" colnum="5"/>
+ <tbody>
+ <row>
+ <entry><b>Keyword</b></entry>
+ <entry><b>Reserved</b><p><b>in</b></p><p><b>SQL:2016</b></p></entry>
+ <entry><b>Reserved</b><p><b>in</b></p><p><b>Impala 2.12 and
+ lower</b></p></entry>
+ <entry><b>Reserved</b><p><b>in</b></p><p><b>Impala 3.0 and
+ higher</b></p></entry>
+ <entry><b>Future Keyword</b></entry>
+ </row>
+ <row>
+ <entry><codeph>abs</codeph></entry>
+ <entry>X</entry>
+ <entry/>
+ <entry/>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>acos</codeph></entry>
+ <entry>X</entry>
+ <entry/>
+ <entry/>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>add</codeph></entry>
+ <entry/>
+ <entry>X</entry>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>aggregate</codeph></entry>
+ <entry/>
+ <entry>X</entry>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>all</codeph></entry>
+ <entry>X</entry>
+ <entry>X</entry>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>allocate</codeph></entry>
+ <entry>X</entry>
+ <entry/>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>alter</codeph></entry>
+ <entry>X</entry>
+ <entry>X</entry>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>analytic</codeph></entry>
+ <entry/>
+ <entry>X</entry>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>and</codeph></entry>
+ <entry>X</entry>
+ <entry>X</entry>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>anti</codeph></entry>
+ <entry/>
+ <entry>X</entry>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>any</codeph></entry>
+ <entry>X</entry>
+ <entry/>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>api_version</codeph></entry>
+ <entry/>
+ <entry/>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>are</codeph></entry>
+ <entry>X</entry>
+ <entry/>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>array</codeph></entry>
+ <entry>X</entry>
+ <entry>X</entry>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>array_agg</codeph></entry>
+ <entry>X</entry>
+ <entry/>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>array_max_cardinality</codeph></entry>
+ <entry>X</entry>
+ <entry/>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>as</codeph></entry>
+ <entry>X</entry>
+ <entry>X</entry>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>asc</codeph></entry>
+ <entry/>
+ <entry>X</entry>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>asensitive</codeph></entry>
+ <entry>X</entry>
+ <entry/>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>asin</codeph></entry>
+ <entry>X</entry>
+ <entry/>
+ <entry/>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>asymmetric</codeph></entry>
+ <entry>X</entry>
+ <entry/>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>at</codeph></entry>
+ <entry>X</entry>
+ <entry/>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>atan</codeph></entry>
+ <entry>X</entry>
+ <entry/>
+ <entry/>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>atomic</codeph></entry>
+ <entry>X</entry>
+ <entry/>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>authorization</codeph></entry>
+ <entry>X</entry>
+ <entry/>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>avg</codeph></entry>
+ <entry>X</entry>
+ <entry/>
+ <entry/>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>avro</codeph></entry>
+ <entry/>
+ <entry>X</entry>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>backup</codeph></entry>
+ <entry/>
+ <entry/>
+ <entry/>
+ <entry>X</entry>
+ </row>
+ <row>
+ <entry><codeph>begin</codeph></entry>
+ <entry>X</entry>
+ <entry/>
+ <entry/>
+ <entry>X</entry>
+ </row>
+ <row>
+ <entry><codeph>begin_frame</codeph></entry>
+ <entry>X</entry>
+ <entry/>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>begin_partition</codeph></entry>
+ <entry>X</entry>
+ <entry/>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>between</codeph></entry>
+ <entry>X</entry>
+ <entry>X</entry>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>bigint</codeph></entry>
+ <entry>X</entry>
+ <entry>X</entry>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>binary</codeph></entry>
+ <entry>X</entry>
+ <entry>X</entry>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>blob</codeph></entry>
+ <entry>X</entry>
+ <entry/>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>block_size</codeph></entry>
+ <entry/>
+ <entry/>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>boolean</codeph></entry>
+ <entry>X</entry>
+ <entry>X</entry>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>both</codeph></entry>
+ <entry>X</entry>
+ <entry/>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>break</codeph></entry>
+ <entry/>
+ <entry/>
+ <entry/>
+ <entry>X</entry>
+ </row>
+ <row>
+ <entry><codeph>browse</codeph></entry>
+ <entry/>
+ <entry/>
+ <entry/>
+ <entry>X</entry>
+ </row>
+ <row>
+ <entry><codeph>bulk</codeph></entry>
+ <entry/>
+ <entry/>
+ <entry/>
+ <entry>X</entry>
+ </row>
+ <row>
+ <entry><codeph>by</codeph></entry>
+ <entry>X</entry>
+ <entry>X</entry>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>cache</codeph></entry>
+ <entry/>
+ <entry/>
+ <entry/>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>cached</codeph></entry>
+ <entry/>
+ <entry>X</entry>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>call</codeph></entry>
+ <entry>X</entry>
+ <entry/>
+ <entry/>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>called</codeph></entry>
+ <entry>X</entry>
+ <entry/>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>cardinality</codeph></entry>
+ <entry>X</entry>
+ <entry/>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>cascade</codeph></entry>
+ <entry/>
+ <entry>X</entry>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>cascaded</codeph></entry>
+ <entry>X</entry>
+ <entry/>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>case</codeph></entry>
+ <entry>X</entry>
+ <entry>X</entry>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>cast</codeph></entry>
+ <entry>X</entry>
+ <entry>X</entry>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>ceil</codeph></entry>
+ <entry>X</entry>
+ <entry/>
+ <entry/>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>ceiling</codeph></entry>
+ <entry>X</entry>
+ <entry/>
+ <entry/>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>change</codeph></entry>
+ <entry/>
+ <entry>X</entry>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>char</codeph></entry>
+ <entry>X</entry>
+ <entry>X</entry>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>char_length</codeph></entry>
+ <entry>X</entry>
+ <entry/>
+ <entry/>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>character</codeph></entry>
+ <entry>X</entry>
+ <entry/>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>character_length</codeph></entry>
+ <entry>X</entry>
+ <entry/>
+ <entry/>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>check</codeph></entry>
+ <entry>X</entry>
+ <entry/>
+ <entry/>
+ <entry>X</entry>
+ </row>
+ <row>
+ <entry><codeph>checkpoint</codeph></entry>
+ <entry/>
+ <entry/>
+ <entry/>
+ <entry>X</entry>
+ </row>
+ <row>
+ <entry><codeph>class</codeph></entry>
+ <entry/>
+ <entry>X</entry>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>classifier</codeph></entry>
+ <entry>X</entry>
+ <entry/>
+ <entry/>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>clob</codeph></entry>
+ <entry>X</entry>
+ <entry/>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>close</codeph></entry>
+ <entry>X</entry>
+ <entry/>
+ <entry/>
+ <entry>X</entry>
+ </row>
+ <row>
+ <entry><codeph>close_fn</codeph></entry>
+ <entry/>
+ <entry/>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>clustered</codeph></entry>
+ <entry/>
+ <entry/>
+ <entry/>
+ <entry>X</entry>
+ </row>
+ <row>
+ <entry><codeph>coalesce</codeph></entry>
+ <entry>X</entry>
+ <entry/>
+ <entry/>
+ <entry>X</entry>
+ </row>
+ <row>
+ <entry><codeph>collate</codeph></entry>
+ <entry>X</entry>
+ <entry/>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>collect</codeph></entry>
+ <entry>X</entry>
+ <entry/>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>column</codeph></entry>
+ <entry>X</entry>
+ <entry>X</entry>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>columns</codeph></entry>
+ <entry/>
+ <entry>X</entry>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>comment</codeph></entry>
+ <entry/>
+ <entry>X</entry>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>commit</codeph></entry>
+ <entry>X</entry>
+ <entry/>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>compression</codeph></entry>
+ <entry/>
+ <entry>X</entry>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>compute</codeph></entry>
+ <entry/>
+ <entry>X</entry>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>condition</codeph></entry>
+ <entry>X</entry>
+ <entry/>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>conf</codeph></entry>
+ <entry/>
+ <entry/>
+ <entry/>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>connect</codeph></entry>
+ <entry>X</entry>
+ <entry/>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>constraint</codeph></entry>
+ <entry>X</entry>
+ <entry/>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>contains</codeph></entry>
+ <entry>X</entry>
+ <entry/>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>continue</codeph></entry>
+ <entry/>
+ <entry/>
+ <entry/>
+ <entry>X</entry>
+ </row>
+ <row>
+ <entry><codeph>convert</codeph></entry>
+ <entry>X</entry>
+ <entry/>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>copy</codeph></entry>
+ <entry>X</entry>
+ <entry/>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>corr</codeph></entry>
+ <entry>X</entry>
+ <entry/>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>corresponding</codeph></entry>
+ <entry>X</entry>
+ <entry/>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>cos</codeph></entry>
+ <entry>X</entry>
+ <entry/>
+ <entry/>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>cosh</codeph></entry>
+ <entry>X</entry>
+ <entry/>
+ <entry/>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>count</codeph></entry>
+ <entry>X</entry>
+ <entry/>
+ <entry/>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>covar_pop</codeph></entry>
+ <entry>X</entry>
+ <entry/>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>covar_samp</codeph></entry>
+ <entry>X</entry>
+ <entry/>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>create</codeph></entry>
+ <entry>X</entry>
+ <entry>X</entry>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>cross</codeph></entry>
+ <entry>X</entry>
+ <entry>X</entry>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>cube</codeph></entry>
+ <entry>X</entry>
+ <entry/>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>cume_dist</codeph></entry>
+ <entry>X</entry>
+ <entry/>
+ <entry/>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>current</codeph></entry>
+ <entry>X</entry>
+ <entry>X</entry>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>current_catalog</codeph></entry>
+ <entry>X</entry>
+ <entry/>
+ <entry/>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>current_date</codeph></entry>
+ <entry>X</entry>
+ <entry/>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>current_default_transform_group</codeph></entry>
+ <entry>X</entry>
+ <entry/>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>current_path</codeph></entry>
+ <entry>X</entry>
+ <entry/>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>current_role</codeph></entry>
+ <entry>X</entry>
+ <entry/>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>current_row</codeph></entry>
+ <entry>X</entry>
+ <entry/>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>current_schema</codeph></entry>
+ <entry>X</entry>
+ <entry/>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>current_time</codeph></entry>
+ <entry>X</entry>
+ <entry/>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>current_timestamp</codeph></entry>
+ <entry>X</entry>
+ <entry/>
+ <entry/>
+ <entry>X</entry>
+ </row>
+ <row>
+ <entry><codeph>current_transform_group_for_type</codeph></entry>
+ <entry>X</entry>
+ <entry/>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>current_user</codeph></entry>
+ <entry>X</entry>
+ <entry/>
+ <entry/>
+ <entry>X</entry>
+ </row>
+ <row>
+ <entry><codeph>cursor</codeph></entry>
+ <entry>X</entry>
+ <entry/>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>cycle</codeph></entry>
+ <entry>X</entry>
+ <entry/>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>data</codeph></entry>
+ <entry/>
+ <entry>X</entry>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>database</codeph></entry>
+ <entry/>
+ <entry>X</entry>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>databases</codeph></entry>
+ <entry/>
+ <entry>X</entry>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>date</codeph></entry>
+ <entry>X</entry>
+ <entry>X</entry>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>datetime</codeph></entry>
+ <entry/>
+ <entry>X</entry>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>day</codeph></entry>
+ <entry>X</entry>
+ <entry/>
+ <entry/>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>dayofweek</codeph></entry>
+ <entry/>
+ <entry/>
+ <entry/>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>dbcc</codeph></entry>
+ <entry/>
+ <entry/>
+ <entry/>
+ <entry>X</entry>
+ </row>
+ <row>
+ <entry><codeph>deallocate</codeph></entry>
+ <entry>X</entry>
+ <entry/>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>dec</codeph></entry>
+ <entry>X</entry>
+ <entry/>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>decfloat</codeph></entry>
+ <entry>X</entry>
+ <entry/>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>decimal</codeph></entry>
+ <entry>X</entry>
+ <entry>X</entry>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>declare</codeph></entry>
+ <entry>X</entry>
+ <entry/>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>default</codeph></entry>
+ <entry>X</entry>
+ <entry>X</entry>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>define</codeph></entry>
+ <entry>X</entry>
+ <entry/>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>delete</codeph></entry>
+ <entry>X</entry>
+ <entry>X</entry>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>delimited</codeph></entry>
+ <entry/>
+ <entry>X</entry>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>dense_rank</codeph></entry>
+ <entry>X</entry>
+ <entry/>
+ <entry/>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>deny</codeph></entry>
+ <entry/>
+ <entry/>
+ <entry/>
+ <entry>X</entry>
+ </row>
+ <row>
+ <entry><codeph>deref</codeph></entry>
+ <entry>X</entry>
+ <entry/>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>desc</codeph></entry>
+ <entry/>
+ <entry>X</entry>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>describe</codeph></entry>
+ <entry>X</entry>
+ <entry>X</entry>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>deterministic</codeph></entry>
+ <entry>X</entry>
+ <entry/>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>disconnect</codeph></entry>
+ <entry>X</entry>
+ <entry/>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>disk</codeph></entry>
+ <entry/>
+ <entry/>
+ <entry/>
+ <entry>X</entry>
+ </row>
+ <row>
+ <entry><codeph>distinct</codeph></entry>
+ <entry>X</entry>
+ <entry>X</entry>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>distributed</codeph></entry>
+ <entry/>
+ <entry/>
+ <entry/>
+ <entry>X</entry>
+ </row>
+ <row>
+ <entry><codeph>div</codeph></entry>
+ <entry/>
+ <entry>X</entry>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>double</codeph></entry>
+ <entry>X</entry>
+ <entry>X</entry>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>drop</codeph></entry>
+ <entry>X</entry>
+ <entry>X</entry>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>dump</codeph></entry>
+ <entry/>
+ <entry/>
+ <entry/>
+ <entry>X</entry>
+ </row>
+ <row>
+ <entry><codeph>dynamic</codeph></entry>
+ <entry>X</entry>
+ <entry/>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>each</codeph></entry>
+ <entry>X</entry>
+ <entry/>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>element</codeph></entry>
+ <entry>X</entry>
+ <entry/>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>else</codeph></entry>
+ <entry>X</entry>
+ <entry>X</entry>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>empty</codeph></entry>
+ <entry>X</entry>
+ <entry/>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>encoding</codeph></entry>
+ <entry/>
+ <entry>X</entry>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>end</codeph></entry>
+ <entry>X</entry>
+ <entry>X</entry>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>end-exec</codeph></entry>
+ <entry>X</entry>
+ <entry/>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>end_frame</codeph></entry>
+ <entry>X</entry>
+ <entry/>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>end_partition</codeph></entry>
+ <entry>X</entry>
+ <entry/>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>equals</codeph></entry>
+ <entry>X</entry>
+ <entry/>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>errlvl</codeph></entry>
+ <entry/>
+ <entry/>
+ <entry/>
+ <entry>X</entry>
+ </row>
+ <row>
+ <entry><codeph>escape</codeph></entry>
+ <entry>X</entry>
+ <entry/>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>escaped</codeph></entry>
+ <entry/>
+ <entry>X</entry>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>every</codeph></entry>
+ <entry>X</entry>
+ <entry/>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>except</codeph></entry>
+ <entry>X</entry>
+ <entry/>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>exchange</codeph></entry>
+ <entry/>
+ <entry/>
+ <entry/>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>exec</codeph></entry>
+ <entry>X</entry>
+ <entry/>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>execute</codeph></entry>
+ <entry>X</entry>
+ <entry/>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>exists</codeph></entry>
+ <entry>X</entry>
+ <entry>X</entry>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>exit</codeph></entry>
+ <entry/>
+ <entry/>
+ <entry/>
+ <entry>X</entry>
+ </row>
+ <row>
+ <entry><codeph>exp</codeph></entry>
+ <entry>X</entry>
+ <entry/>
+ <entry/>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>explain</codeph></entry>
+ <entry/>
+ <entry>X</entry>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>extended</codeph></entry>
+ <entry/>
+ <entry>X</entry>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>external</codeph></entry>
+ <entry>X</entry>
+ <entry>X</entry>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>extract</codeph></entry>
+ <entry>X</entry>
+ <entry/>
+ <entry/>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>false</codeph></entry>
+ <entry>X</entry>
+ <entry>X</entry>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>fetch</codeph></entry>
+ <entry>X</entry>
+ <entry/>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>fields</codeph></entry>
+ <entry/>
+ <entry>X</entry>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>file</codeph></entry>
+ <entry/>
+ <entry/>
+ <entry/>
+ <entry>X</entry>
+ </row>
+ <row>
+ <entry><codeph>filefactor</codeph></entry>
+ <entry/>
+ <entry/>
+ <entry/>
+ <entry>X</entry>
+ </row>
+ <row>
+ <entry><codeph>fileformat</codeph></entry>
+ <entry/>
+ <entry>X</entry>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>files</codeph></entry>
+ <entry/>
+ <entry>X</entry>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>filter</codeph></entry>
+ <entry>X</entry>
+ <entry/>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>finalize_fn</codeph></entry>
+ <entry/>
+ <entry/>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>first</codeph></entry>
+ <entry/>
+ <entry>X</entry>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>first_value</codeph></entry>
+ <entry>X</entry>
+ <entry/>
+ <entry/>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>float</codeph></entry>
+ <entry>X</entry>
+ <entry>X</entry>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>floor</codeph></entry>
+ <entry>X</entry>
+ <entry/>
+ <entry/>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>following</codeph></entry>
+ <entry/>
+ <entry>X</entry>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>for</codeph></entry>
+ <entry>X</entry>
+ <entry>X</entry>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>foreign</codeph></entry>
+ <entry>X</entry>
+ <entry/>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>format</codeph></entry>
+ <entry/>
+ <entry>X</entry>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>formatted</codeph></entry>
+ <entry/>
+ <entry>X</entry>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>frame_row</codeph></entry>
+ <entry>X</entry>
+ <entry/>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>free</codeph></entry>
+ <entry>X</entry>
+ <entry/>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>freetext</codeph></entry>
+ <entry/>
+ <entry/>
+ <entry/>
+ <entry>X</entry>
+ </row>
+ <row>
+ <entry><codeph>from</codeph></entry>
+ <entry>X</entry>
+ <entry>X</entry>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>full</codeph></entry>
+ <entry>X</entry>
+ <entry>X</entry>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>function</codeph></entry>
+ <entry>X</entry>
+ <entry>X</entry>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>functions</codeph></entry>
+ <entry/>
+ <entry>X</entry>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>fusion</codeph></entry>
+ <entry>X</entry>
+ <entry/>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>get</codeph></entry>
+ <entry>X</entry>
+ <entry/>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>global</codeph></entry>
+ <entry>X</entry>
+ <entry/>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>goto</codeph></entry>
+ <entry/>
+ <entry/>
+ <entry/>
+ <entry>X</entry>
+ </row>
+ <row>
+ <entry><codeph>grant</codeph></entry>
+ <entry>X</entry>
+ <entry>X</entry>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>group</codeph></entry>
+ <entry>X</entry>
+ <entry>X</entry>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>grouping</codeph></entry>
+ <entry>X</entry>
+ <entry/>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>groups</codeph></entry>
+ <entry>X</entry>
+ <entry/>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>hash</codeph></entry>
+ <entry/>
+ <entry>X</entry>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>having</codeph></entry>
+ <entry>X</entry>
+ <entry>X</entry>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>hold</codeph></entry>
+ <entry>X</entry>
+ <entry/>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>holdlock</codeph></entry>
+ <entry/>
+ <entry/>
+ <entry/>
+ <entry>X</entry>
+ </row>
+ <row>
+ <entry><codeph>hour</codeph></entry>
+ <entry>X</entry>
+ <entry/>
+ <entry/>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>identity</codeph></entry>
+ <entry>X</entry>
+ <entry/>
+ <entry/>
+ <entry>X</entry>
+ </row>
+ <row>
+ <entry><codeph>if</codeph></entry>
+ <entry/>
+ <entry>X</entry>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>ignore</codeph></entry>
+ <entry/>
+ <entry>X</entry>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>ilike</codeph></entry>
+ <entry/>
+ <entry>X</entry>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>import</codeph></entry>
+ <entry/>
+ <entry/>
+ <entry/>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>in</codeph></entry>
+ <entry>X</entry>
+ <entry>X</entry>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>incremental</codeph></entry>
+ <entry/>
+ <entry>X</entry>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>index</codeph></entry>
+ <entry/>
+ <entry/>
+ <entry/>
+ <entry>X</entry>
+ </row>
+ <row>
+ <entry><codeph>indicator</codeph></entry>
+ <entry>X</entry>
+ <entry/>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>init_fn</codeph></entry>
+ <entry/>
+ <entry/>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>initial</codeph></entry>
+ <entry>X</entry>
+ <entry/>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>inner</codeph></entry>
+ <entry>X</entry>
+ <entry>X</entry>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>inout</codeph></entry>
+ <entry>X</entry>
+ <entry/>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>inpath</codeph></entry>
+ <entry/>
+ <entry>X</entry>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>insensitive</codeph></entry>
+ <entry>X</entry>
+ <entry/>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>insert</codeph></entry>
+ <entry>X</entry>
+ <entry>X</entry>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>int</codeph></entry>
+ <entry>X</entry>
+ <entry>X</entry>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>integer</codeph></entry>
+ <entry>X</entry>
+ <entry>X</entry>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>intermediate</codeph></entry>
+ <entry/>
+ <entry>X</entry>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>intersect</codeph></entry>
+ <entry>X</entry>
+ <entry/>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>intersection</codeph></entry>
+ <entry>X</entry>
+ <entry/>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>interval</codeph></entry>
+ <entry>X</entry>
+ <entry>X</entry>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>into</codeph></entry>
+ <entry>X</entry>
+ <entry>X</entry>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>invalidate</codeph></entry>
+ <entry/>
+ <entry>X</entry>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>iregexp</codeph></entry>
+ <entry/>
+ <entry>X</entry>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>is</codeph></entry>
+ <entry>X</entry>
+ <entry>X</entry>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>join</codeph></entry>
+ <entry>X</entry>
+ <entry>X</entry>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>json_array</codeph></entry>
+ <entry>X</entry>
+ <entry/>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>json_arrayagg</codeph></entry>
+ <entry>X</entry>
+ <entry/>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>json_exists</codeph></entry>
+ <entry>X</entry>
+ <entry/>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>json_object</codeph></entry>
+ <entry>X</entry>
+ <entry/>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>json_objectagg</codeph></entry>
+ <entry>X</entry>
+ <entry/>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>json_query</codeph></entry>
+ <entry>X</entry>
+ <entry/>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>json_table</codeph></entry>
+ <entry>X</entry>
+ <entry/>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>json_table_primitive</codeph></entry>
+ <entry>X</entry>
+ <entry/>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>json_value</codeph></entry>
+ <entry>X</entry>
+ <entry/>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>key</codeph></entry>
+ <entry/>
+ <entry/>
+ <entry/>
+ <entry>X</entry>
+ </row>
+ <row>
+ <entry><codeph>kill</codeph></entry>
+ <entry/>
+ <entry/>
+ <entry/>
+ <entry>X</entry>
+ </row>
+ <row>
+ <entry><codeph>kudu</codeph></entry>
+ <entry/>
+ <entry>X</entry>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>lag</codeph></entry>
+ <entry>X</entry>
+ <entry/>
+ <entry/>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>language</codeph></entry>
+ <entry>X</entry>
+ <entry/>
+ <entry/>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>large</codeph></entry>
+ <entry>X</entry>
+ <entry/>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>last</codeph></entry>
+ <entry/>
+ <entry>X</entry>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>last_value</codeph></entry>
+ <entry>X</entry>
+ <entry/>
+ <entry/>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>lateral</codeph></entry>
+ <entry>X</entry>
+ <entry/>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>lead</codeph></entry>
+ <entry>X</entry>
+ <entry/>
+ <entry/>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>leading</codeph></entry>
+ <entry>X</entry>
+ <entry/>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>left</codeph></entry>
+ <entry>X</entry>
+ <entry>X</entry>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>less</codeph></entry>
+ <entry/>
+ <entry/>
+ <entry/>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>like</codeph></entry>
+ <entry>X</entry>
+ <entry>X</entry>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>like_regex</codeph></entry>
+ <entry>X</entry>
+ <entry/>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>limit</codeph></entry>
+ <entry/>
+ <entry>X</entry>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>lineno</codeph></entry>
+ <entry/>
+ <entry/>
+ <entry/>
+ <entry>X</entry>
+ </row>
+ <row>
+ <entry><codeph>lines</codeph></entry>
+ <entry/>
+ <entry>X</entry>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>listagg</codeph></entry>
+ <entry>X</entry>
+ <entry/>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>ln</codeph></entry>
+ <entry>X</entry>
+ <entry/>
+ <entry/>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>load</codeph></entry>
+ <entry/>
+ <entry>X</entry>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>local</codeph></entry>
+ <entry>X</entry>
+ <entry/>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>localtime</codeph></entry>
+ <entry>X</entry>
+ <entry/>
+ <entry/>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>localtimestamp</codeph></entry>
+ <entry>X</entry>
+ <entry/>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>location</codeph></entry>
+ <entry/>
+ <entry>X</entry>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>log</codeph></entry>
+ <entry>X</entry>
+ <entry/>
+ <entry/>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>log10</codeph></entry>
+ <entry>X</entry>
+ <entry/>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>lower</codeph></entry>
+ <entry>X</entry>
+ <entry/>
+ <entry/>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>macro</codeph></entry>
+ <entry/>
+ <entry/>
+ <entry/>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>map</codeph></entry>
+ <entry/>
+ <entry>X</entry>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>match</codeph></entry>
+ <entry>X</entry>
+ <entry/>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>match_number</codeph></entry>
+ <entry>X</entry>
+ <entry/>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>match_recognize</codeph></entry>
+ <entry>X</entry>
+ <entry/>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>matches</codeph></entry>
+ <entry>X</entry>
+ <entry/>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>max</codeph></entry>
+ <entry>X</entry>
+ <entry/>
+ <entry/>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>member</codeph></entry>
+ <entry>X</entry>
+ <entry/>
+ <entry/>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>merge</codeph></entry>
+ <entry>X</entry>
+ <entry/>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>merge_fn</codeph></entry>
+ <entry/>
+ <entry/>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>metadata</codeph></entry>
+ <entry/>
+ <entry>X</entry>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>method</codeph></entry>
+ <entry>X</entry>
+ <entry/>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>min</codeph></entry>
+ <entry>X</entry>
+ <entry/>
+ <entry/>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>minute</codeph></entry>
+ <entry>X</entry>
+ <entry/>
+ <entry/>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>mod</codeph></entry>
+ <entry>X</entry>
+ <entry/>
+ <entry/>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>modifies</codeph></entry>
+ <entry>X</entry>
+ <entry/>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>module</codeph></entry>
+ <entry>X</entry>
+ <entry/>
+ <entry/>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>month</codeph></entry>
+ <entry>X</entry>
+ <entry/>
+ <entry/>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>more</codeph></entry>
+ <entry/>
+ <entry/>
+ <entry/>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>multiset</codeph></entry>
+ <entry>X</entry>
+ <entry/>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>national</codeph></entry>
+ <entry>X</entry>
+ <entry/>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>natural</codeph></entry>
+ <entry>X</entry>
+ <entry/>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>nchar</codeph></entry>
+ <entry>X</entry>
+ <entry/>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>nclob</codeph></entry>
+ <entry>X</entry>
+ <entry/>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>new</codeph></entry>
+ <entry>X</entry>
+ <entry/>
+ <entry/>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>no</codeph></entry>
+ <entry>X</entry>
+ <entry/>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>nocheck</codeph></entry>
+ <entry/>
+ <entry/>
+ <entry/>
+ <entry>X</entry>
+ </row>
+ <row>
+ <entry><codeph>nonclustered</codeph></entry>
+ <entry/>
+ <entry/>
+ <entry/>
+ <entry>X</entry>
+ </row>
+ <row>
+ <entry><codeph>none</codeph></entry>
+ <entry>X</entry>
+ <entry/>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>normalize</codeph></entry>
+ <entry>X</entry>
+ <entry/>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>not</codeph></entry>
+ <entry>X</entry>
+ <entry>X</entry>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>nth_value</codeph></entry>
+ <entry>X</entry>
+ <entry/>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>ntile</codeph></entry>
+ <entry>X</entry>
+ <entry/>
+ <entry/>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>null</codeph></entry>
+ <entry>X</entry>
+ <entry>X</entry>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>nullif</codeph></entry>
+ <entry>X</entry>
+ <entry/>
+ <entry/>
+ <entry>X</entry>
+ </row>
+ <row>
+ <entry><codeph>nulls</codeph></entry>
+ <entry/>
+ <entry>X</entry>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>numeric</codeph></entry>
+ <entry>X</entry>
+ <entry/>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>occurrences_regex</codeph></entry>
+ <entry>X</entry>
+ <entry/>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>octet_length</codeph></entry>
+ <entry>X</entry>
+ <entry/>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>of</codeph></entry>
+ <entry>X</entry>
+ <entry/>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>off</codeph></entry>
+ <entry/>
+ <entry/>
+ <entry/>
+ <entry>X</entry>
+ </row>
+ <row>
+ <entry><codeph>offset</codeph></entry>
+ <entry>X</entry>
+ <entry>X</entry>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>offsets</codeph></entry>
+ <entry/>
+ <entry/>
+ <entry/>
+ <entry>X</entry>
+ </row>
+ <row>
+ <entry><codeph>old</codeph></entry>
+ <entry>X</entry>
+ <entry/>
+ <entry/>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>omit</codeph></entry>
+ <entry>X</entry>
+ <entry/>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>on</codeph></entry>
+ <entry>X</entry>
+ <entry>X</entry>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>one</codeph></entry>
+ <entry>X</entry>
+ <entry/>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>only</codeph></entry>
+ <entry>X</entry>
+ <entry/>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>open</codeph></entry>
+ <entry>X</entry>
+ <entry/>
+ <entry/>
+ <entry>X</entry>
+ </row>
+ <row>
+ <entry><codeph>option</codeph></entry>
+ <entry/>
+ <entry/>
+ <entry/>
+ <entry>X</entry>
+ </row>
+ <row>
+ <entry><codeph>or</codeph></entry>
+ <entry>X</entry>
+ <entry>X</entry>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>order</codeph></entry>
+ <entry>X</entry>
+ <entry>X</entry>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>out</codeph></entry>
+ <entry>X</entry>
+ <entry/>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>outer</codeph></entry>
+ <entry>X</entry>
+ <entry>X</entry>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>over</codeph></entry>
+ <entry>X</entry>
+ <entry>X</entry>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>overlaps</codeph></entry>
+ <entry>X</entry>
+ <entry/>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>overlay</codeph></entry>
+ <entry>X</entry>
+ <entry/>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>overwrite</codeph></entry>
+ <entry/>
+ <entry>X</entry>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>parameter</codeph></entry>
+ <entry>X</entry>
+ <entry/>
+ <entry/>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>parquet</codeph></entry>
+ <entry/>
+ <entry>X</entry>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>parquetfile</codeph></entry>
+ <entry/>
+ <entry>X</entry>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>partialscan</codeph></entry>
+ <entry/>
+ <entry/>
+ <entry/>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>partition</codeph></entry>
+ <entry>X</entry>
+ <entry>X</entry>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>partitioned</codeph></entry>
+ <entry/>
+ <entry>X</entry>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>partitions</codeph></entry>
+ <entry/>
+ <entry>X</entry>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>pattern</codeph></entry>
+ <entry>X</entry>
+ <entry/>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>per</codeph></entry>
+ <entry>X</entry>
+ <entry/>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>percent</codeph></entry>
+ <entry>X</entry>
+ <entry/>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>percent_rank</codeph></entry>
+ <entry>X</entry>
+ <entry/>
+ <entry/>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>percentile_cont</codeph></entry>
+ <entry>X</entry>
+ <entry/>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>percentile_disc</codeph></entry>
+ <entry>X</entry>
+ <entry/>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>period</codeph></entry>
+ <entry>X</entry>
+ <entry/>
+ <entry/>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>pivot</codeph></entry>
+ <entry/>
+ <entry/>
+ <entry/>
+ <entry>X</entry>
+ </row>
+ <row>
+ <entry><codeph>plan</codeph></entry>
+ <entry/>
+ <entry/>
+ <entry/>
+ <entry>X</entry>
+ </row>
+ <row>
+ <entry><codeph>portion</codeph></entry>
+ <entry>X</entry>
+ <entry/>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>position</codeph></entry>
+ <entry>X</entry>
+ <entry/>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>position_regex</codeph></entry>
+ <entry>X</entry>
+ <entry/>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>power</codeph></entry>
+ <entry>X</entry>
+ <entry/>
+ <entry/>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>precedes</codeph></entry>
+ <entry>X</entry>
+ <entry/>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>preceding</codeph></entry>
+ <entry/>
+ <entry>X</entry>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>precision</codeph></entry>
+ <entry>X</entry>
+ <entry/>
+ <entry/>
+ <entry>X</entry>
+ </row>
+ <row>
+ <entry><codeph>prepare</codeph></entry>
+ <entry>X</entry>
+ <entry/>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>prepare_fn</codeph></entry>
+ <entry/>
+ <entry/>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>preserve</codeph></entry>
+ <entry/>
+ <entry/>
+ <entry/>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>primary</codeph></entry>
+ <entry>X</entry>
+ <entry>X</entry>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>print</codeph></entry>
+ <entry/>
+ <entry/>
+ <entry/>
+ <entry>X</entry>
+ </row>
+ <row>
+ <entry><codeph>proc</codeph></entry>
+ <entry/>
+ <entry/>
+ <entry/>
+ <entry>X</entry>
+ </row>
+ <row>
+ <entry><codeph>procedure</codeph></entry>
+ <entry>X</entry>
+ <entry/>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>produced</codeph></entry>
+ <entry/>
+ <entry>X</entry>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>ptf</codeph></entry>
+ <entry>X</entry>
+ <entry/>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>public</codeph></entry>
+ <entry/>
+ <entry/>
+ <entry/>
+ <entry>X</entry>
+ </row>
+ <row>
+ <entry><codeph>purge</codeph></entry>
+ <entry/>
+ <entry>X</entry>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>raiseerror</codeph></entry>
+ <entry/>
+ <entry/>
+ <entry/>
+ <entry>X</entry>
+ </row>
+ <row>
+ <entry><codeph>range</codeph></entry>
+ <entry>X</entry>
+ <entry>X</entry>
+ <entry>X</entry>
+ <entry/>
+ </row>
+ <row>
+ <entry><codeph>rank</codeph></entry>
+ <entry>X</entry>
+ <entry/>
+ <entry/>
+ <entry/>
+ </row>
+ <row>
+
<TRUNCATED>
[2/8] impala git commit: IMPALA-3916: [DOCS] Reserved keywords
updated for Impala 3.0
Posted by ta...@apache.org.
IMPALA-3916: [DOCS] Reserved keywords updated for Impala 3.0
Change-Id: I09dbaf5e2d22f07a8bca8a601c04e94f3a4b36a0
Cherry-picks: not for 2.x
Reviewed-on: http://gerrit.cloudera.org:8080/9540
Reviewed-by: Alex Behm <al...@cloudera.com>
Tested-by: Impala Public Jenkins
Project: http://git-wip-us.apache.org/repos/asf/impala/repo
Commit: http://git-wip-us.apache.org/repos/asf/impala/commit/6a07076d
Tree: http://git-wip-us.apache.org/repos/asf/impala/tree/6a07076d
Diff: http://git-wip-us.apache.org/repos/asf/impala/diff/6a07076d
Branch: refs/heads/master
Commit: 6a07076de39645dd39b5d752cb7b3643f6b8e696
Parents: 0379438
Author: Alex Rodoni <ar...@cloudera.com>
Authored: Wed Mar 7 14:52:54 2018 -0800
Committer: Impala Public Jenkins <im...@gerrit.cloudera.org>
Committed: Thu Mar 15 00:38:17 2018 +0000
----------------------------------------------------------------------
docs/topics/impala_reserved_words.xml | 4163 +++++++++++++++++++++++++---
1 file changed, 3834 insertions(+), 329 deletions(-)
----------------------------------------------------------------------
[8/8] impala git commit: IMPALA-6488: removes use-after-free bug in
lib_cache
Posted by ta...@apache.org.
IMPALA-6488: removes use-after-free bug in lib_cache
Several recent runs have resulted in a boost mutex
invalid argument exception. The mutex in question is
the one that guards individual lib_cache entries
(LibCacheEntry::lock).
The exception is thrown due to the entry being deleted
by another thread prior to the current thread acquiring
the entry lock. This scenario happens when:
1) thread t1 looks up an existing entry e
a. the lookup is guarded by a cache lock
b. the cache lock is released (L356)
c. e's lock is acquired on the next line and propagated
up the call stack (look for the swaps)
d. while e is locked, its use-count is incremented.
2) thread t2 deletes entry e
a. the cache lock is acquired and e is looked up
b. e's lock is acquired
c. if e's usecount is 0 and was marked for removal, e is deleted.
If t2 runs following (1b), then t1 will acquire a lock on a deleted
entry (1c), causing the mutex exception.
There are two parts to the fix in this change:
(1) don't crash and (2) don't let concurrency regress.
1) remove 1b: keep the cache lock while acquiring e's lock.
The cache lock is then released and all other operations proceed as before.
Note that current lock ordering is still maintained (coarse to fine),
but now, the cache lock can be held while other threads block access
to the entry lock. When files have been copied, these operations
are short. While files are loading, accesses to the same entry
can serialize access to the entire cache.
2) add cache entry after its loaded.
Currently, on a cache miss, a new entry is created, locked,
added to the cache, and finally the cache is unlocked. The intent
is to allow other threads to concurrently load files for other entries.
However, now that the cache lock is held while the entry lock is held,
the loading thread will block other thread's from acquiring the same entry,
which will block access to the cache. The work-around is to release
the cache lock when loading a new entry and add the cache entry only when
its loaded. The workaround avoids expensive work while holding the
cache lock, but may do wasted work if multiple threads miss on the
same entry and load their entries in parallel.
Testing:
- ran core tests + hdfs
- added an end-to-end test that concurrently uses and drops/creates from
the same lib-cache entry. with current settings, the use-after-free
error is reliably reproduced.
- manual testing to examine concurrency of the following cases:
- concurrent function creation from multiple lib files
(stresses coordinator)
- concurrent function use from multiple lib files
(stresses backend)
Change-Id: I1f178a2114cb3969dbb920949ddff4e8220b742e
Reviewed-on: http://gerrit.cloudera.org:8080/9626
Reviewed-by: Dan Hecht <dh...@cloudera.com>
Tested-by: Impala Public Jenkins
Project: http://git-wip-us.apache.org/repos/asf/impala/repo
Commit: http://git-wip-us.apache.org/repos/asf/impala/commit/241031d3
Tree: http://git-wip-us.apache.org/repos/asf/impala/tree/241031d3
Diff: http://git-wip-us.apache.org/repos/asf/impala/diff/241031d3
Branch: refs/heads/master
Commit: 241031d339a7a78ccc75f5c6c6146028bb9c76db
Parents: 69f6bfd
Author: Vuk Ercegovac <ve...@cloudera.com>
Authored: Fri Mar 9 18:53:57 2018 -0800
Committer: Impala Public Jenkins <im...@gerrit.cloudera.org>
Committed: Thu Mar 15 19:50:54 2018 +0000
----------------------------------------------------------------------
be/src/runtime/lib-cache.cc | 217 +++++++++++++++++++++----------------
be/src/runtime/lib-cache.h | 19 +++-
tests/query_test/test_udfs.py | 73 +++++++++++++
3 files changed, 215 insertions(+), 94 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/impala/blob/241031d3/be/src/runtime/lib-cache.cc
----------------------------------------------------------------------
diff --git a/be/src/runtime/lib-cache.cc b/be/src/runtime/lib-cache.cc
index d694c49..c7ca6cb 100644
--- a/be/src/runtime/lib-cache.cc
+++ b/be/src/runtime/lib-cache.cc
@@ -86,42 +86,44 @@ struct LibCacheEntry {
// before it is removed to return the same error.
Status loading_status;
- LibCacheEntry() : use_count(0), should_remove(false), check_needs_refresh(false),
- shared_object_handle(NULL) {}
+ LibCacheEntry()
+ : use_count(0),
+ should_remove(false),
+ check_needs_refresh(false),
+ shared_object_handle(nullptr) {}
~LibCacheEntry();
};
-LibCache::LibCache() : current_process_handle_(NULL) {
-}
+LibCache::LibCache() : current_process_handle_(nullptr) {}
LibCache::~LibCache() {
DropCache();
- if (current_process_handle_ != NULL) DynamicClose(current_process_handle_);
+ if (current_process_handle_ != nullptr) DynamicClose(current_process_handle_);
}
Status LibCache::Init() {
- DCHECK(LibCache::instance_.get() == NULL);
+ DCHECK(LibCache::instance_.get() == nullptr);
LibCache::instance_.reset(new LibCache());
return LibCache::instance_->InitInternal();
}
Status LibCache::InitInternal() {
if (TestInfo::is_fe_test()) {
- // In the FE tests, NULL gives the handle to the java process.
+ // In the FE tests, nullptr gives the handle to the java process.
// Explicitly load the fe-support shared object.
string fe_support_path;
PathBuilder::GetFullBuildPath("service/libfesupport.so", &fe_support_path);
RETURN_IF_ERROR(DynamicOpen(fe_support_path.c_str(), ¤t_process_handle_));
} else {
- RETURN_IF_ERROR(DynamicOpen(NULL, ¤t_process_handle_));
+ RETURN_IF_ERROR(DynamicOpen(nullptr, ¤t_process_handle_));
}
- DCHECK(current_process_handle_ != NULL)
+ DCHECK(current_process_handle_ != nullptr)
<< "We should always be able to get current process handle.";
return Status::OK();
}
LibCacheEntry::~LibCacheEntry() {
- if (shared_object_handle != NULL) {
+ if (shared_object_handle != nullptr) {
DCHECK_EQ(use_count, 0);
DCHECK(should_remove);
DynamicClose(shared_object_handle);
@@ -133,14 +135,14 @@ Status LibCache::GetSoFunctionPtr(const string& hdfs_lib_file, const string& sym
void** fn_ptr, LibCacheEntry** ent, bool quiet) {
if (hdfs_lib_file.empty()) {
// Just loading a function ptr in the current process. No need to take any locks.
- DCHECK(current_process_handle_ != NULL);
+ DCHECK(current_process_handle_ != nullptr);
RETURN_IF_ERROR(DynamicLookup(current_process_handle_, symbol.c_str(), fn_ptr, quiet));
return Status::OK();
}
- LibCacheEntry* entry = NULL;
+ LibCacheEntry* entry = nullptr;
unique_lock<mutex> lock;
- if (ent != NULL && *ent != NULL) {
+ if (ent != nullptr && *ent != nullptr) {
// Reuse already-cached entry provided by user
entry = *ent;
unique_lock<mutex> l(entry->lock);
@@ -148,9 +150,8 @@ Status LibCache::GetSoFunctionPtr(const string& hdfs_lib_file, const string& sym
} else {
RETURN_IF_ERROR(GetCacheEntry(hdfs_lib_file, TYPE_SO, &lock, &entry));
}
- DCHECK(entry != NULL);
+ DCHECK(entry != nullptr);
DCHECK_EQ(entry->type, TYPE_SO);
-
LibCacheEntry::SymbolMap::iterator it = entry->symbol_cache.find(symbol);
if (it != entry->symbol_cache.end()) {
*fn_ptr = it->second;
@@ -160,8 +161,8 @@ Status LibCache::GetSoFunctionPtr(const string& hdfs_lib_file, const string& sym
entry->symbol_cache[symbol] = *fn_ptr;
}
- DCHECK(*fn_ptr != NULL);
- if (ent != NULL && *ent == NULL) {
+ DCHECK(*fn_ptr != nullptr);
+ if (ent != nullptr && *ent == nullptr) {
// Only set and increment user's entry if it wasn't already cached
*ent = entry;
++(*ent)->use_count;
@@ -170,10 +171,10 @@ Status LibCache::GetSoFunctionPtr(const string& hdfs_lib_file, const string& sym
}
void LibCache::DecrementUseCount(LibCacheEntry* entry) {
- if (entry == NULL) return;
+ if (entry == nullptr) return;
bool can_delete = false;
{
- unique_lock<mutex> lock(entry->lock);;
+ unique_lock<mutex> lock(entry->lock);
--entry->use_count;
can_delete = (entry->use_count == 0 && entry->should_remove);
}
@@ -183,9 +184,9 @@ void LibCache::DecrementUseCount(LibCacheEntry* entry) {
Status LibCache::GetLocalLibPath(const string& hdfs_lib_file, LibType type,
string* local_path) {
unique_lock<mutex> lock;
- LibCacheEntry* entry = NULL;
+ LibCacheEntry* entry = nullptr;
RETURN_IF_ERROR(GetCacheEntry(hdfs_lib_file, type, &lock, &entry));
- DCHECK(entry != NULL);
+ DCHECK(entry != nullptr);
DCHECK_EQ(entry->type, type);
*local_path = entry->local_path;
return Status::OK();
@@ -194,13 +195,13 @@ Status LibCache::GetLocalLibPath(const string& hdfs_lib_file, LibType type,
Status LibCache::CheckSymbolExists(const string& hdfs_lib_file, LibType type,
const string& symbol, bool quiet) {
if (type == TYPE_SO) {
- void* dummy_ptr = NULL;
- return GetSoFunctionPtr(hdfs_lib_file, symbol, &dummy_ptr, NULL, quiet);
+ void* dummy_ptr = nullptr;
+ return GetSoFunctionPtr(hdfs_lib_file, symbol, &dummy_ptr, nullptr, quiet);
} else if (type == TYPE_IR) {
unique_lock<mutex> lock;
- LibCacheEntry* entry = NULL;
+ LibCacheEntry* entry = nullptr;
RETURN_IF_ERROR(GetCacheEntry(hdfs_lib_file, type, &lock, &entry));
- DCHECK(entry != NULL);
+ DCHECK(entry != nullptr);
DCHECK_EQ(entry->type, TYPE_IR);
if (entry->symbols.find(symbol) == entry->symbols.end()) {
stringstream ss;
@@ -212,7 +213,7 @@ Status LibCache::CheckSymbolExists(const string& hdfs_lib_file, LibType type,
} else if (type == TYPE_JAR) {
// TODO: figure out how to inspect contents of jars
unique_lock<mutex> lock;
- LibCacheEntry* dummy_entry = NULL;
+ LibCacheEntry* dummy_entry = nullptr;
return GetCacheEntry(hdfs_lib_file, type, &lock, &dummy_entry);
} else {
DCHECK(false);
@@ -238,8 +239,8 @@ void LibCache::RemoveEntry(const string& hdfs_lib_file) {
RemoveEntryInternal(hdfs_lib_file, it);
}
-void LibCache::RemoveEntryInternal(const string& hdfs_lib_file,
- const LibMap::iterator& entry_iter) {
+void LibCache::RemoveEntryInternal(
+ const string& hdfs_lib_file, const LibMap::iterator& entry_iter) {
LibCacheEntry* entry = entry_iter->second;
VLOG(1) << "Removing lib cache entry: " << hdfs_lib_file
<< ", local path: " << entry->local_path;
@@ -291,7 +292,7 @@ Status LibCache::GetCacheEntry(const string& hdfs_lib_file, LibType type,
entry_lock->swap(local_entry_lock);
return status;
}
- if (*entry == NULL) return status;
+ if (*entry == nullptr) return status;
// Set loading_status on the entry so that if another thread calls
// GetCacheEntry() for this lib before this thread is able to acquire lock_ in
@@ -306,77 +307,111 @@ Status LibCache::GetCacheEntry(const string& hdfs_lib_file, LibType type,
Status LibCache::GetCacheEntryInternal(const string& hdfs_lib_file, LibType type,
unique_lock<mutex>* entry_lock, LibCacheEntry** entry) {
DCHECK(!hdfs_lib_file.empty());
- *entry = NULL;
+ *entry = nullptr;
- // Check if this file is already cached or an error occured on another thread while
- // loading the library.
- unique_lock<mutex> lib_cache_lock(lock_);
- LibMap::iterator it = lib_cache_.find(hdfs_lib_file);
- if (it != lib_cache_.end()) {
- {
- unique_lock<mutex> local_entry_lock((it->second)->lock);
- if (!(it->second)->loading_status.ok()) {
- // If loading_status is already set, the returned *entry should be NULL.
- DCHECK(*entry == NULL);
- return (it->second)->loading_status;
- }
+ // Check if this file is already cached. Refresh the entry if needed.
+ {
+ unique_lock<mutex> lib_cache_lock(lock_);
+ LibMap::iterator it = lib_cache_.find(hdfs_lib_file);
+ if (it != lib_cache_.end()) {
+ RETURN_IF_ERROR(RefreshCacheEntry(hdfs_lib_file, type, it, entry_lock, entry));
+ if (*entry != nullptr) return Status::OK();
}
+ }
+
+ // Entry didn't exist. Create a new entry and load it. Note that the cache lock is
+ // *not* held and the entry is not added to the cache until it is loaded. Loading is
+ // expensive, so *not* holding the cache lock and *not* making the entry visible to
+ // other threads avoids blocking other threads with an expensive operation.
+ unique_ptr<LibCacheEntry> new_entry = make_unique<LibCacheEntry>();
+ RETURN_IF_ERROR(LoadCacheEntry(hdfs_lib_file, type, new_entry.get()));
- *entry = it->second;
- if ((*entry)->check_needs_refresh) {
- // Check if file has been modified since loading the cached copy. If so, remove the
- // cached entry and create a new one.
- (*entry)->check_needs_refresh = false;
- time_t last_mod_time;
- hdfsFS hdfs_conn;
- Status status = HdfsFsCache::instance()->GetConnection(hdfs_lib_file, &hdfs_conn);
- if (!status.ok()) {
- RemoveEntryInternal(hdfs_lib_file, it);
- *entry = NULL;
+ // Entry is now loaded. Check that another thread did not already load and add an entry
+ // for the same key. If so, refresh it if needed. If the existing entry is valid, then
+ // use it and discard new_entry.
+ {
+ unique_lock<mutex> lib_cache_lock(lock_);
+ LibMap::iterator it = lib_cache_.find(hdfs_lib_file);
+ if (it != lib_cache_.end()) {
+ Status status = RefreshCacheEntry(hdfs_lib_file, type, it, entry_lock, entry);
+ // The entry lock is held at this point if entry is valid.
+ if (!status.ok() || *entry != nullptr) {
+ // new_entry will be discarded; while wasted work, it avoids holding
+ // the cache lock while loading.
+ new_entry->should_remove = true;
return status;
}
- status = GetLastModificationTime(hdfs_conn, hdfs_lib_file.c_str(), &last_mod_time);
- if (!status.ok() || (*entry)->last_mod_time < last_mod_time) {
- RemoveEntryInternal(hdfs_lib_file, it);
- *entry = NULL;
- }
- RETURN_IF_ERROR(status);
}
+
+ // The entry was not found or was removed for refresh. Use the new entry, so
+ // lock it and add it to the cache.
+ *entry = new_entry.release();
+ unique_lock<mutex> local_entry_lock((*entry)->lock);
+ entry_lock->swap(local_entry_lock);
+ lib_cache_[hdfs_lib_file] = *entry;
}
+ return Status::OK();
+}
- if (*entry != NULL) {
- // Release the lib_cache_ lock. This guarantees other threads looking at other
- // libs can continue.
- lib_cache_lock.unlock();
+Status LibCache::RefreshCacheEntry(const string& hdfs_lib_file, LibType type,
+ const LibMap::iterator& iter, unique_lock<mutex>* entry_lock, LibCacheEntry** entry) {
+ // Check if an error occurred on another thread while loading the library.
+ {
+ unique_lock<mutex> local_entry_lock((iter->second)->lock);
+ if (!(iter->second)->loading_status.ok()) {
+ // If loading_status is already set, the returned *entry should be nullptr.
+ DCHECK(*entry == nullptr);
+ return (iter->second)->loading_status;
+ }
+ }
+
+ // Refresh the cache entry if needed. If refreshed or an error occurred, remove
+ // the entry and set the returned entry to nullptr.
+ *entry = iter->second;
+ if ((*entry)->check_needs_refresh) {
+ // Check if file has been modified since loading the cached copy. If so, remove the
+ // cached entry and create a new one.
+ (*entry)->check_needs_refresh = false;
+ time_t last_mod_time;
+ hdfsFS hdfs_conn;
+ Status status = HdfsFsCache::instance()->GetConnection(hdfs_lib_file, &hdfs_conn);
+ if (!status.ok()) {
+ RemoveEntryInternal(hdfs_lib_file, iter);
+ *entry = nullptr;
+ return status;
+ }
+ status = GetLastModificationTime(hdfs_conn, hdfs_lib_file.c_str(), &last_mod_time);
+ if (!status.ok() || (*entry)->last_mod_time < last_mod_time) {
+ RemoveEntryInternal(hdfs_lib_file, iter);
+ *entry = nullptr;
+ }
+ RETURN_IF_ERROR(status);
+ }
+
+ // No refresh needed, the entry can be used.
+ if (*entry != nullptr) {
+ // The cache level lock continues to be held while the entry lock is obtained
+ // so that some other thread does not access the entry and delete it.
unique_lock<mutex> local_entry_lock((*entry)->lock);
entry_lock->swap(local_entry_lock);
+ // Let the caller propagate any error that occurred when loading the entry.
RETURN_IF_ERROR((*entry)->copy_file_status);
DCHECK_EQ((*entry)->type, type);
DCHECK(!(*entry)->local_path.empty());
- return Status::OK();
}
+ return Status::OK();
+}
- // Entry didn't exist. Add the entry then release lock_ (so other libraries
- // can be accessed).
- *entry = new LibCacheEntry();
-
- // Grab the entry lock before adding it to lib_cache_. We still need to do more
- // work to initialize *entry and we don't want another thread to pick up
- // the uninitialized entry.
- unique_lock<mutex> local_entry_lock((*entry)->lock);
- entry_lock->swap(local_entry_lock);
- lib_cache_[hdfs_lib_file] = *entry;
- lib_cache_lock.unlock();
-
- // At this point we have the entry lock but not the lib cache lock.
- DCHECK(*entry != NULL);
- (*entry)->type = type;
+Status LibCache::LoadCacheEntry(
+ const std::string& hdfs_lib_file, LibType type, LibCacheEntry* entry) {
+ DCHECK(entry != nullptr);
+ entry->type = type;
// Copy the file
- (*entry)->local_path = MakeLocalPath(hdfs_lib_file, FLAGS_local_library_dir);
+ entry->local_path = MakeLocalPath(hdfs_lib_file, FLAGS_local_library_dir);
VLOG(1) << "Adding lib cache entry: " << hdfs_lib_file
- << ", local path: " << (*entry)->local_path;
+ << ", local path: " << entry->local_path;
hdfsFS hdfs_conn, local_conn;
RETURN_IF_ERROR(HdfsFsCache::instance()->GetConnection(hdfs_lib_file, &hdfs_conn));
@@ -385,28 +420,26 @@ Status LibCache::GetCacheEntryInternal(const string& hdfs_lib_file, LibType type
// Note: the file can be updated between getting last_mod_time and copying the file to
// local_path. This can only result in the file unnecessarily being refreshed, and does
// not affect correctness.
- (*entry)->copy_file_status = GetLastModificationTime(
- hdfs_conn, hdfs_lib_file.c_str(), &(*entry)->last_mod_time);
- RETURN_IF_ERROR((*entry)->copy_file_status);
+ entry->copy_file_status =
+ GetLastModificationTime(hdfs_conn, hdfs_lib_file.c_str(), &entry->last_mod_time);
+ RETURN_IF_ERROR(entry->copy_file_status);
- (*entry)->copy_file_status = CopyHdfsFile(
- hdfs_conn, hdfs_lib_file, local_conn, (*entry)->local_path);
- RETURN_IF_ERROR((*entry)->copy_file_status);
+ entry->copy_file_status =
+ CopyHdfsFile(hdfs_conn, hdfs_lib_file, local_conn, entry->local_path);
+ RETURN_IF_ERROR(entry->copy_file_status);
if (type == TYPE_SO) {
// dlopen the local library
- RETURN_IF_ERROR(
- DynamicOpen((*entry)->local_path.c_str(), &(*entry)->shared_object_handle));
+ RETURN_IF_ERROR(DynamicOpen(entry->local_path.c_str(), &entry->shared_object_handle));
} else if (type == TYPE_IR) {
// Load the module temporarily and populate all symbols.
- const string file = (*entry)->local_path;
+ const string file = entry->local_path;
const string module_id = filesystem::path(file).stem().string();
- RETURN_IF_ERROR(LlvmCodeGen::GetSymbols(file, module_id, &(*entry)->symbols));
+ RETURN_IF_ERROR(LlvmCodeGen::GetSymbols(file, module_id, &entry->symbols));
} else {
DCHECK_EQ(type, TYPE_JAR);
// Nothing to do.
}
-
return Status::OK();
}
http://git-wip-us.apache.org/repos/asf/impala/blob/241031d3/be/src/runtime/lib-cache.h
----------------------------------------------------------------------
diff --git a/be/src/runtime/lib-cache.h b/be/src/runtime/lib-cache.h
index 4a564ee..b1b70b1 100644
--- a/be/src/runtime/lib-cache.h
+++ b/be/src/runtime/lib-cache.h
@@ -151,6 +151,21 @@ class LibCache {
Status GetCacheEntryInternal(const std::string& hdfs_lib_file, LibType type,
boost::unique_lock<boost::mutex>* entry_lock, LibCacheEntry** entry);
+ /// Returns iter's cache entry in 'entry' with 'entry_lock' held if entry does not
+ /// need to be refreshed.
+ /// If entry needs to be refreshed, then it is removed and '*entry' is set to nullptr.
+ /// The cache lock must be held prior to calling this method. On return the entry's
+ /// lock is taken and returned in '*entry_lock' if entry does not need to be refreshed.
+ /// TODO: cleanup this method's interface and how the outputs are used.
+ Status RefreshCacheEntry(const std::string& hdfs_lib_file, LibType type,
+ const LibMap::iterator& iter, boost::unique_lock<boost::mutex>* entry_lock,
+ LibCacheEntry** entry);
+
+ /// 'hdfs_lib_file' is copied locally and 'entry' is initialized with its contents.
+ /// No locks are assumed held; 'entry' should be visible only to a single thread.
+ Status LoadCacheEntry(
+ const std::string& hdfs_lib_file, LibType type, LibCacheEntry* entry);
+
/// Utility function for generating a filename unique to this process and
/// 'hdfs_path'. This is to prevent multiple impalad processes or different library files
/// with the same name from clobbering each other. 'hdfs_path' should be the full path
@@ -160,8 +175,8 @@ class LibCache {
/// Implementation to remove an entry from the cache.
/// lock_ must be held. The entry's lock should not be held.
- void RemoveEntryInternal(const std::string& hdfs_lib_file,
- const LibMap::iterator& entry_iterator);
+ void RemoveEntryInternal(
+ const std::string& hdfs_lib_file, const LibMap::iterator& entry_iterator);
};
}
http://git-wip-us.apache.org/repos/asf/impala/blob/241031d3/tests/query_test/test_udfs.py
----------------------------------------------------------------------
diff --git a/tests/query_test/test_udfs.py b/tests/query_test/test_udfs.py
index 5252363..17890a4 100644
--- a/tests/query_test/test_udfs.py
+++ b/tests/query_test/test_udfs.py
@@ -302,6 +302,79 @@ class TestUdfExecution(TestUdfBase):
self.run_test_case('QueryTest/udf-non-deterministic', vector,
use_db=unique_database)
+ def test_native_functions_race(self, vector, unique_database):
+ """ IMPALA-6488: stress concurrent adds, uses, and deletes of native functions.
+ Exposes a crash caused by use-after-free in lib-cache."""
+
+ # Native function used by a query. Stresses lib-cache during analysis and
+ # backend expressions.
+ create_fn_to_use = """create function {0}.use_it(string) returns string
+ LOCATION '{1}'
+ SYMBOL='_Z8IdentityPN10impala_udf15FunctionContextERKNS_9StringValE'"""
+ use_fn = """select * from (select max(int_col) from functional.alltypesagg
+ where {0}.use_it(string_col) = 'blah' union all
+ (select max(int_col) from functional.alltypesagg
+ where {0}.use_it(String_col) > '1' union all
+ (select max(int_col) from functional.alltypesagg
+ where {0}.use_it(string_col) > '1'))) v"""
+ # Reference to another native function from the same 'so' file. Creating/dropping
+ # stresses lib-cache lookup, add, and refresh.
+ create_another_fn = """create function if not exists {0}.other(float)
+ returns float location '{1}' symbol='Identity'"""
+ drop_another_fn = """drop function if exists {0}.other(float)"""
+
+ setup_client = self.create_impala_client()
+ setup_query = create_fn_to_use.format(unique_database, '/test-warehouse/libTestUdfs.so')
+ try:
+ setup_client.execute(setup_query)
+ except Exception as e:
+ print "Unable to create initial function: {0}".format(setup_query)
+ raise
+
+ errors = []
+ def use_fn_method():
+ time.sleep(1 + random.random())
+ client = self.create_impala_client()
+ query = use_fn.format(unique_database)
+ try:
+ client.execute(query)
+ except Exception as e:
+ errors.append(e)
+
+ def load_fn_method():
+ time.sleep(1 + random.random())
+ client = self.create_impala_client()
+ drop = drop_another_fn.format(unique_database)
+ create = create_another_fn.format(unique_database, '/test-warehouse/libTestUdfs.so')
+ try:
+ client.execute(drop)
+ client.execute(create)
+ except Exception as e:
+ errors.append(e)
+
+ # number of uses/loads needed to reliably reproduce the bug.
+ num_uses = 200
+ num_loads = 200
+
+ # create threads to use native function.
+ runner_threads = []
+ for i in xrange(num_uses):
+ runner_threads.append(threading.Thread(target=use_fn_method))
+
+ # create threads to drop/create native functions.
+ for i in xrange(num_loads):
+ runner_threads.append(threading.Thread(target=load_fn_method))
+
+ # launch all runner threads.
+ for t in runner_threads: t.start()
+
+ # join all threads.
+ for t in runner_threads: t.join();
+
+ # Check for any errors.
+ for e in errors: print e
+ assert len(errors) == 0
+
def test_ir_functions(self, vector, unique_database):
if vector.get_value('exec_option')['disable_codegen']:
# IR functions require codegen to be enabled.
[5/8] impala git commit: IMPALA-5270: Pass resolved exprs into
analytic SortInfo.
Posted by ta...@apache.org.
IMPALA-5270: Pass resolved exprs into analytic SortInfo.
The bug was that the SortInfo of analytics was given
ordering exprs that were not fully resolved against their
input (e.g. inline views were not resolved).
As a result, the SortInfo logic did not materialize exprs
like rand() coming from inline views.
The fix is to pass fully resolved exprs to the analytic
SortInfo, and then the existing materialization logic
properly handles non-deterministic built-ins and UDFs.
The code around sort generation was rather convoluted
and difficult to understand. I overhauled SortInfo to
unify the different uses of it under a common codepath
After that cleanup, the fix for this issue was trivial.
Testing:
- Locally ran planner tests
- Locally ran analytic EE tests in test_queries.py
- Core/hdfs run passed
Change-Id: Id2b3f4e5e3f1fd441a63160db3c703c432fbb072
Reviewed-on: http://gerrit.cloudera.org:8080/9631
Reviewed-by: Alex Behm <al...@cloudera.com>
Tested-by: Impala Public Jenkins
Project: http://git-wip-us.apache.org/repos/asf/impala/repo
Commit: http://git-wip-us.apache.org/repos/asf/impala/commit/42abe813
Tree: http://git-wip-us.apache.org/repos/asf/impala/tree/42abe813
Diff: http://git-wip-us.apache.org/repos/asf/impala/diff/42abe813
Branch: refs/heads/master
Commit: 42abe8139ebb16f3284328c8f028b47ae86ab78a
Parents: 45aee12
Author: Alex Behm <al...@cloudera.com>
Authored: Tue Mar 13 10:26:55 2018 -0700
Committer: Impala Public Jenkins <im...@gerrit.cloudera.org>
Committed: Thu Mar 15 02:00:46 2018 +0000
----------------------------------------------------------------------
.../org/apache/impala/analysis/QueryStmt.java | 8 +-
.../org/apache/impala/analysis/SelectStmt.java | 18 +-
.../org/apache/impala/analysis/SortInfo.java | 274 +++++++++----------
.../apache/impala/planner/AnalyticPlanner.java | 57 ++--
.../org/apache/impala/planner/ExchangeNode.java | 6 +-
.../java/org/apache/impala/planner/Planner.java | 7 +-
.../org/apache/impala/planner/SortNode.java | 41 +--
.../PlannerTest/sort-expr-materialization.test | 41 +++
8 files changed, 225 insertions(+), 227 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/impala/blob/42abe813/fe/src/main/java/org/apache/impala/analysis/QueryStmt.java
----------------------------------------------------------------------
diff --git a/fe/src/main/java/org/apache/impala/analysis/QueryStmt.java b/fe/src/main/java/org/apache/impala/analysis/QueryStmt.java
index 1ebee10..7449f19 100644
--- a/fe/src/main/java/org/apache/impala/analysis/QueryStmt.java
+++ b/fe/src/main/java/org/apache/impala/analysis/QueryStmt.java
@@ -19,7 +19,6 @@ package org.apache.impala.analysis;
import java.util.ArrayList;
import java.util.List;
-import java.util.ListIterator;
import java.util.Set;
import org.apache.impala.catalog.Type;
@@ -265,17 +264,16 @@ public abstract class QueryStmt extends StatementBase {
*/
protected void createSortTupleInfo(Analyzer analyzer) throws AnalysisException {
Preconditions.checkState(evaluateOrderBy_);
-
- for (Expr orderingExpr: sortInfo_.getOrderingExprs()) {
+ for (Expr orderingExpr: sortInfo_.getSortExprs()) {
if (orderingExpr.getType().isComplexType()) {
throw new AnalysisException(String.format("ORDER BY expression '%s' with " +
"complex type '%s' is not supported.", orderingExpr.toSql(),
orderingExpr.getType().toSql()));
}
}
+ sortInfo_.createSortTupleInfo(resultExprs_, analyzer);
- ExprSubstitutionMap smap = sortInfo_.createSortTupleInfo(resultExprs_, analyzer);
-
+ ExprSubstitutionMap smap = sortInfo_.getOutputSmap();
for (int i = 0; i < smap.size(); ++i) {
if (!(smap.getLhs().get(i) instanceof SlotRef)
|| !(smap.getRhs().get(i) instanceof SlotRef)) {
http://git-wip-us.apache.org/repos/asf/impala/blob/42abe813/fe/src/main/java/org/apache/impala/analysis/SelectStmt.java
----------------------------------------------------------------------
diff --git a/fe/src/main/java/org/apache/impala/analysis/SelectStmt.java b/fe/src/main/java/org/apache/impala/analysis/SelectStmt.java
index a3bac51..874001d 100644
--- a/fe/src/main/java/org/apache/impala/analysis/SelectStmt.java
+++ b/fe/src/main/java/org/apache/impala/analysis/SelectStmt.java
@@ -270,7 +270,7 @@ public class SelectStmt extends QueryStmt {
if (sortInfo_ != null && hasLimit()) {
// When there is a LIMIT clause in conjunction with an ORDER BY, the ordering exprs
// must be added in the column lineage graph.
- graph.addDependencyPredicates(sortInfo_.getOrderingExprs());
+ graph.addDependencyPredicates(sortInfo_.getSortExprs());
}
if (aggInfo_ != null) {
@@ -563,7 +563,7 @@ public class SelectStmt extends QueryStmt {
&& (havingPred_ == null
|| !havingPred_.contains(Expr.isAggregatePredicate()))
&& (sortInfo_ == null
- || !TreeNode.contains(sortInfo_.getOrderingExprs(),
+ || !TreeNode.contains(sortInfo_.getSortExprs(),
Expr.isAggregatePredicate()))) {
// We're not computing aggregates but we still need to register the HAVING
// clause which could, e.g., contain a constant expression evaluating to false.
@@ -644,7 +644,7 @@ public class SelectStmt extends QueryStmt {
}
if (sortInfo_ != null) {
// TODO: Avoid evaluating aggs in ignored order-bys
- TreeNode.collect(sortInfo_.getOrderingExprs(), Expr.isAggregatePredicate(),
+ TreeNode.collect(sortInfo_.getSortExprs(), Expr.isAggregatePredicate(),
aggExprs);
}
@@ -723,10 +723,10 @@ public class SelectStmt extends QueryStmt {
}
}
if (sortInfo_ != null) {
- sortInfo_.substituteOrderingExprs(combinedSmap, analyzer);
+ sortInfo_.substituteSortExprs(combinedSmap, analyzer);
if (LOG.isTraceEnabled()) {
LOG.trace("post-agg orderingExprs: " +
- Expr.debugString(sortInfo_.getOrderingExprs()));
+ Expr.debugString(sortInfo_.getSortExprs()));
}
}
@@ -742,7 +742,7 @@ public class SelectStmt extends QueryStmt {
}
if (orderByElements_ != null) {
for (int i = 0; i < orderByElements_.size(); ++i) {
- if (!sortInfo_.getOrderingExprs().get(i).isBound(
+ if (!sortInfo_.getSortExprs().get(i).isBound(
finalAggInfo.getOutputTupleId())) {
throw new AnalysisException(
"ORDER BY expression not produced by aggregation output "
@@ -840,7 +840,7 @@ public class SelectStmt extends QueryStmt {
ArrayList<Expr> analyticExprs = Lists.newArrayList();
TreeNode.collect(resultExprs_, AnalyticExpr.class, analyticExprs);
if (sortInfo_ != null) {
- TreeNode.collect(sortInfo_.getOrderingExprs(), AnalyticExpr.class,
+ TreeNode.collect(sortInfo_.getSortExprs(), AnalyticExpr.class,
analyticExprs);
}
if (analyticExprs.isEmpty()) return;
@@ -880,10 +880,10 @@ public class SelectStmt extends QueryStmt {
LOG.trace("post-analytic selectListExprs: " + Expr.debugString(resultExprs_));
}
if (sortInfo_ != null) {
- sortInfo_.substituteOrderingExprs(smap, analyzer);
+ sortInfo_.substituteSortExprs(smap, analyzer);
if (LOG.isTraceEnabled()) {
LOG.trace("post-analytic orderingExprs: " +
- Expr.debugString(sortInfo_.getOrderingExprs()));
+ Expr.debugString(sortInfo_.getSortExprs()));
}
}
}
http://git-wip-us.apache.org/repos/asf/impala/blob/42abe813/fe/src/main/java/org/apache/impala/analysis/SortInfo.java
----------------------------------------------------------------------
diff --git a/fe/src/main/java/org/apache/impala/analysis/SortInfo.java b/fe/src/main/java/org/apache/impala/analysis/SortInfo.java
index 745de6d..fba7286 100644
--- a/fe/src/main/java/org/apache/impala/analysis/SortInfo.java
+++ b/fe/src/main/java/org/apache/impala/analysis/SortInfo.java
@@ -16,13 +16,12 @@
// under the License.
package org.apache.impala.analysis;
-import org.apache.impala.common.TreeNode;
-
-import java.util.ArrayList;
+import java.util.Collection;
import java.util.List;
-import java.util.ListIterator;
import java.util.Set;
+import org.apache.impala.common.TreeNode;
+
import com.google.common.base.Preconditions;
import com.google.common.base.Predicates;
import com.google.common.collect.Lists;
@@ -36,7 +35,7 @@ import com.google.common.collect.Sets;
* particular input row (materialize all row slots)
*/
public class SortInfo {
- // All ordering exprs with cost greater than this will be materialized. Since we don't
+ // All sort exprs with cost greater than this will be materialized. Since we don't
// currently have any information about actual function costs, this value is intended to
// ensure that all expensive functions will be materialized while still leaving simple
// operations unmaterialized, for example 'SlotRef + SlotRef' should have a cost below
@@ -45,74 +44,56 @@ public class SortInfo {
private static final float SORT_MATERIALIZATION_COST_THRESHOLD =
Expr.FUNCTION_CALL_COST;
- private List<Expr> orderingExprs_;
+ private List<Expr> sortExprs_;
private final List<Boolean> isAscOrder_;
// True if "NULLS FIRST", false if "NULLS LAST", null if not specified.
private final List<Boolean> nullsFirstParams_;
- // Subset of ordering exprs that are materialized. Populated in
- // createMaterializedOrderExprs(), used for EXPLAIN output.
- private List<Expr> materializedOrderingExprs_;
- // The single tuple that is materialized, sorted, and output by a sort operator
- // (i.e. SortNode or TopNNode)
+ // Descriptor of tuples materialized, sorted, and output by a SortNode/TopNNode.
private TupleDescriptor sortTupleDesc_;
- // Input expressions materialized into sortTupleDesc_. One expr per slot in
- // sortTupleDesc_.
- private List<Expr> sortTupleSlotExprs_;
+ // List of exprs evaluated against the sort input and materialized into the sort tuple.
+ // One expr per slot in 'sortTupleDesc_'.
+ private final List<Expr> materializedExprs_;
+ // Maps from exprs materialized into the sort tuple to their corresponding SlotRefs.
+ private final ExprSubstitutionMap outputSmap_;
- public SortInfo(List<Expr> orderingExprs, List<Boolean> isAscOrder,
+ public SortInfo(List<Expr> sortExprs, List<Boolean> isAscOrder,
List<Boolean> nullsFirstParams) {
- Preconditions.checkArgument(orderingExprs.size() == isAscOrder.size());
- Preconditions.checkArgument(orderingExprs.size() == nullsFirstParams.size());
- orderingExprs_ = orderingExprs;
+ Preconditions.checkArgument(sortExprs.size() == isAscOrder.size());
+ Preconditions.checkArgument(sortExprs.size() == nullsFirstParams.size());
+ sortExprs_ = sortExprs;
isAscOrder_ = isAscOrder;
nullsFirstParams_ = nullsFirstParams;
- materializedOrderingExprs_ = Lists.newArrayList();
+ materializedExprs_ = Lists.newArrayList();
+ outputSmap_ = new ExprSubstitutionMap();
}
/**
* C'tor for cloning.
*/
private SortInfo(SortInfo other) {
- orderingExprs_ = Expr.cloneList(other.orderingExprs_);
+ sortExprs_ = Expr.cloneList(other.sortExprs_);
isAscOrder_ = Lists.newArrayList(other.isAscOrder_);
nullsFirstParams_ = Lists.newArrayList(other.nullsFirstParams_);
- materializedOrderingExprs_ = Expr.cloneList(other.materializedOrderingExprs_);
+ materializedExprs_ = Expr.cloneList(other.materializedExprs_);
sortTupleDesc_ = other.sortTupleDesc_;
- if (other.sortTupleSlotExprs_ != null) {
- sortTupleSlotExprs_ = Expr.cloneList(other.sortTupleSlotExprs_);
- }
+ outputSmap_ = other.outputSmap_.clone();
}
- /**
- * Sets sortTupleDesc_, which is the internal row representation to be materialized and
- * sorted. The source exprs of the slots in sortTupleDesc_ are changed to those in
- * tupleSlotExprs.
- */
- public void setMaterializedTupleInfo(
- TupleDescriptor tupleDesc, List<Expr> tupleSlotExprs) {
- Preconditions.checkState(tupleDesc.getSlots().size() == tupleSlotExprs.size());
- sortTupleDesc_ = tupleDesc;
- sortTupleSlotExprs_ = tupleSlotExprs;
- for (int i = 0; i < sortTupleDesc_.getSlots().size(); ++i) {
- SlotDescriptor slotDesc = sortTupleDesc_.getSlots().get(i);
- slotDesc.setSourceExpr(sortTupleSlotExprs_.get(i));
- }
- }
- public List<Expr> getOrderingExprs() { return orderingExprs_; }
+ public List<Expr> getSortExprs() { return sortExprs_; }
public List<Boolean> getIsAscOrder() { return isAscOrder_; }
public List<Boolean> getNullsFirstParams() { return nullsFirstParams_; }
- public List<Expr> getMaterializedOrderingExprs() { return materializedOrderingExprs_; }
- public List<Expr> getSortTupleSlotExprs() { return sortTupleSlotExprs_; }
+ public List<Expr> getMaterializedExprs() { return materializedExprs_; }
public TupleDescriptor getSortTupleDescriptor() { return sortTupleDesc_; }
+ public ExprSubstitutionMap getOutputSmap() { return outputSmap_; }
/**
* Gets the list of booleans indicating whether nulls come first or last, independent
* of asc/desc.
*/
public List<Boolean> getNullsFirst() {
- Preconditions.checkState(orderingExprs_.size() == nullsFirstParams_.size());
+ Preconditions.checkState(sortExprs_.size() == nullsFirstParams_.size());
List<Boolean> nullsFirst = Lists.newArrayList();
- for (int i = 0; i < orderingExprs_.size(); ++i) {
+ for (int i = 0; i < sortExprs_.size(); ++i) {
nullsFirst.add(OrderByElement.nullsFirst(nullsFirstParams_.get(i),
isAscOrder_.get(i)));
}
@@ -120,20 +101,19 @@ public class SortInfo {
}
/**
- * Materializes the slots in sortTupleDesc_ referenced in the ordering exprs.
- * Materializes the slots referenced by the corresponding sortTupleSlotExpr after
- * applying the 'smap'.
+ * Materializes the slots in 'sortTupleDesc_' referenced in the sort exprs.
+ * Materializes the slots referenced by the corresponding materialized expr after
+ * applying the 'smap'. Valid to call after createSortTupleInfo().
*/
public void materializeRequiredSlots(Analyzer analyzer, ExprSubstitutionMap smap) {
Preconditions.checkNotNull(sortTupleDesc_);
- Preconditions.checkNotNull(sortTupleSlotExprs_);
Preconditions.checkState(sortTupleDesc_.isMaterialized());
- analyzer.materializeSlots(orderingExprs_);
+ analyzer.materializeSlots(sortExprs_);
List<SlotDescriptor> sortTupleSlotDescs = sortTupleDesc_.getSlots();
List<Expr> materializedExprs = Lists.newArrayList();
for (int i = 0; i < sortTupleSlotDescs.size(); ++i) {
if (sortTupleSlotDescs.get(i).isMaterialized()) {
- materializedExprs.add(sortTupleSlotExprs_.get(i));
+ materializedExprs.add(materializedExprs_.get(i));
}
}
List<Expr> substMaterializedExprs =
@@ -142,20 +122,22 @@ public class SortInfo {
}
/**
- * Replaces orderingExprs_ according to smap. This needs to be called to make sure that
- * the ordering exprs refer to the new tuple materialized by this sort instead of the
+ * Replaces 'sortExprs_' according to smap. This needs to be called to make sure that
+ * the sort exprs refer to the new tuple materialized by this sort instead of the
* original input.
*/
- public void substituteOrderingExprs(ExprSubstitutionMap smap, Analyzer analyzer) {
- orderingExprs_ = Expr.substituteList(orderingExprs_, smap, analyzer, false);
+ public void substituteSortExprs(ExprSubstitutionMap smap, Analyzer analyzer) {
+ sortExprs_ = Expr.substituteList(sortExprs_, smap, analyzer, false);
}
/**
- * Asserts that all ordering exprs are bound by the sort tuple.
+ * Validates internal state. Asserts that all sort exprs are bound by the sort tuple.
*/
public void checkConsistency() {
- for (Expr orderingExpr: orderingExprs_) {
- Preconditions.checkState(orderingExpr.isBound(sortTupleDesc_.getId()));
+ Preconditions.checkState(
+ materializedExprs_.size() == sortTupleDesc_.getSlots().size());
+ for (Expr sortExpr: sortExprs_) {
+ Preconditions.checkState(sortExpr.isBound(sortTupleDesc_.getId()));
}
}
@@ -163,119 +145,111 @@ public class SortInfo {
public SortInfo clone() { return new SortInfo(this); }
/**
+ * Matches SlotRef expressions that do not reference the sort tuple.
+ */
+ private class IsInputSlotRefPred implements com.google.common.base.Predicate<Expr> {
+ private final TupleId sortTid_;
+ public IsInputSlotRefPred(TupleId sortTid) {
+ sortTid_ = sortTid;
+ }
+
+ @Override
+ public boolean apply(Expr e) {
+ return e instanceof SlotRef && !e.isBound(sortTid_);
+ }
+ }
+
+ /**
* Create a tuple descriptor for the single tuple that is materialized, sorted, and
* output by the sort node. Materializes slots required by 'resultExprs' as well as
* non-deterministic and expensive order by exprs. The materialized exprs are
* substituted with slot refs into the new tuple. This simplifies the sorting logic for
- * total and top-n sorts. The substitution map is returned.
+ * total and top-n sorts.
*/
- public ExprSubstitutionMap createSortTupleInfo(
- List<Expr> resultExprs, Analyzer analyzer) {
+ public void createSortTupleInfo(List<Expr> resultExprs, Analyzer analyzer) {
+ Preconditions.checkState(sortTupleDesc_ == null);
+ Preconditions.checkState(outputSmap_.size() == 0);
+
// The descriptor for the tuples on which the sort operates.
- TupleDescriptor sortTupleDesc = analyzer.getDescTbl().createTupleDescriptor("sort");
- sortTupleDesc.setIsMaterialized(true);
- List<Expr> sortTupleExprs = Lists.newArrayList();
+ sortTupleDesc_ = analyzer.getDescTbl().createTupleDescriptor("sort");
+ sortTupleDesc_.setIsMaterialized(true);
- // substOrderBy is a mapping from exprs evaluated on the sort input that get
- // materialized into the sort tuple to their corresponding SlotRefs in the sort tuple.
// The following exprs are materialized:
- // 1. Ordering exprs that we chose to materialize
- // 2. SlotRefs against the sort input contained in the result and ordering exprs after
- // substituting the materialized ordering exprs.
-
- // Case 1:
- ExprSubstitutionMap substOrderBy =
- createMaterializedOrderExprs(sortTupleDesc, analyzer);
- sortTupleExprs.addAll(substOrderBy.getLhs());
-
- // Case 2: SlotRefs in the result and ordering exprs after substituting the
- // materialized ordering exprs. Using a LinkedHashSet prevents the slots getting
- // reordered unnecessarily.
- Set<SlotRef> sourceSlots = Sets.newLinkedHashSet();
- List<Expr> substResultExprs =
- Expr.substituteList(resultExprs, substOrderBy, analyzer, false);
- TreeNode.collect(substResultExprs, Predicates.instanceOf(SlotRef.class), sourceSlots);
- TreeNode.collect(Expr.substituteList(orderingExprs_, substOrderBy, analyzer, false),
- Predicates.instanceOf(SlotRef.class), sourceSlots);
- for (SlotRef origSlotRef: sourceSlots) {
- // Don't rematerialize slots that are already in the sort tuple.
- if (origSlotRef.getDesc().getParent().getId() != sortTupleDesc.getId()) {
- SlotDescriptor origSlotDesc = origSlotRef.getDesc();
- SlotDescriptor materializedDesc =
- analyzer.copySlotDescriptor(origSlotDesc, sortTupleDesc);
- SlotRef cloneRef = new SlotRef(materializedDesc);
- substOrderBy.put(origSlotRef, cloneRef);
- sortTupleExprs.add(origSlotRef);
- }
- }
-
- materializeTupleIsNullPredicates(
- sortTupleDesc, substResultExprs, sortTupleExprs, substOrderBy, analyzer);
+ // 1. Sort exprs that we chose to materialize
+ // 2. SlotRefs against the sort input contained in the result and sort exprs
+ // after substituting the materialized sort exprs.
+ // 3. TupleIsNullPredicates from 'resultExprs' which are not legal to evaluate after
+ // the sort because the tuples referenced by it are gone after the sort.
+
+ // Case 1: Materialize chosen sort exprs.
+ addMaterializedExprs(getMaterializedSortExprs(), analyzer);
+
+ // Case 2: Materialize required input slots. Using a LinkedHashSet prevents the
+ // slots getting reordered unnecessarily.
+ Set<SlotRef> inputSlotRefs = Sets.newLinkedHashSet();
+ IsInputSlotRefPred pred = new IsInputSlotRefPred(sortTupleDesc_.getId());
+ TreeNode.collect(Expr.substituteList(resultExprs, outputSmap_, analyzer, false),
+ pred, inputSlotRefs);
+ TreeNode.collect(Expr.substituteList(sortExprs_, outputSmap_, analyzer, false),
+ pred, inputSlotRefs);
+ addMaterializedExprs(inputSlotRefs, analyzer);
+
+ // Case 3: Materialize TupleIsNullPredicates.
+ List<Expr> tupleIsNullPreds = Lists.newArrayList();
+ TreeNode.collect(resultExprs, Predicates.instanceOf(TupleIsNullPredicate.class),
+ tupleIsNullPreds);
+ Expr.removeDuplicates(tupleIsNullPreds);
+ addMaterializedExprs(tupleIsNullPreds, analyzer);
- // The ordering exprs are evaluated against the sort tuple, so they must reflect the
+ // The sort exprs are evaluated against the sort tuple, so they must reflect the
// materialization decision above.
- substituteOrderingExprs(substOrderBy, analyzer);
-
- // Update the tuple descriptor used to materialize the input of the sort.
- setMaterializedTupleInfo(sortTupleDesc, sortTupleExprs);
-
- return substOrderBy;
+ substituteSortExprs(outputSmap_, analyzer);
+ checkConsistency();
}
/**
- * Materialize ordering exprs by creating slots for them in 'sortTupleDesc' if they:
- * - contain a non-deterministic expr
- * - contain a UDF (since we don't know if they're deterministic)
- * - are more expensive than a cost threshold
- * - don't have a cost set
- *
- * Populates 'materializedOrderingExprs_' and returns a mapping from the original
- * ordering exprs to the new SlotRefs. It is expected that this smap will be passed into
- * substituteOrderingExprs() by the caller.
+ * Materializes each of the given exprs into 'sortTupleDesc' as follows:
+ * - Adds a new slot in 'sortTupleDesc_'
+ * - Adds an entry in 'outputSmap_' mapping from the expr to a SlotRef on the new slot
+ * - Adds an entry in 'materializedExprs_'
+ * Valid to call after createSortTupleInfo().
*/
- public ExprSubstitutionMap createMaterializedOrderExprs(
- TupleDescriptor sortTupleDesc, Analyzer analyzer) {
- ExprSubstitutionMap substOrderBy = new ExprSubstitutionMap();
- for (Expr origOrderingExpr : orderingExprs_) {
- if (!origOrderingExpr.hasCost()
- || origOrderingExpr.getCost() > SORT_MATERIALIZATION_COST_THRESHOLD
- || origOrderingExpr.contains(Expr.IS_NONDETERMINISTIC_BUILTIN_FN_PREDICATE)
- || origOrderingExpr.contains(Expr.IS_UDF_PREDICATE)) {
- SlotDescriptor materializedDesc = analyzer.addSlotDescriptor(sortTupleDesc);
- materializedDesc.initFromExpr(origOrderingExpr);
- materializedDesc.setIsMaterialized(true);
- SlotRef materializedRef = new SlotRef(materializedDesc);
- substOrderBy.put(origOrderingExpr, materializedRef);
- materializedOrderingExprs_.add(origOrderingExpr);
+ public <T extends Expr> void addMaterializedExprs(Collection<T> exprs,
+ Analyzer analyzer) {
+ Preconditions.checkNotNull(sortTupleDesc_);
+ for (Expr srcExpr : exprs) {
+ SlotDescriptor dstSlotDesc;
+ if (srcExpr instanceof SlotRef) {
+ SlotDescriptor srcSlotDesc = ((SlotRef) srcExpr).getDesc();
+ dstSlotDesc = analyzer.copySlotDescriptor(srcSlotDesc, sortTupleDesc_);
+ } else {
+ dstSlotDesc = analyzer.addSlotDescriptor(sortTupleDesc_);
+ dstSlotDesc.initFromExpr(srcExpr);
}
+ dstSlotDesc.setSourceExpr(srcExpr);
+ outputSmap_.put(srcExpr.clone(), new SlotRef(dstSlotDesc));
+ materializedExprs_.add(srcExpr);
}
- return substOrderBy;
}
/**
- * Collects the unique TupleIsNullPredicates from 'exprs' and for each one:
- * - Materializes it into a new slot in 'sortTupleDesc'
- * - Adds it to 'sortSlotExprs'
- * - Adds an entry in 'substOrderBy' mapping it to a SlotRef into the new slot
+ * Returns the subset of 'sortExprs_' that should be materialized. A sort expr is
+ * is materialized if it:
+ * - contains a non-deterministic expr
+ * - contains a UDF (since we don't know if they're deterministic)
+ * - is more expensive than a cost threshold
+ * - does not have a cost set
*/
- public static void materializeTupleIsNullPredicates(TupleDescriptor sortTupleDesc,
- List<Expr> exprs, List<Expr> sortSlotExprs, ExprSubstitutionMap substOrderBy,
- Analyzer analyzer) {
- List<Expr> tupleIsNullPreds = Lists.newArrayList();
- TreeNode.collect(
- exprs, Predicates.instanceOf(TupleIsNullPredicate.class), tupleIsNullPreds);
- Expr.removeDuplicates(tupleIsNullPreds);
-
- // Materialize relevant unique TupleIsNullPredicates.
- for (Expr tupleIsNullPred: tupleIsNullPreds) {
- SlotDescriptor sortSlotDesc = analyzer.addSlotDescriptor(sortTupleDesc);
- sortSlotDesc.setType(tupleIsNullPred.getType());
- sortSlotDesc.setIsMaterialized(true);
- sortSlotDesc.setSourceExpr(tupleIsNullPred);
- sortSlotDesc.setLabel(tupleIsNullPred.toSql());
- SlotRef cloneRef = new SlotRef(sortSlotDesc);
- substOrderBy.put(tupleIsNullPred, cloneRef);
- sortSlotExprs.add(tupleIsNullPred.clone());
+ private List<Expr> getMaterializedSortExprs() {
+ List<Expr> result = Lists.newArrayList();
+ for (Expr sortExpr : sortExprs_) {
+ if (!sortExpr.hasCost()
+ || sortExpr.getCost() > SORT_MATERIALIZATION_COST_THRESHOLD
+ || sortExpr.contains(Expr.IS_NONDETERMINISTIC_BUILTIN_FN_PREDICATE)
+ || sortExpr.contains(Expr.IS_UDF_PREDICATE)) {
+ result.add(sortExpr);
+ }
}
+ return result;
}
}
http://git-wip-us.apache.org/repos/asf/impala/blob/42abe813/fe/src/main/java/org/apache/impala/planner/AnalyticPlanner.java
----------------------------------------------------------------------
diff --git a/fe/src/main/java/org/apache/impala/planner/AnalyticPlanner.java b/fe/src/main/java/org/apache/impala/planner/AnalyticPlanner.java
index f504fda..90d98fd 100644
--- a/fe/src/main/java/org/apache/impala/planner/AnalyticPlanner.java
+++ b/fe/src/main/java/org/apache/impala/planner/AnalyticPlanner.java
@@ -21,9 +21,6 @@ import java.util.Collections;
import java.util.Comparator;
import java.util.List;
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
-
import org.apache.impala.analysis.AggregateInfoBase;
import org.apache.impala.analysis.AnalyticExpr;
import org.apache.impala.analysis.AnalyticInfo;
@@ -44,7 +41,9 @@ import org.apache.impala.analysis.TupleDescriptor;
import org.apache.impala.analysis.TupleId;
import org.apache.impala.analysis.TupleIsNullPredicate;
import org.apache.impala.common.ImpalaException;
-import org.apache.impala.thrift.TPartitionType;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
import com.google.common.base.Preconditions;
import com.google.common.collect.Lists;
@@ -256,25 +255,23 @@ public class AnalyticPlanner {
private SortInfo createSortInfo(
PlanNode input, List<Expr> sortExprs, List<Boolean> isAsc,
List<Boolean> nullsFirst) {
- // create tuple for sort output = the entire materialized input in a single tuple
- TupleDescriptor sortTupleDesc =
- analyzer_.getDescTbl().createTupleDescriptor("sort-tuple");
- ExprSubstitutionMap sortSmap = new ExprSubstitutionMap();
- List<Expr> sortSlotExprs = Lists.newArrayList();
- sortTupleDesc.setIsMaterialized(true);
+ List<Expr> inputSlotRefs = Lists.newArrayList();
for (TupleId tid: input.getTupleIds()) {
TupleDescriptor tupleDesc = analyzer_.getTupleDesc(tid);
for (SlotDescriptor inputSlotDesc: tupleDesc.getSlots()) {
if (!inputSlotDesc.isMaterialized()) continue;
- SlotDescriptor sortSlotDesc =
- analyzer_.copySlotDescriptor(inputSlotDesc, sortTupleDesc);
- // all output slots need to be materialized
- sortSlotDesc.setIsMaterialized(true);
- sortSmap.put(new SlotRef(inputSlotDesc), new SlotRef(sortSlotDesc));
- sortSlotExprs.add(new SlotRef(inputSlotDesc));
+ inputSlotRefs.add(new SlotRef(inputSlotDesc));
}
}
+ // The decision to materialize ordering exprs should be based on exprs that are
+ // fully resolved against our input (IMPALA-5270).
+ ExprSubstitutionMap inputSmap = input.getOutputSmap();
+ List<Expr> resolvedSortExprs =
+ Expr.substituteList(sortExprs, inputSmap, analyzer_, true);
+ SortInfo sortInfo = new SortInfo(resolvedSortExprs, isAsc, nullsFirst);
+ sortInfo.createSortTupleInfo(inputSlotRefs, analyzer_);
+
// Lhs exprs to be substituted in ancestor plan nodes could have a rhs that contains
// TupleIsNullPredicates. TupleIsNullPredicates require specific tuple ids for
// evaluation. Since this sort materializes a new tuple, it's impossible to evaluate
@@ -282,31 +279,17 @@ public class AnalyticPlanner {
// To preserve the information whether an input tuple was null or not this sort node,
// we materialize those rhs TupleIsNullPredicates, which are then substituted
// by a SlotRef into the sort's tuple in ancestor nodes (IMPALA-1519).
- ExprSubstitutionMap inputSmap = input.getOutputSmap();
if (inputSmap != null) {
- List<Expr> relevantRhsExprs = Lists.newArrayList();
- for (int i = 0; i < inputSmap.size(); ++i) {
- Expr rhsExpr = inputSmap.getRhs().get(i);
+ List<Expr> tupleIsNullPreds = Lists.newArrayList();
+ for (Expr rhsExpr: inputSmap.getRhs()) {
// Ignore substitutions that are irrelevant at this plan node and its ancestors.
- if (rhsExpr.isBoundByTupleIds(input.getTupleIds())) {
- relevantRhsExprs.add(rhsExpr);
- }
+ if (!rhsExpr.isBoundByTupleIds(input.getTupleIds())) continue;
+ rhsExpr.collect(TupleIsNullPredicate.class, tupleIsNullPreds);
}
-
- SortInfo.materializeTupleIsNullPredicates(sortTupleDesc, relevantRhsExprs,
- sortSlotExprs, sortSmap, analyzer_);
- }
-
- SortInfo sortInfo = new SortInfo(sortExprs, isAsc, nullsFirst);
- ExprSubstitutionMap smap =
- sortInfo.createMaterializedOrderExprs(sortTupleDesc, analyzer_);
- sortSlotExprs.addAll(smap.getLhs());
- sortSmap = ExprSubstitutionMap.combine(sortSmap, smap);
- sortInfo.substituteOrderingExprs(sortSmap, analyzer_);
- if (LOG.isTraceEnabled()) {
- LOG.trace("sortinfo exprs: " + Expr.debugString(sortInfo.getOrderingExprs()));
+ Expr.removeDuplicates(tupleIsNullPreds);
+ sortInfo.addMaterializedExprs(tupleIsNullPreds, analyzer_);
}
- sortInfo.setMaterializedTupleInfo(sortTupleDesc, sortSlotExprs);
+ sortInfo.getSortTupleDescriptor().materializeSlots();
return sortInfo;
}
http://git-wip-us.apache.org/repos/asf/impala/blob/42abe813/fe/src/main/java/org/apache/impala/planner/ExchangeNode.java
----------------------------------------------------------------------
diff --git a/fe/src/main/java/org/apache/impala/planner/ExchangeNode.java b/fe/src/main/java/org/apache/impala/planner/ExchangeNode.java
index 87d2fd2..05184f2 100644
--- a/fe/src/main/java/org/apache/impala/planner/ExchangeNode.java
+++ b/fe/src/main/java/org/apache/impala/planner/ExchangeNode.java
@@ -138,9 +138,9 @@ public class ExchangeNode extends PlanNode {
if (mergeInfo_ != null && detailLevel.ordinal() > TExplainLevel.MINIMAL.ordinal()) {
output.append(detailPrefix + "order by: ");
- for (int i = 0; i < mergeInfo_.getOrderingExprs().size(); ++i) {
+ for (int i = 0; i < mergeInfo_.getSortExprs().size(); ++i) {
if (i > 0) output.append(", ");
- output.append(mergeInfo_.getOrderingExprs().get(i).toSql() + " ");
+ output.append(mergeInfo_.getSortExprs().get(i).toSql() + " ");
output.append(mergeInfo_.getIsAscOrder().get(i) ? "ASC" : "DESC");
Boolean nullsFirstParam = mergeInfo_.getNullsFirstParams().get(i);
@@ -204,7 +204,7 @@ public class ExchangeNode extends PlanNode {
if (mergeInfo_ != null) {
TSortInfo sortInfo = new TSortInfo(
- Expr.treesToThrift(mergeInfo_.getOrderingExprs()), mergeInfo_.getIsAscOrder(),
+ Expr.treesToThrift(mergeInfo_.getSortExprs()), mergeInfo_.getIsAscOrder(),
mergeInfo_.getNullsFirst());
msg.exchange_node.setSort_info(sortInfo);
msg.exchange_node.setOffset(offset_);
http://git-wip-us.apache.org/repos/asf/impala/blob/42abe813/fe/src/main/java/org/apache/impala/planner/Planner.java
----------------------------------------------------------------------
diff --git a/fe/src/main/java/org/apache/impala/planner/Planner.java b/fe/src/main/java/org/apache/impala/planner/Planner.java
index 6bef39f..c18bc76 100644
--- a/fe/src/main/java/org/apache/impala/planner/Planner.java
+++ b/fe/src/main/java/org/apache/impala/planner/Planner.java
@@ -638,12 +638,9 @@ public class Planner {
List<Boolean> isAscOrder = Collections.nCopies(orderingExprs.size(), true);
List<Boolean> nullsFirstParams = Collections.nCopies(orderingExprs.size(), false);
SortInfo sortInfo = new SortInfo(orderingExprs, isAscOrder, nullsFirstParams);
-
- ExprSubstitutionMap smap = sortInfo.createSortTupleInfo(
- insertStmt.getResultExprs(), analyzer);
+ sortInfo.createSortTupleInfo(insertStmt.getResultExprs(), analyzer);
sortInfo.getSortTupleDescriptor().materializeSlots();
-
- insertStmt.substituteResultExprs(smap, analyzer);
+ insertStmt.substituteResultExprs(sortInfo.getOutputSmap(), analyzer);
PlanNode node = null;
if (partialSort) {
http://git-wip-us.apache.org/repos/asf/impala/blob/42abe813/fe/src/main/java/org/apache/impala/planner/SortNode.java
----------------------------------------------------------------------
diff --git a/fe/src/main/java/org/apache/impala/planner/SortNode.java b/fe/src/main/java/org/apache/impala/planner/SortNode.java
index 3ca50e0..45ee763 100644
--- a/fe/src/main/java/org/apache/impala/planner/SortNode.java
+++ b/fe/src/main/java/org/apache/impala/planner/SortNode.java
@@ -19,9 +19,6 @@ package org.apache.impala.planner;
import java.util.List;
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
-
import org.apache.impala.analysis.Analyzer;
import org.apache.impala.analysis.Expr;
import org.apache.impala.analysis.ExprSubstitutionMap;
@@ -36,6 +33,9 @@ import org.apache.impala.thrift.TQueryOptions;
import org.apache.impala.thrift.TSortInfo;
import org.apache.impala.thrift.TSortNode;
import org.apache.impala.thrift.TSortType;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
import com.google.common.base.Joiner;
import com.google.common.base.Objects;
import com.google.common.base.Preconditions;
@@ -72,7 +72,7 @@ public class SortNode extends PlanNode {
protected long offset_;
// The type of sort. Determines the exec node used in the BE.
- private TSortType type_;
+ private final TSortType type_;
/**
* Creates a new SortNode that implements a partial sort.
@@ -133,8 +133,7 @@ public class SortNode extends PlanNode {
// populate resolvedTupleExprs_ and outputSmap_
List<SlotDescriptor> sortTupleSlots = info_.getSortTupleDescriptor().getSlots();
- List<Expr> slotExprs = info_.getSortTupleSlotExprs();
- Preconditions.checkState(sortTupleSlots.size() == slotExprs.size());
+ List<Expr> slotExprs = info_.getMaterializedExprs();
resolvedTupleExprs_ = Lists.newArrayList();
outputSmap_ = new ExprSubstitutionMap();
for (int i = 0; i < slotExprs.size(); ++i) {
@@ -152,7 +151,7 @@ public class SortNode extends PlanNode {
// Parent nodes have have to do the same so set the composition as the outputSmap_.
outputSmap_ = ExprSubstitutionMap.compose(childSmap, outputSmap_, analyzer);
- info_.substituteOrderingExprs(outputSmap_, analyzer);
+ info_.substituteSortExprs(outputSmap_, analyzer);
info_.checkConsistency();
if (LOG.isTraceEnabled()) {
@@ -179,7 +178,7 @@ public class SortNode extends PlanNode {
}
return Objects.toStringHelper(this)
.add("type_", type_)
- .add("ordering_exprs", Expr.debugString(info_.getOrderingExprs()))
+ .add("ordering_exprs", Expr.debugString(info_.getSortExprs()))
.add("is_asc", "[" + Joiner.on(" ").join(strings) + "]")
.add("nulls_first", "[" + Joiner.on(" ").join(info_.getNullsFirst()) + "]")
.add("offset_", offset_)
@@ -190,7 +189,7 @@ public class SortNode extends PlanNode {
@Override
protected void toThrift(TPlanNode msg) {
msg.node_type = TPlanNodeType.SORT_NODE;
- TSortInfo sort_info = new TSortInfo(Expr.treesToThrift(info_.getOrderingExprs()),
+ TSortInfo sort_info = new TSortInfo(Expr.treesToThrift(info_.getSortExprs()),
info_.getIsAscOrder(), info_.getNullsFirst());
Preconditions.checkState(tupleIds_.size() == 1,
"Incorrect size for tupleIds_ in SortNode");
@@ -208,9 +207,9 @@ public class SortNode extends PlanNode {
displayName_, getNodeExplainDetail(detailLevel)));
if (detailLevel.ordinal() >= TExplainLevel.STANDARD.ordinal()) {
output.append(detailPrefix + "order by: ");
- for (int i = 0; i < info_.getOrderingExprs().size(); ++i) {
+ for (int i = 0; i < info_.getSortExprs().size(); ++i) {
if (i > 0) output.append(", ");
- output.append(info_.getOrderingExprs().get(i).toSql() + " ");
+ output.append(info_.getSortExprs().get(i).toSql() + " ");
output.append(info_.getIsAscOrder().get(i) ? "ASC" : "DESC");
Boolean nullsFirstParam = info_.getNullsFirstParams().get(i);
@@ -221,14 +220,20 @@ public class SortNode extends PlanNode {
output.append("\n");
}
- if (detailLevel.ordinal() >= TExplainLevel.EXTENDED.ordinal()
- && info_.getMaterializedOrderingExprs().size() > 0) {
- output.append(detailPrefix + "materialized: ");
- for (int i = 0; i < info_.getMaterializedOrderingExprs().size(); ++i) {
- if (i > 0) output.append(", ");
- output.append(info_.getMaterializedOrderingExprs().get(i).toSql());
+ if (detailLevel.ordinal() >= TExplainLevel.EXTENDED.ordinal()) {
+ List<Expr> nonSlotRefExprs = Lists.newArrayList();
+ for (Expr e: info_.getMaterializedExprs()) {
+ if (e instanceof SlotRef) continue;
+ nonSlotRefExprs.add(e);
+ }
+ if (!nonSlotRefExprs.isEmpty()) {
+ output.append(detailPrefix + "materialized: ");
+ for (int i = 0; i < nonSlotRefExprs.size(); ++i) {
+ if (i > 0) output.append(", ");
+ output.append(nonSlotRefExprs.get(i).toSql());
+ }
+ output.append("\n");
}
- output.append("\n");
}
return output.toString();
http://git-wip-us.apache.org/repos/asf/impala/blob/42abe813/testdata/workloads/functional-planner/queries/PlannerTest/sort-expr-materialization.test
----------------------------------------------------------------------
diff --git a/testdata/workloads/functional-planner/queries/PlannerTest/sort-expr-materialization.test b/testdata/workloads/functional-planner/queries/PlannerTest/sort-expr-materialization.test
index 146524a..e6dfe72 100644
--- a/testdata/workloads/functional-planner/queries/PlannerTest/sort-expr-materialization.test
+++ b/testdata/workloads/functional-planner/queries/PlannerTest/sort-expr-materialization.test
@@ -199,3 +199,44 @@ PLAN-ROOT SINK
mem-estimate=128.00MB mem-reservation=0B
tuple-ids=0 row-size=41B cardinality=7300
====
+# IMPALA-5270: Rand() and udf inside inline view referenced by analytic function.
+select id, row_number() over (partition by u order by r) from
+ (select id, random() r, u from
+ (select id, TestFn(double_col) u from functional.alltypestiny) v1
+ ) v2
+order by id
+---- PLAN
+F00:PLAN FRAGMENT [UNPARTITIONED] hosts=1 instances=1
+| Per-Host Resources: mem-estimate=42.00MB mem-reservation=16.00MB
+PLAN-ROOT SINK
+| mem-estimate=0B mem-reservation=0B
+|
+03:SORT
+| order by: id ASC
+| mem-estimate=6.00MB mem-reservation=6.00MB spill-buffer=2.00MB
+| tuple-ids=4 row-size=12B cardinality=8
+|
+02:ANALYTIC
+| functions: row_number()
+| partition by: u
+| order by: random() ASC
+| window: ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW
+| mem-estimate=4.00MB mem-reservation=4.00MB spill-buffer=2.00MB
+| tuple-ids=6,5 row-size=32B cardinality=8
+|
+01:SORT
+| order by: default.testfn(double_col) ASC NULLS FIRST, random() ASC
+| materialized: default.testfn(double_col), random()
+| mem-estimate=6.00MB mem-reservation=6.00MB spill-buffer=2.00MB
+| tuple-ids=6 row-size=24B cardinality=8
+|
+00:SCAN HDFS [functional.alltypestiny]
+ partitions=4/4 files=4 size=460B
+ stored statistics:
+ table: rows=8 size=460B
+ partitions: 4/4 rows=8
+ columns: all
+ extrapolated-rows=disabled
+ mem-estimate=32.00MB mem-reservation=0B
+ tuple-ids=0 row-size=12B cardinality=8
+====
[3/8] impala git commit: IMPALA-5191,
IMPALA-6415: [DOCS] Document breaking change of alias and ordinal
substitution
Posted by ta...@apache.org.
IMPALA-5191, IMPALA-6415: [DOCS] Document breaking change of alias and ordinal substitution
The alias and ordinal substitution logic has been
changed by IMPALA-5191. This commit updates the
documentation regarding to the new behavior.
Change-Id: I558230d07212da62d2cd12e07a52ceba03e980a8
Cherry-picks: not for 2.x.
Reviewed-on: http://gerrit.cloudera.org:8080/9211
Reviewed-by: Alex Behm <al...@cloudera.com>
Tested-by: Impala Public Jenkins
Project: http://git-wip-us.apache.org/repos/asf/impala/repo
Commit: http://git-wip-us.apache.org/repos/asf/impala/commit/34fdc590
Tree: http://git-wip-us.apache.org/repos/asf/impala/tree/34fdc590
Diff: http://git-wip-us.apache.org/repos/asf/impala/diff/34fdc590
Branch: refs/heads/master
Commit: 34fdc59079988569084d1ae58cd492550dffcd70
Parents: 6a07076
Author: Zoltan Borok-Nagy <bo...@cloudera.com>
Authored: Mon Feb 5 15:04:55 2018 +0100
Committer: Impala Public Jenkins <im...@gerrit.cloudera.org>
Committed: Thu Mar 15 01:04:57 2018 +0000
----------------------------------------------------------------------
docs/shared/impala_common.xml | 74 ++++++++++++++++++++++++++++++++++++-
docs/topics/impala_aliases.xml | 5 +++
2 files changed, 78 insertions(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/impala/blob/34fdc590/docs/shared/impala_common.xml
----------------------------------------------------------------------
diff --git a/docs/shared/impala_common.xml b/docs/shared/impala_common.xml
index 4c5e57c..e2842a3 100644
--- a/docs/shared/impala_common.xml
+++ b/docs/shared/impala_common.xml
@@ -635,7 +635,7 @@ select concat('abc','mno','xyz');</codeblock>
in some other database systems.
</p>
- <p id="column_aliases">
+ <p id="column_aliases" rev="IMPALA-6415 IMPALA-5191">
You can specify column aliases with or without the <codeph>AS</codeph> keyword, and with no quotation
marks, single quotation marks, or double quotation marks. Some kind of quotation marks are required if the
column alias contains any spaces or other problematic characters. The alias text is displayed in the
@@ -660,6 +660,78 @@ select concat('abc','mno','xyz');</codeblock>
| first column |
+--------------+
...</codeblock>
+ From Impala 3.0, the alias substitution logic in the <codeph>GROUP BY</codeph>, <codeph>HAVING</codeph>,
+ and <codeph>ORDER BY</codeph> clauses has become more consistent with standard SQL behavior, as follows.
+ Aliases are now only legal at the top level, and not in subexpressions. The following statements are
+ allowed:
+<codeblock>
+ SELECT int_col / 2 AS x
+ FROM t
+ GROUP BY x;
+
+ SELECT int_col / 2 AS x
+ FROM t
+ ORDER BY x;
+
+ SELECT NOT bool_col AS nb
+ FROM t
+ GROUP BY nb
+ HAVING nb;
+</codeblock>
+ And the following statements are NOT allowed:
+<codeblock>
+ SELECT int_col / 2 AS x
+ FROM t
+ GROUP BY x / 2;
+
+ SELECT int_col / 2 AS x
+ FROM t
+ ORDER BY -x;
+
+ SELECT int_col / 2 AS x
+ FROM t
+ GROUP BY x
+ HAVING x > 3;
+</codeblock>
+ </p>
+
+ <p id="column_ordinals" rev="IMPALA-6415 IMPALA-5191">
+ You can also refer to <codeph>SELECT</codeph>-list items by their ordinal position. Impala supports
+ ordinals in the <codeph>GROUP BY</codeph>, <codeph>HAVING</codeph>, and <codeph>ORDER BY</codeph>
+ clauses. From Impala 3.0, ordinals can only be used at the top level. For example, the following
+ statements are allowed:
+<codeblock>
+ SELECT int_col / 2, sum(x)
+ FROM t
+ GROUP BY 1;
+
+ SELECT int_col / 2
+ FROM t
+ ORDER BY 1;
+
+ SELECT NOT bool_col
+ FROM t
+ GROUP BY 1
+ HAVING 1;
+</codeblock>
+ Numbers in subexpressions are not interpreted as ordinals:
+<codeblock>
+ SELECT int_col / 2, sum(x)
+ FROM t
+ GROUP BY 1 * 2;
+The above parses OK, however GROUP BY 1 * 2 has no effect.
+
+ SELECT int_col / 2
+ FROM t
+ ORDER BY 1 + 2;
+The above parses OK, however ORDER BY 1 + 2 has no effect.
+
+ SELECT NOT bool_col
+ FROM t
+ GROUP BY 1
+ HAVING not 1;
+The above raises an error at parse-time.
+</codeblock>
</p>
<p id="temp_tables">
http://git-wip-us.apache.org/repos/asf/impala/blob/34fdc590/docs/topics/impala_aliases.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_aliases.xml b/docs/topics/impala_aliases.xml
index f223c27..bcf23e3 100644
--- a/docs/topics/impala_aliases.xml
+++ b/docs/topics/impala_aliases.xml
@@ -72,6 +72,11 @@ SELECT t1.column_n c1, t2.column_n c2 FROM table1 t1, table2 t2
WHERE c1 = c2;
</codeblock>
+ <p rev="IMPALA-6415 IMPALA-5191">
+ From Impala 3.0, the alias substitution logic has changed.
+ <ph conref="../shared/impala_common.xml#common/column_aliases"/>
+ </p>
+
<p>
To use an alias name that matches one of the Impala reserved keywords (listed in
<xref href="impala_reserved_words.xml#reserved_words"/>), surround the identifier with either single or
[7/8] impala git commit: IMPALA-6652: Rename label of MemTracker for
early RPCs
Posted by ta...@apache.org.
IMPALA-6652: Rename label of MemTracker for early RPCs
This change renames the label of the MemTracker in
KrpcDataStreamMgr for tracking payloads of early RPCs
to "Data Stream Manager Early RPCs". This is to distinguish
these RPCs from the deferred RPCs in a receiver. The early
RPCs refer to those RPCs which arrive before a receiver
is ready. The responses to these RPCs are deferred until
the receiver is created. The receiver may also defer
responses to RPCs if the deserialized payloads of RPCs in
an inbound queue exceed FLAGS_exchg_node_buffer_size_bytes.
In this case, the RPCs won't be responded to until the
inbound queue is drained.
Change-Id: I5bb72c28e8d660a6b78543dbc8b5b156e0e7c843
Reviewed-on: http://gerrit.cloudera.org:8080/9633
Reviewed-by: Dan Hecht <dh...@cloudera.com>
Reviewed-by: Michael Ho <kw...@cloudera.com>
Tested-by: Impala Public Jenkins
Project: http://git-wip-us.apache.org/repos/asf/impala/repo
Commit: http://git-wip-us.apache.org/repos/asf/impala/commit/69f6bfd8
Tree: http://git-wip-us.apache.org/repos/asf/impala/tree/69f6bfd8
Diff: http://git-wip-us.apache.org/repos/asf/impala/diff/69f6bfd8
Branch: refs/heads/master
Commit: 69f6bfd85c7a243d40f930b9b713001ff0744f5d
Parents: 6c166bf
Author: Lars Volker <lv...@cloudera.com>
Authored: Wed Mar 14 10:33:48 2018 -0700
Committer: Impala Public Jenkins <im...@gerrit.cloudera.org>
Committed: Thu Mar 15 03:45:16 2018 +0000
----------------------------------------------------------------------
be/src/runtime/krpc-data-stream-mgr.cc | 14 +++++++-------
be/src/runtime/krpc-data-stream-mgr.h | 8 ++++----
tests/custom_cluster/test_krpc_mem_usage.py | 8 ++++----
3 files changed, 15 insertions(+), 15 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/impala/blob/69f6bfd8/be/src/runtime/krpc-data-stream-mgr.cc
----------------------------------------------------------------------
diff --git a/be/src/runtime/krpc-data-stream-mgr.cc b/be/src/runtime/krpc-data-stream-mgr.cc
index b291fee..5a9305f 100644
--- a/be/src/runtime/krpc-data-stream-mgr.cc
+++ b/be/src/runtime/krpc-data-stream-mgr.cc
@@ -78,9 +78,9 @@ KrpcDataStreamMgr::KrpcDataStreamMgr(MetricGroup* metrics)
}
Status KrpcDataStreamMgr::Init(MemTracker* service_mem_tracker) {
- // MemTracker for tracking memory used for buffering deferred RPC calls which
+ // MemTracker for tracking memory used for buffering early RPC calls which
// arrive before the receiver is ready.
- mem_tracker_.reset(new MemTracker(-1, "Data Stream Manager Deferred RPCs",
+ early_rpcs_tracker_.reset(new MemTracker(-1, "Data Stream Manager Early RPCs",
ExecEnv::GetInstance()->process_mem_tracker()));
service_mem_tracker_ = service_mem_tracker;
RETURN_IF_ERROR(Thread::Create("krpc-data-stream-mgr", "maintenance",
@@ -134,14 +134,14 @@ shared_ptr<DataStreamRecvrBase> KrpcDataStreamMgr::CreateRecvr(
// Release memory. The receiver will track it in its instance tracker.
int64_t transfer_size = ctx->rpc_context->GetTransferSize();
recvr->TakeOverEarlySender(move(ctx));
- mem_tracker_->Release(transfer_size);
+ early_rpcs_tracker_->Release(transfer_size);
num_senders_waiting_->Increment(-1);
}
for (const unique_ptr<EndDataStreamCtx>& ctx :
early_senders_for_recvr.closed_sender_ctxs) {
recvr->RemoveSender(ctx->request->sender_id());
DataStreamService::RespondAndReleaseRpc(Status::OK(), ctx->response, ctx->rpc_context,
- mem_tracker_.get());
+ early_rpcs_tracker_.get());
num_senders_waiting_->Increment(-1);
}
return recvr;
@@ -174,7 +174,7 @@ void KrpcDataStreamMgr::AddEarlySender(const TUniqueId& finst_id,
const TransmitDataRequestPB* request, TransmitDataResponsePB* response,
kudu::rpc::RpcContext* rpc_context) {
const int64_t transfer_size = rpc_context->GetTransferSize();
- mem_tracker_->Consume(transfer_size);
+ early_rpcs_tracker_->Consume(transfer_size);
service_mem_tracker_->Release(transfer_size);
RecvrId recvr_id = make_pair(finst_id, request->dest_node_id());
auto payload = make_unique<TransmitDataCtx>(request, response, rpc_context);
@@ -187,7 +187,7 @@ void KrpcDataStreamMgr::AddEarlyClosedSender(const TUniqueId& finst_id,
const EndDataStreamRequestPB* request, EndDataStreamResponsePB* response,
kudu::rpc::RpcContext* rpc_context) {
const int64_t transfer_size = rpc_context->GetTransferSize();
- mem_tracker_->Consume(transfer_size);
+ early_rpcs_tracker_->Consume(transfer_size);
service_mem_tracker_->Release(transfer_size);
RecvrId recvr_id = make_pair(finst_id, request->dest_node_id());
auto payload = make_unique<EndDataStreamCtx>(request, response, rpc_context);
@@ -351,7 +351,7 @@ void KrpcDataStreamMgr::RespondToTimedOutSender(const std::unique_ptr<ContextTyp
ctx->request->dest_node_id());
VLOG_QUERY << msg.msg();
DataStreamService::RespondAndReleaseRpc(Status::Expected(msg), ctx->response,
- ctx->rpc_context, mem_tracker_.get());
+ ctx->rpc_context, early_rpcs_tracker_.get());
num_senders_waiting_->Increment(-1);
num_senders_timedout_->Increment(1);
}
http://git-wip-us.apache.org/repos/asf/impala/blob/69f6bfd8/be/src/runtime/krpc-data-stream-mgr.h
----------------------------------------------------------------------
diff --git a/be/src/runtime/krpc-data-stream-mgr.h b/be/src/runtime/krpc-data-stream-mgr.h
index 889aee5..3cd2191 100644
--- a/be/src/runtime/krpc-data-stream-mgr.h
+++ b/be/src/runtime/krpc-data-stream-mgr.h
@@ -295,10 +295,10 @@ class KrpcDataStreamMgr : public DataStreamMgrBase {
friend class KrpcDataStreamRecvr;
friend class DataStreamTest;
- /// MemTracker for memory used for transmit data requests before we hand them over to a
- /// specific receiver. Used only to track payloads of deferred RPCs (e.g. early
- /// senders).
- std::unique_ptr<MemTracker> mem_tracker_;
+ /// MemTracker for memory used for early transmit data RPCs which arrive before the
+ /// receiver is created. The memory of the RPC payload is transferred to the receiver
+ /// once it's created.
+ std::unique_ptr<MemTracker> early_rpcs_tracker_;
/// MemTracker used by the DataStreamService to track memory for incoming requests.
/// Memory for new incoming requests is initially tracked against this tracker before
http://git-wip-us.apache.org/repos/asf/impala/blob/69f6bfd8/tests/custom_cluster/test_krpc_mem_usage.py
----------------------------------------------------------------------
diff --git a/tests/custom_cluster/test_krpc_mem_usage.py b/tests/custom_cluster/test_krpc_mem_usage.py
index a145a7a..c26b776 100644
--- a/tests/custom_cluster/test_krpc_mem_usage.py
+++ b/tests/custom_cluster/test_krpc_mem_usage.py
@@ -22,7 +22,7 @@ from tests.common.impala_cluster import ImpalaCluster
from tests.common.skip import SkipIf, SkipIfBuildType
from tests.verifiers.mem_usage_verifier import MemUsageVerifier
-DATA_STREAM_MGR_METRIC = "Data Stream Manager Deferred RPCs"
+DATA_STREAM_MGR_METRIC = "Data Stream Manager Early RPCs"
DATA_STREAM_SVC_METRIC = "Data Stream Service Queue"
ALL_METRICS = [ DATA_STREAM_MGR_METRIC, DATA_STREAM_SVC_METRIC ]
@@ -74,7 +74,7 @@ class TestKrpcMemUsage(CustomClusterTestSuite):
@SkipIfBuildType.not_dev_build
@pytest.mark.execute_serially
@CustomClusterTestSuite.with_args("--stress_datastream_recvr_delay_ms=1000")
- def test_krpc_deferred_memory_usage(self, vector):
+ def test_krpc_early_sender_memory_usage(self, vector):
"""Executes a simple query. The cluster is started with delayed receiver creation to
trigger RPC queueing.
"""
@@ -83,9 +83,9 @@ class TestKrpcMemUsage(CustomClusterTestSuite):
@SkipIfBuildType.not_dev_build
@pytest.mark.execute_serially
@CustomClusterTestSuite.with_args("--stress_datastream_recvr_delay_ms=1000")
- def test_krpc_deferred_memory_cancellation(self, vector):
+ def test_krpc_early_sender_memory_cancellation(self, vector):
"""Executes a query and cancels it while RPCs are still queued up. This exercises the
- code to flush the deferred RPC queue in the receiver.
+ code to flush the early sender RPC queue in the receiver.
"""
query = "select count(*) from tpch_parquet.lineitem l1 join tpch_parquet.lineitem l2 \
where l1.l_orderkey = l2.l_orderkey"
[4/8] impala git commit: Removing (broken) retries from
split-hbase.sh.
Posted by ta...@apache.org.
Removing (broken) retries from split-hbase.sh.
The retries in split-hbase.sh don't work in the common case,
because $MINIKDC_PRINC_HIVE is not set in non-kerberized (common)
environments. The regular data load scripts (create-load-data.sh)
have code to manage that, but split-hbase.sh blindly forges ahead,
leading to errors like:
/home/impdev/Impala/testdata/bin/split-hbase.sh: line 49: MINIKDC_PRINC_HIVE: unbound variable
Error in /home/impdev/Impala/testdata/bin/create-load-data.sh at line 48: LOAD_DATA_ARGS=""
Since this hasn't been working, I opted to remove it entirely, as a failure on
the line where HBase splitting actually failed would be significantly more
useful than the error here. A search of mailing lists suggested that I was at
least the second person to have run into this. (In my case, I did break HBase
splitting, but it took me a second to identify the error, since the log was
spammed with unrelated information relating to the cluster restart.)
Testing: core tests.
Change-Id: I715891c9e744f21002330c3ae3ebc14095d94ffd
Reviewed-on: http://gerrit.cloudera.org:8080/9588
Reviewed-by: Philip Zeyliger <ph...@cloudera.com>
Tested-by: Impala Public Jenkins
Project: http://git-wip-us.apache.org/repos/asf/impala/repo
Commit: http://git-wip-us.apache.org/repos/asf/impala/commit/45aee121
Tree: http://git-wip-us.apache.org/repos/asf/impala/tree/45aee121
Diff: http://git-wip-us.apache.org/repos/asf/impala/diff/45aee121
Branch: refs/heads/master
Commit: 45aee121eb95505396c63f08c48f2cd0e0182e41
Parents: 34fdc59
Author: Philip Zeyliger <ph...@cloudera.com>
Authored: Mon Mar 12 13:55:37 2018 -0700
Committer: Impala Public Jenkins <im...@gerrit.cloudera.org>
Committed: Thu Mar 15 01:32:14 2018 +0000
----------------------------------------------------------------------
testdata/bin/split-hbase.sh | 17 +++--------------
1 file changed, 3 insertions(+), 14 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/impala/blob/45aee121/testdata/bin/split-hbase.sh
----------------------------------------------------------------------
diff --git a/testdata/bin/split-hbase.sh b/testdata/bin/split-hbase.sh
index 2d95b25..8ede905 100755
--- a/testdata/bin/split-hbase.sh
+++ b/testdata/bin/split-hbase.sh
@@ -38,17 +38,6 @@ mvn $IMPALA_MAVEN_OPTIONS -q dependency:copy-dependencies
export CLASSPATH=$IMPALA_HOME/testdata/target/impala-testdata-0.1-SNAPSHOT.jar:$CLASSPATH
: ${JAVA_KERBEROS_MAGIC=}
-for ATTEMPT in {1..10}; do
- if "$JAVA" ${JAVA_KERBEROS_MAGIC} \
- org.apache.impala.datagenerator.HBaseTestDataRegionAssigment \
- functional_hbase.alltypesagg functional_hbase.alltypessmall; then
- break
- fi
- # Hopefully reloading the data will somehow help the splitting succeed.
- $IMPALA_HOME/bin/start-impala-cluster.py
- $IMPALA_HOME/bin/load-data.py -w functional-query \
- --table_names=alltypesagg,alltypessmall --table_formats=hbase/none --force \
- ${KERB_ARGS} --principal=${MINIKDC_PRINC_HIVE}
- $IMPALA_HOME/tests/util/compute_table_stats.py --db_names=functional_hbase \
- --table_names=alltypesagg,alltypessmall ${KERB_ARGS}
-done
+"$JAVA" ${JAVA_KERBEROS_MAGIC} \
+ org.apache.impala.datagenerator.HBaseTestDataRegionAssigment \
+ functional_hbase.alltypesagg functional_hbase.alltypessmall
[6/8] impala git commit: IMPALA-6498:
test_query_profile_thrift_timestamps causes following tests to fail.
Posted by ta...@apache.org.
IMPALA-6498: test_query_profile_thrift_timestamps causes following
tests to fail.
test_query_profile_thrift_timestamps uses ImapaTestSuite.client.close()
to force cancellation/unregistration of the query, so that 'EndTime'
of the query shows up in the profile. Since other test cases also need
a valid ImpalaTestSuite.client, we move the test case in question to
its own test suite.
Have also reduced the query to 'select sleep(5)', as the earlier
'select sleep(10000)' is just really excessively long.
Change-Id: I287a05f3c90b1a71a5b7ee0f5c06a8840ebac4c9
Reviewed-on: http://gerrit.cloudera.org:8080/9590
Reviewed-by: Thomas Tauber-Marshall <tm...@cloudera.com>
Reviewed-by: Michael Ho <kw...@cloudera.com>
Tested-by: Impala Public Jenkins
Project: http://git-wip-us.apache.org/repos/asf/impala/repo
Commit: http://git-wip-us.apache.org/repos/asf/impala/commit/6c166bf0
Tree: http://git-wip-us.apache.org/repos/asf/impala/tree/6c166bf0
Diff: http://git-wip-us.apache.org/repos/asf/impala/diff/6c166bf0
Branch: refs/heads/master
Commit: 6c166bf015b67b71375850f6c816e53366919b72
Parents: 42abe81
Author: Zoram Thanga <zo...@cloudera.com>
Authored: Mon Mar 12 11:16:27 2018 -0700
Committer: Impala Public Jenkins <im...@gerrit.cloudera.org>
Committed: Thu Mar 15 02:13:42 2018 +0000
----------------------------------------------------------------------
tests/query_test/test_observability.py | 105 +++++++++++++++-------------
1 file changed, 56 insertions(+), 49 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/impala/blob/6c166bf0/tests/query_test/test_observability.py
----------------------------------------------------------------------
diff --git a/tests/query_test/test_observability.py b/tests/query_test/test_observability.py
index 2915225..bba154a 100644
--- a/tests/query_test/test_observability.py
+++ b/tests/query_test/test_observability.py
@@ -137,55 +137,6 @@ class TestObservability(ImpalaTestSuite):
assert results.runtime_profile.count("AGGREGATION_NODE") == 2
assert results.runtime_profile.count("PLAN_ROOT_SINK") == 2
- # IMPALA-6399: Run this test serially to avoid a delay over the wait time in fetching
- # the profile.
- @pytest.mark.execute_serially
- def test_query_profile_thrift_timestamps(self):
- """Test that the query profile start and end time date-time strings have
- nanosecond precision. Nanosecond precision is expected by management API clients
- that consume Impala debug webpages."""
- query = "select sleep(1000)"
- handle = self.client.execute_async(query)
- query_id = handle.get_handle().id
- results = self.client.fetch(query, handle)
- self.client.close()
-
- MAX_WAIT = 300
- start = time.time()
- end = start + MAX_WAIT
- while time.time() <= end:
- # Sleep before trying to fetch the profile. This helps to prevent a warning when the
- # profile is not yet available immediately. It also makes it less likely to
- # introduce an error below in future changes by forgetting to sleep.
- time.sleep(1)
- tree = self.impalad_test_service.get_thrift_profile(query_id)
- if not tree:
- continue
-
- # tree.nodes[1] corresponds to ClientRequestState::summary_profile_
- # See be/src/service/client-request-state.[h|cc].
- start_time = tree.nodes[1].info_strings["Start Time"]
- end_time = tree.nodes[1].info_strings["End Time"]
- # Start and End Times are of the form "2017-12-07 22:26:52.167711000"
- start_time_sub_sec_str = start_time.split('.')[-1]
- end_time_sub_sec_str = end_time.split('.')[-1]
- if len(end_time_sub_sec_str) == 0:
- elapsed = time.time() - start
- logging.info("end_time_sub_sec_str hasn't shown up yet, elapsed=%d", elapsed)
- continue
-
- assert len(end_time_sub_sec_str) == 9, end_time
- assert len(start_time_sub_sec_str) == 9, start_time
- return True
-
- # If we're here, we didn't get the final thrift profile from the debug web page.
- # This could happen due to heavy system load. The test is then inconclusive.
- # Log a message and fail this run.
-
- dbg_str = "Debug thrift profile for query {0} not available in {1} seconds".format(
- query_id, MAX_WAIT)
- assert False, dbg_str
-
def test_query_profile_contains_query_events(self):
"""Test that the expected events show up in a query profile."""
event_regexes = [r'Query Timeline:',
@@ -236,3 +187,59 @@ class TestObservability(ImpalaTestSuite):
event_regexes[event_regex_index] + " not in " + line + "\n" + runtime_profile
assert event_regex_index == len(event_regexes), \
"Didn't find all events in profile: \n" + runtime_profile
+
+class TestThriftProfile(ImpalaTestSuite):
+ @classmethod
+ def get_workload(self):
+ return 'functional-query'
+
+ # IMPALA-6399: Run this test serially to avoid a delay over the wait time in fetching
+ # the profile.
+ # This test needs to call self.client.close() to force computation of query end time,
+ # so it has to be in its own suite (IMPALA-6498).
+ @pytest.mark.execute_serially
+ def test_query_profile_thrift_timestamps(self):
+ """Test that the query profile start and end time date-time strings have
+ nanosecond precision. Nanosecond precision is expected by management API clients
+ that consume Impala debug webpages."""
+ query = "select sleep(5)"
+ handle = self.client.execute_async(query)
+ query_id = handle.get_handle().id
+ results = self.client.fetch(query, handle)
+ self.client.close()
+
+ MAX_WAIT = 300
+ start = time.time()
+ end = start + MAX_WAIT
+ while time.time() <= end:
+ # Sleep before trying to fetch the profile. This helps to prevent a warning when the
+ # profile is not yet available immediately. It also makes it less likely to
+ # introduce an error below in future changes by forgetting to sleep.
+ time.sleep(1)
+ tree = self.impalad_test_service.get_thrift_profile(query_id)
+ if not tree:
+ continue
+
+ # tree.nodes[1] corresponds to ClientRequestState::summary_profile_
+ # See be/src/service/client-request-state.[h|cc].
+ start_time = tree.nodes[1].info_strings["Start Time"]
+ end_time = tree.nodes[1].info_strings["End Time"]
+ # Start and End Times are of the form "2017-12-07 22:26:52.167711000"
+ start_time_sub_sec_str = start_time.split('.')[-1]
+ end_time_sub_sec_str = end_time.split('.')[-1]
+ if len(end_time_sub_sec_str) == 0:
+ elapsed = time.time() - start
+ logging.info("end_time_sub_sec_str hasn't shown up yet, elapsed=%d", elapsed)
+ continue
+
+ assert len(end_time_sub_sec_str) == 9, end_time
+ assert len(start_time_sub_sec_str) == 9, start_time
+ return True
+
+ # If we're here, we didn't get the final thrift profile from the debug web page.
+ # This could happen due to heavy system load. The test is then inconclusive.
+ # Log a message and fail this run.
+
+ dbg_str = "Debug thrift profile for query {0} not available in {1} seconds".format(
+ query_id, MAX_WAIT)
+ assert False, dbg_str