You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@impala.apache.org by tm...@apache.org on 2018/10/17 17:02:56 UTC
[01/11] impala git commit: IMPALA-5031: Undefined uninitialized bool
read
Repository: impala
Updated Branches:
refs/heads/master fe6dabda6 -> 5e92d139b
IMPALA-5031: Undefined uninitialized bool read
This fixes a bug in cpp-mustache that was already fixed in its
upstream repository
(https://github.com/henryr/cpp-mustache/commit/7efdb810e). Without
this fix, the code reads something like:
bool foo;
MaybeSetValue(&foo);
ReadValue(foo);
Because MaybeSetValue might not set the vale of foo, foo might be
uninitialized in the call ReadValue(foo), which is undefined behavior
at the call site when foo is read. The standard says, in
[basic.fundamental]:
Using a bool value in ways described by this International
Standard as "undefined," such as by examining the value of an
uninitialized automatic object, might cause it to behave as if it
is neither true nor false.
Change-Id: I7b32a5dd3afc3f2d847be37766aa0f7e53af32a3
Reviewed-on: http://gerrit.cloudera.org:8080/11682
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
Project: http://git-wip-us.apache.org/repos/asf/impala/repo
Commit: http://git-wip-us.apache.org/repos/asf/impala/commit/2737e22e
Tree: http://git-wip-us.apache.org/repos/asf/impala/tree/2737e22e
Diff: http://git-wip-us.apache.org/repos/asf/impala/diff/2737e22e
Branch: refs/heads/master
Commit: 2737e22eeb0dc7b1aa7064e92eb25b67928d033b
Parents: fe6dabd
Author: Jim Apple <jb...@apache.org>
Authored: Sat Oct 13 17:03:42 2018 -0700
Committer: Impala Public Jenkins <im...@cloudera.com>
Committed: Tue Oct 16 00:42:24 2018 +0000
----------------------------------------------------------------------
be/src/thirdparty/mustache/mustache.cc | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/impala/blob/2737e22e/be/src/thirdparty/mustache/mustache.cc
----------------------------------------------------------------------
diff --git a/be/src/thirdparty/mustache/mustache.cc b/be/src/thirdparty/mustache/mustache.cc
index 7d5a840..fc6c4f6 100644
--- a/be/src/thirdparty/mustache/mustache.cc
+++ b/be/src/thirdparty/mustache/mustache.cc
@@ -358,7 +358,7 @@ void RenderTemplate(const string& document, const string& document_root,
while (idx < document.size() && idx != -1) {
string tag_name;
TagOperator tag_op;
- bool is_triple;
+ bool is_triple = true;
idx = FindNextTag(document, idx, &tag_op, &tag_name, &is_triple, out);
idx = EvaluateTag(document, document_root, idx, &context, tag_op, tag_name, is_triple,
out);
[05/11] impala git commit: IMPALA-7545: Add queuing reason to query
log
Posted by tm...@apache.org.
IMPALA-7545: Add queuing reason to query log
After this change, the HS2 GetLog() function returns the queuing
reason for a query when it is queued by the AdmissionController.
Testing: Added an end-to-end test to test_admission_controller.py
to verify the query logs returned.
Change-Id: I2e5d8de4f6691a9ba2594ca68c54ea4dca760545
Reviewed-on: http://gerrit.cloudera.org:8080/11669
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
Project: http://git-wip-us.apache.org/repos/asf/impala/repo
Commit: http://git-wip-us.apache.org/repos/asf/impala/commit/ec2dabaf
Tree: http://git-wip-us.apache.org/repos/asf/impala/tree/ec2dabaf
Diff: http://git-wip-us.apache.org/repos/asf/impala/diff/ec2dabaf
Branch: refs/heads/master
Commit: ec2dabafb989e2aca0fddb8f6c467e6f551d0424
Parents: 97731e5
Author: poojanilangekar <po...@cloudera.com>
Authored: Fri Oct 12 10:48:49 2018 -0700
Committer: Impala Public Jenkins <im...@cloudera.com>
Committed: Tue Oct 16 01:47:05 2018 +0000
----------------------------------------------------------------------
be/src/scheduling/admission-controller.cc | 28 +++++++------
be/src/scheduling/admission-controller.h | 13 +++++++
be/src/service/impala-hs2-server.cc | 20 +++++++++-
.../custom_cluster/test_admission_controller.py | 41 ++++++++++++++++++++
4 files changed, 89 insertions(+), 13 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/impala/blob/ec2dabaf/be/src/scheduling/admission-controller.cc
----------------------------------------------------------------------
diff --git a/be/src/scheduling/admission-controller.cc b/be/src/scheduling/admission-controller.cc
index e2b5415..000429f 100644
--- a/be/src/scheduling/admission-controller.cc
+++ b/be/src/scheduling/admission-controller.cc
@@ -102,17 +102,23 @@ const string QUERY_EVENT_QUEUED = "Queued";
const string QUERY_EVENT_COMPLETED_ADMISSION = "Completed admission";
// Profile info strings
-const string PROFILE_INFO_KEY_ADMISSION_RESULT = "Admission result";
-const string PROFILE_INFO_VAL_ADMIT_IMMEDIATELY = "Admitted immediately";
-const string PROFILE_INFO_VAL_QUEUED = "Queued";
-const string PROFILE_INFO_VAL_CANCELLED_IN_QUEUE= "Cancelled (queued)";
-const string PROFILE_INFO_VAL_ADMIT_QUEUED = "Admitted (queued)";
-const string PROFILE_INFO_VAL_REJECTED = "Rejected";
-const string PROFILE_INFO_VAL_TIME_OUT = "Timed out (queued)";
-const string PROFILE_INFO_KEY_INITIAL_QUEUE_REASON = "Initial admission queue reason";
-const string PROFILE_INFO_VAL_INITIAL_QUEUE_REASON = "waited $0 ms, reason: $1";
-const string PROFILE_INFO_KEY_LAST_QUEUED_REASON = "Latest admission queue reason";
-const string PROFILE_INFO_KEY_ADMITTED_MEM = "Cluster Memory Admitted";
+const string AdmissionController::PROFILE_INFO_KEY_ADMISSION_RESULT = "Admission result";
+const string AdmissionController::PROFILE_INFO_VAL_ADMIT_IMMEDIATELY =
+ "Admitted immediately";
+const string AdmissionController::PROFILE_INFO_VAL_QUEUED = "Queued";
+const string AdmissionController::PROFILE_INFO_VAL_CANCELLED_IN_QUEUE =
+ "Cancelled (queued)";
+const string AdmissionController::PROFILE_INFO_VAL_ADMIT_QUEUED = "Admitted (queued)";
+const string AdmissionController::PROFILE_INFO_VAL_REJECTED = "Rejected";
+const string AdmissionController::PROFILE_INFO_VAL_TIME_OUT = "Timed out (queued)";
+const string AdmissionController::PROFILE_INFO_KEY_INITIAL_QUEUE_REASON =
+ "Initial admission queue reason";
+const string AdmissionController::PROFILE_INFO_VAL_INITIAL_QUEUE_REASON =
+ "waited $0 ms, reason: $1";
+const string AdmissionController::PROFILE_INFO_KEY_LAST_QUEUED_REASON =
+ "Latest admission queue reason";
+const string AdmissionController::PROFILE_INFO_KEY_ADMITTED_MEM =
+ "Cluster Memory Admitted";
// Error status string details
const string REASON_MEM_LIMIT_TOO_LOW_FOR_RESERVATION =
http://git-wip-us.apache.org/repos/asf/impala/blob/ec2dabaf/be/src/scheduling/admission-controller.h
----------------------------------------------------------------------
diff --git a/be/src/scheduling/admission-controller.h b/be/src/scheduling/admission-controller.h
index b88356c..5b741b9 100644
--- a/be/src/scheduling/admission-controller.h
+++ b/be/src/scheduling/admission-controller.h
@@ -214,6 +214,19 @@ enum class AdmissionOutcome {
class AdmissionController {
public:
+ // Profile info strings
+ static const string PROFILE_INFO_KEY_ADMISSION_RESULT;
+ static const string PROFILE_INFO_VAL_ADMIT_IMMEDIATELY;
+ static const string PROFILE_INFO_VAL_QUEUED;
+ static const string PROFILE_INFO_VAL_CANCELLED_IN_QUEUE;
+ static const string PROFILE_INFO_VAL_ADMIT_QUEUED;
+ static const string PROFILE_INFO_VAL_REJECTED;
+ static const string PROFILE_INFO_VAL_TIME_OUT;
+ static const string PROFILE_INFO_KEY_INITIAL_QUEUE_REASON;
+ static const string PROFILE_INFO_VAL_INITIAL_QUEUE_REASON;
+ static const string PROFILE_INFO_KEY_LAST_QUEUED_REASON;
+ static const string PROFILE_INFO_KEY_ADMITTED_MEM;
+
AdmissionController(StatestoreSubscriber* subscriber,
RequestPoolService* request_pool_service, MetricGroup* metrics,
const TNetworkAddress& host_addr);
http://git-wip-us.apache.org/repos/asf/impala/blob/ec2dabaf/be/src/service/impala-hs2-server.cc
----------------------------------------------------------------------
diff --git a/be/src/service/impala-hs2-server.cc b/be/src/service/impala-hs2-server.cc
index 881b19a..f2e401e 100644
--- a/be/src/service/impala-hs2-server.cc
+++ b/be/src/service/impala-hs2-server.cc
@@ -35,10 +35,11 @@
#include "common/version.h"
#include "rpc/thrift-util.h"
#include "runtime/coordinator.h"
-#include "runtime/raw-value.h"
#include "runtime/exec-env.h"
-#include "service/hs2-util.h"
+#include "runtime/raw-value.h"
+#include "scheduling/admission-controller.h"
#include "service/client-request-state.h"
+#include "service/hs2-util.h"
#include "service/query-options.h"
#include "service/query-result-set.h"
#include "util/auth-util.h"
@@ -826,6 +827,21 @@ void ImpalaServer::GetLog(TGetLogResp& return_val, const TGetLogReq& request) {
}
// Report analysis errors
ss << join(request_state->GetAnalysisWarnings(), "\n");
+ // Report queuing reason if the admission controller queued the query.
+ const string* admission_result = request_state->summary_profile()->GetInfoString(
+ AdmissionController::PROFILE_INFO_KEY_ADMISSION_RESULT);
+ if (admission_result != nullptr) {
+ if (*admission_result == AdmissionController::PROFILE_INFO_VAL_QUEUED) {
+ ss << AdmissionController::PROFILE_INFO_KEY_ADMISSION_RESULT << " : "
+ << *admission_result << "\n";
+ const string* queued_reason = request_state->summary_profile()->GetInfoString(
+ AdmissionController::PROFILE_INFO_KEY_LAST_QUEUED_REASON);
+ if (queued_reason != nullptr) {
+ ss << AdmissionController::PROFILE_INFO_KEY_LAST_QUEUED_REASON << " : "
+ << *queued_reason << "\n";
+ }
+ }
+ }
if (coord != nullptr) {
// Report execution errors
ss << coord->GetErrorLog();
http://git-wip-us.apache.org/repos/asf/impala/blob/ec2dabaf/tests/custom_cluster/test_admission_controller.py
----------------------------------------------------------------------
diff --git a/tests/custom_cluster/test_admission_controller.py b/tests/custom_cluster/test_admission_controller.py
index 56d2b0b..fc74f00 100644
--- a/tests/custom_cluster/test_admission_controller.py
+++ b/tests/custom_cluster/test_admission_controller.py
@@ -817,6 +817,47 @@ class TestAdmissionController(TestAdmissionControllerBase, HS2TestSuite):
assert False, "Timed out waiting for change to profile\nSearch " \
"String: {0}\nProfile:\n{1}".format(search_string, str(profile))
+ @pytest.mark.execute_serially
+ @CustomClusterTestSuite.with_args(
+ impalad_args=impalad_admission_ctrl_flags(max_requests=1, max_queued=10,
+ pool_max_mem=1024 * 1024 * 1024))
+ @needs_session()
+ def test_hs2_admission_controller_logs(self):
+ """Test to verify that the GetLog() function invoked by the HS2 client returns the
+ reason for queuing of the query."""
+ # Start a long running query.
+ long_query_req = TCLIService.TExecuteStatementReq()
+ long_query_req.sessionHandle = self.session_handle
+ long_query_req.statement = "select sleep(1000000)"
+ long_query_resp = self.hs2_client.ExecuteStatement(long_query_req)
+ HS2TestSuite.check_response(long_query_resp)
+ # Ensure that the query is running.
+ self.wait_for_admission_control(long_query_resp.operationHandle)
+ # Submit another query.
+ execute_statement_req = TCLIService.TExecuteStatementReq()
+ execute_statement_req.sessionHandle = self.session_handle
+ execute_statement_req.statement = "select 1"
+ execute_statement_resp = self.hs2_client.ExecuteStatement(execute_statement_req)
+ HS2TestSuite.check_response(execute_statement_resp)
+ # Wait until the query is queued.
+ self.wait_for_operation_state(execute_statement_resp.operationHandle,
+ TCLIService.TOperationState.PENDING_STATE)
+ # Ensure that the log message contains the queuing reason.
+ get_log_req = TCLIService.TGetLogReq()
+ get_log_req.operationHandle = execute_statement_resp.operationHandle
+ log = self.hs2_client.GetLog(get_log_req).log
+ assert "Admission result : Queued" in log, log
+ assert "Latest admission queue reason : number of running queries 1 is at or over "
+ "limit 1" in log, log
+ # Close the running query.
+ close_operation_req = TCLIService.TCloseOperationReq()
+ close_operation_req.operationHandle = long_query_resp.operationHandle
+ HS2TestSuite.check_response(self.hs2_client.CloseOperation(close_operation_req))
+ # Close the queued query.
+ close_operation_req = TCLIService.TCloseOperationReq()
+ close_operation_req.operationHandle = execute_statement_resp.operationHandle
+ HS2TestSuite.check_response(self.hs2_client.CloseOperation(close_operation_req))
+
class TestAdmissionControllerStress(TestAdmissionControllerBase):
"""Submits a number of queries (parameterized) with some delay between submissions
[07/11] impala git commit: IMPALA-7076: [DOCS] Document ALTER TABLE /
VIEW SET OWNER statement
Posted by tm...@apache.org.
IMPALA-7076: [DOCS] Document ALTER TABLE / VIEW SET OWNER statement
Change-Id: I203a800855a413069a40c728dfa157939ea15caf
Reviewed-on: http://gerrit.cloudera.org:8080/11673
Tested-by: Impala Public Jenkins <im...@cloudera.com>
Reviewed-by: Fredy Wijaya <fw...@cloudera.com>
Project: http://git-wip-us.apache.org/repos/asf/impala/repo
Commit: http://git-wip-us.apache.org/repos/asf/impala/commit/96debe0a
Tree: http://git-wip-us.apache.org/repos/asf/impala/tree/96debe0a
Diff: http://git-wip-us.apache.org/repos/asf/impala/diff/96debe0a
Branch: refs/heads/master
Commit: 96debe0a5c701781029908848437b66fcc5fff25
Parents: ad26584
Author: Alex Rodoni <ar...@cloudera.com>
Authored: Fri Oct 12 15:37:42 2018 -0700
Committer: Alex Rodoni <ar...@cloudera.com>
Committed: Wed Oct 17 01:34:31 2018 +0000
----------------------------------------------------------------------
docs/topics/impala_alter_table.xml | 741 ++++++++++++++++++--------------
docs/topics/impala_alter_view.xml | 20 +-
2 files changed, 444 insertions(+), 317 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/impala/blob/96debe0a/docs/topics/impala_alter_table.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_alter_table.xml b/docs/topics/impala_alter_table.xml
index 2d0d1c6..6667f84 100644
--- a/docs/topics/impala_alter_table.xml
+++ b/docs/topics/impala_alter_table.xml
@@ -21,7 +21,13 @@ under the License.
<concept id="alter_table">
<title>ALTER TABLE Statement</title>
- <titlealts audience="PDF"><navtitle>ALTER TABLE</navtitle></titlealts>
+
+ <titlealts audience="PDF">
+
+ <navtitle>ALTER TABLE</navtitle>
+
+ </titlealts>
+
<prolog>
<metadata>
<data name="Category" value="Impala"/>
@@ -42,14 +48,18 @@ under the License.
<p>
<indexterm audience="hidden">ALTER TABLE statement</indexterm>
- The <codeph>ALTER TABLE</codeph> statement changes the structure or properties of an existing Impala table.
+ The <codeph>ALTER TABLE</codeph> statement changes the structure or properties of an
+ existing Impala table.
</p>
+
<p>
- In Impala, this is primarily a logical operation that updates the table metadata in the metastore database that Impala
- shares with Hive. Most <codeph>ALTER TABLE</codeph> operations do not actually rewrite, move, and so on the actual data
- files. (The <codeph>RENAME TO</codeph> clause is the one exception; it can cause HDFS files to be moved to different paths.)
- When you do an <codeph>ALTER TABLE</codeph> operation, you typically need to perform corresponding physical filesystem operations,
- such as rewriting the data files to include extra fields, or converting them to a different file format.
+ In Impala, this is primarily a logical operation that updates the table metadata in the
+ metastore database that Impala shares with Hive. Most <codeph>ALTER TABLE</codeph>
+ operations do not actually rewrite, move, and so on the actual data files. (The
+ <codeph>RENAME TO</codeph> clause is the one exception; it can cause HDFS files to be
+ moved to different paths.) When you do an <codeph>ALTER TABLE</codeph> operation, you
+ typically need to perform corresponding physical filesystem operations, such as rewriting
+ the data files to include extra fields, or converting them to a different file format.
</p>
<p conref="../shared/impala_common.xml#common/syntax_blurb"/>
@@ -62,6 +72,9 @@ ALTER TABLE <varname>name</varname> CHANGE <varname>column_name</varname> <varna
ALTER TABLE <varname>name</varname> REPLACE COLUMNS (<varname>col_spec</varname>[, <varname>col_spec</varname> ...])
+<ph rev="3.1 IMPALA-6988">ALTER TABLE <varname>name</varname> SET OWNER USER <varname>user_name</varname>
+ALTER TABLE <varname>name</varname> SET OWNER ROLE <varname>role_name</varname>
+</ph>
<ph rev="2.10.0 IMPALA-4622">-- Kudu tables only.
ALTER TABLE <varname>name</varname> ALTER [COLUMN] <varname>column_name</varname>
{ SET <varname>kudu_storage_attr</varname> <varname>attr_value</varname>
@@ -132,51 +145,54 @@ statsKey ::= numDVs | numNulls | avgSize | maxSize</ph>
<p conref="../shared/impala_common.xml#common/complex_types_blurb"/>
<p rev="2.3.0">
- In <keyword keyref="impala23_full"/> and higher, the <codeph>ALTER TABLE</codeph> statement can
- change the metadata for tables containing complex types (<codeph>ARRAY</codeph>,
- <codeph>STRUCT</codeph>, and <codeph>MAP</codeph>).
- For example, you can use an <codeph>ADD COLUMNS</codeph>, <codeph>DROP COLUMN</codeph>, or <codeph>CHANGE</codeph>
- clause to modify the table layout for complex type columns.
- Although Impala queries only work for complex type columns in Parquet tables, the complex type support in the
- <codeph>ALTER TABLE</codeph> statement applies to all file formats.
- For example, you can use Impala to update metadata for a staging table in a non-Parquet file format where the
- data is populated by Hive. Or you can use <codeph>ALTER TABLE SET FILEFORMAT</codeph> to change the format
- of an existing table to Parquet so that Impala can query it. Remember that changing the file format for a table does
- not convert the data files within the table; you must prepare any Parquet data files containing complex types
- outside Impala, and bring them into the table using <codeph>LOAD DATA</codeph> or updating the table's
- <codeph>LOCATION</codeph> property.
- See <xref href="impala_complex_types.xml#complex_types"/> for details about using complex types.
+ In <keyword keyref="impala23_full"/> and higher, the <codeph>ALTER TABLE</codeph>
+ statement can change the metadata for tables containing complex types
+ (<codeph>ARRAY</codeph>, <codeph>STRUCT</codeph>, and <codeph>MAP</codeph>). For example,
+ you can use an <codeph>ADD COLUMNS</codeph>, <codeph>DROP COLUMN</codeph>, or
+ <codeph>CHANGE</codeph> clause to modify the table layout for complex type columns.
+ Although Impala queries only work for complex type columns in Parquet tables, the complex
+ type support in the <codeph>ALTER TABLE</codeph> statement applies to all file formats.
+ For example, you can use Impala to update metadata for a staging table in a non-Parquet
+ file format where the data is populated by Hive. Or you can use <codeph>ALTER TABLE SET
+ FILEFORMAT</codeph> to change the format of an existing table to Parquet so that Impala
+ can query it. Remember that changing the file format for a table does not convert the data
+ files within the table; you must prepare any Parquet data files containing complex types
+ outside Impala, and bring them into the table using <codeph>LOAD DATA</codeph> or updating
+ the table's <codeph>LOCATION</codeph> property. See
+ <xref
+ href="impala_complex_types.xml#complex_types"/> for details about using
+ complex types.
</p>
<p conref="../shared/impala_common.xml#common/usage_notes_blurb"/>
<p>
- Whenever you specify partitions in an <codeph>ALTER TABLE</codeph> statement, through the <codeph>PARTITION
- (<varname>partition_spec</varname>)</codeph> clause, you must include all the partitioning columns in the
- specification.
+ Whenever you specify partitions in an <codeph>ALTER TABLE</codeph> statement, through the
+ <codeph>PARTITION (<varname>partition_spec</varname>)</codeph> clause, you must include
+ all the partitioning columns in the specification.
</p>
<p>
- Most of the <codeph>ALTER TABLE</codeph> operations work the same for internal tables (managed by Impala) as
- for external tables (with data files located in arbitrary locations). The exception is renaming a table; for
- an external table, the underlying data directory is not renamed or moved.
+ Most of the <codeph>ALTER TABLE</codeph> operations work the same for internal tables
+ (managed by Impala) as for external tables (with data files located in arbitrary
+ locations). The exception is renaming a table; for an external table, the underlying data
+ directory is not renamed or moved.
</p>
<p>
- <b>Dropping or altering multiple partitions:</b>
+ <b>To drop or alter multiple partitions:</b>
</p>
<p rev="IMPALA-1654">
- In <keyword keyref="impala28_full"/> and higher,
- the expression for the partition clause with a <codeph>DROP</codeph> or <codeph>SET</codeph>
- operation can include comparison operators such as <codeph><</codeph>, <codeph>IN</codeph>,
- or <codeph>BETWEEN</codeph>, and Boolean operators such as <codeph>AND</codeph>
- and <codeph>OR</codeph>.
+ In <keyword keyref="impala28_full"/> and higher, the expression for the partition clause
+ with a <codeph>DROP</codeph> or <codeph>SET</codeph> operation can include comparison
+ operators such as <codeph><</codeph>, <codeph>IN</codeph>, or <codeph>BETWEEN</codeph>,
+ and Boolean operators such as <codeph>AND</codeph> and <codeph>OR</codeph>.
</p>
<p rev="IMPALA-1654">
- For example, you might drop a group of partitions corresponding to a particular date
- range after the data <q>ages out</q>:
+ For example, you might drop a group of partitions corresponding to a particular date range
+ after the data <q>ages out</q>:
</p>
<codeblock><![CDATA[
@@ -186,9 +202,9 @@ alter table historical_data drop partition (year = 1996 and month between 1 and
</codeblock>
<p rev="IMPALA-1654">
- For tables with multiple partition keys columns, you can specify multiple
- conditions separated by commas, and the operation only applies to the partitions
- that match all the conditions (similar to using an <codeph>AND</codeph> clause):
+ For tables with multiple partition keys columns, you can specify multiple conditions
+ separated by commas, and the operation only applies to the partitions that match all the
+ conditions (similar to using an <codeph>AND</codeph> clause):
</p>
<codeblock><![CDATA[
@@ -197,9 +213,9 @@ alter table historical_data drop partition (year < 1995, last_name like 'A%');
</codeblock>
<p rev="IMPALA-1654">
- This technique can also be used to change the file format of groups of partitions,
- as part of an ETL pipeline that periodically consolidates and rewrites the underlying
- data files in a different file format:
+ This technique can also be used to change the file format of groups of partitions, as part
+ of an ETL pipeline that periodically consolidates and rewrites the underlying data files
+ in a different file format:
</p>
<codeblock><![CDATA[
@@ -209,40 +225,43 @@ alter table fast_growing_data partition (year = 2016, month in (10,11,12)) set f
<note>
<p rev="IMPALA-1654">
- The extended syntax involving comparison operators and multiple partitions
- applies to the <codeph>SET FILEFORMAT</codeph>, <codeph>SET TBLPROPERTIES</codeph>,
- <codeph>SET SERDEPROPERTIES</codeph>, and <codeph>SET [UN]CACHED</codeph> clauses.
- You can also use this syntax with the <codeph>PARTITION</codeph> clause
- in the <codeph>COMPUTE INCREMENTAL STATS</codeph> statement, and with the
- <codeph>PARTITION</codeph> clause of the <codeph>SHOW FILES</codeph> statement.
- Some forms of <codeph>ALTER TABLE</codeph> still only apply to one partition
- at a time: the <codeph>SET LOCATION</codeph> and <codeph>ADD PARTITION</codeph>
- clauses. The <codeph>PARTITION</codeph> clauses in the <codeph>LOAD DATA</codeph>
- and <codeph>INSERT</codeph> statements also only apply to one partition at a time.
+ The extended syntax involving comparison operators and multiple partitions applies to
+ the <codeph>SET FILEFORMAT</codeph>, <codeph>SET TBLPROPERTIES</codeph>, <codeph>SET
+ SERDEPROPERTIES</codeph>, and <codeph>SET [UN]CACHED</codeph> clauses. You can also use
+ this syntax with the <codeph>PARTITION</codeph> clause in the <codeph>COMPUTE
+ INCREMENTAL STATS</codeph> statement, and with the <codeph>PARTITION</codeph> clause of
+ the <codeph>SHOW FILES</codeph> statement. Some forms of <codeph>ALTER TABLE</codeph>
+ still only apply to one partition at a time: the <codeph>SET LOCATION</codeph> and
+ <codeph>ADD PARTITION</codeph> clauses. The <codeph>PARTITION</codeph> clauses in the
+ <codeph>LOAD DATA</codeph> and <codeph>INSERT</codeph> statements also only apply to one
+ partition at a time.
</p>
+
<p>
- A DDL statement that applies to multiple partitions is considered successful
- (resulting in no changes) even if no partitions match the conditions.
- The results are the same as if the <codeph>IF EXISTS</codeph> clause was specified.
+ A DDL statement that applies to multiple partitions is considered successful (resulting
+ in no changes) even if no partitions match the conditions. The results are the same as
+ if the <codeph>IF EXISTS</codeph> clause was specified.
</p>
+
<p>
- The performance and scalability of this technique is similar to
- issuing a sequence of single-partition <codeph>ALTER TABLE</codeph>
- statements in quick succession. To minimize bottlenecks due to
- communication with the metastore database, or causing other
- DDL operations on the same table to wait, test the effects of
- performing <codeph>ALTER TABLE</codeph> statements that affect
- large numbers of partitions.
+ The performance and scalability of this technique is similar to issuing a sequence of
+ single-partition <codeph>ALTER TABLE</codeph> statements in quick succession. To
+ minimize bottlenecks due to communication with the metastore database, or causing other
+ DDL operations on the same table to wait, test the effects of performing <codeph>ALTER
+ TABLE</codeph> statements that affect large numbers of partitions.
</p>
</note>
<p conref="../shared/impala_common.xml#common/s3_blurb"/>
<p rev="2.6.0 IMPALA-1878">
- You can specify an <codeph>s3a://</codeph> prefix on the <codeph>LOCATION</codeph> attribute of a table or partition
- to make Impala query data from the Amazon S3 filesystem. In <keyword keyref="impala26_full"/> and higher, Impala automatically
- handles creating or removing the associated folders when you issue <codeph>ALTER TABLE</codeph> statements
- with the <codeph>ADD PARTITION</codeph> or <codeph>DROP PARTITION</codeph> clauses.
+ You can specify an <codeph>s3a://</codeph> prefix on the <codeph>LOCATION</codeph>
+ attribute of a table or partition to make Impala query data from the Amazon S3 filesystem.
+ In <keyword
+ keyref="impala26_full"/> and higher, Impala automatically handles
+ creating or removing the associated folders when you issue <codeph>ALTER TABLE</codeph>
+ statements with the <codeph>ADD PARTITION</codeph> or <codeph>DROP PARTITION</codeph>
+ clauses.
</p>
<p conref="../shared/impala_common.xml#common/s3_ddl"/>
@@ -252,21 +271,20 @@ alter table fast_growing_data partition (year = 2016, month in (10,11,12)) set f
</p>
<p rev="1.4.0">
- If you specify the <codeph>CACHED IN</codeph> clause, any existing or future data files in the table
- directory or the partition subdirectories are designated to be loaded into memory with the HDFS caching
- mechanism. See <xref href="impala_perf_hdfs_caching.xml#hdfs_caching"/> for details about using the HDFS
- caching feature.
+ If you specify the <codeph>CACHED IN</codeph> clause, any existing or future data files in
+ the table directory or the partition subdirectories are designated to be loaded into
+ memory with the HDFS caching mechanism. See
+ <xref
+ href="impala_perf_hdfs_caching.xml#hdfs_caching"/> for details about using
+ the HDFS caching feature.
</p>
- <p conref="../shared/impala_common.xml#common/impala_cache_replication_factor"/>
+ <p
+ conref="../shared/impala_common.xml#common/impala_cache_replication_factor"/>
<p conref="../shared/impala_common.xml#common/sync_ddl_blurb"/>
<p>
- The following sections show examples of the use cases for various <codeph>ALTER TABLE</codeph> clauses.
- </p>
-
- <p>
<b>To rename a table (RENAME TO clause):</b>
</p>
@@ -275,15 +293,15 @@ alter table fast_growing_data partition (year = 2016, month in (10,11,12)) set f
-->
<p>
- The <codeph>RENAME TO</codeph> clause lets you change the name of an existing table, and optionally which
- database it is located in.
+ The <codeph>RENAME TO</codeph> clause lets you change the name of an existing table, and
+ optionally which database it is located in.
</p>
<p>
- For internal tables, this operation physically renames the directory within HDFS that contains the data files;
- the original directory name no longer exists. By qualifying the table names with database names, you can use
- this technique to move an internal table (and its associated data directory) from one database to another.
- For example:
+ For internal tables, this operation physically renames the directory within HDFS that
+ contains the data files; the original directory name no longer exists. By qualifying the
+ table names with database names, you can use this technique to move an internal table (and
+ its associated data directory) from one database to another. For example:
</p>
<codeblock>create database d1;
@@ -298,27 +316,46 @@ use d1;
-- Move table from one database to another.
alter table d2.mobile rename to d3.mobile;</codeblock>
+ <p rev="3.1 IMPALA-6988">
+ <b>To change the owner of a table:</b>
+ </p>
+
+<codeblock>ALTER TABLE <varname>name</varname> SET OWNER USER <varname>user_name;</varname>
+ALTER TABLE <varname>name</varname> SET OWNER ROLE <varname>role_name;</varname>
+</codeblock>
+
<p>
- For external tables,
+ The table owner is originally set to the user who creates the table. When object ownership
+ is enabled in Sentry, an owner of a table can have the <codeph>ALL</codeph> with
+ <codeph>GRANT</codeph> or <codeph>ALL</codeph> without <codeph>GRANT</codeph> privilege.
+ The term <codeph>OWNER</codeph> is used to differentiate between the <codeph>ALL</codeph>
+ privilege that is explicitly granted via the <codeph>GRANT</codeph> statement and a
+ privilege that is implicitly granted by the <codeph>CREATE TABLE</codeph> statement.
</p>
<p>
- <b>To change the physical location where Impala looks for data files associated with a table or
- partition:</b>
+ Use the <codeph>ALTER TABLE SET OWNER</codeph> to transfer the ownership from the current
+ owner to another user or a role.
+ </p>
+
+ <p>
+ <b>To change the physical location where Impala looks for data files associated with a
+ table or partition:</b>
</p>
<codeblock>ALTER TABLE <varname>table_name</varname> [PARTITION (<varname>partition_spec</varname>)] SET LOCATION '<varname>hdfs_path_of_directory</varname>';</codeblock>
<p>
- The path you specify is the full HDFS path where the data files reside, or will be created. Impala does not
- create any additional subdirectory named after the table. Impala does not move any data files to this new
- location or change any data files that might already exist in that directory.
+ The path you specify is the full HDFS path where the data files reside, or will be
+ created. Impala does not create any additional subdirectory named after the table. Impala
+ does not move any data files to this new location or change any data files that might
+ already exist in that directory.
</p>
<p>
- To set the location for a single partition, include the <codeph>PARTITION</codeph> clause. Specify all the
- same partitioning columns for the table, with a constant value for each, to precisely identify the single
- partition affected by the statement:
+ To set the location for a single partition, include the <codeph>PARTITION</codeph> clause.
+ Specify all the same partitioning columns for the table, with a constant value for each,
+ to precisely identify the single partition affected by the statement:
</p>
<codeblock>create table p1 (s string) partitioned by (month int, day int);
@@ -335,22 +372,24 @@ alter table p1 partition (month=1, day=1) set location '/usr/external_data/new_y
<note conref="../shared/impala_common.xml#common/add_partition_set_location"/>
<p rev="2.3.0 IMPALA-1568">
- <b>To automatically detect new partition directories added through Hive or HDFS operations:</b>
+ <b>To automatically detect new partition directories added through Hive or HDFS
+ operations:</b>
</p>
<p rev="2.3.0 IMPALA-1568">
- In <keyword keyref="impala23_full"/> and higher, the <codeph>RECOVER PARTITIONS</codeph> clause scans
- a partitioned table to detect if any new partition directories were added outside of Impala,
- such as by Hive <codeph>ALTER TABLE</codeph> statements or by <cmdname>hdfs dfs</cmdname>
- or <cmdname>hadoop fs</cmdname> commands. The <codeph>RECOVER PARTITIONS</codeph> clause
- automatically recognizes any data files present in these new directories, the same as
- the <codeph>REFRESH</codeph> statement does.
+ In <keyword keyref="impala23_full"/> and higher, the <codeph>RECOVER PARTITIONS</codeph>
+ clause scans a partitioned table to detect if any new partition directories were added
+ outside of Impala, such as by Hive <codeph>ALTER TABLE</codeph> statements or by
+ <cmdname>hdfs dfs</cmdname> or <cmdname>hadoop fs</cmdname> commands. The <codeph>RECOVER
+ PARTITIONS</codeph> clause automatically recognizes any data files present in these new
+ directories, the same as the <codeph>REFRESH</codeph> statement does.
</p>
<p rev="2.3.0 IMPALA-1568">
- For example, here is a sequence of examples showing how you might create a partitioned table in Impala,
- create new partitions through Hive, copy data files into the new partitions with the <cmdname>hdfs</cmdname>
- command, and have Impala recognize the new partitions and new data:
+ For example, here is a sequence of examples showing how you might create a partitioned
+ table in Impala, create new partitions through Hive, copy data files into the new
+ partitions with the <cmdname>hdfs</cmdname> command, and have Impala recognize the new
+ partitions and new data:
</p>
<p rev="2.3.0 IMPALA-1568">
@@ -374,8 +413,8 @@ quit;
</codeblock>
<p rev="2.3.0 IMPALA-1568">
- In Hive, create some new partitions. In a real use case, you might create the
- partitions and populate them with data as the final stages of an ETL pipeline.
+ In Hive, create some new partitions. In a real use case, you might create the partitions
+ and populate them with data as the final stages of an ETL pipeline.
</p>
<codeblock rev="2.3.0 IMPALA-1568">
@@ -391,8 +430,8 @@ hive> quit;
</codeblock>
<p rev="2.3.0 IMPALA-1568">
- For demonstration purposes, manually copy data (a single row) into these
- new partitions, using manual HDFS operations:
+ For demonstration purposes, manually copy data (a single row) into these new partitions,
+ using manual HDFS operations:
</p>
<codeblock rev="2.3.0 IMPALA-1568">
@@ -422,10 +461,9 @@ hive> quit;
</codeblock>
<p rev="2.3.0 IMPALA-1568">
- In Impala, initially the partitions and data are not visible.
- Running <codeph>ALTER TABLE</codeph> with the <codeph>RECOVER PARTITIONS</codeph>
- clause scans the table data directory to find any new partition directories, and
- the data files inside them:
+ In Impala, initially the partitions and data are not visible. Running <codeph>ALTER
+ TABLE</codeph> with the <codeph>RECOVER PARTITIONS</codeph> clause scans the table data
+ directory to find any new partition directories, and the data files inside them:
</p>
<codeblock rev="2.3.0 IMPALA-1568">
@@ -457,27 +495,26 @@ select * from t1;
ALTER TABLE <varname>table_name</varname> SET SERDEPROPERTIES ('<varname>key1</varname>'='<varname>value1</varname>', '<varname>key2</varname>'='<varname>value2</varname>'[, ...]);</codeblock>
<p>
- The <codeph>TBLPROPERTIES</codeph> clause is primarily a way to associate arbitrary user-specified data items
- with a particular table.
+ The <codeph>TBLPROPERTIES</codeph> clause is primarily a way to associate arbitrary
+ user-specified data items with a particular table.
</p>
<p>
- The <codeph>SERDEPROPERTIES</codeph> clause sets up metadata defining
- how tables are read or written, needed in some cases by Hive but not used
- extensively by Impala. You would use this clause primarily to change the
- delimiter in an existing text table or partition, by setting the
- <codeph>'serialization.format'</codeph> and
- <codeph>'field.delim'</codeph> property values to the new delimiter
- character: The <codeph>SERDEPROPERTIES</codeph> clause does not change the
- existing data in the table. The change only affects the future insert into
- the table.
+ The <codeph>SERDEPROPERTIES</codeph> clause sets up metadata defining how tables are read
+ or written, needed in some cases by Hive but not used extensively by Impala. You would use
+ this clause primarily to change the delimiter in an existing text table or partition, by
+ setting the <codeph>'serialization.format'</codeph> and <codeph>'field.delim'</codeph>
+ property values to the new delimiter character: The <codeph>SERDEPROPERTIES</codeph>
+ clause does not change the existing data in the table. The change only affects the future
+ insert into the table.
</p>
<p>
- Use the <codeph>DESCRIBE FORMATTED</codeph> statement to see the current
- values of these properties for an existing table. See <xref
- href="impala_create_table.xml#create_table"/> for more details about
- these clauses.
+ Use the <codeph>DESCRIBE FORMATTED</codeph> statement to see the current values of these
+ properties for an existing table. See
+ <xref
+ href="impala_create_table.xml#create_table"/> for more details about these
+ clauses.
</p>
<p>
@@ -485,27 +522,26 @@ ALTER TABLE <varname>table_name</varname> SET SERDEPROPERTIES ('<varname>key1</v
</p>
<p>
- Although for most tables the <codeph>COMPUTE STATS</codeph> or <codeph>COMPUTE INCREMENTAL STATS</codeph>
- statement is all you need to keep table and column statistics up to date for a table,
- sometimes for a very large table or one that is updated frequently, the length of time to recompute
- all the statistics might make it impractical to run those statements as often as needed.
- As a workaround, you can use the <codeph>ALTER TABLE</codeph> statement to set table statistics
- at the level of the entire table or a single partition, or column statistics at the level of
- the entire table.
+ Although for most tables the <codeph>COMPUTE STATS</codeph> or <codeph>COMPUTE INCREMENTAL
+ STATS</codeph> statement is all you need to keep table and column statistics up to date
+ for a table, sometimes for a very large table or one that is updated frequently, the
+ length of time to recompute all the statistics might make it impractical to run those
+ statements as often as needed. As a workaround, you can use the <codeph>ALTER
+ TABLE</codeph> statement to set table statistics at the level of the entire table or a
+ single partition, or column statistics at the level of the entire table.
</p>
<p>
You can set the <codeph>numrows</codeph> value for table statistics by changing the
- <codeph>TBLPROPERTIES</codeph> setting for a table or partition.
- For example:
+ <codeph>TBLPROPERTIES</codeph> setting for a table or partition. For example:
<codeblock conref="../shared/impala_common.xml#common/set_numrows_example"/>
<codeblock conref="../shared/impala_common.xml#common/set_numrows_partitioned_example"/>
See <xref href="impala_perf_stats.xml#perf_table_stats_manual"/> for details.
</p>
<p rev="2.6.0 IMPALA-3369">
- In <keyword keyref="impala26_full"/> and higher, you can use the <codeph>SET COLUMN STATS</codeph> clause
- to set a specific stats value for a particular column.
+ In <keyword keyref="impala26_full"/> and higher, you can use the <codeph>SET COLUMN
+ STATS</codeph> clause to set a specific stats value for a particular column.
</p>
<p conref="../shared/impala_common.xml#common/set_column_stats_example"/>
@@ -520,22 +556,23 @@ ALTER TABLE <varname>table_name</varname> CHANGE <varname>column_name</varname>
ALTER TABLE <varname>table_name</varname> DROP <varname>column_name</varname>;</codeblock>
<p>
- The <varname>column_spec</varname> is the same as in the <codeph>CREATE TABLE</codeph> statement: the column
- name, then its data type, then an optional comment. You can add multiple columns at a time. The parentheses
- are required whether you add a single column or multiple columns. When you replace columns, all the original
- column definitions are discarded. You might use this technique if you receive a new set of data files with
- different data types or columns in a different order. (The data files are retained, so if the new columns are
- incompatible with the old ones, use <codeph>INSERT OVERWRITE</codeph> or <codeph>LOAD DATA OVERWRITE</codeph>
- to replace all the data before issuing any further queries.)
+ The <varname>column_spec</varname> is the same as in the <codeph>CREATE TABLE</codeph>
+ statement: the column name, then its data type, then an optional comment. You can add
+ multiple columns at a time. The parentheses are required whether you add a single column
+ or multiple columns. When you replace columns, all the original column definitions are
+ discarded. You might use this technique if you receive a new set of data files with
+ different data types or columns in a different order. (The data files are retained, so if
+ the new columns are incompatible with the old ones, use <codeph>INSERT OVERWRITE</codeph>
+ or <codeph>LOAD DATA OVERWRITE</codeph> to replace all the data before issuing any further
+ queries.)
</p>
<p rev="">
- For example, here is how you might add columns to an existing table.
- The first <codeph>ALTER TABLE</codeph> adds two new columns, and the second
- <codeph>ALTER TABLE</codeph> adds one new column.
- A single Impala query reads both the old and new data files, containing different numbers of columns.
- For any columns not present in a particular data file, all the column values are
- considered to be <codeph>NULL</codeph>.
+ For example, here is how you might add columns to an existing table. The first
+ <codeph>ALTER TABLE</codeph> adds two new columns, and the second <codeph>ALTER
+ TABLE</codeph> adds one new column. A single Impala query reads both the old and new data
+ files, containing different numbers of columns. For any columns not present in a
+ particular data file, all the column values are considered to be <codeph>NULL</codeph>.
</p>
<codeblock rev="">
@@ -560,20 +597,23 @@ select * from t1 order by x;
</codeblock>
<p>
- You might use the <codeph>CHANGE</codeph> clause to rename a single column, or to treat an existing column as
- a different type than before, such as to switch between treating a column as <codeph>STRING</codeph> and
- <codeph>TIMESTAMP</codeph>, or between <codeph>INT</codeph> and <codeph>BIGINT</codeph>. You can only drop a
- single column at a time; to drop multiple columns, issue multiple <codeph>ALTER TABLE</codeph> statements, or
- define the new set of columns with a single <codeph>ALTER TABLE ... REPLACE COLUMNS</codeph> statement.
+ You might use the <codeph>CHANGE</codeph> clause to rename a single column, or to treat an
+ existing column as a different type than before, such as to switch between treating a
+ column as <codeph>STRING</codeph> and <codeph>TIMESTAMP</codeph>, or between
+ <codeph>INT</codeph> and <codeph>BIGINT</codeph>. You can only drop a single column at a
+ time; to drop multiple columns, issue multiple <codeph>ALTER TABLE</codeph> statements, or
+ define the new set of columns with a single <codeph>ALTER TABLE ... REPLACE
+ COLUMNS</codeph> statement.
</p>
<p rev="">
- The following examples show some safe operations to drop or change columns. Dropping the final column
- in a table lets Impala ignore the data causing any disruption to existing data files. Changing the type
- of a column works if existing data values can be safely converted to the new type. The type conversion
- rules depend on the file format of the underlying table. For example, in a text table, the same value
- can be interpreted as a <codeph>STRING</codeph> or a numeric value, while in a binary format such as
- Parquet, the rules are stricter and type conversions only work between certain sizes of integers.
+ The following examples show some safe operations to drop or change columns. Dropping the
+ final column in a table lets Impala ignore the data causing any disruption to existing
+ data files. Changing the type of a column works if existing data values can be safely
+ converted to the new type. The type conversion rules depend on the file format of the
+ underlying table. For example, in a text table, the same value can be interpreted as a
+ <codeph>STRING</codeph> or a numeric value, while in a binary format such as Parquet, the
+ rules are stricter and type conversions only work between certain sizes of integers.
</p>
<codeblock rev="">
@@ -618,21 +658,24 @@ select s, upper(x) from int_to_string;
</codeblock>
<p rev="">
- Remember that Impala does not actually do any conversion for the underlying data files as a result of
- <codeph>ALTER TABLE</codeph> statements. If you use <codeph>ALTER TABLE</codeph> to create a table
- layout that does not agree with the contents of the underlying files, you must replace the files
- yourself, such as using <codeph>LOAD DATA</codeph> to load a new set of data files, or
- <codeph>INSERT OVERWRITE</codeph> to copy from another table and replace the original data.
+ Remember that Impala does not actually do any conversion for the underlying data files as
+ a result of <codeph>ALTER TABLE</codeph> statements. If you use <codeph>ALTER
+ TABLE</codeph> to create a table layout that does not agree with the contents of the
+ underlying files, you must replace the files yourself, such as using <codeph>LOAD
+ DATA</codeph> to load a new set of data files, or <codeph>INSERT OVERWRITE</codeph> to
+ copy from another table and replace the original data.
</p>
<p rev="">
- The following example shows what happens if you delete the middle column from a Parquet table containing three columns.
- The underlying data files still contain three columns of data. Because the columns are interpreted based on their positions in
- the data file instead of the specific column names, a <codeph>SELECT *</codeph> query now reads the first and second
- columns from the data file, potentially leading to unexpected results or conversion errors.
- For this reason, if you expect to someday drop a column, declare it as the last column in the table, where its data
- can be ignored by queries after the column is dropped. Or, re-run your ETL process and create new data files
- if you drop or change the type of a column in a way that causes problems with existing data files.
+ The following example shows what happens if you delete the middle column from a Parquet
+ table containing three columns. The underlying data files still contain three columns of
+ data. Because the columns are interpreted based on their positions in the data file
+ instead of the specific column names, a <codeph>SELECT *</codeph> query now reads the
+ first and second columns from the data file, potentially leading to unexpected results or
+ conversion errors. For this reason, if you expect to someday drop a column, declare it as
+ the last column in the table, where its data can be ignored by queries after the column is
+ dropped. Or, re-run your ETL process and create new data files if you drop or change the
+ type of a column in a way that causes problems with existing data files.
</p>
<codeblock rev="">
@@ -692,37 +735,38 @@ optional int32 x [i:1 d:1 r:0]
</codeblock>
<p rev="IMPALA-3092">
- In <keyword keyref="impala26_full"/> and higher, if an Avro table is created without column definitions in the
- <codeph>CREATE TABLE</codeph> statement, and columns are later
- added through <codeph>ALTER TABLE</codeph>, the resulting
- table is now queryable. Missing values from the newly added
- columns now default to <codeph>NULL</codeph>.
+ In <keyword keyref="impala26_full"/> and higher, if an Avro table is created without
+ column definitions in the <codeph>CREATE TABLE</codeph> statement, and columns are later
+ added through <codeph>ALTER TABLE</codeph>, the resulting table is now queryable. Missing
+ values from the newly added columns now default to <codeph>NULL</codeph>.
</p>
<p>
- <b>To change the file format that Impala expects data to be in, for a table or partition:</b>
+ <b>To change the file format that Impala expects data to be in, for a table or
+ partition:</b>
</p>
<p>
- Use an <codeph>ALTER TABLE ... SET FILEFORMAT</codeph> clause. You can include an optional <codeph>PARTITION
- (<varname>col1</varname>=<varname>val1</varname>, <varname>col2</varname>=<varname>val2</varname>,
- ...</codeph> clause so that the file format is changed for a specific partition rather than the entire table.
+ Use an <codeph>ALTER TABLE ... SET FILEFORMAT</codeph> clause. You can include an optional
+ <codeph>PARTITION (<varname>col1</varname>=<varname>val1</varname>,
+ <varname>col2</varname>=<varname>val2</varname>, ...</codeph> clause so that the file
+ format is changed for a specific partition rather than the entire table.
</p>
<p>
- Because this operation only changes the table metadata, you must do any conversion of existing data using
- regular Hadoop techniques outside of Impala. Any new data created by the Impala <codeph>INSERT</codeph>
- statement will be in the new format. You cannot specify the delimiter for Text files; the data files must be
- comma-delimited.
+ Because this operation only changes the table metadata, you must do any conversion of
+ existing data using regular Hadoop techniques outside of Impala. Any new data created by
+ the Impala <codeph>INSERT</codeph> statement will be in the new format. You cannot specify
+ the delimiter for Text files; the data files must be comma-delimited.
<!-- Although Impala can read Avro tables
created through Hive, you cannot specify the Avro file format in an Impala
<codeph>ALTER TABLE</codeph> statement. -->
</p>
<p>
- To set the file format for a single partition, include the <codeph>PARTITION</codeph> clause. Specify all the
- same partitioning columns for the table, with a constant value for each, to precisely identify the single
- partition affected by the statement:
+ To set the file format for a single partition, include the <codeph>PARTITION</codeph>
+ clause. Specify all the same partitioning columns for the table, with a constant value for
+ each, to precisely identify the single partition affected by the statement:
</p>
<codeblock>create table p1 (s string) partitioned by (month int, day int);
@@ -735,72 +779,113 @@ alter table p1 add partition (month=2, day=2);
-- in this format for this specific partition.
alter table p1 partition (month=2, day=2) set fileformat parquet;
</codeblock>
- <p><b>To change the row format with different delimiter characters:</b></p>
- <p> Use the <codeph>SET ROW FORMAT DELIMITED</codeph> clause to ingest data
- files that use a different delimiter character or a different line end
- character. When specifying delimiter and line end characters with the
- <codeph>FIELDS TERMINATED BY</codeph>, <codeph>ESCAPED BY</codeph>, and
- <codeph>LINES TERMINATED BY</codeph> clauses, you can use the following:<ul>
- <li>A regular ASCII character surrounded by single or double quotation
- marks.</li>
- <li>An octal sequence, such as <codeph>'\054'</codeph> representing a
- comma or <codeph>'\0'</codeph> for ASCII null (hex 00).</li>
- <li>Special characters, such as:<ul>
- <li><codeph>'\t'</codeph> for tab</li>
- <li><codeph>'\n'</codeph> for newline or linefeed</li>
- <li><codeph>'\r'</codeph> for carriage return</li>
- </ul></li>
- <li>An integer in the range '-127'..'128' (with quotation marks but no
- backslash)<p>Negative values are subtracted from 256. For example,
- <codeph>FIELDS TERMINATED BY '-2'</codeph> sets the field
- delimiter to ASCII code 254.</p></li>
- </ul></p>
- <p>For more examples of text tables, see <xref
- href="impala_txtfile.xml#txtfile"/>. </p>
- <p> For the <codeph>ESCAPED BY</codeph> clause, choose an escape character
- that is not used anywhere else in the file. The character following the
- escape character is taken literally as part of a field value. </p>
- <p>Surrounding field values with quotation marks does not help Impala to
- parse fields with embedded delimiter characters as the quotation marks are
- considered to be part of the column value. </p>
- <p>If you want to use <codeph>\</codeph> as the escape character, specify
- the clause in <cmdname>impala-shell</cmdname> as <codeph>ESCAPED BY
- '\\'</codeph>. </p>
<p>
- <b>To add or drop partitions for a table</b>, the table must already be partitioned (that is, created with a
- <codeph>PARTITIONED BY</codeph> clause). The partition is a physical directory in HDFS, with a name that
- encodes a particular column value (the <b>partition key</b>). The Impala <codeph>INSERT</codeph> statement
- already creates the partition if necessary, so the <codeph>ALTER TABLE ... ADD PARTITION</codeph> is
- primarily useful for importing data by moving or copying existing data files into the HDFS directory
- corresponding to a partition. (You can use the <codeph>LOAD DATA</codeph> statement to move files into the
- partition directory, or <codeph>ALTER TABLE ... PARTITION (...) SET LOCATION</codeph> to point a partition at
- a directory that already contains data files.
+ <b>To change the row format with different delimiter characters:</b>
+ </p>
+
+ <p>
+ Use the <codeph>SET ROW FORMAT DELIMITED</codeph> clause to ingest data files that use a
+ different delimiter character or a different line end character. When specifying delimiter
+ and line end characters with the <codeph>FIELDS TERMINATED BY</codeph>, <codeph>ESCAPED
+ BY</codeph>, and <codeph>LINES TERMINATED BY</codeph> clauses, you can use the following:
+ <ul>
+ <li>
+ A regular ASCII character surrounded by single or double quotation marks.
+ </li>
+
+ <li>
+ An octal sequence, such as <codeph>'\054'</codeph> representing a comma or
+ <codeph>'\0'</codeph> for ASCII null (hex 00).
+ </li>
+
+ <li>
+ Special characters, such as:
+ <ul>
+ <li>
+ <codeph>'\t'</codeph> for tab
+ </li>
+
+ <li>
+ <codeph>'\n'</codeph> for newline or linefeed
+ </li>
+
+ <li>
+ <codeph>'\r'</codeph> for carriage return
+ </li>
+ </ul>
+ </li>
+
+ <li>
+ An integer in the range '-127'..'128' (with quotation marks but no backslash)
+ <p>
+ Negative values are subtracted from 256. For example, <codeph>FIELDS TERMINATED BY
+ '-2'</codeph> sets the field delimiter to ASCII code 254.
+ </p>
+ </li>
+ </ul>
+ </p>
+
+ <p>
+ For more examples of text tables, see <xref
+ href="impala_txtfile.xml#txtfile"/>.
</p>
<p>
- The <codeph>DROP PARTITION</codeph> clause is used to remove the HDFS directory and associated data files for
- a particular set of partition key values; for example, if you always analyze the last 3 months worth of data,
- at the beginning of each month you might drop the oldest partition that is no longer needed. Removing
- partitions reduces the amount of metadata associated with the table and the complexity of calculating the
- optimal query plan, which can simplify and speed up queries on partitioned tables, particularly join queries.
- Here is an example showing the <codeph>ADD PARTITION</codeph> and <codeph>DROP PARTITION</codeph> clauses.
+ For the <codeph>ESCAPED BY</codeph> clause, choose an escape character that is not used
+ anywhere else in the file. The character following the escape character is taken literally
+ as part of a field value.
</p>
<p>
- To avoid errors while adding or dropping partitions whose existence is not certain,
- add the optional <codeph>IF [NOT] EXISTS</codeph> clause between the <codeph>ADD</codeph> or
- <codeph>DROP</codeph> keyword and the <codeph>PARTITION</codeph> keyword. That is, the entire
- clause becomes <codeph>ADD IF NOT EXISTS PARTITION</codeph> or <codeph>DROP IF EXISTS PARTITION</codeph>.
- The following example shows how partitions can be created automatically through <codeph>INSERT</codeph>
- statements, or manually through <codeph>ALTER TABLE</codeph> statements. The <codeph>IF [NOT] EXISTS</codeph>
- clauses let the <codeph>ALTER TABLE</codeph> statements succeed even if a new requested partition already
- exists, or a partition to be dropped does not exist.
+ Surrounding field values with quotation marks does not help Impala to parse fields with
+ embedded delimiter characters as the quotation marks are considered to be part of the
+ column value.
</p>
-<p>
-Inserting 2 year values creates 2 partitions:
-</p>
+ <p>
+ If you want to use <codeph>\</codeph> as the escape character, specify the clause in
+ <cmdname>impala-shell</cmdname> as <codeph>ESCAPED BY '\\'</codeph>.
+ </p>
+
+ <p>
+ <b>To add or drop partitions for a table</b>, the table must already be partitioned (that
+ is, created with a <codeph>PARTITIONED BY</codeph> clause). The partition is a physical
+ directory in HDFS, with a name that encodes a particular column value (the <b>partition
+ key</b>). The Impala <codeph>INSERT</codeph> statement already creates the partition if
+ necessary, so the <codeph>ALTER TABLE ... ADD PARTITION</codeph> is primarily useful for
+ importing data by moving or copying existing data files into the HDFS directory
+ corresponding to a partition. (You can use the <codeph>LOAD DATA</codeph> statement to
+ move files into the partition directory, or <codeph>ALTER TABLE ... PARTITION (...) SET
+ LOCATION</codeph> to point a partition at a directory that already contains data files.
+ </p>
+
+ <p>
+ The <codeph>DROP PARTITION</codeph> clause is used to remove the HDFS directory and
+ associated data files for a particular set of partition key values; for example, if you
+ always analyze the last 3 months worth of data, at the beginning of each month you might
+ drop the oldest partition that is no longer needed. Removing partitions reduces the amount
+ of metadata associated with the table and the complexity of calculating the optimal query
+ plan, which can simplify and speed up queries on partitioned tables, particularly join
+ queries. Here is an example showing the <codeph>ADD PARTITION</codeph> and <codeph>DROP
+ PARTITION</codeph> clauses.
+ </p>
+
+ <p>
+ To avoid errors while adding or dropping partitions whose existence is not certain, add
+ the optional <codeph>IF [NOT] EXISTS</codeph> clause between the <codeph>ADD</codeph> or
+ <codeph>DROP</codeph> keyword and the <codeph>PARTITION</codeph> keyword. That is, the
+ entire clause becomes <codeph>ADD IF NOT EXISTS PARTITION</codeph> or <codeph>DROP IF
+ EXISTS PARTITION</codeph>. The following example shows how partitions can be created
+ automatically through <codeph>INSERT</codeph> statements, or manually through
+ <codeph>ALTER TABLE</codeph> statements. The <codeph>IF [NOT] EXISTS</codeph> clauses let
+ the <codeph>ALTER TABLE</codeph> statements succeed even if a new requested partition
+ already exists, or a partition to be dropped does not exist.
+ </p>
+
+ <p>
+ Inserting 2 year values creates 2 partitions:
+ </p>
<codeblock>
create table partition_t (s string) partitioned by (y int);
@@ -815,19 +900,20 @@ show partitions partition_t;
+-------+-------+--------+------+--------------+-------------------+--------+-------+
</codeblock>
-<p>
-Without the <codeph>IF NOT EXISTS</codeph> clause, an attempt to add a new partition might fail:
-</p>
+ <p>
+ Without the <codeph>IF NOT EXISTS</codeph> clause, an attempt to add a new partition might
+ fail:
+ </p>
<codeblock>
alter table partition_t add partition (y=2000);
ERROR: AnalysisException: Partition spec already exists: (y=2000).
</codeblock>
-<p>
-The <codeph>IF NOT EXISTS</codeph> clause makes the statement succeed whether or not there was already a
-partition with the specified key value:
-</p>
+ <p>
+ The <codeph>IF NOT EXISTS</codeph> clause makes the statement succeed whether or not there
+ was already a partition with the specified key value:
+ </p>
<codeblock>
alter table partition_t add if not exists partition (y=2000);
@@ -843,10 +929,10 @@ show partitions partition_t;
+-------+-------+--------+------+--------------+-------------------+--------+-------+
</codeblock>
-<p>
-Likewise, the <codeph>IF EXISTS</codeph> clause lets <codeph>DROP PARTITION</codeph> succeed whether or not the partition is already
-in the table:
-</p>
+ <p>
+ Likewise, the <codeph>IF EXISTS</codeph> clause lets <codeph>DROP PARTITION</codeph>
+ succeed whether or not the partition is already in the table:
+ </p>
<codeblock>
alter table partition_t drop if exists partition (y=2000);
@@ -861,16 +947,17 @@ show partitions partition_t;
+-------+-------+--------+------+--------------+-------------------+--------+-------+
</codeblock>
- <p rev="2.3.0"> The optional <codeph>PURGE</codeph> keyword, available in
+ <p rev="2.3.0">
+ The optional <codeph>PURGE</codeph> keyword, available in
<keyword keyref="impala23_full"/> and higher, is used with the <codeph>DROP
- PARTITION</codeph> clause to remove associated HDFS data files
- immediately rather than going through the HDFS trashcan mechanism. Use
- this keyword when dropping a partition if it is crucial to remove the data
- as quickly as possible to free up space, or if there is a problem with the
- trashcan, such as the trash cannot being configured or being in a
- different HDFS encryption zone than the data files. </p>
-
- <!--
+ PARTITION</codeph> clause to remove associated HDFS data files immediately rather than
+ going through the HDFS trashcan mechanism. Use this keyword when dropping a partition if
+ it is crucial to remove the data as quickly as possible to free up space, or if there is a
+ problem with the trashcan, such as the trash cannot being configured or being in a
+ different HDFS encryption zone than the data files.
+ </p>
+
+<!--
To do: Make example more general by partitioning by year/month/day.
Then could show inserting into fixed year, variable month and day;
dropping particular year/month/day partition.
@@ -897,8 +984,8 @@ alter table part_t add partition (month=3) set fileformat=parquet;
</codeblock>
<p>
- The value specified for a partition key can be an arbitrary constant expression, without any references to
- columns. For example:
+ The value specified for a partition key can be an arbitrary constant expression, without
+ any references to columns. For example:
</p>
<codeblock>alter table time_data add partition (month=concat('Decem','ber'));
@@ -906,11 +993,12 @@ alter table sales_data add partition (zipcode = cast(9021 * 10 as string));</cod
<note>
<p>
- An alternative way to reorganize a table and its associated data files is to use <codeph>CREATE
- TABLE</codeph> to create a variation of the original table, then use <codeph>INSERT</codeph> to copy the
- transformed or reordered data to the new table. The advantage of <codeph>ALTER TABLE</codeph> is that it
- avoids making a duplicate copy of the data files, allowing you to reorganize huge volumes of data in a
- space-efficient way using familiar Hadoop techniques.
+ An alternative way to reorganize a table and its associated data files is to use
+ <codeph>CREATE TABLE</codeph> to create a variation of the original table, then use
+ <codeph>INSERT</codeph> to copy the transformed or reordered data to the new table. The
+ advantage of <codeph>ALTER TABLE</codeph> is that it avoids making a duplicate copy of
+ the data files, allowing you to reorganize huge volumes of data in a space-efficient way
+ using familiar Hadoop techniques.
</p>
</note>
@@ -918,59 +1006,62 @@ alter table sales_data add partition (zipcode = cast(9021 * 10 as string));</cod
<b>To switch a table between internal and external:</b>
</p>
- <p conref="../shared/impala_common.xml#common/switch_internal_external_table"/>
+ <p
+ conref="../shared/impala_common.xml#common/switch_internal_external_table"/>
<p conref="../shared/impala_common.xml#common/cancel_blurb_no"/>
<p conref="../shared/impala_common.xml#common/permissions_blurb"/>
+
<p rev="">
- Most <codeph>ALTER TABLE</codeph> clauses do not actually
- read or write any HDFS files, and so do not depend on
- specific HDFS permissions. For example, the <codeph>SET FILEFORMAT</codeph>
- clause does not actually check the file format existing data files or
- convert them to the new format, and the <codeph>SET LOCATION</codeph> clause
- does not require any special permissions on the new location.
- (Any permission-related failures would come later, when you
- actually query or insert into the table.)
+ Most <codeph>ALTER TABLE</codeph> clauses do not actually read or write any HDFS files,
+ and so do not depend on specific HDFS permissions. For example, the <codeph>SET
+ FILEFORMAT</codeph> clause does not actually check the file format existing data files or
+ convert them to the new format, and the <codeph>SET LOCATION</codeph> clause does not
+ require any special permissions on the new location. (Any permission-related failures
+ would come later, when you actually query or insert into the table.)
</p>
+
<!-- Haven't rigorously tested all the assertions in the following paragraph. -->
+
<!-- Most testing so far has been around RENAME TO clause. -->
+
<p>
- In general, <codeph>ALTER TABLE</codeph> clauses that do touch
- HDFS files and directories require the same HDFS permissions
- as corresponding <codeph>CREATE</codeph>, <codeph>INSERT</codeph>,
- or <codeph>SELECT</codeph> statements.
- The permissions allow
- the user ID that the <cmdname>impalad</cmdname> daemon runs under,
- typically the <codeph>impala</codeph> user, to read or write
- files or directories, or (in the case of the execute bit) descend into a directory.
- The <codeph>RENAME TO</codeph> clause requires read, write, and execute permission in the
- source and destination database directories and in the table data directory,
- and read and write permission for the data files within the table.
- The <codeph>ADD PARTITION</codeph> and <codeph>DROP PARTITION</codeph> clauses
+ In general, <codeph>ALTER TABLE</codeph> clauses that do touch HDFS files and directories
+ require the same HDFS permissions as corresponding <codeph>CREATE</codeph>,
+ <codeph>INSERT</codeph>, or <codeph>SELECT</codeph> statements. The permissions allow the
+ user ID that the <cmdname>impalad</cmdname> daemon runs under, typically the
+ <codeph>impala</codeph> user, to read or write files or directories, or (in the case of
+ the execute bit) descend into a directory. The <codeph>RENAME TO</codeph> clause requires
+ read, write, and execute permission in the source and destination database directories and
+ in the table data directory, and read and write permission for the data files within the
+ table. The <codeph>ADD PARTITION</codeph> and <codeph>DROP PARTITION</codeph> clauses
require write and execute permissions for the associated partition directory.
</p>
<p conref="../shared/impala_common.xml#common/kudu_blurb"/>
<p rev="kudu IMPALA-2890">
- Because of the extra constraints and features of Kudu tables, such as the <codeph>NOT NULL</codeph>
- and <codeph>DEFAULT</codeph> attributes for columns, <codeph>ALTER TABLE</codeph> has specific
- requirements related to Kudu tables:
+ Because of the extra constraints and features of Kudu tables, such as the <codeph>NOT
+ NULL</codeph> and <codeph>DEFAULT</codeph> attributes for columns, <codeph>ALTER
+ TABLE</codeph> has specific requirements related to Kudu tables:
<ul>
<li>
<p>
- In an <codeph>ADD COLUMNS</codeph> operation, you can specify the <codeph>NULL</codeph>,
- <codeph>NOT NULL</codeph>, and <codeph>DEFAULT <varname>default_value</varname></codeph>
- column attributes.
+ In an <codeph>ADD COLUMNS</codeph> operation, you can specify the
+ <codeph>NULL</codeph>, <codeph>NOT NULL</codeph>, and <codeph>DEFAULT
+ <varname>default_value</varname></codeph> column attributes.
</p>
</li>
+
<li>
<p rev="2.9.0 IMPALA-4616">
- In <keyword keyref="impala29_full"/> and higher, you can also specify the <codeph>ENCODING</codeph>,
- <codeph>COMPRESSION</codeph>, and <codeph>BLOCK_SIZE</codeph> attributes when adding a column.
+ In <keyword keyref="impala29_full"/> and higher, you can also specify the
+ <codeph>ENCODING</codeph>, <codeph>COMPRESSION</codeph>, and
+ <codeph>BLOCK_SIZE</codeph> attributes when adding a column.
</p>
</li>
+
<li>
<p>
If you add a column with a <codeph>NOT NULL</codeph> attribute, it must also have a
@@ -978,23 +1069,27 @@ alter table sales_data add partition (zipcode = cast(9021 * 10 as string));</cod
column for all existing rows.
</p>
</li>
+
<li>
<p>
The <codeph>DROP COLUMN</codeph> clause works the same for a Kudu table as for other
kinds of tables.
</p>
</li>
+
<li>
<p>
- Although you can change the name of a column with the <codeph>CHANGE</codeph> clause,
- you cannot change the type of a column in a Kudu table.
+ Although you can change the name of a column with the <codeph>CHANGE</codeph>
+ clause, you cannot change the type of a column in a Kudu table.
</p>
</li>
+
<li>
<p>
You cannot change the nullability of existing columns in a Kudu table.
</p>
</li>
+
<li>
<p rev="2.10.0 IMPALA-4622">
In <keyword keyref="impala210_full"/>, you can change the default value, encoding,
@@ -1002,24 +1097,27 @@ alter table sales_data add partition (zipcode = cast(9021 * 10 as string));</cod
<codeph>SET</codeph> clause.
</p>
</li>
+
<li>
<p>
You cannot use the <codeph>REPLACE COLUMNS</codeph> clause with a Kudu table.
</p>
</li>
- <li> The <codeph>RENAME TO</codeph> clause for a Kudu table only affects
- the name stored in the metastore database that Impala uses to refer to
- the table. To change which underlying Kudu table is associated with an
- Impala table name, you must change the <codeph>TBLPROPERTIES</codeph>
- property of the table: <codeph>SET
- TBLPROPERTIES('kudu.table_name'='<varname>kudu_tbl_name</varname>)</codeph>.
- You can only change underlying Kudu tables for the external
- tables.</li>
+
+ <li>
+ The <codeph>RENAME TO</codeph> clause for a Kudu table only affects the name stored in
+ the metastore database that Impala uses to refer to the table. To change which
+ underlying Kudu table is associated with an Impala table name, you must change the
+ <codeph>TBLPROPERTIES</codeph> property of the table: <codeph>SET
+ TBLPROPERTIES('kudu.table_name'='<varname>kudu_tbl_name</varname>)</codeph>. You can
+ only change underlying Kudu tables for the external tables.
+ </li>
</ul>
</p>
<p>
- The following are some examples of using the <codeph>ADD COLUMNS</codeph> clause for a Kudu table:
+ The following are some examples of using the <codeph>ADD COLUMNS</codeph> clause for a
+ Kudu table:
</p>
<codeblock rev="2.9.0 IMPALA-4616">
@@ -1033,7 +1131,8 @@ ALTER TABLE t1 ADD COLUMNS (a STRING NOT NULL DEFAULT '', t TIMESTAMP COMPRESSIO
</codeblock>
<p rev="2.10.0 IMPALA-4622">
- The following are some examples of modifying column defaults and storage attributes for a Kudu table:
+ The following are some examples of modifying column defaults and storage attributes for a
+ Kudu table:
</p>
<codeblock rev="2.10.0 IMPALA-4622">
@@ -1082,19 +1181,29 @@ desc kt;
</codeblock>
<p rev="kudu">
- Kudu tables all use an underlying partitioning mechanism. The partition syntax is different than for non-Kudu
- tables. You can use the <codeph>ALTER TABLE</codeph> statement to add and drop <term>range partitions</term>
- from a Kudu table. Any new range must not overlap with any existing ranges. Dropping a range removes all the associated
- rows from the table. See <xref href="impala_kudu.xml#kudu_partitioning"/> for details.
+ Kudu tables all use an underlying partitioning mechanism. The partition syntax is
+ different than for non-Kudu tables. You can use the <codeph>ALTER TABLE</codeph> statement
+ to add and drop <term>range partitions</term> from a Kudu table. Any new range must not
+ overlap with any existing ranges. Dropping a range removes all the associated rows from
+ the table. See <xref href="impala_kudu.xml#kudu_partitioning"/> for details.
</p>
<p conref="../shared/impala_common.xml#common/related_info"/>
<p>
<xref href="impala_tables.xml#tables"/>,
- <xref href="impala_create_table.xml#create_table"/>, <xref href="impala_drop_table.xml#drop_table"/>,
- <xref href="impala_partitioning.xml#partitioning"/>, <xref href="impala_tables.xml#internal_tables"/>,
- <xref href="impala_tables.xml#external_tables"/>
+ <xref
+ href="impala_create_table.xml#create_table"/>,
+ <xref
+ href="impala_drop_table.xml#drop_table"/>,
+ <xref
+ href="impala_partitioning.xml#partitioning"/>,
+ <xref
+ href="impala_tables.xml#internal_tables"/>,
+ <xref
+ href="impala_tables.xml#external_tables"/>
</p>
+
</conbody>
+
</concept>
http://git-wip-us.apache.org/repos/asf/impala/blob/96debe0a/docs/topics/impala_alter_view.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_alter_view.xml b/docs/topics/impala_alter_view.xml
index 09558a2..f63152f 100644
--- a/docs/topics/impala_alter_view.xml
+++ b/docs/topics/impala_alter_view.xml
@@ -65,7 +65,11 @@ under the License.
AS <varname>select_statement</varname>;
ALTER VIEW [<varname>database_name</varname>.]<varname>view_name</varname>
- RENAME TO [<varname>database_name</varname>.]<varname>view_name</varname>;</codeblock>
+ RENAME TO [<varname>database_name</varname>.]<varname>view_name</varname>;
+
+ALTER VIEW [<varname>database_name</varname>.]<varname>view_name</varname> SET OWNER USER user_name;
+ALTER TABLE [<varname>database_name</varname>.]<varname>view_name</varname> SET OWNER ROLE role_name;
+</codeblock>
<ul>
<li>
@@ -95,6 +99,20 @@ ALTER VIEW db1.v1 RENAME TO db1.v2; -- Rename the view in the same database.
ALTER VIEW db1.v1 RENAME TO db2.v1; -- Move the view to a difference database with the same view name.</codeblock>
</p>
</li>
+
+ <li rev="3.1 IMPALA-6988">
+ The <codeph>SET OWNER</codeph> clause transfers the ownership of the view from the
+ current owner to another user or a role.
+ <p>
+ The view owner is originally set to the user who creates the view. When object
+ ownership is enabled in Sentry, an owner of a view can have the <codeph>ALL</codeph>
+ with <codeph>GRANT</codeph> or <codeph>ALL</codeph> without <codeph>GRANT</codeph>
+ privilege. The term <codeph>OWNER</codeph> is used to differentiate between the
+ <codeph>ALL</codeph> privilege that is explicitly granted via the
+ <codeph>GRANT</codeph> statement and a privilege that is implicitly granted by the
+ <codeph>CREATE VIEW</codeph> statement.
+ </p>
+ </li>
</ul>
<p conref="../shared/impala_common.xml#common/ddl_blurb"/>
[04/11] impala git commit: IMPALA-7709: Add options to restart
catalogd and statestored in start-impala-cluster.py
Posted by tm...@apache.org.
IMPALA-7709: Add options to restart catalogd and statestored in start-impala-cluster.py
This patch adds two options start-impala-cluster.py:
--restart_catalogd_only to restart catalogd process
--restart_statestored_only to restart statestored process
Testing:
- Manually tested the two new options
Change-Id: Ide26902f6bce11718708d5ab0174282dd94400a3
Reviewed-on: http://gerrit.cloudera.org:8080/11687
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
Project: http://git-wip-us.apache.org/repos/asf/impala/repo
Commit: http://git-wip-us.apache.org/repos/asf/impala/commit/97731e5b
Tree: http://git-wip-us.apache.org/repos/asf/impala/tree/97731e5b
Diff: http://git-wip-us.apache.org/repos/asf/impala/diff/97731e5b
Branch: refs/heads/master
Commit: 97731e5b85c75ae6fa06af74456428890069ca12
Parents: 3b6c0f6
Author: Fredy Wijaya <fw...@cloudera.com>
Authored: Mon Oct 15 11:52:32 2018 -0700
Committer: Impala Public Jenkins <im...@cloudera.com>
Committed: Tue Oct 16 01:23:55 2018 +0000
----------------------------------------------------------------------
bin/start-impala-cluster.py | 38 ++++++++++++++++++++++++++++++++------
1 file changed, 32 insertions(+), 6 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/impala/blob/97731e5b/bin/start-impala-cluster.py
----------------------------------------------------------------------
diff --git a/bin/start-impala-cluster.py b/bin/start-impala-cluster.py
index ebf8fa5..7e22ca9 100755
--- a/bin/start-impala-cluster.py
+++ b/bin/start-impala-cluster.py
@@ -68,6 +68,12 @@ parser.add_option("--force_kill", dest="force_kill", action="store_true", defaul
parser.add_option("-r", "--restart_impalad_only", dest="restart_impalad_only",
action="store_true", default=False,
help="Restarts only the impalad processes")
+parser.add_option("--restart_catalogd_only", dest="restart_catalogd_only",
+ action="store_true", default=False,
+ help="Restarts only the catalogd process")
+parser.add_option("--restart_statestored_only", dest="restart_statestored_only",
+ action="store_true", default=False,
+ help="Restarts only the statestored process")
parser.add_option("--in-process", dest="inprocess", action="store_true", default=False,
help="Start all Impala backends and state store in a single process.")
parser.add_option("--log_dir", dest="log_dir",
@@ -459,13 +465,26 @@ if __name__ == "__main__":
log_dir=options.log_dir))
sys.exit(1)
- # Kill existing cluster processes based on the current configuration.
- if options.restart_impalad_only:
+ restart_only_count = len([opt for opt in [options.restart_impalad_only,
+ options.restart_statestored_only,
+ options.restart_catalogd_only] if opt])
+ if restart_only_count > 1:
+ LOG.error("--restart_impalad_only, --restart_catalogd_only, and "
+ "--restart_statestored_only options are mutually exclusive")
+ sys.exit(1)
+ elif restart_only_count == 1:
if options.inprocess:
LOG.error(
- "Cannot perform individual component restarts using an in-process cluster")
+ "Cannot perform individual component restarts using an in-process cluster")
sys.exit(1)
+
+ # Kill existing cluster processes based on the current configuration.
+ if options.restart_impalad_only:
kill_matching_processes(["impalad"], force=options.force_kill)
+ elif options.restart_catalogd_only:
+ kill_matching_processes(["catalogd"], force=options.force_kill)
+ elif options.restart_statestored_only:
+ kill_matching_processes(["statestored"], force=options.force_kill)
else:
kill_cluster_processes(force=options.force_kill)
@@ -496,11 +515,18 @@ if __name__ == "__main__":
wait_for_cluster = wait_for_cluster_cmdline
try:
- if not options.restart_impalad_only:
+ if options.restart_catalogd_only:
+ start_catalogd()
+ elif options.restart_statestored_only:
+ start_statestore()
+ elif options.restart_impalad_only:
+ start_impalad_instances(options.cluster_size, options.num_coordinators,
+ options.use_exclusive_coordinators)
+ else:
start_statestore()
start_catalogd()
- start_impalad_instances(options.cluster_size, options.num_coordinators,
- options.use_exclusive_coordinators)
+ start_impalad_instances(options.cluster_size, options.num_coordinators,
+ options.use_exclusive_coordinators)
# Sleep briefly to reduce log spam: the cluster takes some time to start up.
sleep(3)
[11/11] impala git commit: IMPALA-7678: Reapply "IMPALA-7660: Support
ECDH ciphers for debug webserver"
Posted by tm...@apache.org.
IMPALA-7678: Reapply "IMPALA-7660: Support ECDH ciphers for debug webserver"
This patch reverses the revert of IMPALA-7660.
The problem with IMPALA-7660 was that urllib.urlopen added the
'context' parameter in 2.7.9, so it isn't present on rhel7, which uses
2.7.5
The fix is to switch to using the 'requests' library, which supports
ssl connections on all the platforms Impala is supported on.
This patch also adds more info to the error message printed by
start-impala-cluster.py when the debug webserver cannot be reached yet
to help with debugging these issues in the future.
Testing:
- Ran full builds on rhel7, rhel6, and ubuntu16.
Change-Id: I679469ed7f27944f75004ec4b16d513e6ea6b544
Reviewed-on: http://gerrit.cloudera.org:8080/11625
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
Project: http://git-wip-us.apache.org/repos/asf/impala/repo
Commit: http://git-wip-us.apache.org/repos/asf/impala/commit/5e92d139
Tree: http://git-wip-us.apache.org/repos/asf/impala/tree/5e92d139
Diff: http://git-wip-us.apache.org/repos/asf/impala/diff/5e92d139
Branch: refs/heads/master
Commit: 5e92d139b951e77d3f9f355c1e46736454a654b0
Parents: 0cd9151
Author: Thomas Tauber-Marshall <tm...@cloudera.com>
Authored: Mon Oct 8 15:43:53 2018 -0700
Committer: Impala Public Jenkins <im...@cloudera.com>
Committed: Wed Oct 17 05:39:32 2018 +0000
----------------------------------------------------------------------
be/src/thirdparty/squeasel/squeasel.c | 36 ++++++++++++++++++++++++----
tests/common/impala_cluster.py | 27 +++++++++++++--------
tests/common/impala_service.py | 36 +++++++++++++++++-----------
tests/custom_cluster/test_client_ssl.py | 35 ++++++++++++++++++++++-----
tests/custom_cluster/test_redaction.py | 22 +++++++----------
tests/run-tests.py | 6 ++---
6 files changed, 110 insertions(+), 52 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/impala/blob/5e92d139/be/src/thirdparty/squeasel/squeasel.c
----------------------------------------------------------------------
diff --git a/be/src/thirdparty/squeasel/squeasel.c b/be/src/thirdparty/squeasel/squeasel.c
index 2149497..045740d 100644
--- a/be/src/thirdparty/squeasel/squeasel.c
+++ b/be/src/thirdparty/squeasel/squeasel.c
@@ -4298,11 +4298,37 @@ static int set_ssl_option(struct sq_context *ctx) {
(void) SSL_CTX_use_certificate_chain_file(ctx->ssl_ctx, pem);
}
- if (ctx->config[SSL_CIPHERS] != NULL &&
- (SSL_CTX_set_cipher_list(ctx->ssl_ctx, ctx->config[SSL_CIPHERS]) == 0)) {
- cry(fc(ctx), "SSL_CTX_set_cipher_list: error setting ciphers (%s): %s",
- ctx->config[SSL_CIPHERS], ssl_error());
- return 0;
+ if (ctx->config[SSL_CIPHERS] != NULL) {
+ if (SSL_CTX_set_cipher_list(ctx->ssl_ctx, ctx->config[SSL_CIPHERS]) == 0) {
+ cry(fc(ctx), "SSL_CTX_set_cipher_list: error setting ciphers (%s): %s",
+ ctx->config[SSL_CIPHERS], ssl_error());
+ return 0;
+ }
+#ifndef OPENSSL_NO_ECDH
+#if OPENSSL_VERSION_NUMBER < 0x10002000L
+ // OpenSSL 1.0.1 and below only support setting a single ECDH curve at once.
+ // We choose prime256v1 because it's the first curve listed in the "modern
+ // compatibility" section of the Mozilla Server Side TLS recommendations,
+ // accessed Feb. 2017.
+ EC_KEY* ecdh = EC_KEY_new_by_curve_name(NID_X9_62_prime256v1);
+ if (ecdh == NULL) {
+ cry(fc(ctx), "EC_KEY_new_by_curve_name: %s", ssl_error());
+ }
+
+ int rc = SSL_CTX_set_tmp_ecdh(ctx->ssl_ctx, ecdh);
+ if (rc <= 0) {
+ cry(fc(ctx), "SSL_CTX_set_tmp_ecdh: %s", ssl_error());
+ }
+#elif OPENSSL_VERSION_NUMBER < 0x10100000L
+ // OpenSSL 1.0.2 provides the set_ecdh_auto API which internally figures out
+ // the best curve to use.
+ int rc = SSL_CTX_set_ecdh_auto(ctx->ssl_ctx, 1);
+ if (rc <= 0) {
+ cry(fc(ctx), "SSL_CTX_set_ecdh_auto: %s", ssl_error());
+ }
+#endif
+#endif
+
}
return 1;
http://git-wip-us.apache.org/repos/asf/impala/blob/5e92d139/tests/common/impala_cluster.py
----------------------------------------------------------------------
diff --git a/tests/common/impala_cluster.py b/tests/common/impala_cluster.py
index f25c8ed..f0c1f8f 100644
--- a/tests/common/impala_cluster.py
+++ b/tests/common/impala_cluster.py
@@ -215,13 +215,19 @@ class Process(object):
# Base class for all Impala processes
class BaseImpalaProcess(Process):
- def __init__(self, cmd, hostname):
+ def __init__(self, cmd):
super(BaseImpalaProcess, self).__init__(cmd)
- self.hostname = hostname
+ self.hostname = self._get_hostname()
def _get_webserver_port(self, default=None):
return int(self._get_arg_value('webserver_port', default))
+ def _get_webserver_certificate_file(self):
+ return self._get_arg_value("webserver_certificate_file", "")
+
+ def _get_hostname(self):
+ return self._get_arg_value("hostname", socket.gethostname())
+
def _get_arg_value(self, arg_name, default=None):
"""Gets the argument value for given argument name"""
for arg in self.cmd:
@@ -235,11 +241,12 @@ class BaseImpalaProcess(Process):
# Represents an impalad process
class ImpaladProcess(BaseImpalaProcess):
def __init__(self, cmd):
- super(ImpaladProcess, self).__init__(cmd, socket.gethostname())
+ super(ImpaladProcess, self).__init__(cmd)
self.service = ImpaladService(self.hostname, self._get_webserver_port(default=25000),
self.__get_beeswax_port(default=21000),
self.__get_be_port(default=22000),
- self.__get_hs2_port(default=21050))
+ self.__get_hs2_port(default=21050),
+ self._get_webserver_certificate_file())
def __get_beeswax_port(self, default=None):
return int(self._get_arg_value('beeswax_port', default))
@@ -263,17 +270,17 @@ class ImpaladProcess(BaseImpalaProcess):
# Represents a statestored process
class StateStoreProcess(BaseImpalaProcess):
def __init__(self, cmd):
- super(StateStoreProcess, self).__init__(cmd, socket.gethostname())
- self.service =\
- StateStoredService(self.hostname, self._get_webserver_port(default=25010))
+ super(StateStoreProcess, self).__init__(cmd)
+ self.service = StateStoredService(self.hostname,
+ self._get_webserver_port(default=25010), self._get_webserver_certificate_file())
# Represents a catalogd process
class CatalogdProcess(BaseImpalaProcess):
def __init__(self, cmd):
- super(CatalogdProcess, self).__init__(cmd, socket.gethostname())
- self.service = CatalogdService(self.hostname,
- self._get_webserver_port(default=25020), self.__get_port(default=26000))
+ super(CatalogdProcess, self).__init__(cmd)
+ self.service = CatalogdService(self.hostname, self._get_webserver_port(default=25020),
+ self._get_webserver_certificate_file(), self.__get_port(default=26000))
def __get_port(self, default=None):
return int(self._get_arg_value('catalog_service_port', default))
http://git-wip-us.apache.org/repos/asf/impala/blob/5e92d139/tests/common/impala_service.py
----------------------------------------------------------------------
diff --git a/tests/common/impala_service.py b/tests/common/impala_service.py
index 16419ae..cfd4b55 100644
--- a/tests/common/impala_service.py
+++ b/tests/common/impala_service.py
@@ -22,7 +22,7 @@
import json
import logging
import re
-import urllib
+import requests
from time import sleep, time
from tests.common.impala_connection import create_connection, create_ldap_connection
@@ -41,31 +41,36 @@ LOG.setLevel(level=logging.DEBUG)
# Base class for all Impala services
# TODO: Refactor the retry/timeout logic into a common place.
class BaseImpalaService(object):
- def __init__(self, hostname, webserver_port):
+ def __init__(self, hostname, webserver_port, webserver_certificate_file):
self.hostname = hostname
self.webserver_port = webserver_port
+ self.webserver_certificate_file = webserver_certificate_file
def open_debug_webpage(self, page_name, timeout=10, interval=1):
start_time = time()
while (time() - start_time < timeout):
try:
- return urllib.urlopen("http://%s:%d/%s" %
- (self.hostname, int(self.webserver_port), page_name))
- except Exception:
- LOG.info("Debug webpage not yet available.")
+ protocol = "http"
+ if self.webserver_certificate_file != "":
+ protocol = "https"
+ url = "%s://%s:%d/%s" % \
+ (protocol, self.hostname, int(self.webserver_port), page_name)
+ return requests.get(url, verify=self.webserver_certificate_file)
+ except Exception as e:
+ LOG.info("Debug webpage not yet available: %s", str(e))
sleep(interval)
assert 0, 'Debug webpage did not become available in expected time.'
def read_debug_webpage(self, page_name, timeout=10, interval=1):
- return self.open_debug_webpage(page_name, timeout=timeout, interval=interval).read()
+ return self.open_debug_webpage(page_name, timeout=timeout, interval=interval).text
def get_thrift_profile(self, query_id, timeout=10, interval=1):
"""Returns thrift profile of the specified query ID, if available"""
page_name = "query_profile_encoded?query_id=%s" % (query_id)
try:
response = self.open_debug_webpage(page_name, timeout=timeout, interval=interval)
- tbuf = response.read()
+ tbuf = response.text
except Exception as e:
LOG.info("Thrift profile for query %s not yet available: %s", query_id, str(e))
return None
@@ -166,8 +171,9 @@ class BaseImpalaService(object):
# new connections or accessing the debug webpage.
class ImpaladService(BaseImpalaService):
def __init__(self, hostname, webserver_port=25000, beeswax_port=21000, be_port=22000,
- hs2_port=21050):
- super(ImpaladService, self).__init__(hostname, webserver_port)
+ hs2_port=21050, webserver_certificate_file=""):
+ super(ImpaladService, self).__init__(
+ hostname, webserver_port, webserver_certificate_file)
self.beeswax_port = beeswax_port
self.be_port = be_port
self.hs2_port = hs2_port
@@ -327,8 +333,9 @@ class ImpaladService(BaseImpalaService):
# Allows for interacting with the StateStore service to perform operations such as
# accessing the debug webpage.
class StateStoredService(BaseImpalaService):
- def __init__(self, hostname, webserver_port):
- super(StateStoredService, self).__init__(hostname, webserver_port)
+ def __init__(self, hostname, webserver_port, webserver_certificate_file):
+ super(StateStoredService, self).__init__(
+ hostname, webserver_port, webserver_certificate_file)
def wait_for_live_subscribers(self, num_subscribers, timeout=15, interval=1):
self.wait_for_metric_value('statestore.live-backends', num_subscribers,
@@ -338,8 +345,9 @@ class StateStoredService(BaseImpalaService):
# Allows for interacting with the Catalog service to perform operations such as
# accessing the debug webpage.
class CatalogdService(BaseImpalaService):
- def __init__(self, hostname, webserver_port, service_port):
- super(CatalogdService, self).__init__(hostname, webserver_port)
+ def __init__(self, hostname, webserver_port, webserver_certificate_file, service_port):
+ super(CatalogdService, self).__init__(
+ hostname, webserver_port, webserver_certificate_file)
self.service_port = service_port
def get_catalog_version(self, timeout=10, interval=1):
http://git-wip-us.apache.org/repos/asf/impala/blob/5e92d139/tests/custom_cluster/test_client_ssl.py
----------------------------------------------------------------------
diff --git a/tests/custom_cluster/test_client_ssl.py b/tests/custom_cluster/test_client_ssl.py
index b6f7e04..edea640 100644
--- a/tests/custom_cluster/test_client_ssl.py
+++ b/tests/custom_cluster/test_client_ssl.py
@@ -19,6 +19,7 @@
import logging
import os
import pytest
+import requests
import signal
import ssl
import socket
@@ -110,13 +111,28 @@ class TestClientSsl(CustomClusterTestSuite):
assert "Query Status: Cancelled" in result.stdout
assert impalad.wait_for_num_in_flight_queries(0)
+ WEBSERVER_SSL_ARGS = ("--webserver_certificate_file=%(cert_dir)s/server-cert.pem "
+ "--webserver_private_key_file=%(cert_dir)s/server-key.pem "
+ "--hostname=localhost" # Must match hostname in certificate
+ % {'cert_dir': CERT_DIR})
+
+ @pytest.mark.execute_serially
+ @CustomClusterTestSuite.with_args(impalad_args=WEBSERVER_SSL_ARGS,
+ statestored_args=WEBSERVER_SSL_ARGS,
+ catalogd_args=WEBSERVER_SSL_ARGS)
+ def test_webserver_ssl(self):
+ "Tests that the debug web pages are reachable when run with ssl."
+ self._verify_ssl_webserver()
+
# Test that the shell can connect to a ECDH only cluster.
- TLS_ECDH_ARGS = ("--ssl_client_ca_certificate=%s/server-cert.pem "
- "--ssl_server_certificate=%s/server-cert.pem "
- "--ssl_private_key=%s/server-key.pem "
- "--hostname=localhost " # Required to match hostname in certificate"
- "--ssl_cipher_list=ECDHE-RSA-AES128-GCM-SHA256 "
- % (CERT_DIR, CERT_DIR, CERT_DIR))
+ TLS_ECDH_ARGS = ("--ssl_client_ca_certificate=%(cert_dir)s/server-cert.pem "
+ "--ssl_server_certificate=%(cert_dir)s/server-cert.pem "
+ "--ssl_private_key=%(cert_dir)s/server-key.pem "
+ "--hostname=localhost " # Must match hostname in certificate
+ "--ssl_cipher_list=ECDHE-RSA-AES128-GCM-SHA256 "
+ "--webserver_certificate_file=%(cert_dir)s/server-cert.pem "
+ "--webserver_private_key_file=%(cert_dir)s/server-key.pem "
+ % {'cert_dir': CERT_DIR})
@pytest.mark.execute_serially
@CustomClusterTestSuite.with_args(impalad_args=TLS_ECDH_ARGS,
@@ -128,6 +144,7 @@ class TestClientSsl(CustomClusterTestSuite):
def test_tls_ecdh(self, vector):
self._verify_negative_cases()
self._validate_positive_cases("%s/server-cert.pem" % self.CERT_DIR)
+ self._verify_ssl_webserver()
# Test that the shell can connect to a TLS1.2 only cluster, and for good measure
# restrict the cipher suite to just one choice.
@@ -209,3 +226,9 @@ class TestClientSsl(CustomClusterTestSuite):
result = run_impala_shell_cmd(shell_options)
for msg in [self.SSL_ENABLED, self.CONNECTED, self.FETCHED]:
assert msg in result.stderr
+
+ def _verify_ssl_webserver(self):
+ for port in ["25000", "25010", "25020"]:
+ url = "https://localhost:%s" % port
+ response = requests.get(url, verify="%s/server-cert.pem" % self.CERT_DIR)
+ assert response.status_code == requests.codes.ok, url
http://git-wip-us.apache.org/repos/asf/impala/blob/5e92d139/tests/custom_cluster/test_redaction.py
----------------------------------------------------------------------
diff --git a/tests/custom_cluster/test_redaction.py b/tests/custom_cluster/test_redaction.py
index 7789e06..3bf831e 100644
--- a/tests/custom_cluster/test_redaction.py
+++ b/tests/custom_cluster/test_redaction.py
@@ -109,10 +109,9 @@ class TestRedaction(CustomClusterTestSuite, unittest.TestCase):
# TODO: The HS2 interface may be better about exposing the query handle even if a
# query fails. Maybe investigate that after the switch to HS2.
regex = re.compile(r'query_id=(\w+:\w+)')
- for line in self.create_impala_service().open_debug_webpage('queries'):
- match = regex.search(line)
- if match:
- return match.group(1)
+ match = regex.search(self.create_impala_service().read_debug_webpage('queries'))
+ if match:
+ return match.group(1)
raise Exception('Unable to find any query id')
def assert_server_fails_to_start(self, rules, start_options, expected_error_message):
@@ -147,9 +146,8 @@ class TestRedaction(CustomClusterTestSuite, unittest.TestCase):
for response_format in ('html', 'json'):
# The 'html' param is actually ignored by the server.
url = page + '?query_id=' + query_id + "&" + response_format
- results = grep_file(impala_service.open_debug_webpage(url), unredacted_value)
- assert not results, "Web page %s should not contain '%s' but does" \
- % (url, unredacted_value)
+ assert unredacted_value not in impala_service.read_debug_webpage(url), \
+ "Web page %s should not contain '%s' but does" % (url, unredacted_value)
# But the redacted value should be shown.
self.assert_web_ui_contains(query_id, redacted_value)
@@ -159,17 +157,15 @@ class TestRedaction(CustomClusterTestSuite, unittest.TestCase):
impala_service = self.create_impala_service()
for page in ('queries', 'query_stmt', 'query_plan_text', 'query_profile'):
url = '%s?query_id=%s' % (page, query_id)
- results = grep_file(impala_service.open_debug_webpage(url), search)
- assert results, "Web page %s should contain '%s' but does not" \
- % (url, search)
+ assert search in impala_service.read_debug_webpage(url), \
+ "Web page %s should contain '%s' but does not" % (url, search)
def assert_query_profile_contains(self, query_id, search):
''' Asserts that the query profile for 'query_id' contains 'search' string'''
impala_service = self.create_impala_service()
url = 'query_profile?query_id=%s' % query_id
- results = grep_file(impala_service.open_debug_webpage(url), search)
- assert results, "Query profile %s should contain '%s' but does not" \
- % (url, search)
+ assert search in impala_service.read_debug_webpage(url), \
+ "Query profile %s should contain '%s' but does not" % (url, search)
@pytest.mark.execute_serially
def test_bad_rules(self):
http://git-wip-us.apache.org/repos/asf/impala/blob/5e92d139/tests/run-tests.py
----------------------------------------------------------------------
diff --git a/tests/run-tests.py b/tests/run-tests.py
index c2fd420..9aaeaa5 100755
--- a/tests/run-tests.py
+++ b/tests/run-tests.py
@@ -214,10 +214,8 @@ def print_metrics(substring):
print ">" * 80
port = impalad._get_webserver_port()
print "connections metrics for impalad at port {0}:".format(port)
- debug_info = json.loads(ImpaladService(
- impalad.hostname,
- webserver_port=port)
- .open_debug_webpage('metrics?json').read())
+ debug_info = json.loads(ImpaladService(impalad.hostname, webserver_port=port)
+ .read_debug_webpage('metrics?json'))
for metric in debug_info['metric_group']['metrics']:
if substring in metric['name']:
print json.dumps(metric, indent=1)
[09/11] impala git commit: IMPALA-7669: Gracefully handle concurrent
invalidate/partial fetch RPCs
Posted by tm...@apache.org.
IMPALA-7669: Gracefully handle concurrent invalidate/partial fetch RPCs
The bug here was that any partial RPC on an IncompleteTable was throwing
an NPE.
Ideally, we attempt to load the table (if we find that it is not loaded)
before making the partial info request, but a concurrent invalidate could
reset the table state and move it back to an uninitialized state.
This patch handles this case better by propagating a meaningful error to
the caller.
Testing:
-------
- Added a test that fails consistently with an NPE without this patch.
Change-Id: I8533f73f25ca42a20f146ddfd95d4213add9b705
Reviewed-on: http://gerrit.cloudera.org:8080/11638
Reviewed-by: Bharath Vissapragada <bh...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
Project: http://git-wip-us.apache.org/repos/asf/impala/repo
Commit: http://git-wip-us.apache.org/repos/asf/impala/commit/2b2cf8d9
Tree: http://git-wip-us.apache.org/repos/asf/impala/tree/2b2cf8d9
Diff: http://git-wip-us.apache.org/repos/asf/impala/diff/2b2cf8d9
Branch: refs/heads/master
Commit: 2b2cf8d96617320d184d070f9319c2463aa0d84f
Parents: 28aecd6
Author: Bharath Vissapragada <bh...@cloudera.com>
Authored: Tue Oct 9 16:40:59 2018 -0700
Committer: Impala Public Jenkins <im...@cloudera.com>
Committed: Wed Oct 17 04:03:53 2018 +0000
----------------------------------------------------------------------
common/thrift/CatalogService.thrift | 1 +
.../impala/catalog/CatalogServiceCatalog.java | 4 ++
.../apache/impala/catalog/IncompleteTable.java | 2 +
.../catalog/local/CatalogdMetaProvider.java | 1 +
.../impala/catalog/PartialCatalogInfoTest.java | 3 --
tests/custom_cluster/test_local_catalog.py | 52 ++++++++++++++++++++
6 files changed, 60 insertions(+), 3 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/impala/blob/2b2cf8d9/common/thrift/CatalogService.thrift
----------------------------------------------------------------------
diff --git a/common/thrift/CatalogService.thrift b/common/thrift/CatalogService.thrift
index 6b94697..c0792b3 100644
--- a/common/thrift/CatalogService.thrift
+++ b/common/thrift/CatalogService.thrift
@@ -378,6 +378,7 @@ enum CatalogLookupStatus {
OK,
DB_NOT_FOUND,
TABLE_NOT_FOUND,
+ TABLE_NOT_LOADED,
FUNCTION_NOT_FOUND
}
http://git-wip-us.apache.org/repos/asf/impala/blob/2b2cf8d9/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
----------------------------------------------------------------------
diff --git a/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java b/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
index b3c714f..9533507 100644
--- a/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
+++ b/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
@@ -2195,6 +2195,10 @@ public class CatalogServiceCatalog extends Catalog {
}
if (table == null) {
return createGetPartialCatalogObjectError(CatalogLookupStatus.TABLE_NOT_FOUND);
+ } else if (!table.isLoaded()) {
+ // Table can still remain in an incomplete state if there was a concurrent
+ // invalidate request.
+ return createGetPartialCatalogObjectError(CatalogLookupStatus.TABLE_NOT_LOADED);
}
// TODO(todd): consider a read-write lock here.
table.getLock().lock();
http://git-wip-us.apache.org/repos/asf/impala/blob/2b2cf8d9/fe/src/main/java/org/apache/impala/catalog/IncompleteTable.java
----------------------------------------------------------------------
diff --git a/fe/src/main/java/org/apache/impala/catalog/IncompleteTable.java b/fe/src/main/java/org/apache/impala/catalog/IncompleteTable.java
index ba3e5cf..7de3d89 100644
--- a/fe/src/main/java/org/apache/impala/catalog/IncompleteTable.java
+++ b/fe/src/main/java/org/apache/impala/catalog/IncompleteTable.java
@@ -20,6 +20,7 @@ package org.apache.impala.catalog;
import java.util.List;
import java.util.Set;
+import com.google.common.base.Preconditions;
import org.apache.hadoop.hive.metastore.IMetaStoreClient;
import org.apache.impala.common.ImpalaException;
@@ -136,6 +137,7 @@ public class IncompleteTable extends Table {
@Override
public TGetPartialCatalogObjectResponse getPartialInfo(
TGetPartialCatalogObjectRequest req) throws TableLoadingException {
+ Preconditions.checkNotNull(cause_);
Throwables.propagateIfPossible(cause_, TableLoadingException.class);
throw new TableLoadingException(cause_.getMessage());
}
http://git-wip-us.apache.org/repos/asf/impala/blob/2b2cf8d9/fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java
----------------------------------------------------------------------
diff --git a/fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java b/fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java
index 62f1d3e..e099b53 100644
--- a/fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java
+++ b/fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java
@@ -344,6 +344,7 @@ public class CatalogdMetaProvider implements MetaProvider {
case DB_NOT_FOUND:
case FUNCTION_NOT_FOUND:
case TABLE_NOT_FOUND:
+ case TABLE_NOT_LOADED:
invalidateCacheForObject(req.object_desc);
throw new InconsistentMetadataFetchException(
String.format("Fetching %s failed. Could not find %s",
http://git-wip-us.apache.org/repos/asf/impala/blob/2b2cf8d9/fe/src/test/java/org/apache/impala/catalog/PartialCatalogInfoTest.java
----------------------------------------------------------------------
diff --git a/fe/src/test/java/org/apache/impala/catalog/PartialCatalogInfoTest.java b/fe/src/test/java/org/apache/impala/catalog/PartialCatalogInfoTest.java
index 07d3309..63e0fda 100644
--- a/fe/src/test/java/org/apache/impala/catalog/PartialCatalogInfoTest.java
+++ b/fe/src/test/java/org/apache/impala/catalog/PartialCatalogInfoTest.java
@@ -34,7 +34,6 @@ import avro.shaded.com.google.common.collect.Lists;
import com.google.common.base.Preconditions;
import org.apache.hadoop.hive.metastore.api.ColumnStatisticsObj;
import org.apache.impala.common.InternalException;
-import org.apache.impala.common.RuntimeEnv;
import org.apache.impala.service.BackendConfig;
import org.apache.impala.testutil.CatalogServiceTestCatalog;
import org.apache.impala.thrift.TCatalogInfoSelector;
@@ -50,8 +49,6 @@ import org.apache.impala.thrift.TTableInfoSelector;
import org.apache.thrift.TDeserializer;
import org.apache.thrift.TException;
import org.apache.thrift.TSerializer;
-import org.junit.AfterClass;
-import org.junit.BeforeClass;
import org.junit.Test;
import com.google.common.collect.ImmutableList;
http://git-wip-us.apache.org/repos/asf/impala/blob/2b2cf8d9/tests/custom_cluster/test_local_catalog.py
----------------------------------------------------------------------
diff --git a/tests/custom_cluster/test_local_catalog.py b/tests/custom_cluster/test_local_catalog.py
index 14a9a54..916443b 100644
--- a/tests/custom_cluster/test_local_catalog.py
+++ b/tests/custom_cluster/test_local_catalog.py
@@ -18,6 +18,7 @@
# Test behaviors specific to --use_local_catalog being enabled.
import pytest
+import Queue
import random
import threading
import time
@@ -230,6 +231,57 @@ class TestCompactCatalogUpdates(CustomClusterTestSuite):
@pytest.mark.execute_serially
@CustomClusterTestSuite.with_args(
+ impalad_args="--use_local_catalog=true",
+ catalogd_args="--catalog_topic_mode=minimal")
+ def test_concurrent_invalidate_with_queries(self, unique_database):
+ """
+ Tests that the queries are replanned when they clash with concurrent invalidates.
+ """
+ # TODO: Merge this with the above test after fixing IMPALA-7717
+ try:
+ impalad1 = self.cluster.impalads[0]
+ impalad2 = self.cluster.impalads[1]
+ client1 = impalad1.service.create_beeswax_client()
+ client2 = impalad2.service.create_beeswax_client()
+
+ # Track the number of replans.
+ replans_seen = [0]
+ replans_seen_lock = threading.Lock()
+
+ # Queue to propagate exceptions from failed queries, if any.
+ failed_queries = Queue.Queue()
+
+ def stress_thread(client):
+ while replans_seen[0] == 0:
+ q = random.choice([
+ 'invalidate metadata functional.alltypesnopart',
+ 'select count(*) from functional.alltypesnopart',
+ 'select count(*) from functional.alltypesnopart'])
+ try:
+ ret = self.execute_query_expect_success(client, q)
+ except Exception as e:
+ failed_queries.put((q, str(e)))
+ if RETRY_PROFILE_MSG in ret.runtime_profile:
+ with replans_seen_lock:
+ replans_seen[0] += 1
+
+ threads = [threading.Thread(target=stress_thread, args=(c,))
+ for c in [client1, client2]]
+ for t in threads:
+ t.start()
+ for t in threads:
+ t.join(30)
+ assert failed_queries.empty(),\
+ "Failed query count non zero: %s" % list(failed_queries.queue)
+ assert replans_seen[0] > 0, "Did not trigger any re-plans"
+
+ finally:
+ client1.close()
+ client2.close()
+
+
+ @pytest.mark.execute_serially
+ @CustomClusterTestSuite.with_args(
impalad_args="--use_local_catalog=true --local_catalog_max_fetch_retries=0",
catalogd_args="--catalog_topic_mode=minimal")
def test_replan_limit(self, unique_database):
[02/11] impala git commit: IMPALA-7673: Support values from other
variables in Impala shell --var
Posted by tm...@apache.org.
IMPALA-7673: Support values from other variables in Impala shell --var
Prior to this patch, Impala shell --var could not accept values from
other variables unlike the one in Impala interactive shell with the SET
command. This patch refactors the logic of variable substitution to
use the same logic in both interactive and command line shells.
Example:
$ impala-shell.sh \
--var="msg1=1" \
--var="msg2=\${var:msg1}2" \
--var="msg3=\${var:msg1}\${var:msg2}"
[localhost:21000] default> select ${var:msg3};
Query: select 112
+-----+
| 112 |
+-----+
| 112 |
+-----+
Testing:
- Added a new shell test
- Ran all shell tests
Change-Id: Ib5b9fda329c45f2e5682f3cbc76d29ceca2e226a
Reviewed-on: http://gerrit.cloudera.org:8080/11623
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
Project: http://git-wip-us.apache.org/repos/asf/impala/repo
Commit: http://git-wip-us.apache.org/repos/asf/impala/commit/31dfa3e2
Tree: http://git-wip-us.apache.org/repos/asf/impala/tree/31dfa3e2
Diff: http://git-wip-us.apache.org/repos/asf/impala/diff/31dfa3e2
Branch: refs/heads/master
Commit: 31dfa3e28c2dadb6567c1bc812011286024efa4c
Parents: 2737e22
Author: Fredy Wijaya <fw...@cloudera.com>
Authored: Mon Oct 8 20:36:56 2018 -0700
Committer: Impala Public Jenkins <im...@cloudera.com>
Committed: Tue Oct 16 00:50:26 2018 +0000
----------------------------------------------------------------------
shell/impala_shell.py | 90 ++++++++++++++++--------------
tests/shell/test_shell_commandline.py | 25 +++++++++
2 files changed, 73 insertions(+), 42 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/impala/blob/31dfa3e2/shell/impala_shell.py
----------------------------------------------------------------------
diff --git a/shell/impala_shell.py b/shell/impala_shell.py
index 2abee3f..a7b1ac8 100755
--- a/shell/impala_shell.py
+++ b/shell/impala_shell.py
@@ -543,32 +543,6 @@ class ImpalaShell(object, cmd.Cmd):
print_to_stderr("Failed to reconnect and close (try %i/%i): %s" % (
cancel_try + 1, ImpalaShell.CANCELLATION_TRIES, err_msg))
- def _replace_variables(self, query):
- """Replaces variable within the query text with their corresponding values"""
- errors = False
- matches = set(map(lambda v: v.upper(), re.findall(r'(?<!\\)\${([^}]+)}', query)))
- for name in matches:
- value = None
- # Check if syntax is correct
- var_name = self._get_var_name(name)
- if var_name is None:
- print_to_stderr('Error: Unknown substitution syntax (%s). ' % (name,) + \
- 'Use ${VAR:var_name}.')
- errors = True
- else:
- # Replaces variable value
- if self.set_variables and var_name in self.set_variables:
- value = self.set_variables[var_name]
- regexp = re.compile(r'(?<!\\)\${%s}' % (name,), re.IGNORECASE)
- query = regexp.sub(value, query)
- else:
- print_to_stderr('Error: Unknown variable %s' % (var_name))
- errors = True
- if errors:
- return None
- else:
- return query
-
def set_prompt(self, db):
self.prompt = ImpalaShell.PROMPT_FORMAT.format(
host=self.impalad[0], port=self.impalad[1], db=db)
@@ -598,7 +572,7 @@ class ImpalaShell(object, cmd.Cmd):
the interactive case, when cmdloop is called.
"""
# Replace variables in the statement before it's executed
- line = self._replace_variables(line)
+ line = replace_variables(self.set_variables, line)
# Cmd is an old-style class, hence we need to call the method directly
# instead of using super()
# TODO: This may have to be changed to a super() call once we move to Python 3
@@ -673,18 +647,6 @@ class ImpalaShell(object, cmd.Cmd):
except KeyError:
return False
- def _get_var_name(self, name):
- """Look for a namespace:var_name pattern in an option name.
- Return the variable name if it's a match or None otherwise.
- """
- ns_match = re.match(r'^([^:]*):(.*)', name)
- if ns_match is not None:
- ns = ns_match.group(1)
- var_name = ns_match.group(2)
- if ns in ImpalaShell.VAR_PREFIXES:
- return var_name
- return None
-
def _print_with_set(self, print_level):
self._print_options(print_level)
print "\nVariables:"
@@ -721,7 +683,7 @@ class ImpalaShell(object, cmd.Cmd):
return CmdStatus.ERROR
option_upper = tokens[0].upper()
# Check if it's a variable
- var_name = self._get_var_name(option_upper)
+ var_name = get_var_name(option_upper)
if var_name is not None:
# Set the variable
self.set_variables[var_name] = tokens[1]
@@ -745,7 +707,7 @@ class ImpalaShell(object, cmd.Cmd):
return CmdStatus.ERROR
option = args.upper()
# Check if it's a variable
- var_name = self._get_var_name(option)
+ var_name = get_var_name(option)
if var_name is not None:
if self.set_variables.get(var_name):
print 'Unsetting variable %s' % var_name
@@ -1539,9 +1501,53 @@ def parse_variables(keyvals):
parser.print_help()
sys.exit(1)
else:
- vars[match.groups()[0].upper()] = match.groups()[1]
+ vars[match.groups()[0].upper()] = replace_variables(vars, match.groups()[1])
return vars
+
+def replace_variables(set_variables, string):
+ """Replaces variable within the string with their corresponding values using the
+ given set_variables."""
+ errors = False
+ matches = set(map(lambda v: v.upper(), re.findall(r'(?<!\\)\${([^}]+)}', string)))
+ for name in matches:
+ value = None
+ # Check if syntax is correct
+ var_name = get_var_name(name)
+ if var_name is None:
+ print_to_stderr('Error: Unknown substitution syntax (%s). ' % (name,) +
+ 'Use ${VAR:var_name}.')
+ errors = True
+ else:
+ # Replaces variable value
+ if set_variables and var_name in set_variables:
+ value = set_variables[var_name]
+ if value is None:
+ errors = True
+ else:
+ regexp = re.compile(r'(?<!\\)\${%s}' % (name,), re.IGNORECASE)
+ string = regexp.sub(value, string)
+ else:
+ print_to_stderr('Error: Unknown variable %s' % (var_name))
+ errors = True
+ if errors:
+ return None
+ else:
+ return string
+
+
+def get_var_name(name):
+ """Look for a namespace:var_name pattern in an option name.
+ Return the variable name if it's a match or None otherwise.
+ """
+ ns_match = re.match(r'^([^:]*):(.*)', name)
+ if ns_match is not None:
+ ns = ns_match.group(1)
+ var_name = ns_match.group(2)
+ if ns in ImpalaShell.VAR_PREFIXES:
+ return var_name
+ return None
+
def execute_queries_non_interactive_mode(options, query_options):
"""Run queries in non-interactive mode."""
if options.query_file:
http://git-wip-us.apache.org/repos/asf/impala/blob/31dfa3e2/tests/shell/test_shell_commandline.py
----------------------------------------------------------------------
diff --git a/tests/shell/test_shell_commandline.py b/tests/shell/test_shell_commandline.py
index 0f73620..b74fbc2 100644
--- a/tests/shell/test_shell_commandline.py
+++ b/tests/shell/test_shell_commandline.py
@@ -535,6 +535,31 @@ class TestImpalaShell(ImpalaTestSuite):
assert ("Error: Could not parse key-value \"foo\". It must follow the pattern "
"\"KEY=VALUE\".") in result.stderr
+ # IMPALA-7673: Test that variable substitution in command line can accept values
+ # from other variables just like the one in interactive shell.
+ result = run_impala_shell_cmd('--var="msg1=1" --var="msg2=${var:msg1}2" '
+ '--var="msg3=${var:msg1}${var:msg2}" '
+ '--query="select ${var:msg3}"')
+ self._validate_shell_messages(result.stderr, ['112', 'Fetched 1 row(s)'],
+ should_exist=True)
+
+ # Test with an escaped variable.
+ result = run_impala_shell_cmd('--var="msg1=1" --var="msg2=${var:msg1}2" '
+ '--var="msg3=\${var:msg1}${var:msg2}" '
+ '--query="select \'${var:msg3}\'"')
+ self._validate_shell_messages(result.stderr, ['${var:msg1}12', 'Fetched 1 row(s)'],
+ should_exist=True)
+
+ # Referencing a non-existent variable will result in an error.
+ result = run_impala_shell_cmd('--var="msg1=1" --var="msg2=${var:doesnotexist}2" '
+ '--var="msg3=\${var:msg1}${var:msg2}" '
+ '--query="select \'${var:msg3}\'"',
+ expect_success=False)
+ self._validate_shell_messages(result.stderr,
+ ['Error: Unknown variable DOESNOTEXIST',
+ 'Could not execute command: select \'${var:msg3}\''],
+ should_exist=True)
+
# Checks if 'messages' exists/does not exist in 'result_stderr' based on the value of
# 'should_exist'
def _validate_shell_messages(self, result_stderr, messages, should_exist=True):
[10/11] impala git commit: IMPALA-7713: Add test coverage for
catalogd restart when authorization is enabled
Posted by tm...@apache.org.
IMPALA-7713: Add test coverage for catalogd restart when authorization is enabled
This patch adds a test coverage for catalogd restart when authorization
is enabled to ensure all privileges in the impalad's catalogs get reset
after the catalogd restart to avoid stale privileges in the impalad's
catalogs, which can pose a security issue.
Testing:
- Ran all E2E authorization tests
- Added a new test
Change-Id: Ib9a168697401cf0b83c7a193fa477888b48cb369
Reviewed-on: http://gerrit.cloudera.org:8080/11696
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
Project: http://git-wip-us.apache.org/repos/asf/impala/repo
Commit: http://git-wip-us.apache.org/repos/asf/impala/commit/0cd91518
Tree: http://git-wip-us.apache.org/repos/asf/impala/tree/0cd91518
Diff: http://git-wip-us.apache.org/repos/asf/impala/diff/0cd91518
Branch: refs/heads/master
Commit: 0cd9151801cf446330b129b0609d6cedd4b98f06
Parents: 2b2cf8d
Author: Fredy Wijaya <fw...@cloudera.com>
Authored: Tue Oct 16 12:30:34 2018 -0700
Committer: Impala Public Jenkins <im...@cloudera.com>
Committed: Wed Oct 17 05:33:05 2018 +0000
----------------------------------------------------------------------
tests/authorization/test_authorization.py | 55 ++++++++++++++++++++++++++
tests/conftest.py | 12 ++++++
2 files changed, 67 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/impala/blob/0cd91518/tests/authorization/test_authorization.py
----------------------------------------------------------------------
diff --git a/tests/authorization/test_authorization.py b/tests/authorization/test_authorization.py
index a508440..cf5b2e9 100644
--- a/tests/authorization/test_authorization.py
+++ b/tests/authorization/test_authorization.py
@@ -383,3 +383,58 @@ class TestAuthorization(CustomClusterTestSuite):
def test_deprecated_flags(self):
assert_file_in_dir_contains(self.impala_log_dir, "authorization_policy_file flag" +
" is deprecated. Object Ownership feature is not supported")
+
+ @pytest.mark.execute_serially
+ @CustomClusterTestSuite.with_args(
+ impalad_args="--server_name=server1 --sentry_config=%s" % SENTRY_CONFIG_FILE,
+ catalogd_args="--sentry_config=%s" % SENTRY_CONFIG_FILE,
+ impala_log_dir=tempfile.mkdtemp(prefix="test_catalog_restart_",
+ dir=os.getenv("LOG_DIR")))
+ def test_catalog_restart(self, unique_role):
+ """IMPALA-7713: Tests that a catalogd restart when authorization is enabled should
+ reset the previous privileges stored in impalad's catalog to avoid stale privilege
+ data in the impalad's catalog."""
+ def assert_privileges():
+ result = self.client.execute("show grant role %s_foo" % unique_role)
+ TestAuthorization._check_privileges(result, [["database", "functional",
+ "", "", "", "all", "false"]])
+
+ result = self.client.execute("show grant role %s_bar" % unique_role)
+ TestAuthorization._check_privileges(result, [["database", "functional_kudu",
+ "", "", "", "all", "false"]])
+
+ result = self.client.execute("show grant role %s_baz" % unique_role)
+ TestAuthorization._check_privileges(result, [["database", "functional_avro",
+ "", "", "", "all", "false"]])
+
+ self.role_cleanup(unique_role)
+ try:
+ self.client.execute("create role %s_foo" % unique_role)
+ self.client.execute("create role %s_bar" % unique_role)
+ self.client.execute("create role %s_baz" % unique_role)
+ self.client.execute("grant all on database functional to role %s_foo" %
+ unique_role)
+ self.client.execute("grant all on database functional_kudu to role %s_bar" %
+ unique_role)
+ self.client.execute("grant all on database functional_avro to role %s_baz" %
+ unique_role)
+
+ assert_privileges()
+ self._start_impala_cluster(["--catalogd_args=--sentry_config=%s" %
+ SENTRY_CONFIG_FILE, "--restart_catalogd_only"])
+ assert_privileges()
+ finally:
+ self.role_cleanup(unique_role)
+
+ def role_cleanup(self, role_name_match):
+ """Cleans up any roles that match the given role name."""
+ for role_name in self.client.execute("show roles").data:
+ if role_name_match in role_name:
+ self.client.execute("drop role %s" % role_name)
+
+ @staticmethod
+ def _check_privileges(result, expected):
+ def columns(row):
+ cols = row.split("\t")
+ return cols[0:len(cols) - 1]
+ assert map(columns, result.data) == expected
http://git-wip-us.apache.org/repos/asf/impala/blob/0cd91518/tests/conftest.py
----------------------------------------------------------------------
diff --git a/tests/conftest.py b/tests/conftest.py
index 1eb5c76..4e1c837 100644
--- a/tests/conftest.py
+++ b/tests/conftest.py
@@ -329,6 +329,18 @@ def unique_database(request, testid_checksum):
return first_db_name
+@pytest.fixture
+def unique_role(request, testid_checksum):
+ """Returns a unique role to any test using the fixture. The fixture does not create
+ a role."""
+ role_name_prefix = request.function.__name__
+ fixture_params = getattr(request, 'param', None)
+ if fixture_params is not None:
+ if 'name_prefix' in fixture_params:
+ role_name_prefix = fixture_params['name_prefix']
+ return '{0}_{1}_role'.format(role_name_prefix, testid_checksum)
+
+
@pytest.yield_fixture
def kudu_client():
"""Provides a new Kudu client as a pytest fixture. The client only exists for the
[03/11] impala git commit: IMPALA-7708: Switch to faster deflater
compression level for incr stats
Posted by tm...@apache.org.
IMPALA-7708: Switch to faster deflater compression level for incr stats
On a table with 3000 partitions and ~150 columns, we noticed that the
BEST_SPEED deflater strategy is ~8x faster with ~4% compression ratio
penalty. Given these results, this patch switches the default to
BEST_SPEED from BEST_COMPRESSION.
Change-Id: Ife688aca3aed0e1e8af26c8348b850175d84b4ad
Reviewed-on: http://gerrit.cloudera.org:8080/11685
Reviewed-by: Philip Zeyliger <ph...@cloudera.com>
Reviewed-by: Vuk Ercegovac <ve...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
Project: http://git-wip-us.apache.org/repos/asf/impala/repo
Commit: http://git-wip-us.apache.org/repos/asf/impala/commit/3b6c0f62
Tree: http://git-wip-us.apache.org/repos/asf/impala/tree/3b6c0f62
Diff: http://git-wip-us.apache.org/repos/asf/impala/diff/3b6c0f62
Branch: refs/heads/master
Commit: 3b6c0f6296e807b25f3e40bd614b2571f4f01d48
Parents: 31dfa3e
Author: Bharath Vissapragada <bh...@cloudera.com>
Authored: Mon Oct 15 10:47:19 2018 -0700
Committer: Impala Public Jenkins <im...@cloudera.com>
Committed: Tue Oct 16 01:13:16 2018 +0000
----------------------------------------------------------------------
fe/src/main/java/org/apache/impala/util/CompressionUtil.java | 6 ++++--
1 file changed, 4 insertions(+), 2 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/impala/blob/3b6c0f62/fe/src/main/java/org/apache/impala/util/CompressionUtil.java
----------------------------------------------------------------------
diff --git a/fe/src/main/java/org/apache/impala/util/CompressionUtil.java b/fe/src/main/java/org/apache/impala/util/CompressionUtil.java
index 86d8477..f6e11a1 100644
--- a/fe/src/main/java/org/apache/impala/util/CompressionUtil.java
+++ b/fe/src/main/java/org/apache/impala/util/CompressionUtil.java
@@ -40,9 +40,11 @@ public class CompressionUtil {
public static byte[] deflateCompress(byte[] input) {
if (input == null) return null;
ByteArrayOutputStream bos = new ByteArrayOutputStream(input.length);
- // TODO: Benchmark other compression levels.
+ // Experiments on a wide partitioned table with incremental stats showed that the
+ // Deflater with 'BEST_SPEED' level provided reasonable compression ratios at much
+ // faster speeds compared to other modes like BEST_COMPRESSION/DEFAULT_COMPRESSION.
DeflaterOutputStream stream =
- new DeflaterOutputStream(bos, new Deflater(Deflater.BEST_COMPRESSION));
+ new DeflaterOutputStream(bos, new Deflater(Deflater.BEST_SPEED));
try {
stream.write(input);
stream.close();
[06/11] impala git commit: IMPALA-7702: enable fetch incremental
stats by default
Posted by tm...@apache.org.
IMPALA-7702: enable fetch incremental stats by default
Flips the default from always off to always on for
--pull_incremental_statistics. With this setting, the default
is for coordinators to fetch incremental stats from catalogd
directly (only when computing incremental stats) instead of
receiving it from the statestore broadcast.
Fetching incremental stats is not applicable when using a
CatalogMetaProvider. By making fetch the default, it would
require that --pull_incremental_statistics is set to false
when enabling CatalogMetaProvider. This change makes
--use_local_catalog to take priority over --pull_incremental_statistics
so that when both are turned on, only the local catalog setting
is enabled.
Testing:
- manual testing
- moved the testing for pull incremental stats out of custom cluster
tests since the default flipped
- added tests that run with local catalog and pulling incremental stats.
Change-Id: I5601a24f81bb3466cff5308c7093d2765bb1c481
Reviewed-on: http://gerrit.cloudera.org:8080/11677
Reviewed-by: Vuk Ercegovac <ve...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
Project: http://git-wip-us.apache.org/repos/asf/impala/repo
Commit: http://git-wip-us.apache.org/repos/asf/impala/commit/ad265842
Tree: http://git-wip-us.apache.org/repos/asf/impala/tree/ad265842
Diff: http://git-wip-us.apache.org/repos/asf/impala/diff/ad265842
Branch: refs/heads/master
Commit: ad265842b691927e1c204203390172a92dc38a68
Parents: ec2daba
Author: Vuk Ercegovac <ve...@cloudera.com>
Authored: Thu Oct 11 22:41:20 2018 -0700
Committer: Impala Public Jenkins <im...@cloudera.com>
Committed: Tue Oct 16 10:12:03 2018 +0000
----------------------------------------------------------------------
be/src/common/global-flags.cc | 4 +-
.../impala/analysis/ComputeStatsStmt.java | 5 ++
tests/custom_cluster/test_incremental_stats.py | 46 +++++++++++
tests/custom_cluster/test_pull_stats.py | 84 --------------------
tests/metadata/test_compute_stats.py | 49 ++++++++++++
5 files changed, 102 insertions(+), 86 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/impala/blob/ad265842/be/src/common/global-flags.cc
----------------------------------------------------------------------
diff --git a/be/src/common/global-flags.cc b/be/src/common/global-flags.cc
index 2ea1ca5..d6f1d04 100644
--- a/be/src/common/global-flags.cc
+++ b/be/src/common/global-flags.cc
@@ -210,11 +210,11 @@ DEFINE_bool_hidden(disable_catalog_data_ops_debug_only, false,
// same way, is error prone. One fix for this flag is to set it only on
// catalogd, propagate the setting as a property of the Catalog object, and let
// impalad uses act on this setting.
-DEFINE_bool(pull_incremental_statistics, false,
+DEFINE_bool(pull_incremental_statistics, true,
"When set, impalad coordinators pull incremental statistics from catalogd on-demand "
"and catalogd does not broadcast incremental statistics via statestored to "
"coordinators. If used, the flag must be set on both catalogd and all impalad "
- "coordinators.");
+ "coordinators. This feature should not be used when --use_local_catalog is true.");
DEFINE_int32(invalidate_tables_timeout_s, 0, "If a table has not been referenced in a "
"SQL statement for more than the configured amount of time, the catalog server will "
http://git-wip-us.apache.org/repos/asf/impala/blob/ad265842/fe/src/main/java/org/apache/impala/analysis/ComputeStatsStmt.java
----------------------------------------------------------------------
diff --git a/fe/src/main/java/org/apache/impala/analysis/ComputeStatsStmt.java b/fe/src/main/java/org/apache/impala/analysis/ComputeStatsStmt.java
index 24f387c..a85a946 100644
--- a/fe/src/main/java/org/apache/impala/analysis/ComputeStatsStmt.java
+++ b/fe/src/main/java/org/apache/impala/analysis/ComputeStatsStmt.java
@@ -615,7 +615,12 @@ public class ComputeStatsStmt extends StatementBase {
int expectedNumStats = partitions.size() - excludedPartitions.size();
Preconditions.checkArgument(expectedNumStats >= 0);
+ // Incremental stats are fetched only when configured to do so except
+ // when also using a local catalog or when testing. When using a local
+ // catalog, it makes more sense to use the getPartitions api which is
+ // designed to fetch specific fields and specific partitions.
if (BackendConfig.INSTANCE.pullIncrementalStatistics()
+ && !BackendConfig.INSTANCE.getBackendCfg().use_local_catalog
&& !RuntimeEnv.INSTANCE.isTestEnv()) {
// We're configured to fetch the statistics from catalogd, so collect the relevant
// partition ids.
http://git-wip-us.apache.org/repos/asf/impala/blob/ad265842/tests/custom_cluster/test_incremental_stats.py
----------------------------------------------------------------------
diff --git a/tests/custom_cluster/test_incremental_stats.py b/tests/custom_cluster/test_incremental_stats.py
new file mode 100644
index 0000000..e58d5fd
--- /dev/null
+++ b/tests/custom_cluster/test_incremental_stats.py
@@ -0,0 +1,46 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import pytest
+from tests.common.custom_cluster_test_suite import CustomClusterTestSuite
+
+class TestIncrementalStatistics(CustomClusterTestSuite):
+ """Tests flag that controls pulling directly from catalogd using the
+ --pull_incremental_statistics flag."""
+
+ @classmethod
+ def get_workload(self):
+ return 'functional-query'
+
+ @pytest.mark.execute_serially
+ @CustomClusterTestSuite.with_args(impalad_args="--pull_incremental_statistics=false",
+ catalogd_args="--pull_incremental_statistics=false")
+ def test_push_stats(self, vector, unique_database):
+ """
+ Tests compute incremental stats when incremental stats are pushed via statestore.
+ """
+ self.run_test_case('QueryTest/compute-stats-incremental', vector, unique_database)
+
+ @pytest.mark.execute_serially
+ @CustomClusterTestSuite.with_args(
+ impalad_args="--pull_incremental_statistics=true --use_local_catalog=true",
+ catalogd_args="--pull_incremental_statistics=true --catalog_topic_mode=minimal")
+ def test_with_local_catalog(self, vector, unique_database):
+ """
+ Tests that when local catalog is used, the pull incremental stats flag has no effect.
+ """
+ self.run_test_case('QueryTest/compute-stats-incremental', vector, unique_database)
http://git-wip-us.apache.org/repos/asf/impala/blob/ad265842/tests/custom_cluster/test_pull_stats.py
----------------------------------------------------------------------
diff --git a/tests/custom_cluster/test_pull_stats.py b/tests/custom_cluster/test_pull_stats.py
deleted file mode 100644
index b852f3d..0000000
--- a/tests/custom_cluster/test_pull_stats.py
+++ /dev/null
@@ -1,84 +0,0 @@
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing,
-# software distributed under the License is distributed on an
-# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-# KIND, either express or implied. See the License for the
-# specific language governing permissions and limitations
-# under the License.
-
-import pytest
-from tests.common.custom_cluster_test_suite import CustomClusterTestSuite
-
-class TestPullStatistics(CustomClusterTestSuite):
- """Tests incremental statistics when pulling directly from catalogd
- using the --pull_incremental_statistics flag."""
-
- @classmethod
- def get_workload(self):
- return 'functional-query'
-
- @pytest.mark.execute_serially
- @CustomClusterTestSuite.with_args(impalad_args="--pull_incremental_statistics=true",
- catalogd_args="--pull_incremental_statistics=true")
- def test_pull_stats(self, vector, unique_database):
- self.run_test_case('QueryTest/compute-stats-incremental', vector, unique_database)
-
- @pytest.mark.execute_serially
- @CustomClusterTestSuite.with_args(impalad_args="--pull_incremental_statistics=true",
- catalogd_args="--pull_incremental_statistics=true")
- def test_pull_stats_profile(self, vector, unique_database):
- """Checks that the frontend profile includes metrics when computing
- incremental statistics.
- """
- try:
- client = self.cluster.impalads[0].service.create_beeswax_client()
- create = "create table test like functional.alltypes"
- load = "insert into test partition(year, month) select * from functional.alltypes"
- insert = """insert into test partition(year=2009, month=1) values
- (29349999, true, 4, 4, 4, 40,4.400000095367432,40.4,
- "10/21/09","4","2009-10-21 03:24:09.600000000")"""
- stats_all = "compute incremental stats test"
- stats_part = "compute incremental stats test partition (year=2009,month=1)"
-
- # Checks that profile does not have metrics for incremental stats when
- # the operation is not 'compute incremental stats'.
- self.execute_query_expect_success(client, "use %s" % unique_database)
- profile = self.execute_query_expect_success(client, create).runtime_profile
- assert profile.count("StatsFetch") == 0
- # Checks that incremental stats metrics are present when 'compute incremental
- # stats' is run. Since the table has no stats, expect that no bytes are fetched.
- self.execute_query_expect_success(client, load)
- profile = self.execute_query_expect_success(client, stats_all).runtime_profile
- assert profile.count("StatsFetch") > 1
- assert profile.count("StatsFetch.CompressedBytes: 0") == 1
- # Checks that bytes fetched is non-zero since incremental stats are present now
- # and should have been fetched.
- self.execute_query_expect_success(client, insert)
- profile = self.execute_query_expect_success(client, stats_part).runtime_profile
- assert profile.count("StatsFetch") > 1
- assert profile.count("StatsFetch.CompressedBytes") == 1
- assert profile.count("StatsFetch.CompressedBytes: 0") == 0
- # Adds a partition, computes stats, and checks that the metrics in the profile
- # reflect the operation.
- alter = "alter table test add partition(year=2011, month=1)"
- insert_new_partition = """
- insert into test partition(year=2011, month=1) values
- (29349999, true, 4, 4, 4, 40,4.400000095367432,40.4,
- "10/21/09","4","2009-10-21 03:24:09.600000000")
- """
- self.execute_query_expect_success(client, alter)
- self.execute_query_expect_success(client, insert_new_partition)
- profile = self.execute_query_expect_success(client, stats_all).runtime_profile
- assert profile.count("StatsFetch.TotalPartitions: 25") == 1
- assert profile.count("StatsFetch.NumPartitionsWithStats: 24") == 1
- finally:
- client.close()
http://git-wip-us.apache.org/repos/asf/impala/blob/ad265842/tests/metadata/test_compute_stats.py
----------------------------------------------------------------------
diff --git a/tests/metadata/test_compute_stats.py b/tests/metadata/test_compute_stats.py
index 3a13859..c3d182d 100644
--- a/tests/metadata/test_compute_stats.py
+++ b/tests/metadata/test_compute_stats.py
@@ -18,6 +18,7 @@
import pytest
from subprocess import check_call
+from tests.common.impala_cluster import ImpalaCluster
from tests.common.impala_test_suite import ImpalaTestSuite
from tests.common.skip import SkipIfS3, SkipIfADLS, SkipIfIsilon, SkipIfLocal
from tests.common.test_dimensions import (
@@ -118,6 +119,54 @@ class TestComputeStats(ImpalaTestSuite):
assert(len(show_result.data) == 2)
assert("1\tpval\t8" in show_result.data[0])
+ def test_pull_stats_profile(self, vector, unique_database):
+ """Checks that the frontend profile includes metrics when computing
+ incremental statistics.
+ """
+ try:
+ client = ImpalaCluster().impalads[0].service.create_beeswax_client()
+ create = "create table test like functional.alltypes"
+ load = "insert into test partition(year, month) select * from functional.alltypes"
+ insert = """insert into test partition(year=2009, month=1) values
+ (29349999, true, 4, 4, 4, 40,4.400000095367432,40.4,
+ "10/21/09","4","2009-10-21 03:24:09.600000000")"""
+ stats_all = "compute incremental stats test"
+ stats_part = "compute incremental stats test partition (year=2009,month=1)"
+
+ # Checks that profile does not have metrics for incremental stats when
+ # the operation is not 'compute incremental stats'.
+ self.execute_query_expect_success(client, "use %s" % unique_database)
+ profile = self.execute_query_expect_success(client, create).runtime_profile
+ assert profile.count("StatsFetch") == 0
+ # Checks that incremental stats metrics are present when 'compute incremental
+ # stats' is run. Since the table has no stats, expect that no bytes are fetched.
+ self.execute_query_expect_success(client, load)
+ profile = self.execute_query_expect_success(client, stats_all).runtime_profile
+ assert profile.count("StatsFetch") > 1
+ assert profile.count("StatsFetch.CompressedBytes: 0") == 1
+ # Checks that bytes fetched is non-zero since incremental stats are present now
+ # and should have been fetched.
+ self.execute_query_expect_success(client, insert)
+ profile = self.execute_query_expect_success(client, stats_part).runtime_profile
+ assert profile.count("StatsFetch") > 1
+ assert profile.count("StatsFetch.CompressedBytes") == 1
+ assert profile.count("StatsFetch.CompressedBytes: 0") == 0
+ # Adds a partition, computes stats, and checks that the metrics in the profile
+ # reflect the operation.
+ alter = "alter table test add partition(year=2011, month=1)"
+ insert_new_partition = """
+ insert into test partition(year=2011, month=1) values
+ (29349999, true, 4, 4, 4, 40,4.400000095367432,40.4,
+ "10/21/09","4","2009-10-21 03:24:09.600000000")
+ """
+ self.execute_query_expect_success(client, alter)
+ self.execute_query_expect_success(client, insert_new_partition)
+ profile = self.execute_query_expect_success(client, stats_all).runtime_profile
+ assert profile.count("StatsFetch.TotalPartitions: 25") == 1
+ assert profile.count("StatsFetch.NumPartitionsWithStats: 24") == 1
+ finally:
+ client.close()
+
# Tests compute stats on HBase tables. This test is separate from TestComputeStats,
# because we want to use the existing machanism to disable running tests on hbase/none
# based on the filesystem type (S3, Isilon, etc.).
[08/11] impala git commit: [DOCS] Additional note in
impala_alter_database
Posted by tm...@apache.org.
[DOCS] Additional note in impala_alter_database
- Added a note about the effective ALL with GRANT and ALL without
GRANT privileges.
Change-Id: Idf2cb58993adadb7fb32ec188f5e4dff216c5897
Reviewed-on: http://gerrit.cloudera.org:8080/11703
Tested-by: Impala Public Jenkins <im...@cloudera.com>
Reviewed-by: Fredy Wijaya <fw...@cloudera.com>
Project: http://git-wip-us.apache.org/repos/asf/impala/repo
Commit: http://git-wip-us.apache.org/repos/asf/impala/commit/28aecd6d
Tree: http://git-wip-us.apache.org/repos/asf/impala/tree/28aecd6d
Diff: http://git-wip-us.apache.org/repos/asf/impala/diff/28aecd6d
Branch: refs/heads/master
Commit: 28aecd6db8e9c330c73ee906eac88ec5f2bb2056
Parents: 96debe0
Author: Alex Rodoni <ar...@cloudera.com>
Authored: Tue Oct 16 15:05:44 2018 -0700
Committer: Alex Rodoni <ar...@cloudera.com>
Committed: Wed Oct 17 01:41:13 2018 +0000
----------------------------------------------------------------------
docs/topics/impala_alter_database.xml | 12 +++++++-----
1 file changed, 7 insertions(+), 5 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/impala/blob/28aecd6d/docs/topics/impala_alter_database.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_alter_database.xml b/docs/topics/impala_alter_database.xml
index a768346..6aa742c 100644
--- a/docs/topics/impala_alter_database.xml
+++ b/docs/topics/impala_alter_database.xml
@@ -52,11 +52,13 @@ under the License.
</p>
<p>
- The database owner is originally set to the user who creates the database. An owner of a
- database has the <codeph>ALL</codeph> privilege, but the term <codeph>OWNER</codeph> is
- used to differentiate between the <codeph>ALL</codeph> privilege that is explicitly
- granted via the <codeph>GRANT</codeph> statement and a privilege that is implicitly
- granted by the <codeph>CREATE DATABASE</codeph> statement.
+ The database owner is originally set to the user who creates the database. When object
+ ownership is enabled in Sentry, an owner of a database can have the <codeph>ALL</codeph>
+ with <codeph>GRANT</codeph> or <codeph>ALL</codeph> without <codeph>GRANT</codeph>
+ privilege. The term <codeph>OWNER</codeph> is used to differentiate between the
+ <codeph>ALL</codeph> privilege that is explicitly granted via the <codeph>GRANT</codeph>
+ statement and a privilege that is implicitly granted by the <codeph>CREATE
+ DATABASE</codeph> statement.
</p>
<p conref="../shared/impala_common.xml#common/syntax_blurb"/>