You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@impala.apache.org by mi...@apache.org on 2018/08/04 22:00:18 UTC
[1/3] impala git commit: IMPALA-7395: [DOCS] The TABLE keyword is
optional in the TRUNCATE statement
Repository: impala
Updated Branches:
refs/heads/master 2b4d06710 -> 34fd9732e
IMPALA-7395: [DOCS] The TABLE keyword is optional in the TRUNCATE statement
Change-Id: Iec64cd225449cea07b353201a84ac2fb8bc07a00
Reviewed-on: http://gerrit.cloudera.org:8080/11120
Tested-by: Impala Public Jenkins <im...@cloudera.com>
Reviewed-by: Tim Armstrong <ta...@cloudera.com>
Project: http://git-wip-us.apache.org/repos/asf/impala/repo
Commit: http://git-wip-us.apache.org/repos/asf/impala/commit/2aa9ca25
Tree: http://git-wip-us.apache.org/repos/asf/impala/tree/2aa9ca25
Diff: http://git-wip-us.apache.org/repos/asf/impala/diff/2aa9ca25
Branch: refs/heads/master
Commit: 2aa9ca25f4e33537cd1da3d373804866d5335262
Parents: 2b4d067
Author: Alex Rodoni <ar...@cloudera.com>
Authored: Fri Aug 3 16:13:19 2018 -0700
Committer: Alex Rodoni <ar...@cloudera.com>
Committed: Fri Aug 3 23:35:07 2018 +0000
----------------------------------------------------------------------
docs/topics/impala_truncate_table.xml | 133 +++++++++++------------------
1 file changed, 51 insertions(+), 82 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/impala/blob/2aa9ca25/docs/topics/impala_truncate_table.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_truncate_table.xml b/docs/topics/impala_truncate_table.xml
index cf0e0a4..e40802c 100644
--- a/docs/topics/impala_truncate_table.xml
+++ b/docs/topics/impala_truncate_table.xml
@@ -35,85 +35,58 @@ under the License.
</prolog>
<conbody>
-
<p rev="2.3.0">
- <indexterm audience="hidden">TRUNCATE TABLE statement</indexterm>
- Removes the data from an Impala table while leaving the table itself.
- </p>
-
+ <indexterm audience="hidden">TRUNCATE TABLE statement</indexterm> Removes
+ the data from an Impala table while leaving the table itself. </p>
<p conref="../shared/impala_common.xml#common/syntax_blurb"/>
-
-<codeblock>TRUNCATE TABLE <ph rev="2.5.0 IMPALA-2641">[IF EXISTS]</ph> [<varname>db_name</varname>.]<varname>table_name</varname></codeblock>
-
+ <codeblock>TRUNCATE [TABLE] <ph rev="2.5.0 IMPALA-2641">[IF EXISTS]</ph> [<varname>db_name</varname>.]<varname>table_name</varname></codeblock>
<p conref="../shared/impala_common.xml#common/ddl_blurb"/>
-
<p conref="../shared/impala_common.xml#common/usage_notes_blurb"/>
-
- <p>
- Often used to empty tables that are used during ETL cycles, after the data has been copied to another
- table for the next stage of processing. This statement is a low-overhead alternative to dropping and
- recreating the table, or using <codeph>INSERT OVERWRITE</codeph> to replace the data during the
- next ETL cycle.
- </p>
-
- <p>
- This statement removes all the data and associated data files in the table. It can remove data files from internal tables,
- external tables, partitioned tables, and tables mapped to HBase or the Amazon Simple Storage Service (S3).
- The data removal applies to the entire table, including all partitions of a partitioned table.
- </p>
-
- <p>
- Any statistics produced by the <codeph>COMPUTE STATS</codeph> statement are reset when the data is removed.
- </p>
-
- <p>
- Make sure that you are in the correct database before truncating a table, either by issuing a
- <codeph>USE</codeph> statement first or by using a fully qualified name
- <codeph><varname>db_name</varname>.<varname>table_name</varname></codeph>.
- </p>
-
- <p rev="2.5.0 IMPALA-2641">
- The optional <codeph>IF EXISTS</codeph> clause makes the statement succeed whether or not the table exists.
- If the table does exist, it is truncated; if it does not exist, the statement has no effect. This capability is
- useful in standardized setup scripts that are might be run both before and after some of the tables exist.
- This clause is available in <keyword keyref="impala25_full"/> and higher.
- </p>
-
- <p>
- Any HDFS data files removed by this statement go into the HDFS trashcan, from which you can recover them
- within a defined time interval if this operation turns out to be a mistake.
- </p>
-
+ <p> Often used to empty tables that are used during ETL cycles, after the
+ data has been copied to another table for the next stage of processing.
+ This statement is a low-overhead alternative to dropping and recreating
+ the table, or using <codeph>INSERT OVERWRITE</codeph> to replace the data
+ during the next ETL cycle. </p>
+ <p> This statement removes all the data and associated data files in the
+ table. It can remove data files from internal tables, external tables,
+ partitioned tables, and tables mapped to HBase or the Amazon Simple
+ Storage Service (S3). The data removal applies to the entire table,
+ including all partitions of a partitioned table. </p>
+ <p> Any statistics produced by the <codeph>COMPUTE STATS</codeph> statement
+ are reset when the data is removed. </p>
+ <p> Make sure that you are in the correct database before truncating a
+ table, either by issuing a <codeph>USE</codeph> statement first or by
+ using a fully qualified name
+ <codeph><varname>db_name</varname>.<varname>table_name</varname></codeph>. </p>
+ <p>The optional <codeph>TABLE</codeph> keyword does not affect the behavior
+ of the statement.</p>
+ <p rev="2.5.0 IMPALA-2641"> The optional <codeph>IF EXISTS</codeph> clause
+ makes the statement succeed whether or not the table exists. If the table
+ does exist, it is truncated; if it does not exist, the statement has no
+ effect. This capability is useful in standardized setup scripts that are
+ might be run both before and after some of the tables exist. This clause
+ is available in <keyword keyref="impala25_full"/> and higher. </p>
+ <p> Any HDFS data files removed by this statement go into the HDFS trashcan,
+ from which you can recover them within a defined time interval if this
+ operation turns out to be a mistake. </p>
<p conref="../shared/impala_common.xml#common/disk_space_blurb"/>
-
<p conref="../shared/impala_common.xml#common/s3_blurb"/>
- <p rev="2.2.0">
- Although Impala cannot write new data to a table stored in the Amazon
- S3 filesystem, the <codeph>TRUNCATE TABLE</codeph> statement can remove data files from S3.
- See <xref href="impala_s3.xml#s3"/> for details about working with S3 tables.
- </p>
-
+ <p rev="2.2.0"> Although Impala cannot write new data to a table stored in
+ the Amazon S3 filesystem, the <codeph>TRUNCATE TABLE</codeph> statement
+ can remove data files from S3. See <xref href="impala_s3.xml#s3"/> for
+ details about working with S3 tables. </p>
<p conref="../shared/impala_common.xml#common/cancel_blurb_no"/>
-
<p conref="../shared/impala_common.xml#common/permissions_blurb"/>
- <p rev="">
- The user ID that the <cmdname>impalad</cmdname> daemon runs under,
- typically the <codeph>impala</codeph> user, must have write
- permission for all the files and directories that make up the table.
- </p>
-
+ <p rev=""> The user ID that the <cmdname>impalad</cmdname> daemon runs
+ under, typically the <codeph>impala</codeph> user, must have write
+ permission for all the files and directories that make up the table. </p>
<p conref="../shared/impala_common.xml#common/kudu_blurb"/>
<p conref="../shared/impala_common.xml#common/kudu_no_truncate_table"/>
-
<p conref="../shared/impala_common.xml#common/example_blurb"/>
-
- <p>
- The following example shows a table containing some data and with table and column statistics.
- After the <codeph>TRUNCATE TABLE</codeph> statement, the data is removed and the statistics
- are reset.
- </p>
-
-<codeblock>CREATE TABLE truncate_demo (x INT);
+ <p> The following example shows a table containing some data and with table
+ and column statistics. After the <codeph>TRUNCATE TABLE</codeph>
+ statement, the data is removed and the statistics are reset. </p>
+ <codeblock>CREATE TABLE truncate_demo (x INT);
INSERT INTO truncate_demo VALUES (1), (2), (4), (8);
SELECT COUNT(*) FROM truncate_demo;
+----------+
@@ -162,13 +135,10 @@ SHOW COLUMN STATS truncate_demo;
| x | INT | -1 | -1 | 4 | 4 |
+--------+------+------------------+--------+----------+----------+
</codeblock>
-
- <p>
- The following example shows how the <codeph>IF EXISTS</codeph> clause allows the <codeph>TRUNCATE TABLE</codeph>
- statement to be run without error whether or not the table exists:
- </p>
-
-<codeblock rev="2.5.0 IMPALA-2641">CREATE TABLE staging_table1 (x INT, s STRING);
+ <p> The following example shows how the <codeph>IF EXISTS</codeph> clause
+ allows the <codeph>TRUNCATE TABLE</codeph> statement to be run without
+ error whether or not the table exists: </p>
+ <codeblock rev="2.5.0 IMPALA-2641">CREATE TABLE staging_table1 (x INT, s STRING);
Fetched 0 row(s) in 0.33s
SHOW TABLES LIKE 'staging*';
@@ -191,15 +161,14 @@ Fetched 0 row(s) in 0.25s
TRUNCATE TABLE IF EXISTS staging_table3;
Fetched 0 row(s) in 0.25s
</codeblock>
-
<p conref="../shared/impala_common.xml#common/related_info"/>
-
<p>
- <xref href="impala_tables.xml#tables"/>,
- <xref href="impala_alter_table.xml#alter_table"/>, <xref href="impala_create_table.xml#create_table"/>,
- <xref href="impala_partitioning.xml#partitioning"/>, <xref href="impala_tables.xml#internal_tables"/>,
- <xref href="impala_tables.xml#external_tables"/>
+ <xref href="impala_tables.xml#tables"/>, <xref
+ href="impala_alter_table.xml#alter_table"/>, <xref
+ href="impala_create_table.xml#create_table"/>, <xref
+ href="impala_partitioning.xml#partitioning"/>, <xref
+ href="impala_tables.xml#internal_tables"/>, <xref
+ href="impala_tables.xml#external_tables"/>
</p>
-
</conbody>
</concept>
[3/3] impala git commit: IMPALA-7394: Don't print stack trace in
ExpireSessions()
Posted by mi...@apache.org.
IMPALA-7394: Don't print stack trace in ExpireSessions()
When sessions expire, ExpireSessions() may create excessive amount
of log spew due to printing of the stack traces esp when there are
a massive amount of sessions expiring at the same time. This stack
crawling also unnecessarily extends the critical section under the
session_state_map_lock_, increasing the contention for that lock.
This change converts the use of Status() to Status::Expected() to
remove the unnecessary stack crawl.
Change-Id: I29cb143af96e2eef45479a03365d8f49a2ee2dfa
Reviewed-on: http://gerrit.cloudera.org:8080/11117
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
Project: http://git-wip-us.apache.org/repos/asf/impala/repo
Commit: http://git-wip-us.apache.org/repos/asf/impala/commit/34fd9732
Tree: http://git-wip-us.apache.org/repos/asf/impala/tree/34fd9732
Diff: http://git-wip-us.apache.org/repos/asf/impala/diff/34fd9732
Branch: refs/heads/master
Commit: 34fd9732e6fa66c7eec41317bbec3486063559eb
Parents: 504e995
Author: Michael Ho <kw...@cloudera.com>
Authored: Fri Aug 3 14:42:25 2018 -0700
Committer: Impala Public Jenkins <im...@cloudera.com>
Committed: Sat Aug 4 21:23:18 2018 +0000
----------------------------------------------------------------------
be/src/service/impala-server.cc | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/impala/blob/34fd9732/be/src/service/impala-server.cc
----------------------------------------------------------------------
diff --git a/be/src/service/impala-server.cc b/be/src/service/impala-server.cc
index 89b7e41..34293f6 100644
--- a/be/src/service/impala-server.cc
+++ b/be/src/service/impala-server.cc
@@ -1942,7 +1942,7 @@ void ImpalaServer::UnregisterSessionTimeout(int32_t session_timeout) {
session_state.second->inflight_queries.end());
}
// Unregister all open queries from this session.
- Status status("Session expired due to inactivity");
+ Status status = Status::Expected("Session expired due to inactivity");
for (const TUniqueId& query_id: inflight_queries) {
cancellation_thread_pool_->Offer(CancellationWork(query_id, status, true));
}
[2/3] impala git commit: IMPALA-7325: Incorrect SHOW CREATE VIEW with
built-in functions
Posted by mi...@apache.org.
IMPALA-7325: Incorrect SHOW CREATE VIEW with built-in functions
In the prior code, the authorization checker for the masked privilege
requests skips the check for system database access. As a result, certain
commands, such as SHOW CREATE VIEW that references built-in database
requires permission to access to the built-in database where accessing
built-in database should always be allowed. The patch fixes it by using
the authorizePrivilegeRequest() method that does a check on the system
database similar to how other authorization checks are performed.
Testing:
- Added new authorization test
- Ran all FE tests
Change-Id: Ia164c55fd9459cf5f11eb72561e9cd4ffe1d5367
Reviewed-on: http://gerrit.cloudera.org:8080/10995
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
Project: http://git-wip-us.apache.org/repos/asf/impala/repo
Commit: http://git-wip-us.apache.org/repos/asf/impala/commit/504e9955
Tree: http://git-wip-us.apache.org/repos/asf/impala/tree/504e9955
Diff: http://git-wip-us.apache.org/repos/asf/impala/diff/504e9955
Branch: refs/heads/master
Commit: 504e9955a1481425c9fde64f9259d4b16b0f27c5
Parents: 2aa9ca2
Author: Fredy Wijaya <fw...@cloudera.com>
Authored: Thu Jul 19 13:44:27 2018 -0700
Committer: Impala Public Jenkins <im...@cloudera.com>
Committed: Sat Aug 4 17:42:07 2018 +0000
----------------------------------------------------------------------
.../apache/impala/analysis/AnalysisContext.java | 4 ++-
.../impala/analysis/AuthorizationStmtTest.java | 29 ++++++++++++++++++++
.../apache/impala/common/FrontendTestBase.java | 11 +++++++-
3 files changed, 42 insertions(+), 2 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/impala/blob/504e9955/fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java
----------------------------------------------------------------------
diff --git a/fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java b/fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java
index eeda2eb..47fe8a7 100644
--- a/fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java
+++ b/fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java
@@ -564,7 +564,9 @@ public class AnalysisContext {
// These checks don't result in an AuthorizationException but set the
// 'user_has_profile_access' flag in queryCtx_.
for (Pair<PrivilegeRequest, String> maskedReq: analyzer.getMaskedPrivilegeReqs()) {
- if (!authzChecker.hasAccess(analyzer.getUser(), maskedReq.first)) {
+ try {
+ authorizePrivilegeRequest(authzChecker, maskedReq.first);
+ } catch (AuthorizationException e) {
analysisResult_.setUserHasProfileAccess(false);
if (!Strings.isNullOrEmpty(maskedReq.second)) {
throw new AuthorizationException(maskedReq.second);
http://git-wip-us.apache.org/repos/asf/impala/blob/504e9955/fe/src/test/java/org/apache/impala/analysis/AuthorizationStmtTest.java
----------------------------------------------------------------------
diff --git a/fe/src/test/java/org/apache/impala/analysis/AuthorizationStmtTest.java b/fe/src/test/java/org/apache/impala/analysis/AuthorizationStmtTest.java
index 627ec1a..b27227a 100644
--- a/fe/src/test/java/org/apache/impala/analysis/AuthorizationStmtTest.java
+++ b/fe/src/test/java/org/apache/impala/analysis/AuthorizationStmtTest.java
@@ -993,6 +993,30 @@ public class AuthorizationStmtTest extends FrontendTestBase {
// Show create view on non-existent table.
authorize("show create view functional.notbl").error(accessError("functional.notbl"));
+ // IMPALA-7325: show create view that references built-in function(s) should not
+ // require access to the _impala_builtins database this is because reading metadata
+ // in system database should always be allowed.
+ addTestView(authzCatalog_, "create view functional.test_view as " +
+ "select count(*) from functional.alltypes");
+ test = authorize("show create view functional.test_view");
+ for (TPrivilegeLevel privilege: viewMetadataPrivileges()) {
+ test.ok(onServer(privilege, TPrivilegeLevel.SELECT))
+ .ok(onDatabase("functional", privilege, TPrivilegeLevel.SELECT))
+ .ok(onTable("functional", "test_view", privilege),
+ onTable("functional", "alltypes", privilege, TPrivilegeLevel.SELECT));
+ }
+ test.error(accessError("functional.test_view"))
+ .error(accessError("functional.test_view"), onServer(
+ allExcept(viewMetadataPrivileges())))
+ .error(accessError("functional.test_view"), onDatabase("functional",
+ allExcept(viewMetadataPrivileges())))
+ .error(accessError("functional.test_view"), onTable("functional", "test_view",
+ allExcept(viewMetadataPrivileges())), onTable("functional", "alltypes",
+ TPrivilegeLevel.SELECT))
+ .error(viewDefError("functional.test_view"), onTable("functional", "test_view",
+ TPrivilegeLevel.SELECT), onTable("functional", "alltypes", allExcept(
+ viewMetadataPrivileges())));
+
// Show create function.
ScalarFunction fn = addFunction("functional", "f");
try {
@@ -2217,6 +2241,11 @@ public class AuthorizationStmtTest extends FrontendTestBase {
return "Cannot modify system database.";
}
+ private static String viewDefError(String object) {
+ return "User '%s' does not have privileges to see the definition of view '" +
+ object + "'.";
+ }
+
private static String createFunctionError(String object) {
return "User '%s' does not have privileges to CREATE functions in: " + object;
}
http://git-wip-us.apache.org/repos/asf/impala/blob/504e9955/fe/src/test/java/org/apache/impala/common/FrontendTestBase.java
----------------------------------------------------------------------
diff --git a/fe/src/test/java/org/apache/impala/common/FrontendTestBase.java b/fe/src/test/java/org/apache/impala/common/FrontendTestBase.java
index 7b22551..cbb48ce 100644
--- a/fe/src/test/java/org/apache/impala/common/FrontendTestBase.java
+++ b/fe/src/test/java/org/apache/impala/common/FrontendTestBase.java
@@ -211,8 +211,17 @@ public class FrontendTestBase {
* Returns the new view.
*/
protected Table addTestView(String createViewSql) {
+ return addTestView(catalog_, createViewSql);
+ }
+
+ /**
+ * Adds a test-local view to the specified catalog based on the given CREATE VIEW sql.
+ * The test views are registered in testTables_ and removed in the @After method.
+ * Returns the new view.
+ */
+ protected Table addTestView(Catalog catalog, String createViewSql) {
CreateViewStmt createViewStmt = (CreateViewStmt) AnalyzesOk(createViewSql);
- Db db = catalog_.getDb(createViewStmt.getDb());
+ Db db = catalog.getDb(createViewStmt.getDb());
Preconditions.checkNotNull(db, "Test views must be created in an existing db.");
// Do not analyze the stmt to avoid applying rewrites that would alter the view
// definition. We want to model real views as closely as possible.