You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@impala.apache.org by ta...@apache.org on 2019/09/17 17:53:44 UTC
[impala] branch master updated (34d132c -> 05e1a4f)
This is an automated email from the ASF dual-hosted git repository.
tarmstrong pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/impala.git.
from 34d132c IMPALA-8825: Add additional counters to PlanRootSink
new e74c294 IMPALA-8930: [DOCS] Object ownership support when integrated with Ranger
new e38d57f IMPALA-8945: [DOCS] Fixed an incorrect example of DISTINT FROM and its equivalent
new 451d31f IMPALA-8947: scratch alloc error uses wrong metric
new 05e1a4f IMPALA-8903: [DOCS] TRUNCATE is supported for Insert-only transactional tables
The 4 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
be/src/runtime/tmp-file-mgr-test.cc | 10 ++++++++
be/src/runtime/tmp-file-mgr.cc | 3 ++-
be/src/util/metrics.h | 2 ++
docs/shared/impala_common.xml | 17 +++++++-------
docs/topics/impala_authorization.xml | 45 +++++++++++++++++++++++++++++++++++-
docs/topics/impala_grant.xml | 2 +-
docs/topics/impala_operators.xml | 38 +++++++++++++++---------------
docs/topics/impala_transactions.xml | 4 ++--
8 files changed, 89 insertions(+), 32 deletions(-)
[impala] 02/04: IMPALA-8945: [DOCS] Fixed an incorrect example of
DISTINT FROM and its equivalent
Posted by ta...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
tarmstrong pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/impala.git
commit e38d57fe5d0266d0423a04b7f0b7350a3fd300f2
Author: Alex Rodoni <ar...@cloudera.com>
AuthorDate: Mon Sep 16 13:30:24 2019 -0700
IMPALA-8945: [DOCS] Fixed an incorrect example of DISTINT FROM and its equivalent
Change-Id: I9bee4c0935ee21d70a0964507c477a2fccb1c7cc
Reviewed-on: http://gerrit.cloudera.org:8080/14239
Tested-by: Impala Public Jenkins <im...@cloudera.com>
Reviewed-by: Tim Armstrong <ta...@cloudera.com>
---
docs/topics/impala_operators.xml | 38 +++++++++++++++++++-------------------
1 file changed, 19 insertions(+), 19 deletions(-)
diff --git a/docs/topics/impala_operators.xml b/docs/topics/impala_operators.xml
index 731022f..0666457 100644
--- a/docs/topics/impala_operators.xml
+++ b/docs/topics/impala_operators.xml
@@ -1045,19 +1045,20 @@ SELECT COUNT(DISTINCT(visitor_id)) FROM web_traffic WHERE month IN ('January','J
<conbody>
- <p>
- <indexterm audience="hidden">IS DISTINCT FROM operator</indexterm>
-
- <indexterm audience="hidden">IS NOT DISTINCT FROM operator</indexterm>
- The <codeph>IS DISTINCT FROM</codeph> operator, and its converse the <codeph>IS NOT DISTINCT FROM</codeph> operator, test whether or
- not values are identical. <codeph>IS NOT DISTINCT FROM</codeph> is similar to the <codeph>=</codeph> operator, and <codeph>IS
- DISTINCT FROM</codeph> is similar to the <codeph>!=</codeph> operator, except that <codeph>NULL</codeph> values are treated as
- identical. Therefore, <codeph>IS NOT DISTINCT FROM</codeph> returns <codeph>true</codeph> rather than <codeph>NULL</codeph>, and
- <codeph>IS DISTINCT FROM</codeph> returns <codeph>false</codeph> rather than <codeph>NULL</codeph>, when comparing two
- <codeph>NULL</codeph> values. If one of the values being compared is <codeph>NULL</codeph> and the other is not, <codeph>IS DISTINCT
- FROM</codeph> returns <codeph>true</codeph> and <codeph>IS NOT DISTINCT FROM</codeph> returns <codeph>false</codeph>, again instead
- of returning <codeph>NULL</codeph> in both cases.
- </p>
+ <p> The <codeph>IS DISTINCT FROM</codeph> operator, and its converse the
+ <codeph>IS NOT DISTINCT FROM</codeph> operator, test whether or not
+ values are identical. <codeph>IS NOT DISTINCT FROM</codeph> is similar
+ to the <codeph>=</codeph> operator, and <codeph>IS DISTINCT
+ FROM</codeph> is similar to the <codeph>!=</codeph> operator, except
+ that <codeph>NULL</codeph> values are treated as identical. Therefore,
+ <codeph>IS NOT DISTINCT FROM</codeph> returns <codeph>true</codeph>
+ rather than <codeph>NULL</codeph>, and <codeph>IS DISTINCT FROM</codeph>
+ returns <codeph>false</codeph> rather than <codeph>NULL</codeph>, when
+ comparing two <codeph>NULL</codeph> values. If one of the values being
+ compared is <codeph>NULL</codeph> and the other is not, <codeph>IS
+ DISTINCT FROM</codeph> returns <codeph>true</codeph> and <codeph>IS
+ NOT DISTINCT FROM</codeph> returns <codeph>false</codeph>, again
+ instead of returning <codeph>NULL</codeph> in both cases. </p>
<p conref="../shared/impala_common.xml#common/syntax_blurb"/>
@@ -1083,12 +1084,11 @@ SELECT COUNT(DISTINCT(visitor_id)) FROM web_traffic WHERE month IN ('January','J
<codeph>=</codeph> or <codeph>!=</codeph> operator.
</p>
- <p>
- The <codeph><=></codeph> operator, used like an equality operator in a join query,
- is more efficient than the equivalent clause: <codeph>A = B OR (A IS NULL AND B IS NULL)</codeph>.
- The <codeph><=></codeph> operator can use a hash join, while the <codeph>OR</codeph> expression
- cannot.
- </p>
+ <p> The <codeph><=></codeph> operator, used like an equality
+ operator in a join query, is more efficient than the equivalent clause:
+ <codeph>IF (A IS NULL OR B IS NULL, A IS NULL AND B IS NULL, A =
+ B)</codeph>. The <codeph><=></codeph> operator can use a hash
+ join, while the <codeph>IF</codeph> expression cannot. </p>
<p conref="../shared/impala_common.xml#common/example_blurb"/>
[impala] 03/04: IMPALA-8947: scratch alloc error uses wrong metric
Posted by ta...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
tarmstrong pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/impala.git
commit 451d31f2b45d0bfdfab8d8110cc63b06a061849a
Author: Tim Armstrong <ta...@cloudera.com>
AuthorDate: Mon Sep 16 12:42:39 2019 -0700
IMPALA-8947: scratch alloc error uses wrong metric
Use the global metric instead of the per-FileGroup
counter.
Testing:
Updated unit test to validate the error message in
a case where there are two FileGroups.
Change-Id: I2732dcd49c277d5d278fad68efa6ef381bc0eb81
Reviewed-on: http://gerrit.cloudera.org:8080/14236
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
be/src/runtime/tmp-file-mgr-test.cc | 10 ++++++++++
be/src/runtime/tmp-file-mgr.cc | 3 ++-
be/src/util/metrics.h | 2 ++
3 files changed, 14 insertions(+), 1 deletion(-)
diff --git a/be/src/runtime/tmp-file-mgr-test.cc b/be/src/runtime/tmp-file-mgr-test.cc
index 9b852a6..075ee75 100644
--- a/be/src/runtime/tmp-file-mgr-test.cc
+++ b/be/src/runtime/tmp-file-mgr-test.cc
@@ -787,8 +787,18 @@ TEST_F(TmpFileMgrTest, TestDirectoryLimitsExhausted) {
// The directories are at capacity, so allocations should fail.
Status err1 = GroupAllocateSpace(&file_group_1, ALLOC_SIZE, &alloc_file, &offset);
ASSERT_EQ(err1.code(), TErrorCode::SCRATCH_ALLOCATION_FAILED);
+ EXPECT_STR_CONTAINS(err1.GetDetail(), "Could not create files in any configured "
+ "scratch directories (--scratch_dirs=/tmp/tmp-file-mgr-test.1/impala-scratch,"
+ "/tmp/tmp-file-mgr-test.2/impala-scratch)");
+ EXPECT_STR_CONTAINS(err1.GetDetail(), "1.25 MB of scratch is currently in use by "
+ "this Impala Daemon (1.25 MB by this query)");
Status err2 = GroupAllocateSpace(&file_group_2, ALLOC_SIZE, &alloc_file, &offset);
ASSERT_EQ(err2.code(), TErrorCode::SCRATCH_ALLOCATION_FAILED);
+ EXPECT_STR_CONTAINS(err2.GetDetail(), "Could not create files in any configured "
+ "scratch directories (--scratch_dirs=/tmp/tmp-file-mgr-test.1/impala-scratch,"
+ "/tmp/tmp-file-mgr-test.2/impala-scratch)");
+ EXPECT_STR_CONTAINS(err2.GetDetail(), "1.25 MB of scratch is currently in use by "
+ "this Impala Daemon (0 by this query)");
// A FileGroup should recover once allocations are released, i.e. it does not
// permanently block allocating files from the group.
diff --git a/be/src/runtime/tmp-file-mgr.cc b/be/src/runtime/tmp-file-mgr.cc
index a27ae2e..b803a55 100644
--- a/be/src/runtime/tmp-file-mgr.cc
+++ b/be/src/runtime/tmp-file-mgr.cc
@@ -566,7 +566,8 @@ Status TmpFileMgr::FileGroup::ScratchAllocationFailedStatus(
}
Status status(TErrorCode::SCRATCH_ALLOCATION_FAILED, join(tmp_dir_paths, ","),
GetBackendString(),
- PrettyPrinter::PrintBytes(scratch_space_bytes_used_counter_->value()),
+ PrettyPrinter::PrintBytes(
+ tmp_file_mgr_->scratch_bytes_used_metric_->current_value()->GetValue()),
PrettyPrinter::PrintBytes(current_bytes_allocated_),
join(at_capacity_dir_paths, ","));
// Include all previous errors that may have caused the failure.
diff --git a/be/src/util/metrics.h b/be/src/util/metrics.h
index 68f213f..c2c895e 100644
--- a/be/src/util/metrics.h
+++ b/be/src/util/metrics.h
@@ -274,6 +274,8 @@ class AtomicHighWaterMarkGauge : public ScalarMetric<int64_t, TMetricKind::GAUGE
UpdateMax(new_val);
}
+ IntGauge* current_value() const { return current_value_; }
+
private:
FRIEND_TEST(MetricsTest, AtomicHighWaterMarkGauge);
friend class TmpFileMgrTest;
[impala] 01/04: IMPALA-8930: [DOCS] Object ownership support when
integrated with Ranger
Posted by ta...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
tarmstrong pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/impala.git
commit e74c294b133dc8fcf523b50ffa9cbb7986ca8790
Author: Alex Rodoni <ar...@cloudera.com>
AuthorDate: Fri Sep 13 19:27:45 2019 -0700
IMPALA-8930: [DOCS] Object ownership support when integrated with Ranger
Change-Id: Ie4fdaf05953373c8d1870b7eface257830c7c6e5
Reviewed-on: http://gerrit.cloudera.org:8080/14229
Reviewed-by: Bharath Vissapragada <bh...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
docs/shared/impala_common.xml | 17 +++++++-------
docs/topics/impala_authorization.xml | 45 +++++++++++++++++++++++++++++++++++-
docs/topics/impala_grant.xml | 2 +-
3 files changed, 54 insertions(+), 10 deletions(-)
diff --git a/docs/shared/impala_common.xml b/docs/shared/impala_common.xml
index 6d96090..70dc7f4 100644
--- a/docs/shared/impala_common.xml
+++ b/docs/shared/impala_common.xml
@@ -110,21 +110,22 @@ under the License.
</section>
- <section id="sentry">
+ <section id="authz">
- <title>Sentry-Related Content</title>
+ <title>Authorization Content</title>
- <p>
- Material related to Sentry security, intended to be reused between Hive and Impala.
- Complicated by the fact that most of it will probably be multi-paragraph or involve
- subheads, might need to be represented as nested topics at the end of this file.
- </p>
+ <p> Material related to Sentry and Ranger security, intended to be reused
+ between Hive and Impala. Complicated by the fact that most of it will
+ probably be multi-paragraph or involve subheads, might need to be
+ represented as nested topics at the end of this file. </p>
- <p id="sentry_privileges_objects">
+ <p id="privileges_objects">
The table below lists the minimum level of privileges and the scope required to execute
SQL statements in <keyword keyref="impala30_full"/> and higher. The following notations
are used:
<ul>
+ <li>The <b>SERVER</b> resource type in Ranger implies all databases,
+ all tables, all columns, all UDFs, and all URIs.</li>
<li>
<b>ANY</b> denotes the <codeph>SELECT</codeph>, <codeph>INSERT</codeph>,
<codeph>CREATE</codeph>, <codeph>ALTER</codeph>, <codeph>DROP</codeph>,
diff --git a/docs/topics/impala_authorization.xml b/docs/topics/impala_authorization.xml
index f4be80b..9c45970 100644
--- a/docs/topics/impala_authorization.xml
+++ b/docs/topics/impala_authorization.xml
@@ -88,7 +88,7 @@ under the License.
individual columns.
</p>
- <p conref="../shared/impala_common.xml#common/sentry_privileges_objects"/>
+ <p conref="../shared/impala_common.xml#common/privileges_objects"/>
<p>
Privileges are managed via the <codeph>GRANT</codeph> and <codeph>REVOKE</codeph> SQL
@@ -145,6 +145,49 @@ under the License.
</concept>
+ <concept id="concept_fgf_smj_bjb">
+
+ <title>Object Ownership in Ranger</title>
+
+ <conbody>
+
+ <p>
+ Object ownership for tables, views and databases is enabled by default in Impala.
+ </p>
+
+ <p>
+ To define owner specific privileges, go to ranger UI and define appropriate policies on
+ the <codeph>{OWNER}</codeph> user.
+ </p>
+
+ <p>
+ The <codeph>CREATE</codeph> statements implicitly make the user running the statement
+ the owner of the object. For example, if <varname>User A</varname> creates a database,
+ <varname>foo</varname>, via the <codeph>CREATE DATABASE</codeph> statement,
+ <varname>User A</varname> now owns the <varname>foo</varname> database and is authorized
+ to perform any operation on the <varname>foo</varname> database.
+ </p>
+
+ <p>
+ An ownership can be transferred to another user or role via the <codeph>ALTER
+ DATABASE</codeph>, <codeph>ALTER TABLE</codeph>, or <codeph>ALTER VIEW</codeph> with the
+ <codeph>SET OWNER</codeph> clause.
+ </p>
+
+ <note id="impala-8937">
+ Currently, due to a known issue
+ (<xref
+ href="https://issues.apache.org/jira/browse/IMPALA-8937" format="html"
+ scope="external">IMPALA-8937</xref>),
+ until the ownership information is fully loaded in the coordinator catalog cache, the
+ owner of a table might not be able to see the table when executing the <codeph>SHOW
+ TABLES</codeph> statement The owner can still query the table.
+ </note>
+
+ </conbody>
+
+ </concept>
+
<concept id="secure_startup">
<title>Starting Impala with Sentry Authorization Enabled</title>
diff --git a/docs/topics/impala_grant.xml b/docs/topics/impala_grant.xml
index 669e112..1b05c6a 100644
--- a/docs/topics/impala_grant.xml
+++ b/docs/topics/impala_grant.xml
@@ -120,7 +120,7 @@ GRANT <varname>privilege</varname> ON <varname>object_type</varname> <varname>ob
Finer-grained privileges mentioned below on a <codeph>URI</codeph> are not supported.
</p>
- <p conref="../shared/impala_common.xml#common/sentry_privileges_objects"/>
+ <p conref="../shared/impala_common.xml#common/privileges_objects"/>
<p conref="../shared/impala_common.xml#common/compatibility_blurb"/>
[impala] 04/04: IMPALA-8903: [DOCS] TRUNCATE is supported for
Insert-only transactional tables
Posted by ta...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
tarmstrong pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/impala.git
commit 05e1a4f2185c892226497b4592fe0eac3e2747d2
Author: Alex Rodoni <ar...@cloudera.com>
AuthorDate: Mon Sep 16 12:46:42 2019 -0700
IMPALA-8903: [DOCS] TRUNCATE is supported for Insert-only transactional tables
Change-Id: Ib457775fab03d7f30430fd7dea6404dfcf0783a8
Reviewed-on: http://gerrit.cloudera.org:8080/14235
Tested-by: Impala Public Jenkins <im...@cloudera.com>
Reviewed-by: Zoltan Borok-Nagy <bo...@cloudera.com>
---
docs/topics/impala_transactions.xml | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/docs/topics/impala_transactions.xml b/docs/topics/impala_transactions.xml
index 8753ca6..45f4492 100644
--- a/docs/topics/impala_transactions.xml
+++ b/docs/topics/impala_transactions.xml
@@ -50,8 +50,8 @@ under the License.
<p>
For insert-only transactional tables, you can perform the following statements:
- <codeph>CREATE TABLE</codeph>, <codeph>DROP TABLE</codeph>, <codeph>INSERT</codeph>,
- <codeph>SELECT</codeph>
+ <codeph>CREATE TABLE</codeph>, <codeph>DROP TABLE</codeph>, <codeph>TRUNCATE</codeph>,
+ <codeph>INSERT</codeph>, <codeph>SELECT</codeph>
</p>
<p>