You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@impala.apache.org by ta...@apache.org on 2018/08/06 20:20:05 UTC
[1/3] impala git commit: IMPALA-4690: [DOCS] More content for CONV()
Repository: impala
Updated Branches:
refs/heads/master 3334c167a -> a0673c058
IMPALA-4690: [DOCS] More content for CONV()
Change-Id: I4e424fd5a009ff5aa2d35a403e08fcd33c75fec5
Reviewed-on: http://gerrit.cloudera.org:8080/11075
Reviewed-by: Jim Apple <jb...@apache.org>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
Project: http://git-wip-us.apache.org/repos/asf/impala/repo
Commit: http://git-wip-us.apache.org/repos/asf/impala/commit/332d59b1
Tree: http://git-wip-us.apache.org/repos/asf/impala/tree/332d59b1
Diff: http://git-wip-us.apache.org/repos/asf/impala/diff/332d59b1
Branch: refs/heads/master
Commit: 332d59b1152014d09bec2974a1152facf0d589fc
Parents: 3334c16
Author: Alex Rodoni <ar...@cloudera.com>
Authored: Fri Jul 27 17:34:55 2018 -0700
Committer: Alex Rodoni <ar...@cloudera.com>
Committed: Mon Aug 6 19:01:20 2018 +0000
----------------------------------------------------------------------
docs/topics/impala_math_functions.xml | 73 ++++++++++++++++++++++++++----
1 file changed, 63 insertions(+), 10 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/impala/blob/332d59b1/docs/topics/impala_math_functions.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_math_functions.xml b/docs/topics/impala_math_functions.xml
index 9ac8d33..430afe4 100644
--- a/docs/topics/impala_math_functions.xml
+++ b/docs/topics/impala_math_functions.xml
@@ -195,19 +195,72 @@ under the License.
<dlentry id="conv">
<dt>
- <codeph>conv(bigint num, int from_base, int to_base), conv(string num, int from_base, int
- to_base)</codeph>
+ <codeph>conv(bigint n, int from_base, int to_base), conv(string s, int
+ from_base, int to_base)</codeph>
</dt>
<dd>
- <indexterm audience="hidden">conv() function</indexterm>
- <b>Purpose:</b> Returns a string representation of an integer value in a particular base. The input value
- can be a string, for example to convert a hexadecimal number such as <codeph>fce2</codeph> to decimal. To
- use the return value as a number (for example, when converting to base 10), use <codeph>CAST()</codeph>
- to convert to the appropriate type.
- <p>
- <b>Return type:</b> <codeph>string</codeph>
- </p>
+ <b>Purpose:</b> Returns a string representation of the first argument
+ converted from <codeph>from_base</codeph> to <codeph>to_base</codeph>.
+ The first argument can be specified as a number or a string. For
+ example, <codeph>conv(100, 2, 10)</codeph> and <codeph>conv('100', 2,
+ 10)</codeph> both return <codeph>'4'</codeph>. <p>
+ <b>Return type:</b>
+ <codeph>string</codeph>
+ </p>
+ <p>
+ <b>Usage notes:</b>
+ </p>
+ <p> If <codeph>to_base</codeph> is negative, the first argument is
+ treated as signed, and otherwise, it is treated as unsigned. For
+ example: </p>
+ <ul>
+ <li>
+ <codeph>conv(-17, 10, -2) </codeph>returns
+ <codeph>'-10001'</codeph>,<codeph> -17</codeph> in base 2. </li>
+ <li>
+ <codeph>conv(-17, 10, 10)</codeph> returns
+ <codeph>'18446744073709551599'</codeph>. <codeph>-17</codeph> is
+ interpreted as an unsigned, 2^64-17, and then the value is
+ returned in base 10.</li>
+ </ul><p>The function returns <codeph>NULL</codeph> when the following
+ illegal arguments are specified: </p>
+ <ul>
+ <li> Any argument is <codeph>NULL</codeph>. </li>
+ <li>
+ <codeph>from_base</codeph> or <codeph>to_base</codeph> is below
+ <codeph>-36</codeph> or above <codeph>36</codeph>. </li>
+ <li>
+ <codeph>from_base</codeph> or <codeph>to_base</codeph> is
+ <codeph>-1</codeph>, <codeph>0</codeph>, or <codeph>1</codeph>. </li>
+ <li> The first argument represents a positive number and
+ <codeph>from_base</codeph> is a negative number.</li>
+ </ul>
+ <p>If the first argument represents a negative number and
+ <codeph>from_base</codeph> is a negative number, the function
+ returns <codeph>0</codeph>.</p><p> If the first argument represents
+ a number larger than the maximum <codeph>bigint</codeph>, the
+ function returns: </p>
+ <ul>
+ <li> The string representation of -1 in <codeph>to_base</codeph> if
+ <codeph>to_base</codeph> is negative. </li>
+ <li> The string representation of 18446744073709551615' (2^64 - 1)
+ in <codeph>to_base</codeph> if <codeph>to_base</codeph> is
+ positive.</li>
+ </ul>
+ <p> If the first argument does not represent a valid number in
+ <codeph>from_base</codeph>, e.g. 3 in base 2 or '1a23' in base 10,
+ the digits in the first argument are evaluated from left-to-right
+ and used if a valid digit in <codeph>from_base</codeph>. The invalid
+ digit and the digits to the right are ignored. </p>
+ <p> For example:<ul>
+ <li><codeph> conv(445, 5, 10)</codeph> is converted to
+ <codeph>conv(44, 5, 10)</codeph> and returns
+ <codeph>'24'</codeph>. </li>
+ <li><codeph> conv('1a23', 10, 16)</codeph> is converted to
+ <codeph>conv('1', 10 , 16)</codeph> and returns
+ <codeph>'1'</codeph>. </li>
+ </ul></p>
</dd>
</dlentry>
[2/3] impala git commit: IMPALA-6789: disable impersonation in hive
in minicluster
Posted by ta...@apache.org.
IMPALA-6789: disable impersonation in hive in minicluster
Due to bug of HADOOP-7050, users with dots in their username can't
launch HiveServer2 in minicluster. To work arround this, we can set
hive.server2.enable.doAs to false to disable impersonation in hive.
Impala's authorization only depends on Sentry, so we can disable this
without breaking any tests.
This patch also quotes the group name in AuthorizationStmtTest#testShow
to avoid syntax errors when group name contains dots.
Test:
* Build succeed with username quanlong.huang locally
Change-Id: I39d8945e0fe90baf7e9e4b26eebab08d2058a14a
Reviewed-on: http://gerrit.cloudera.org:8080/11119
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
Project: http://git-wip-us.apache.org/repos/asf/impala/repo
Commit: http://git-wip-us.apache.org/repos/asf/impala/commit/8bd9f175
Tree: http://git-wip-us.apache.org/repos/asf/impala/tree/8bd9f175
Diff: http://git-wip-us.apache.org/repos/asf/impala/diff/8bd9f175
Branch: refs/heads/master
Commit: 8bd9f175b79f117c425def065dbeaa92098bdecd
Parents: 332d59b
Author: stiga-huang <hu...@gmail.com>
Authored: Wed Aug 1 18:29:55 2018 -0700
Committer: Impala Public Jenkins <im...@cloudera.com>
Committed: Mon Aug 6 19:38:45 2018 +0000
----------------------------------------------------------------------
.../impala/analysis/AuthorizationStmtTest.java | 2 +-
fe/src/test/resources/mysql-hive-site.xml.template | 9 +++++++++
.../test/resources/postgresql-hive-site.xml.template | 15 +++++++++------
3 files changed, 19 insertions(+), 7 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/impala/blob/8bd9f175/fe/src/test/java/org/apache/impala/analysis/AuthorizationStmtTest.java
----------------------------------------------------------------------
diff --git a/fe/src/test/java/org/apache/impala/analysis/AuthorizationStmtTest.java b/fe/src/test/java/org/apache/impala/analysis/AuthorizationStmtTest.java
index b27227a..9a38300 100644
--- a/fe/src/test/java/org/apache/impala/analysis/AuthorizationStmtTest.java
+++ b/fe/src/test/java/org/apache/impala/analysis/AuthorizationStmtTest.java
@@ -960,7 +960,7 @@ public class AuthorizationStmtTest extends FrontendTestBase {
authorize("show roles").ok();
// Show role grant group should always be allowed.
- authorize(String.format("show role grant group %s", USER.getName())).ok();
+ authorize(String.format("show role grant group `%s`", USER.getName())).ok();
// Show grant role should always be allowed.
authorize(String.format("show grant role authz_test_role")).ok();
http://git-wip-us.apache.org/repos/asf/impala/blob/8bd9f175/fe/src/test/resources/mysql-hive-site.xml.template
----------------------------------------------------------------------
diff --git a/fe/src/test/resources/mysql-hive-site.xml.template b/fe/src/test/resources/mysql-hive-site.xml.template
index add10dc..9fc67ff 100644
--- a/fe/src/test/resources/mysql-hive-site.xml.template
+++ b/fe/src/test/resources/mysql-hive-site.xml.template
@@ -135,6 +135,15 @@
</property>
<property>
+ <name>hive.server2.enable.doAs</name>
+ <description>
+ Disable user impersonation for HiveServer2 to avoid launch failure
+ if username contains dots (IMPALA-6789)
+ </description>
+ <value>false</value>
+</property>
+
+<property>
<!-- TODO: Remove this once Hive has changed their default back to ColumnarSerDe -->
<name>hive.default.rcfile.serde</name>
<value>org.apache.hadoop.hive.serde2.columnar.ColumnarSerDe</value>
http://git-wip-us.apache.org/repos/asf/impala/blob/8bd9f175/fe/src/test/resources/postgresql-hive-site.xml.template
----------------------------------------------------------------------
diff --git a/fe/src/test/resources/postgresql-hive-site.xml.template b/fe/src/test/resources/postgresql-hive-site.xml.template
index c89abca..5ff2491 100644
--- a/fe/src/test/resources/postgresql-hive-site.xml.template
+++ b/fe/src/test/resources/postgresql-hive-site.xml.template
@@ -142,12 +142,6 @@
<value>${KRB5_KTNAME}</value>
</property>
-<property>
- <name>hive.server2.enable.impersonation</name>
- <description>Enable user impersonation for HiveServer2</description>
- <value>true</value>
-</property>
-
<!-- Having problems getting the metastore up with Kerberos. Defer for now.
<property>
<name>hive.metastore.sasl.enabled</name>
@@ -197,6 +191,15 @@
<!-- END Kerberos settings -->
<property>
+ <name>hive.server2.enable.doAs</name>
+ <description>
+ Disable user impersonation for HiveServer2 to avoid launch failure
+ if username contains dots (IMPALA-6789)
+ </description>
+ <value>false</value>
+</property>
+
+<property>
<!-- TODO: Remove this once Hive has changed their default back to ColumnarSerDe -->
<name>hive.default.rcfile.serde</name>
<value>org.apache.hadoop.hive.serde2.columnar.ColumnarSerDe</value>
[3/3] impala git commit: IMPALA-7396: unblock testing with
--thread_creation_fault_injection
Posted by ta...@apache.org.
IMPALA-7396: unblock testing with --thread_creation_fault_injection
These are minimal fixes to allow testing new patches with the fault
injection.
ClientRequestState: fixing this seems somewhat involved, so disabled the
fault injection for now to unblock further testing.
HdfsScanNode: thread_state_.DecrementNumActive() was called on the error
path before the count had actually been incremented.
Testing:
Ran some of the end-to-end tests for 20-30 minutes and confirmed that
no Impalads crashed.
start-impala-cluster.py \
--impalad_args=--thread_creation_fault_injection=true
impala-py.test tests/query_test/*.py -n2 --verbose
Change-Id: Idc9da645c449a266e3a54d0e35b18a2823019cb7
Reviewed-on: http://gerrit.cloudera.org:8080/11123
Reviewed-by: Tim Armstrong <ta...@cloudera.com>
Tested-by: Tim Armstrong <ta...@cloudera.com>
Project: http://git-wip-us.apache.org/repos/asf/impala/repo
Commit: http://git-wip-us.apache.org/repos/asf/impala/commit/a0673c05
Tree: http://git-wip-us.apache.org/repos/asf/impala/tree/a0673c05
Diff: http://git-wip-us.apache.org/repos/asf/impala/diff/a0673c05
Branch: refs/heads/master
Commit: a0673c058754a323ef88459532f0c19881f3c143
Parents: 8bd9f17
Author: Tim Armstrong <ta...@cloudera.com>
Authored: Fri Aug 3 16:41:46 2018 -0700
Committer: Tim Armstrong <ta...@cloudera.com>
Committed: Mon Aug 6 20:19:43 2018 +0000
----------------------------------------------------------------------
be/src/exec/hdfs-scan-node.cc | 1 -
be/src/service/client-request-state.cc | 4 +++-
2 files changed, 3 insertions(+), 2 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/impala/blob/a0673c05/be/src/exec/hdfs-scan-node.cc
----------------------------------------------------------------------
diff --git a/be/src/exec/hdfs-scan-node.cc b/be/src/exec/hdfs-scan-node.cc
index 5d4f9b0..d4cef94 100644
--- a/be/src/exec/hdfs-scan-node.cc
+++ b/be/src/exec/hdfs-scan-node.cc
@@ -265,7 +265,6 @@ void HdfsScanNode::ThreadTokenAvailableCb(ThreadResourcePool* pool) {
Thread::Create(FragmentInstanceState::FINST_THREAD_GROUP_NAME, name, fn, &t, true);
if (!status.ok()) {
ReturnReservationFromScannerThread(lock, scanner_thread_reservation);
- thread_state_.DecrementNumActive();
// Release the token and skip running callbacks to find a replacement. Skipping
// serves two purposes. First, it prevents a mutual recursion between this function
// and ReleaseThreadToken()->InvokeCallbacks(). Second, Thread::Create() failed and
http://git-wip-us.apache.org/repos/asf/impala/blob/a0673c05/be/src/service/client-request-state.cc
----------------------------------------------------------------------
diff --git a/be/src/service/client-request-state.cc b/be/src/service/client-request-state.cc
index 398ab59..5839e9c 100644
--- a/be/src/service/client-request-state.cc
+++ b/be/src/service/client-request-state.cc
@@ -651,8 +651,10 @@ Status ClientRequestState::Exec(const TMetadataOpRequest& exec_request) {
}
Status ClientRequestState::WaitAsync() {
+ // TODO: IMPALA-7396: thread creation fault inject is disabled because it is not
+ // handled correctly.
return Thread::Create("query-exec-state", "wait-thread",
- &ClientRequestState::Wait, this, &wait_thread_, true);
+ &ClientRequestState::Wait, this, &wait_thread_, false);
}
void ClientRequestState::BlockOnWait() {