You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@impala.apache.org by ta...@apache.org on 2018/04/03 17:20:41 UTC
[1/5] impala git commit: [DOCS] Updates to the load balancing
algorithms section
Repository: impala
Updated Branches:
refs/heads/2.x 0197b17fa -> 5166270bf
[DOCS] Updates to the load balancing algorithms section
Further refined the Load Balancing Algorithm section
with reviews and comments from SME.
Change-Id: Ia697aafc799b2a3414a208aa85e1de4bf0214317
Reviewed-on: http://gerrit.cloudera.org:8080/9869
Reviewed-by: Tim Armstrong <ta...@cloudera.com>
Tested-by: Impala Public Jenkins
Project: http://git-wip-us.apache.org/repos/asf/impala/repo
Commit: http://git-wip-us.apache.org/repos/asf/impala/commit/5166270b
Tree: http://git-wip-us.apache.org/repos/asf/impala/tree/5166270b
Diff: http://git-wip-us.apache.org/repos/asf/impala/diff/5166270b
Branch: refs/heads/2.x
Commit: 5166270bfcfca424be023b5495e9dcf0d344fca2
Parents: 0c4e1a1
Author: Alex Rodoni <ar...@cloudera.com>
Authored: Fri Mar 30 11:23:16 2018 -0700
Committer: Impala Public Jenkins <im...@gerrit.cloudera.org>
Committed: Tue Apr 3 00:55:30 2018 +0000
----------------------------------------------------------------------
docs/topics/impala_proxy.xml | 111 +++++++++++++++++++-------------------
1 file changed, 54 insertions(+), 57 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/impala/blob/5166270b/docs/topics/impala_proxy.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_proxy.xml b/docs/topics/impala_proxy.xml
index 653f7f4..1f5bb4b 100644
--- a/docs/topics/impala_proxy.xml
+++ b/docs/topics/impala_proxy.xml
@@ -95,9 +95,12 @@ under the License.
<ol>
<li>
- Download the load-balancing proxy software. It should only need to be installed and configured on a
- single host. Pick a host other than the DataNodes where <cmdname>impalad</cmdname> is running,
- because the intention is to protect against the possibility of one or more of these DataNodes becoming unavailable.
+ Select and download the load-balancing proxy software or other
+ load-balancing hardware appliance. It should only need to be installed
+ and configured on a single host, typically on an edge node. Pick a
+ host other than the DataNodes where <cmdname>impalad</cmdname> is
+ running, because the intention is to protect against the possibility
+ of one or more of these DataNodes becoming unavailable.
</li>
<li>
@@ -105,42 +108,27 @@ under the License.
In particular:
<ul>
<li>
- <p>
- Set up a port that the load balancer will listen on to relay Impala requests back and forth.
- </p>
- </li>
+ Set up a port that the load balancer will listen on to relay
+ Impala requests back and forth. </li>
<li>
- <p rev="DOCS-690">
- Consider using the <i>source affinity</i>
- algorithm to ensure the sticky sessions. Where practical, enable
- this setting so that stateless client applications such as
- <cmdname>impalad</cmdname> and Hue are not disconnected from
- long-running queries. Evaluate whether this setting is
- appropriate for your combination of workload and client
- applications. See <xref href="#proxy_balancing" format="dita"/>
- for load balancing algorithm options.
- </p>
+ See <xref href="#proxy_balancing" format="dita"/> for load
+ balancing algorithm options.
</li>
<li>
- <p>
- For Kerberized clusters, follow the instructions in <xref href="impala_proxy.xml#proxy_kerberos"/>.
- </p>
+ For Kerberized clusters, follow the instructions in <xref
+ href="impala_proxy.xml#proxy_kerberos"/>.
</li>
</ul>
</li>
<li>
- Specify the host and port settings for each Impala node. These are the hosts that the load balancer will
- choose from when relaying each Impala query. See <xref href="impala_ports.xml#ports"/> for when to use
- port 21000, 21050, or another value depending on what type of connections you are load balancing.
- <note rev="">
- <p rev="">
- In particular, if you are using Hue or JDBC-based applications,
- you typically set up load balancing for both ports 21000 and 21050, because
- these client applications connect through port 21050 while the <cmdname>impala-shell</cmdname>
- command connects through port 21000.
- </p>
- </note>
+ If you are using Hue or JDBC-based applications, you typically set
+ up load balancing for both ports 21000 and 21050, because these client
+ applications connect through port 21050 while the
+ <cmdname>impala-shell</cmdname> command connects through port
+ 21000. See <xref href="impala_ports.xml#ports"/> for when to use port
+ 21000, 21050, or another value depending on what type of connections
+ you are load balancing.
</li>
<li>
@@ -148,9 +136,11 @@ under the License.
</li>
<li>
- For any scripts, jobs, or configuration settings for applications that formerly connected to a specific
- datanode to run Impala SQL statements, change the connection information (such as the <codeph>-i</codeph>
- option in <cmdname>impala-shell</cmdname>) to point to the load balancer instead.
+ For any scripts, jobs, or configuration settings for applications
+ that formerly connected to a specific DataNode to run Impala SQL
+ statements, change the connection information (such as the
+ <codeph>-i</codeph> option in <cmdname>impala-shell</cmdname>) to
+ point to the load balancer instead.
</li>
</ol>
@@ -174,49 +164,56 @@ under the License.
<dl>
<dlentry>
- <dt>leastconn</dt>
+ <dt>Leastconn</dt>
<dd>
Connects sessions to the coordinator with the fewest connections,
to balance the load evenly. Typically used for workloads consisting
of many independent, short-running queries. In configurations with
only a few client machines, this setting can avoid having all
- requests go to only a small set of coordinators. Recommended for
- Impala with F5.
+ requests go to only a small set of coordinators.
+ </dd>
+ <dd>
+ Recommended for Impala with F5.
</dd>
</dlentry>
<dlentry>
- <dt>source affinity</dt>
+ <dt>Source IP Persistence</dt>
<dd>
- Sessions from the same IP address always go to the same coordinator.
- A good choice for Impala workloads containing a mix of queries and
- DDL statements, such as <codeph>CREATE TABLE</codeph> and <codeph>ALTER TABLE</codeph>.
- Because the metadata changes from a DDL statement take time to propagate across the cluster,
- prefer to use source affinity in this case. If necessary, run the DDL and subsequent
- queries that depend on the results of the DDL through the same session, for example
- by running <codeph>impala-shell -f <varname>script_file</varname></codeph> to submit
- several statements through a single session.
- An alternative is to set the query option <codeph>SYNC_DDL=1</codeph>
- to hold back subsequent queries until the results of a DDL operation have propagated
- throughout the cluster, but that is a relatively expensive setting.
+ <p>
+ Sessions from the same IP address always go to the same
+ coordinator. A good choice for Impala workloads containing a mix
+ of queries and DDL statements, such as <codeph>CREATE TABLE</codeph>
+ and <codeph>ALTER TABLE</codeph>. Because the metadata changes from
+ a DDL statement take time to propagate across the cluster, prefer
+ to use the Source IP Persistence in this case. If you are unable
+ to choose Source IP Persistence, run the DDL and subsequent queries
+ that depend on the results of the DDL through the same session,
+ for example by running <codeph>impala-shell -f <varname>script_file</varname></codeph>
+ to submit several statements through a single session.
+ </p>
</dd>
<dd>
- Recommended for use with Hue.
+ <p>
+ Required for setting up high availability with Hue.
+ </p>
</dd>
</dlentry>
<dlentry>
- <dt>round-robin</dt>
+ <dt>Round-robin</dt>
<dd>
- Distributes connections to all coordinator nodes.
- Typically not recommended for Impala.
+ <p>
+ Distributes connections to all coordinator nodes.
+ Typically not recommended for Impala.
+ </p>
</dd>
</dlentry>
</dl>
<p>
- You might need to perform benchmarks and load testing to determine which setting is optimal for your
- use case. If some client applications have special characteristics, such as long-running Hue queries
- working best with source affinity, you might configure multiple virtual IP addresses with a
- different load-balancing algorithm for each.
+ You might need to perform benchmarks and load testing to determine
+ which setting is optimal for your use case. Always set up using two
+ load-balancing algorithms: Source IP Persistence for Hue and Leastconn
+ for others.
</p>
</conbody>
[4/5] impala git commit: Do clean as part of
"bootstrap_development.sh"
Posted by ta...@apache.org.
Do clean as part of "bootstrap_development.sh"
This commit instructs bootstrap_development.sh, which is
used in Gerrity-Verify-Dryrun jobs, to clean before
running the build. In the common case, this is essentially
a no-op, because the workspace is new to begin with.
This handles an issue where GVDs would fail if they
happened to run on a machine where a previous build ran
against a workspace where the fe/generated-sources
directories were incompatible. I confirmed that
our clean script does clean out fe/generated-sources
nicely.
Change-Id: I238528ca3938200f0750505a62113630290e8c96
Reviewed-on: http://gerrit.cloudera.org:8080/9841
Reviewed-by: Philip Zeyliger <ph...@cloudera.com>
Tested-by: Impala Public Jenkins
Project: http://git-wip-us.apache.org/repos/asf/impala/repo
Commit: http://git-wip-us.apache.org/repos/asf/impala/commit/0c4e1a10
Tree: http://git-wip-us.apache.org/repos/asf/impala/tree/0c4e1a10
Diff: http://git-wip-us.apache.org/repos/asf/impala/diff/0c4e1a10
Branch: refs/heads/2.x
Commit: 0c4e1a10906d2c96d113edd2a22c0b892233bc8d
Parents: 456aa5d
Author: Philip Zeyliger <ph...@cloudera.com>
Authored: Wed Mar 28 14:30:07 2018 -0700
Committer: Impala Public Jenkins <im...@gerrit.cloudera.org>
Committed: Tue Apr 3 00:55:30 2018 +0000
----------------------------------------------------------------------
bin/bootstrap_development.sh | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/impala/blob/0c4e1a10/bin/bootstrap_development.sh
----------------------------------------------------------------------
diff --git a/bin/bootstrap_development.sh b/bin/bootstrap_development.sh
index 0e31a3f..b6ea369 100755
--- a/bin/bootstrap_development.sh
+++ b/bin/bootstrap_development.sh
@@ -46,7 +46,7 @@ source "${BINDIR}/bootstrap_system.sh"
export MAX_PYTEST_FAILURES=0
source bin/impala-config.sh
export NUM_CONCURRENT_TESTS=$(nproc)
-time -p ./buildall.sh -noclean -format -testdata -skiptests
+time -p ./buildall.sh -format -testdata -skiptests
# To then run the tests:
# time -p bin/run-all-tests.sh
[5/5] impala git commit: IMPALA-6546: [DOCS] Add the new ODBC
functions to docs
Posted by ta...@apache.org.
IMPALA-6546: [DOCS] Add the new ODBC functions to docs
- Added the following new functions to the string functions doc:
left() and right()
- Added the following new functions to the date time functions doc:
monthname(), quarter(), week()
- Added quarter to extract() and date_part() as a unit
Change-Id: Icf31b50584628603c0c86ff0772a12ac6ac5c7b6
Reviewed-on: http://gerrit.cloudera.org:8080/9880
Reviewed-by: Alex Behm <al...@cloudera.com>
Tested-by: Impala Public Jenkins
Project: http://git-wip-us.apache.org/repos/asf/impala/repo
Commit: http://git-wip-us.apache.org/repos/asf/impala/commit/6dacdcf1
Tree: http://git-wip-us.apache.org/repos/asf/impala/tree/6dacdcf1
Diff: http://git-wip-us.apache.org/repos/asf/impala/diff/6dacdcf1
Branch: refs/heads/2.x
Commit: 6dacdcf1c4f0468f2297437c37c9ad85017e8111
Parents: 1d0af16
Author: Alex Rodoni <ar...@cloudera.com>
Authored: Fri Mar 30 18:35:23 2018 -0700
Committer: Impala Public Jenkins <im...@gerrit.cloudera.org>
Committed: Tue Apr 3 00:55:30 2018 +0000
----------------------------------------------------------------------
docs/topics/impala_datetime_functions.xml | 91 +++++++++++++++++---------
docs/topics/impala_string_functions.xml | 16 +++++
2 files changed, 76 insertions(+), 31 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/impala/blob/6dacdcf1/docs/topics/impala_datetime_functions.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_datetime_functions.xml b/docs/topics/impala_datetime_functions.xml
index d40e28b..cb2f03f 100644
--- a/docs/topics/impala_datetime_functions.xml
+++ b/docs/topics/impala_datetime_functions.xml
@@ -565,8 +565,8 @@ select now() as right_now, datediff(now(), now() - interval 18 hours) as 18_hour
<dlentry id="day">
<dt>
- <!-- <codeph>day(string date), <ph id="dayofmonth">dayofmonth(string date)</ph></codeph> -->
- <codeph>day(timestamp date), <ph id="dayofmonth">dayofmonth(timestamp date)</ph></codeph>
+ <codeph>day(timestamp date), <ph id="dayofmonth"
+ >dayofmonth(timestamp date)</ph></codeph>
</dt>
<dd>
@@ -684,7 +684,6 @@ select now() + interval 1 day as tomorrow,
<dlentry rev="1.1" id="dayofweek">
<dt>
- <!-- <codeph>dayofweek(string date)</codeph> -->
<codeph>dayofweek(timestamp date)</codeph>
</dt>
@@ -811,41 +810,47 @@ select now() as right_now, days_sub(now(), 31) as 31_days_ago;
<dlentry rev="1.4.0" id="extract">
<dt>
- <codeph>extract(timestamp, string unit)</codeph><codeph rev="2.0.0">extract(unit FROM timestamp)</codeph>
+ <codeph>extract(timestamp, string unit)</codeph>, <codeph rev="2.0.0">extract(unit FROM timestamp)</codeph>
</dt>
<dd>
<indexterm audience="hidden">extract() function</indexterm>
- <b>Purpose:</b> Returns one of the numeric date or time fields from a <codeph>TIMESTAMP</codeph> value.
- <p>
- <b>Unit argument:</b> The <codeph>unit</codeph> string can be one of <codeph>epoch</codeph>,
- <codeph>year</codeph>, <codeph>month</codeph>, <codeph>day</codeph>, <codeph>hour</codeph>,
- <codeph>minute</codeph>, <codeph>second</codeph>, or <codeph>millisecond</codeph>.
- This argument value is case-insensitive.
- </p>
- <p rev="2.0.0">
- In Impala 2.0 and higher, you can use special syntax rather than a regular function call, for
- compatibility with code that uses the SQL-99 format with the <codeph>FROM</codeph> keyword. With this
- style, the unit names are identifiers rather than <codeph>STRING</codeph> literals. For example, the
- following calls are both equivalent:
-<codeblock>extract(year from now());
+ <b>Purpose:</b> Returns one of the numeric date or time fields from a
+ <codeph>TIMESTAMP</codeph> value.
+ <p>
+ <b>Unit argument:</b> The <codeph>unit</codeph> string can be one of
+ <codeph>epoch</codeph>, <codeph>year</codeph>,
+ <codeph>quarter</codeph>, <codeph>month</codeph>,
+ <codeph>day</codeph>, <codeph>hour</codeph>,
+ <codeph>minute</codeph>, <codeph>second</codeph>, or
+ <codeph>millisecond</codeph>. This argument value is
+ case-insensitive.
+ </p>
+ <p rev="2.0.0"> In Impala 2.0 and higher, you can use special syntax
+ rather than a regular function call, for compatibility with code
+ that uses the SQL-99 format with the <codeph>FROM</codeph> keyword.
+ With this style, the unit names are identifiers rather than
+ <codeph>STRING</codeph> literals. For example, the following calls
+ are both equivalent:
+ <codeblock>extract(year from now());
extract(now(), "year");
</codeblock>
</p>
<p conref="../shared/impala_common.xml#common/usage_notes_blurb"/>
- <p>
- Typically used in <codeph>GROUP BY</codeph> queries to arrange results by hour,
- day, month, and so on. You can also use this function in an <codeph>INSERT ... SELECT</codeph> into a
- partitioned table to split up <codeph>TIMESTAMP</codeph> values into individual parts, if the
- partitioned table has separate partition key columns representing year, month, day, and so on. If you
- need to divide by more complex units of time, such as by week or by quarter, use the
- <codeph>TRUNC()</codeph> function instead.
- </p>
- <p>
- <b>Return type:</b> <codeph>int</codeph>
+ <p> Typically used in <codeph>GROUP BY</codeph> queries to arrange
+ results by hour, day, month, and so on. You can also use this
+ function in an <codeph>INSERT ... SELECT</codeph> into a partitioned
+ table to split up <codeph>TIMESTAMP</codeph> values into individual
+ parts, if the partitioned table has separate partition key columns
+ representing year, month, day, and so on. If you need to divide by
+ more complex units of time, such as by week or by quarter, use the
+ <codeph>TRUNC()</codeph> function instead. </p>
+ <p>
+ <b>Return type:</b>
+ <codeph>int</codeph>
</p>
<p conref="../shared/impala_common.xml#common/example_blurb"/>
-<codeblock>
+ <codeblock>
select now() as right_now,
extract(year from now()) as this_year,
extract(month from now()) as this_month;
@@ -1532,7 +1537,6 @@ select now() as right_now, minutes_sub(now(), 90) as 90_minutes_ago;
<dlentry id="month">
<dt>
- <!-- <codeph>month(string date)</codeph> -->
<codeph>month(timestamp date)</codeph>
</dt>
@@ -1554,6 +1558,19 @@ select now() as right_now, month(now()) as current_month;
</dd>
</dlentry>
+ <dlentry id="monthname">
+ <dt>
+ <codeph>monthname(timestamp date)</codeph>
+ </dt>
+ <dd>
+ <b>Purpose:</b> Returns the month field from a
+ <codeph>TIMESTAMP</codeph> value, converted to the string
+ corresponding to that month name.
+ <p>
+ <b>Return type:</b> <codeph>string</codeph>
+ </p>
+ </dd>
+ </dlentry>
<dlentry rev="1.3.0" id="months_add">
@@ -1953,6 +1970,19 @@ select now() as right_now,
</dd>
</dlentry>
+ <dlentry id="quarter">
+ <dt>
+ <codeph>quarter(timestamp date)</codeph>
+ </dt>
+ <dd>
+ <b>Purpose:</b> Returns the quarter in the input
+ <codeph>TIMESTAMP</codeph> expression as an integer value, 1, 2, 3, or
+ 4, where 1 represents January 1 through March 31.
+ <p>
+ <b>Return type:</b> <codeph>int</codeph>
+ </p>
+ </dd>
+ </dlentry>
<dlentry id="second">
@@ -2686,8 +2716,7 @@ select current_timestamp(), utc_timestamp();
<dlentry id="weekofyear">
<dt>
- <!-- <codeph>weekofyear(string date)</codeph> -->
- <codeph>weekofyear(timestamp date)</codeph>
+ <codeph>week(timestamp date)</codeph>, <codeph>weekofyear(timestamp date)</codeph>
</dt>
<dd>
http://git-wip-us.apache.org/repos/asf/impala/blob/6dacdcf1/docs/topics/impala_string_functions.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_string_functions.xml b/docs/topics/impala_string_functions.xml
index a424683..fe08c8e 100644
--- a/docs/topics/impala_string_functions.xml
+++ b/docs/topics/impala_string_functions.xml
@@ -550,6 +550,14 @@ select instr('foo bar bletch', 'b', 1, null);
</dd>
</dlentry>
+ <dlentry id="left">
+ <dt>
+ <codeph>left(string a, int num_chars)</codeph>
+ </dt>
+ <dd>
+ See the <codeph>strleft</codeph> function.
+ </dd>
+ </dlentry>
<dlentry id="length">
@@ -1001,6 +1009,14 @@ select replace('hello world','xyz','abc');
</dd>
</dlentry>
+ <dlentry id="right">
+ <dt>
+ <codeph>right(string a, int num_chars)</codeph>
+ </dt>
+ <dd>
+ See the <codeph>strright</codeph> function.
+ </dd>
+ </dlentry>
<dlentry id="rpad">
[2/5] impala git commit: IMPALA-6739: Exception in ALTER TABLE SET
statements
Posted by ta...@apache.org.
IMPALA-6739: Exception in ALTER TABLE SET statements
The patch fixes issues with executing ALTER TABLE SET statements when
there are no matching partitions.
The patch also removes incorrect precondition i.e.
(partitionSet == null || !partitionSet.isEmpty()) in ALTER TABLE SET
statements because a partitionSet can be null when PARTITION is not
specified in the ALTER TABLE SET statement and partitionSet can be
empty when there is no matching partition. For example:
Matching partitions (partitionSet != null && !partitionSet.isEmpty()):
> alter table functional.alltypesagg partition(year=2009, month=1)
set fileformat parquet;
No matching partitions (partitionSet != null && partitionSet.isEmpty()):
> alter table functional.alltypesagg partition(year=2009, month=1)
set fileformat parquet;
No partition specified (partitionSet == null):
> alter table functional.alltypesagg set fileformat parquet;
Testing:
- Added a new test
- Ran all front-end tests
Change-Id: I793e827d5cf5b7986bd150dd9706df58da3417f3
Reviewed-on: http://gerrit.cloudera.org:8080/9819
Reviewed-by: Alex Behm <al...@cloudera.com>
Tested-by: Impala Public Jenkins
Project: http://git-wip-us.apache.org/repos/asf/impala/repo
Commit: http://git-wip-us.apache.org/repos/asf/impala/commit/456aa5d5
Tree: http://git-wip-us.apache.org/repos/asf/impala/tree/456aa5d5
Diff: http://git-wip-us.apache.org/repos/asf/impala/diff/456aa5d5
Branch: refs/heads/2.x
Commit: 456aa5d56e862a86e31bfdf88ed542efd58349aa
Parents: 6dacdcf
Author: Fredy Wijaya <fw...@cloudera.com>
Authored: Tue Mar 27 09:37:31 2018 -0700
Committer: Impala Public Jenkins <im...@gerrit.cloudera.org>
Committed: Tue Apr 3 00:55:30 2018 +0000
----------------------------------------------------------------------
.../analysis/AlterTableSetLocationStmt.java | 6 ++-
.../impala/service/CatalogOpExecutor.java | 2 -
.../apache/impala/analysis/AnalyzeDDLTest.java | 12 +++++
.../partition-ddl-predicates-all-fs.test | 52 ++++++++++++++++++++
4 files changed, 68 insertions(+), 4 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/impala/blob/456aa5d5/fe/src/main/java/org/apache/impala/analysis/AlterTableSetLocationStmt.java
----------------------------------------------------------------------
diff --git a/fe/src/main/java/org/apache/impala/analysis/AlterTableSetLocationStmt.java b/fe/src/main/java/org/apache/impala/analysis/AlterTableSetLocationStmt.java
index f076312..ae31a76 100644
--- a/fe/src/main/java/org/apache/impala/analysis/AlterTableSetLocationStmt.java
+++ b/fe/src/main/java/org/apache/impala/analysis/AlterTableSetLocationStmt.java
@@ -59,8 +59,9 @@ public class AlterTableSetLocationStmt extends AlterTableSetStmt {
params.setAlter_type(TAlterTableType.SET_LOCATION);
TAlterTableSetLocationParams locationParams =
new TAlterTableSetLocationParams(location_.toString());
- if (getPartitionSet() != null) {
- List<List<TPartitionKeyValue>> tPartitionSet = getPartitionSet().toThrift();
+ PartitionSet partitionSet = getPartitionSet();
+ if (partitionSet != null && !partitionSet.getPartitions().isEmpty()) {
+ List<List<TPartitionKeyValue>> tPartitionSet = partitionSet.toThrift();
Preconditions.checkState(tPartitionSet.size() == 1);
locationParams.setPartition_spec(tPartitionSet.get(0));
}
@@ -80,6 +81,7 @@ public class AlterTableSetLocationStmt extends AlterTableSetStmt {
if (getPartitionSet() != null) {
// Targeting a partition rather than a table.
List<HdfsPartition> partitions = getPartitionSet().getPartitions();
+ if (partitions.isEmpty()) { return; }
if (partitions.size() != 1) {
// Sort the partitions to get a consistent error reporting.
List<HdfsPartition> sortedPartitions = Lists.newArrayList(partitions);
http://git-wip-us.apache.org/repos/asf/impala/blob/456aa5d5/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
----------------------------------------------------------------------
diff --git a/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java b/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
index 9ccfa50..184a839 100644
--- a/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
+++ b/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
@@ -2238,7 +2238,6 @@ public class CatalogOpExecutor {
Reference<Long> numUpdatedPartitions)
throws ImpalaException {
Preconditions.checkState(tbl.getLock().isHeldByCurrentThread());
- Preconditions.checkState(partitionSet == null || !partitionSet.isEmpty());
boolean reloadFileMetadata = false;
if (partitionSet == null) {
org.apache.hadoop.hive.metastore.api.Table msTbl =
@@ -2275,7 +2274,6 @@ public class CatalogOpExecutor {
Reference<Long> numUpdatedPartitions)
throws ImpalaException {
Preconditions.checkState(tbl.getLock().isHeldByCurrentThread());
- Preconditions.checkState(partitionSet == null || !partitionSet.isEmpty());
Preconditions.checkArgument(tbl instanceof HdfsTable);
boolean reloadFileMetadata = false;
RowFormat rowFormat = RowFormat.fromThrift(tRowFormat);
http://git-wip-us.apache.org/repos/asf/impala/blob/456aa5d5/fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java
----------------------------------------------------------------------
diff --git a/fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java b/fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java
index ce49828..8eaf674 100644
--- a/fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java
+++ b/fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java
@@ -535,6 +535,18 @@ public class AnalyzeDDLTest extends FrontendTestBase {
"'sort.columns'='ID, foo')",
"Could not find SORT BY column 'foo' in table.");
+ // Table is partitioned, but it has no matching partitions.
+ AnalyzesOk("alter table functional.alltypesagg partition(year=2009, month=1) " +
+ "set location 'hdfs://localhost:20500/test-warehouse/new_table'");
+ AnalyzesOk("alter table functional.alltypesagg partition(year=2009, month=1) " +
+ "set fileformat parquet");
+ AnalyzesOk("alter table functional.alltypesagg partition(year=2009, month=1) " +
+ "set tblproperties ('key'='value')");
+ AnalyzesOk("alter table functional.alltypesagg partition(year=2009, month=1) " +
+ "set serdeproperties ('key'='value')");
+ AnalyzesOk("alter table functional.alltypesagg partition(year=2009, month=1) " +
+ "set row format delimited fields terminated by '|'");
+
{
// Check that long_properties fail at the analysis layer
String long_property_key = "";
http://git-wip-us.apache.org/repos/asf/impala/blob/456aa5d5/testdata/workloads/functional-query/queries/QueryTest/partition-ddl-predicates-all-fs.test
----------------------------------------------------------------------
diff --git a/testdata/workloads/functional-query/queries/QueryTest/partition-ddl-predicates-all-fs.test b/testdata/workloads/functional-query/queries/QueryTest/partition-ddl-predicates-all-fs.test
index 17113aa..f90dcff 100644
--- a/testdata/workloads/functional-query/queries/QueryTest/partition-ddl-predicates-all-fs.test
+++ b/testdata/workloads/functional-query/queries/QueryTest/partition-ddl-predicates-all-fs.test
@@ -153,3 +153,55 @@ show partitions p1
---- TYPES
STRING, STRING, BIGINT, BIGINT, STRING, STRING, STRING, STRING, STRING, STRING
====
+---- QUERY
+# Tests no matching partition.
+alter table p1 partition (j=100) set location '$FILESYSTEM_PREFIX/test-warehouse/newtable';
+---- RESULTS
+# TODO: IMPALA-6775
+==== QUERY
+# Check nothing was updated.
+show partitions p1
+---- RESULTS
+'NULL','g',-1,0,regex:.+,regex:.+,regex:.+,regex:.+,regex:.+,regex:.*/test-warehouse/.+/p1/j=__HIVE_DEFAULT_PARTITION__/k=g
+'2','D',-1,0,regex:.+,regex:.+,regex:.+,'TEXT',regex:.+,regex:.*/test-warehouse/.+/p1/j=2/k=D
+'2','E',-1,0,regex:.+,regex:.+,regex:.+,'TEXT',regex:.+,regex:.*/test-warehouse/.+/p1/j=2/k=E
+'2','F',-1,0,regex:.+,regex:.+,regex:.+,'TEXT',regex:.+,regex:.*/test-warehouse/.+/p1/j=2/k=F
+'2','d',-1,0,regex:.+,regex:.+,regex:.+,'PARQUET',regex:.+,regex:.*/test-warehouse/.+/p1/j=2/k=d
+'2','e',-1,0,regex:.+,regex:.+,regex:.+,'PARQUET',regex:.+,regex:.*/test-warehouse/.+/p1/j=2/k=e
+'2','f',-1,0,regex:.+,regex:.+,regex:.+,'PARQUET',regex:.+,regex:.*/test-warehouse/.+/p1/j=2/k=f
+'Total','',3,0,regex:.+,regex:.+,'','','',''
+---- TYPES
+STRING, STRING, BIGINT, BIGINT, STRING, STRING, STRING, STRING, STRING, STRING
+====
+---- QUERY
+# Tests no matching partition.
+alter table p1 partition (j=100) set fileformat parquet;
+---- RESULTS
+'Updated 0 partition(s).'
+---- TYPES
+STRING
+====
+---- QUERY
+# Tests no matching partition.
+alter table p1 partition (j=100) set tblproperties ('key'='value');
+---- RESULTS
+'Updated 0 partition(s).'
+---- TYPES
+STRING
+====
+---- QUERY
+# Tests no matching partition.
+alter table p1 partition (j=100) set serdeproperties ('key'='value');
+---- RESULTS
+'Updated 0 partition(s).'
+---- TYPES
+STRING
+====
+---- QUERY
+# Tests no matching partition.
+alter table p1 partition (j=100) set row format delimited fields terminated by '|';
+---- RESULTS
+'Updated 0 partition(s).'
+---- TYPES
+STRING
+====
\ No newline at end of file
[3/5] impala git commit: IMPALA-6445: Test for kudu master address
with whitespace
Posted by ta...@apache.org.
IMPALA-6445: Test for kudu master address with whitespace
A concern was brought up that Impala might not handle kudu master
addresses containing whitespace correctly. Turns out that the Kudu
client takes care of stripping whitespace, so it works, but it would
be good to have a test to ensure it continues to work.
Change-Id: I1857b8dbcb5af66d69f7620368cd3b9b85ae7576
Reviewed-on: http://gerrit.cloudera.org:8080/9876
Reviewed-by: Alex Behm <al...@cloudera.com>
Tested-by: Impala Public Jenkins
Project: http://git-wip-us.apache.org/repos/asf/impala/repo
Commit: http://git-wip-us.apache.org/repos/asf/impala/commit/1d0af162
Tree: http://git-wip-us.apache.org/repos/asf/impala/tree/1d0af162
Diff: http://git-wip-us.apache.org/repos/asf/impala/diff/1d0af162
Branch: refs/heads/2.x
Commit: 1d0af16263d120071bbd6c6456955579df0edac4
Parents: 0197b17
Author: Thomas Tauber-Marshall <tm...@cloudera.com>
Authored: Fri Mar 30 14:52:23 2018 -0700
Committer: Impala Public Jenkins <im...@gerrit.cloudera.org>
Committed: Tue Apr 3 00:55:30 2018 +0000
----------------------------------------------------------------------
.../functional-query/queries/QueryTest/kudu_create.test | 12 ++++++++++++
1 file changed, 12 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/impala/blob/1d0af162/testdata/workloads/functional-query/queries/QueryTest/kudu_create.test
----------------------------------------------------------------------
diff --git a/testdata/workloads/functional-query/queries/QueryTest/kudu_create.test b/testdata/workloads/functional-query/queries/QueryTest/kudu_create.test
index 24dfdf1..204fc68 100644
--- a/testdata/workloads/functional-query/queries/QueryTest/kudu_create.test
+++ b/testdata/workloads/functional-query/queries/QueryTest/kudu_create.test
@@ -103,6 +103,18 @@ create table tdata_non_existing_host
Couldn't resolve this master's address bogus.host.name:7051
====
---- QUERY
+# Valid host with whitespace
+create table tdata_master_addresses_whitespace (id int primary key) stored as kudu
+ tblproperties('kudu.master_addresses' = ' localhost ')
+---- RESULTS
+====
+---- QUERY
+insert into tdata_master_addresses_whitespace values (0), (1)
+---- DML_RESULTS: tdata_master_addresses_whitespace
+0
+1
+====
+---- QUERY
# Test that string case is ignored
create table ignore_column_case (Id int, NAME string, vAlf float, vali bigint,
primary key (Id, NAME)) PARTITION BY RANGE (PARTITION VALUE = (1, 'Martin'))