You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@impala.apache.org by ta...@apache.org on 2018/06/08 15:49:58 UTC
[1/5] impala git commit: IMPALA-3040: Print logs around
removeDirective()
Repository: impala
Updated Branches:
refs/heads/2.x b8f3d3cc6 -> a0541cd83
IMPALA-3040: Print logs around removeDirective()
I observed that some caching directives on partitions don't get
removed. HDFS would throw an exception if removeDirective() operation
fails on its side, so I suspect that removeDirective() isn't called by
catalogd. This patch prints logs around removeDirective() if it's
skipped.
Change-Id: I9d9f6eb1951c77e095d5029e8bbdc67e9b39b814
Reviewed-on: http://gerrit.cloudera.org:8080/10640
Reviewed-by: Tim Armstrong <ta...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
Project: http://git-wip-us.apache.org/repos/asf/impala/repo
Commit: http://git-wip-us.apache.org/repos/asf/impala/commit/7f0a7b13
Tree: http://git-wip-us.apache.org/repos/asf/impala/tree/7f0a7b13
Diff: http://git-wip-us.apache.org/repos/asf/impala/diff/7f0a7b13
Branch: refs/heads/2.x
Commit: 7f0a7b1384d8c833bda68cfd1eb5ea766cf9b324
Parents: b8f3d3c
Author: Tianyi Wang <ti...@apache.org>
Authored: Thu Jun 7 11:42:14 2018 -0700
Committer: Impala Public Jenkins <im...@gerrit.cloudera.org>
Committed: Thu Jun 7 22:49:43 2018 +0000
----------------------------------------------------------------------
.../org/apache/impala/util/HdfsCachingUtil.java | 26 +++++++++++++++++---
1 file changed, 22 insertions(+), 4 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/impala/blob/7f0a7b13/fe/src/main/java/org/apache/impala/util/HdfsCachingUtil.java
----------------------------------------------------------------------
diff --git a/fe/src/main/java/org/apache/impala/util/HdfsCachingUtil.java b/fe/src/main/java/org/apache/impala/util/HdfsCachingUtil.java
index 1846834..63e4924 100644
--- a/fe/src/main/java/org/apache/impala/util/HdfsCachingUtil.java
+++ b/fe/src/main/java/org/apache/impala/util/HdfsCachingUtil.java
@@ -133,8 +133,17 @@ public class HdfsCachingUtil {
if (LOG.isTraceEnabled()) {
LOG.trace("Uncaching table: " + table.getDbName() + "." + table.getTableName());
}
- Long id = getCacheDirectiveId(table.getParameters());
- if (id == null) return;
+ Map<String, String> parameters = table.getParameters();
+ if (parameters == null) {
+ LOG.warn("removePartitionCacheDirective(): table " + table.getTableName() +
+ "has a null parameter map.");
+ }
+ Long id = getCacheDirectiveId(parameters);
+ if (id == null) {
+ LOG.warn("removePartitionCacheDirective(): table " + table.getTableName() +
+ "doesn't have a cache directive id.");
+ return;
+ }
HdfsCachingUtil.removeDirective(id);
table.getParameters().remove(CACHE_DIR_ID_PROP_NAME);
table.getParameters().remove(CACHE_DIR_REPLICATION_PROP_NAME);
@@ -148,8 +157,17 @@ public class HdfsCachingUtil {
public static void removePartitionCacheDirective(HdfsPartition part)
throws ImpalaException {
Preconditions.checkNotNull(part);
- Long id = getCacheDirectiveId(part.getParameters());
- if (id == null) return;
+ Map<String, String> parameters = part.getParameters();
+ if (parameters == null) {
+ LOG.warn("removePartitionCacheDirective(): partition " + part.getPartitionName() +
+ "has a null parameter map.");
+ }
+ Long id = getCacheDirectiveId(parameters);
+ if (id == null) {
+ LOG.warn("removePartitionCacheDirective(): partition " + part.getPartitionName() +
+ "doesn't have a cache directive id.");
+ return;
+ }
HdfsCachingUtil.removeDirective(id);
part.getParameters().remove(CACHE_DIR_ID_PROP_NAME);
part.getParameters().remove(CACHE_DIR_REPLICATION_PROP_NAME);
[2/5] impala git commit: IMPALA-7062: fix unsafe
RuntimeProfile::SortChildren() function
Posted by ta...@apache.org.
IMPALA-7062: fix unsafe RuntimeProfile::SortChildren() function
The API is inherently unsafe, because std::sort() can crash if a
comparator returns unstable results, e.g. because a profile
counter value changes. The generality of this function is both
unnecessary and dangerous.
The fix is to take a snapshot of the counter values and sort
based on the snapshot.
Change-Id: Ie456c87c34e7d75eb7482f499e90d50de740df40
Reviewed-on: http://gerrit.cloudera.org:8080/10488
Reviewed-by: Tim Armstrong <ta...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
Project: http://git-wip-us.apache.org/repos/asf/impala/repo
Commit: http://git-wip-us.apache.org/repos/asf/impala/commit/99f43a2c
Tree: http://git-wip-us.apache.org/repos/asf/impala/tree/99f43a2c
Diff: http://git-wip-us.apache.org/repos/asf/impala/diff/99f43a2c
Branch: refs/heads/2.x
Commit: 99f43a2c4103e69e54aaae89032439aca0f538ac
Parents: 7f0a7b1
Author: Tim Armstrong <ta...@cloudera.com>
Authored: Wed May 23 10:43:31 2018 -0700
Committer: Impala Public Jenkins <im...@gerrit.cloudera.org>
Committed: Fri Jun 8 02:29:25 2018 +0000
----------------------------------------------------------------------
be/src/runtime/coordinator-backend-state.cc | 14 +-------------
be/src/util/runtime-profile-test.cc | 10 +++++-----
be/src/util/runtime-profile.cc | 20 ++++++++++++++++++++
be/src/util/runtime-profile.h | 12 ++++--------
4 files changed, 30 insertions(+), 26 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/impala/blob/99f43a2c/be/src/runtime/coordinator-backend-state.cc
----------------------------------------------------------------------
diff --git a/be/src/runtime/coordinator-backend-state.cc b/be/src/runtime/coordinator-backend-state.cc
index b9b2f15..c871a48 100644
--- a/be/src/runtime/coordinator-backend-state.cc
+++ b/be/src/runtime/coordinator-backend-state.cc
@@ -542,20 +542,8 @@ void Coordinator::FragmentStats::AddSplitStats() {
avg_profile_->AddInfoString("split sizes", ss.str());
}
-// Comparator to order RuntimeProfiles by descending total time
-typedef struct {
- typedef pair<RuntimeProfile*, bool> Profile;
- bool operator()(const Profile& a, const Profile& b) const {
- // Reverse ordering: we want the longest first
- return
- a.first->total_time_counter()->value() > b.first->total_time_counter()->value();
- }
-} InstanceComparator;
-
void Coordinator::FragmentStats::AddExecStats() {
- InstanceComparator comparator;
- root_profile_->SortChildren(comparator);
-
+ root_profile_->SortChildrenByTotalTime();
stringstream times_label;
times_label
<< "min:" << PrettyPrinter::Print(
http://git-wip-us.apache.org/repos/asf/impala/blob/99f43a2c/be/src/util/runtime-profile-test.cc
----------------------------------------------------------------------
diff --git a/be/src/util/runtime-profile-test.cc b/be/src/util/runtime-profile-test.cc
index b194cc9..3a49042 100644
--- a/be/src/util/runtime-profile-test.cc
+++ b/be/src/util/runtime-profile-test.cc
@@ -266,11 +266,11 @@ TEST(CountersTest, MergeAndUpdateChildOrder) {
EXPECT_EQ("Child2", tmp_children[1]->name());
// Test that changes in order of children is handled gracefully by preserving the
- // order from the previous update.
- profile1->SortChildren([](
- const pair<RuntimeProfile*, bool>& p1, const pair<RuntimeProfile*, bool>& p2) {
- return p1.first->name() > p2.first->name();
- });
+ // order from the previous update. Sorting puts the children in descending total time
+ // order.
+ p1_child1->total_time_counter()->Set(1);
+ p1_child2->total_time_counter()->Set(2);
+ profile1->SortChildrenByTotalTime();
profile1->GetChildren(&tmp_children);
EXPECT_EQ("Child2", tmp_children[0]->name());
EXPECT_EQ("Child1", tmp_children[1]->name());
http://git-wip-us.apache.org/repos/asf/impala/blob/99f43a2c/be/src/util/runtime-profile.cc
----------------------------------------------------------------------
diff --git a/be/src/util/runtime-profile.cc b/be/src/util/runtime-profile.cc
index c7d50de..b43bbbe 100644
--- a/be/src/util/runtime-profile.cc
+++ b/be/src/util/runtime-profile.cc
@@ -17,8 +17,10 @@
#include "util/runtime-profile-counters.h"
+#include <algorithm>
#include <iomanip>
#include <iostream>
+#include <utility>
#include <boost/bind.hpp>
#include <boost/thread/locks.hpp>
@@ -501,6 +503,24 @@ void RuntimeProfile::GetAllChildren(vector<RuntimeProfile*>* children) {
}
}
+void RuntimeProfile::SortChildrenByTotalTime() {
+ lock_guard<SpinLock> l(children_lock_);
+ // Create a snapshot of total time values so that they don't change while we're
+ // sorting. Sort the <total_time, index> pairs, then reshuffle children_.
+ vector<pair<int64_t, int64_t>> total_times;
+ for (int i = 0; i < children_.size(); ++i) {
+ total_times.emplace_back(children_[i].first->total_time_counter()->value(), i);
+ }
+ // Order by descending total time.
+ sort(total_times.begin(), total_times.end(),
+ [](const pair<int64_t, int64_t>& p1, const pair<int64_t, int64_t>& p2) {
+ return p1.first > p2.first;
+ });
+ ChildVector new_children;
+ for (const auto& p : total_times) new_children.emplace_back(children_[p.second]);
+ children_ = move(new_children);
+}
+
void RuntimeProfile::AddInfoString(const string& key, const string& value) {
return AddInfoStringInternal(key, value, false);
}
http://git-wip-us.apache.org/repos/asf/impala/blob/99f43a2c/be/src/util/runtime-profile.h
----------------------------------------------------------------------
diff --git a/be/src/util/runtime-profile.h b/be/src/util/runtime-profile.h
index a7f9a95..d0e296f 100644
--- a/be/src/util/runtime-profile.h
+++ b/be/src/util/runtime-profile.h
@@ -40,7 +40,7 @@ class ObjectPool;
/// single thread per process that will convert an amount (i.e. bytes) counter to a
/// corresponding rate based counter. This thread wakes up at fixed intervals and updates
/// all of the rate counters.
-/// Thread-safe.
+/// All methods are thread-safe unless otherwise mentioned.
class RuntimeProfile { // NOLINT: This struct is not packed, but there are not so many
// of them that it makes a performance difference
public:
@@ -116,7 +116,7 @@ class RuntimeProfile { // NOLINT: This struct is not packed, but there are not s
static RuntimeProfile* CreateFromThrift(ObjectPool* pool,
const TRuntimeProfileTree& profiles);
- /// Adds a child profile. This is thread safe.
+ /// Adds a child profile.
/// Checks if 'child' is already added by searching for its name in the
/// child map, and only adds it if the name doesn't exist.
/// 'indent' indicates whether the child will be printed w/ extra indentation
@@ -136,13 +136,9 @@ class RuntimeProfile { // NOLINT: This struct is not packed, but there are not s
RuntimeProfile* CreateChild(
const std::string& name, bool indent = true, bool prepend = false);
- /// Sorts all children according to a custom comparator. Does not
+ /// Sorts all children according to descending total time. Does not
/// invalidate pointers to profiles.
- template <class Compare>
- void SortChildren(const Compare& cmp) {
- boost::lock_guard<SpinLock> l(children_lock_);
- std::sort(children_.begin(), children_.end(), cmp);
- }
+ void SortChildrenByTotalTime();
/// Updates the AveragedCounter counters in this profile with the counters from the
/// 'src' profile. If a counter is present in 'src' but missing in this profile, a new
[5/5] impala git commit: IMPALA-5740: [DOCS] Correct the max length
of STRING
Posted by ta...@apache.org.
IMPALA-5740: [DOCS] Correct the max length of STRING
Change-Id: I43c5a2819c8a3db33a8ce3a6bbde6a1d823ec9b2
Reviewed-on: http://gerrit.cloudera.org:8080/10511
Reviewed-by: Tim Armstrong <ta...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
Project: http://git-wip-us.apache.org/repos/asf/impala/repo
Commit: http://git-wip-us.apache.org/repos/asf/impala/commit/a0541cd8
Tree: http://git-wip-us.apache.org/repos/asf/impala/tree/a0541cd8
Diff: http://git-wip-us.apache.org/repos/asf/impala/diff/a0541cd8
Branch: refs/heads/2.x
Commit: a0541cd835b345c60d147a9b930db2903ff6899c
Parents: 7662172
Author: Alex Rodoni <ar...@cloudera.com>
Authored: Thu May 24 11:35:43 2018 -0700
Committer: Impala Public Jenkins <im...@gerrit.cloudera.org>
Committed: Fri Jun 8 02:29:26 2018 +0000
----------------------------------------------------------------------
docs/topics/impala_string.xml | 238 ++++++++++++++++++++++++-------------
1 file changed, 157 insertions(+), 81 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/impala/blob/a0541cd8/docs/topics/impala_string.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_string.xml b/docs/topics/impala_string.xml
index 758ce6a..fa3c316 100644
--- a/docs/topics/impala_string.xml
+++ b/docs/topics/impala_string.xml
@@ -34,165 +34,241 @@ under the License.
</prolog>
<conbody>
-
<p>
- A data type used in <codeph>CREATE TABLE</codeph> and <codeph>ALTER TABLE</codeph> statements.
+ A data type used in <codeph>CREATE TABLE</codeph> and <codeph>ALTER
+ TABLE</codeph> statements.
</p>
<p conref="../shared/impala_common.xml#common/syntax_blurb"/>
<p>
- In the column definition of a <codeph>CREATE TABLE</codeph> statement:
+ In the column definition of a <codeph>CREATE TABLE</codeph> and
+ <codeph>ALTER TABLE</codeph> statements:
</p>
-<codeblock><varname>column_name</varname> STRING</codeblock>
+ <codeblock><varname>column_name</varname> STRING</codeblock>
<p>
- <b>Length:</b> Maximum of 32,767 bytes. Do not use any length constraint when declaring
- <codeph>STRING</codeph> columns, as you might be familiar with from <codeph>VARCHAR</codeph>,
- <codeph>CHAR</codeph>, or similar column types from relational database systems. <ph rev="2.0.0">If you do
- need to manipulate string values with precise or maximum lengths, in Impala 2.0 and higher you can declare
- columns as <codeph>VARCHAR(<varname>max_length</varname>)</codeph> or
- <codeph>CHAR(<varname>length</varname>)</codeph>, but for best performance use <codeph>STRING</codeph>
- where practical.</ph>
+ <b>Length:</b>
+ </p>
+ <p><ph rev="2.0.0">
+ If you need to manipulate string values with precise or
+ maximum lengths, in Impala 2.0 and higher you can declare columns as
+ <codeph>VARCHAR(<varname>max_length</varname>)</codeph> or
+ <codeph>CHAR(<varname>length</varname>)</codeph>, but for best
+ performance use <codeph>STRING</codeph> where practical.</ph>
</p>
<p>
- <b>Character sets:</b> For full support in all Impala subsystems, restrict string values to the ASCII
- character set. Although some UTF-8 character data can be stored in Impala and retrieved through queries, UTF-8 strings
- containing non-ASCII characters are not guaranteed to work properly in combination with many SQL aspects,
- including but not limited to:
+ Take the following considerations for <codeph>STRING</codeph>
+ lengths:
</p>
+
+ <ul>
+ <li>
+ The hard limit on the size of a <codeph>STRING</codeph> and the total
+ size of a row is 2 GB.
+
+ <p>
+ If a query tries to process or create a string
+ larger than this limit, it will return an error to the user.
+ </p>
+ </li>
+
+ <li>
+ The limit is 1 GB on <codeph>STRING</codeph> when writing to Parquet
+ files.
+ </li>
+
+ <li>
+ Queries operating on strings with 32 KB or less will work reliably and
+ will not hit significant performance or memory problems (unless you have
+ very complex queries, very many columns, etc.)
+ </li>
+
+ <li>
+ Performance and memory consumption may degrade with strings larger
+ than 32 KB.
+ </li>
+
+ <li>
+ The row size, i.e. the total size of all string and other columns, is
+ subject to lower limits at various points in query execution that
+ support spill-to-disk. A few examples for lower row size limits are:
+
+ <ul>
+ <li>
+ Rows coming from the right side of any hash join
+ </li>
+
+ <li>
+ Rows coming from either side of a hash join that spills to disk
+ </li>
+
+ <li>
+ Rows being sorted by the <codeph>SORT</codeph> operator without a
+ limit
+ </li>
+
+ <li>
+ Rows in a grouping aggregation
+ </li>
+
+ </ul>
+
+ <p>
+ In <keyword keyref="impala29"/> and lower, the default limit of
+ the row size in the above cases is 8 MB.
+ </p>
+
+ <p>
+ In <keyword
+ keyref="impala210"/> and higher, the max row size is configurable on
+ a per-query basis with the <codeph>MAX_ROW_SIZE</codeph> query option.
+ Rows up to <codeph>MAX_ROW_SIZE</codeph> (which defaults to 512 KB)
+ can always be processed in the above cases. Rows larger than
+ <codeph>MAX_ROW_SIZE</codeph> are processed on a best-effort basis.
+ See <keyword keyref="max_row_size">MAX_ROW_SIZE</keyword> for more
+ details.
+ </p>
+ </li>
+ </ul>
+
+ <p>
+ <b>Character sets:</b>
+ </p>
+
+ <p>
+ For full support in all Impala subsystems, restrict string values to the
+ ASCII character set. Although some UTF-8 character data can be stored in
+ Impala and retrieved through queries, UTF-8 strings containing non-ASCII
+ characters are not guaranteed to work properly in combination with many
+ SQL aspects, including but not limited to:
+ </p>
+
<ul>
<li>
String manipulation functions.
</li>
+
<li>
Comparison operators.
</li>
+
<li>
The <codeph>ORDER BY</codeph> clause.
</li>
- <li>
- Values in partition key columns.
+
+ <li> Values in partition key columns.
</li>
</ul>
<p>
- For any national language aspects such as
- collation order or interpreting extended ASCII variants such as ISO-8859-1 or ISO-8859-2 encodings, Impala
- does not include such metadata with the table definition. If you need to sort, manipulate, or display data
- depending on those national language characteristics of string data, use logic on the application side.
+ For any national language aspects such as collation order or
+ interpreting extended ASCII variants such as ISO-8859-1 or ISO-8859-2
+ encodings, Impala does not include such metadata with the table
+ definition. If you need to sort, manipulate, or display data depending on
+ those national language characteristics of string data, use logic on the
+ application side.
</p>
-
<p>
<b>Conversions:</b>
</p>
-
<ul>
<li>
<p>
- Impala does not automatically convert <codeph>STRING</codeph> to any numeric type. Impala does
- automatically convert <codeph>STRING</codeph> to <codeph>TIMESTAMP</codeph> if the value matches one of
- the accepted <codeph>TIMESTAMP</codeph> formats; see <xref href="impala_timestamp.xml#timestamp"/> for
- details.
+ Impala does not automatically convert <codeph>STRING</codeph> to any
+ numeric type. Impala does automatically convert
+ <codeph>STRING</codeph> to <codeph>TIMESTAMP</codeph> if the value
+ matches one of the accepted <codeph>TIMESTAMP</codeph> formats; see
+ <xref href="impala_timestamp.xml#timestamp"/> for details.
</p>
</li>
-
<li>
<p>
- You can use <codeph>CAST()</codeph> to convert <codeph>STRING</codeph> values to
- <codeph>TINYINT</codeph>, <codeph>SMALLINT</codeph>, <codeph>INT</codeph>, <codeph>BIGINT</codeph>,
- <codeph>FLOAT</codeph>, <codeph>DOUBLE</codeph>, or <codeph>TIMESTAMP</codeph>.
+ You can use <codeph>CAST()</codeph> to convert
+ <codeph>STRING</codeph> values to <codeph>TINYINT</codeph>,
+ <codeph>SMALLINT</codeph>, <codeph>INT</codeph>,
+ <codeph>BIGINT</codeph>, <codeph>FLOAT</codeph>,
+ <codeph>DOUBLE</codeph>, or <codeph>TIMESTAMP</codeph>.
</p>
</li>
-
<li>
<p>
- You cannot directly cast a <codeph>STRING</codeph> value to <codeph>BOOLEAN</codeph>. You can use a
- <codeph>CASE</codeph> expression to evaluate string values such as <codeph>'T'</codeph>,
- <codeph>'true'</codeph>, and so on and return Boolean <codeph>true</codeph> and <codeph>false</codeph>
- values as appropriate.
+ You cannot directly cast a <codeph>STRING</codeph> value to
+ <codeph>BOOLEAN</codeph>. You can use a <codeph>CASE</codeph>
+ expression to evaluate string values such as <codeph>'T'</codeph>,
+ <codeph>'true'</codeph>, and so on and return Boolean
+ <codeph>true</codeph> and <codeph>false</codeph> values as
+ appropriate.
</p>
</li>
-
<li>
<p>
- You can cast a <codeph>BOOLEAN</codeph> value to <codeph>STRING</codeph>, returning <codeph>'1'</codeph>
- for <codeph>true</codeph> values and <codeph>'0'</codeph> for <codeph>false</codeph> values.
+ You can cast a <codeph>BOOLEAN</codeph> value to
+ <codeph>STRING</codeph>, returning <codeph>'1'</codeph> for
+ <codeph>true</codeph> values and <codeph>'0'</codeph> for
+ <codeph>false</codeph> values.
</p>
</li>
</ul>
-
<p conref="../shared/impala_common.xml#common/partitioning_blurb"/>
-
<p>
- Although it might be convenient to use <codeph>STRING</codeph> columns for partition keys, even when those
- columns contain numbers, for performance and scalability it is much better to use numeric columns as
- partition keys whenever practical. Although the underlying HDFS directory name might be the same in either
- case, the in-memory storage for the partition key columns is more compact, and computations are faster, if
- partition key columns such as <codeph>YEAR</codeph>, <codeph>MONTH</codeph>, <codeph>DAY</codeph> and so on
- are declared as <codeph>INT</codeph>, <codeph>SMALLINT</codeph>, and so on.
+ Although it might be convenient to use <codeph>STRING</codeph> columns
+ for partition keys, even when those columns contain numbers, for
+ performance and scalability it is much better to use numeric columns as
+ partition keys whenever practical. Although the underlying HDFS directory
+ name might be the same in either case, the in-memory storage for the
+ partition key columns is more compact, and computations are faster, if
+ partition key columns such as <codeph>YEAR</codeph>,
+ <codeph>MONTH</codeph>, <codeph>DAY</codeph> and so on are declared as
+ <codeph>INT</codeph>, <codeph>SMALLINT</codeph>, and so on.
</p>
-
<p conref="../shared/impala_common.xml#common/zero_length_strings"/>
-
-<!-- <p conref="../shared/impala_common.xml#common/hbase_blurb"/> -->
-
-<!-- <p conref="../shared/impala_common.xml#common/parquet_blurb"/> -->
-
+ <!-- <p conref="../shared/impala_common.xml#common/hbase_blurb"/> -->
+ <!-- <p conref="../shared/impala_common.xml#common/parquet_blurb"/> -->
<p conref="../shared/impala_common.xml#common/text_bulky"/>
-
<p><b>Avro considerations:</b></p>
<p conref="../shared/impala_common.xml#common/avro_2gb_strings"/>
-
-<!-- <p conref="../shared/impala_common.xml#common/compatibility_blurb"/> -->
-
-<!-- <p conref="../shared/impala_common.xml#common/internals_blurb"/> -->
-
-<!-- <p conref="../shared/impala_common.xml#common/added_in_20"/> -->
-
+ <!-- <p conref="../shared/impala_common.xml#common/compatibility_blurb"/> -->
+ <!-- <p conref="../shared/impala_common.xml#common/internals_blurb"/> -->
+ <!-- <p conref="../shared/impala_common.xml#common/added_in_20"/> -->
<p conref="../shared/impala_common.xml#common/column_stats_variable"/>
-
<p conref="../shared/impala_common.xml#common/example_blurb"/>
-
<p>
- The following examples demonstrate double-quoted and single-quoted string literals, and required escaping for
- quotation marks within string literals:
+ The following examples demonstrate double-quoted and single-quoted
+ string literals, and required escaping for quotation marks within string
+ literals:
</p>
-
-<codeblock>SELECT 'I am a single-quoted string';
+ <codeblock>SELECT 'I am a single-quoted string';
SELECT "I am a double-quoted string";
SELECT 'I\'m a single-quoted string with an apostrophe';
SELECT "I\'m a double-quoted string with an apostrophe";
SELECT 'I am a "short" single-quoted string containing quotes';
SELECT "I am a \"short\" double-quoted string containing quotes";
</codeblock>
-
<p>
- The following examples demonstrate calls to string manipulation functions to concatenate strings, convert
- numbers to strings, or pull out substrings:
+ The following examples demonstrate calls to string manipulation
+ functions to concatenate strings, convert numbers to strings, or pull out
+ substrings:
</p>
-
-<codeblock>SELECT CONCAT("Once upon a time, there were ", CAST(3 AS STRING), ' little pigs.');
+ <codeblock>SELECT CONCAT("Once upon a time, there were ", CAST(3 AS STRING), ' little pigs.');
SELECT SUBSTR("hello world",7,5);
</codeblock>
-
<p>
- The following examples show how to perform operations on <codeph>STRING</codeph> columns within a table:
+ The following examples show how to perform operations on
+ <codeph>STRING</codeph> columns within a table:
</p>
-
-<codeblock>CREATE TABLE t1 (s1 STRING, s2 STRING);
+ <codeblock>CREATE TABLE t1 (s1 STRING, s2 STRING);
INSERT INTO t1 VALUES ("hello", 'world'), (CAST(7 AS STRING), "wonders");
SELECT s1, s2, length(s1) FROM t1 WHERE s2 LIKE 'w%';
</codeblock>
-
<p conref="../shared/impala_common.xml#common/related_info"/>
-
<p>
- <xref href="impala_literals.xml#string_literals"/>, <xref href="impala_char.xml#char"/>,
- <xref href="impala_varchar.xml#varchar"/>, <xref href="impala_string_functions.xml#string_functions"/>,
- <xref href="impala_datetime_functions.xml#datetime_functions"/>
+ <xref href="impala_literals.xml#string_literals"/>, <xref
+ href="impala_char.xml#char"/>, <xref href="impala_varchar.xml#varchar"
+ />, <xref href="impala_string_functions.xml#string_functions"/>, <xref
+ href="impala_datetime_functions.xml#datetime_functions"/>
</p>
</conbody>
</concept>
[3/5] impala git commit: IMPALA-7071: make get_fs_path() idempotent
Posted by ta...@apache.org.
IMPALA-7071: make get_fs_path() idempotent
This avoids future errors like IMPALA-7068.
Change-Id: I10c7cbb36119883718e60660db2ac7313f14d388
Reviewed-on: http://gerrit.cloudera.org:8080/10517
Reviewed-by: Dan Hecht <dh...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
Project: http://git-wip-us.apache.org/repos/asf/impala/repo
Commit: http://git-wip-us.apache.org/repos/asf/impala/commit/6cdd4bdf
Tree: http://git-wip-us.apache.org/repos/asf/impala/tree/6cdd4bdf
Diff: http://git-wip-us.apache.org/repos/asf/impala/diff/6cdd4bdf
Branch: refs/heads/2.x
Commit: 6cdd4bdf0fcc652181b255d2f5b14d4a37edd789
Parents: 99f43a2
Author: Dan Hecht <dh...@cloudera.com>
Authored: Thu May 24 14:03:50 2018 -0700
Committer: Impala Public Jenkins <im...@gerrit.cloudera.org>
Committed: Fri Jun 8 02:29:26 2018 +0000
----------------------------------------------------------------------
tests/infra/test_utils.py | 31 +++++++++++++++++++++++++++++++
tests/util/filesystem_utils.py | 8 ++++++--
2 files changed, 37 insertions(+), 2 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/impala/blob/6cdd4bdf/tests/infra/test_utils.py
----------------------------------------------------------------------
diff --git a/tests/infra/test_utils.py b/tests/infra/test_utils.py
new file mode 100644
index 0000000..c998703
--- /dev/null
+++ b/tests/infra/test_utils.py
@@ -0,0 +1,31 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# This module contains tests for some of the tests/util code.
+
+from tests.util.filesystem_utils import prepend_with_fs
+
+def test_filesystem_utils():
+ # Verify that empty FS prefix gives back the same path.
+ path = "/fake-warehouse"
+ assert prepend_with_fs("", path) == path
+
+ # Verify that prepend_with_fs() is idempotent.
+ fs = "fakeFs://bucket"
+ path = "/fake-warehouse"
+ assert prepend_with_fs(fs, path) == fs + path
+ assert prepend_with_fs(fs, prepend_with_fs(fs, path)) == fs + path
http://git-wip-us.apache.org/repos/asf/impala/blob/6cdd4bdf/tests/util/filesystem_utils.py
----------------------------------------------------------------------
diff --git a/tests/util/filesystem_utils.py b/tests/util/filesystem_utils.py
index 77112be..e107e57 100644
--- a/tests/util/filesystem_utils.py
+++ b/tests/util/filesystem_utils.py
@@ -49,10 +49,14 @@ ADLS_CLIENT_ID = os.getenv("azure_client_id")
ADLS_TENANT_ID = os.getenv("azure_tenant_id")
ADLS_CLIENT_SECRET = os.getenv("azure_client_secret")
+def prepend_with_fs(fs, path):
+ """Prepend 'path' with 'fs' if it's not already the prefix."""
+ return path if path.startswith(fs) else "%s%s" % (fs, path)
+
def get_fs_path(path):
- return "%s%s" % (FILESYSTEM_PREFIX, path)
+ return prepend_with_fs(FILESYSTEM_PREFIX, path)
def get_secondary_fs_path(path):
- return "%s%s" % (SECONDARY_FILESYSTEM, path)
+ return prepend_with_fs(SECONDARY_FILESYSTEM, path)
WAREHOUSE = get_fs_path('/test-warehouse')
[4/5] impala git commit: IMPALA-7099: Don't set FILESYSTEM_PREFIX for
s3
Posted by ta...@apache.org.
IMPALA-7099: Don't set FILESYSTEM_PREFIX for s3
When S3 support was originally added to Impala, Impala still required
that the defaultFs point to HDFS, and so the S3 tests were originally
run as a "secondary" filesystem (which meant that all paths had to be
fully qualified).
Since then, the restriction on defaultFs has been addressed, and
additionally, the S3 test configuration had been changed to run with S3
as the defaultFs. So, setting FILESYSTEM_PREFIX for S3 is no longer
necessary.
Let's remove that so that S3 tests run in a configuration more similar
to HDFS, which will help prevent them from breaking as often (since
precheckin tests run only in the HDFS configuration).
Incidently, this addresses the problem with
test_unsupported_text_compression on S3 since the problem is with the
handling of fully qualified paths.
Testing:
- core on S3
Change-Id: Iace5fc9557ff6e7d184573beafea694191424742
Reviewed-on: http://gerrit.cloudera.org:8080/10595
Reviewed-by: Dan Hecht <dh...@cloudera.com>
Tested-by: Dan Hecht <dh...@cloudera.com>
Project: http://git-wip-us.apache.org/repos/asf/impala/repo
Commit: http://git-wip-us.apache.org/repos/asf/impala/commit/7662172c
Tree: http://git-wip-us.apache.org/repos/asf/impala/tree/7662172c
Diff: http://git-wip-us.apache.org/repos/asf/impala/diff/7662172c
Branch: refs/heads/2.x
Commit: 7662172c9eb810968af0ce9119d96484e3d3c55a
Parents: 6cdd4bd
Author: Dan Hecht <dh...@cloudera.com>
Authored: Fri Jun 1 10:02:23 2018 -0700
Committer: Impala Public Jenkins <im...@gerrit.cloudera.org>
Committed: Fri Jun 8 02:29:26 2018 +0000
----------------------------------------------------------------------
bin/impala-config.sh | 1 -
1 file changed, 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/impala/blob/7662172c/bin/impala-config.sh
----------------------------------------------------------------------
diff --git a/bin/impala-config.sh b/bin/impala-config.sh
index c8c99d1..779cf10 100755
--- a/bin/impala-config.sh
+++ b/bin/impala-config.sh
@@ -369,7 +369,6 @@ if [ "${TARGET_FILESYSTEM}" = "s3" ]; then
if ${IMPALA_HOME}/bin/check-s3-access.sh; then
export S3_ACCESS_VALIDATED=1
export DEFAULT_FS="s3a://${S3_BUCKET}"
- export FILESYSTEM_PREFIX="${DEFAULT_FS}"
else
return 1
fi