You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@impala.apache.org by sa...@apache.org on 2016/08/11 03:45:38 UTC
[1/4] incubator-impala git commit: IMPALA-3843: Update warning for
non-SSSE3 CPUs
Repository: incubator-impala
Updated Branches:
refs/heads/master 27b3b4d47 -> ac4f22b1b
IMPALA-3843: Update warning for non-SSSE3 CPUs
As described in IMPALA-3843, the old message for non-SSSE3 CPUs implies
that Impala will exit without SSSE3. This is no longer the case after
IMPALA-1399 and IMPALA-1646 have been resolved. The new message merely
warns that there might be problems (and SSSE3 might be the culprit).
Change-Id: Id242b310715c7abea091a35e6a89647bded26c4b
Reviewed-on: http://gerrit.cloudera.org:8080/3884
Reviewed-by: Tim Armstrong <ta...@cloudera.com>
Tested-by: Internal Jenkins
Project: http://git-wip-us.apache.org/repos/asf/incubator-impala/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-impala/commit/b6934f0f
Tree: http://git-wip-us.apache.org/repos/asf/incubator-impala/tree/b6934f0f
Diff: http://git-wip-us.apache.org/repos/asf/incubator-impala/diff/b6934f0f
Branch: refs/heads/master
Commit: b6934f0fcd4cafc737ee44d9676657406511c162
Parents: 27b3b4d
Author: Tomas Gavenciak <to...@nic.cz>
Authored: Wed Aug 3 07:16:25 2016 -0700
Committer: Internal Jenkins <cl...@gerrit.cloudera.org>
Committed: Wed Aug 10 21:31:38 2016 +0000
----------------------------------------------------------------------
be/src/util/cpu-info.cc | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/b6934f0f/be/src/util/cpu-info.cc
----------------------------------------------------------------------
diff --git a/be/src/util/cpu-info.cc b/be/src/util/cpu-info.cc
index 573b131..2a98102 100644
--- a/be/src/util/cpu-info.cc
+++ b/be/src/util/cpu-info.cc
@@ -137,8 +137,8 @@ void CpuInfo::Init() {
void CpuInfo::VerifyCpuRequirements() {
if (!CpuInfo::IsSupported(CpuInfo::SSSE3)) {
- LOG(ERROR) << "CPU does not support the Supplemental SSE3 (SSSE3) instruction set, "
- << "which is required. Exiting if Supplemental SSE3 is not functional...";
+ LOG(ERROR) << "CPU does not support the Supplemental SSE3 (SSSE3) instruction set. "
+ << "This setup is generally unsupported and Impala might be unstable.";
}
}
[2/4] incubator-impala git commit: IMPALA-3786: Remove "Cloudera"
from impalad webpage title
Posted by sa...@apache.org.
IMPALA-3786: Remove "Cloudera" from impalad webpage title
Change-Id: Idc4d57792bcce65ad1756d018ed9f245ff732b37
Reviewed-on: http://gerrit.cloudera.org:8080/3885
Reviewed-by: Henry Robinson <he...@cloudera.com>
Tested-by: Internal Jenkins
Project: http://git-wip-us.apache.org/repos/asf/incubator-impala/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-impala/commit/16a50f79
Tree: http://git-wip-us.apache.org/repos/asf/incubator-impala/tree/16a50f79
Diff: http://git-wip-us.apache.org/repos/asf/incubator-impala/diff/16a50f79
Branch: refs/heads/master
Commit: 16a50f79ee76936734bd7a8f80d5665428988247
Parents: b6934f0
Author: Jim Apple <jb...@cloudera.com>
Authored: Wed Aug 10 10:44:58 2016 -0700
Committer: Internal Jenkins <cl...@gerrit.cloudera.org>
Committed: Wed Aug 10 22:48:47 2016 +0000
----------------------------------------------------------------------
www/common-header.tmpl | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/16a50f79/www/common-header.tmpl
----------------------------------------------------------------------
diff --git a/www/common-header.tmpl b/www/common-header.tmpl
index d5897df..5536bda 100644
--- a/www/common-header.tmpl
+++ b/www/common-header.tmpl
@@ -20,7 +20,7 @@ under the License.
common-footer.tmpl) }}
<!DOCTYPE html>
<html>
- <head><title>Cloudera Impala</title>
+ <head><title>Apache Impala (incubating)</title>
<link href='/www/bootstrap/css/bootstrap.min.css' rel='stylesheet' media='screen'>
<style>
</style>
[3/4] incubator-impala git commit: IMPALA-3923: fix overflow in
BufferedTupleStream::GetRows()
Posted by sa...@apache.org.
IMPALA-3923: fix overflow in BufferedTupleStream::GetRows()
Testing:
Added a targeted unit test.
Change-Id: Iabb977014d855d6edb035bf2e9519ac8b3234767
Reviewed-on: http://gerrit.cloudera.org:8080/3878
Reviewed-by: Tim Armstrong <ta...@cloudera.com>
Tested-by: Internal Jenkins
Project: http://git-wip-us.apache.org/repos/asf/incubator-impala/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-impala/commit/e6702c67
Tree: http://git-wip-us.apache.org/repos/asf/incubator-impala/tree/e6702c67
Diff: http://git-wip-us.apache.org/repos/asf/incubator-impala/diff/e6702c67
Branch: refs/heads/master
Commit: e6702c67309cae4b46da6de935bcf873800a2f3c
Parents: 16a50f7
Author: Tim Armstrong <ta...@cloudera.com>
Authored: Wed Jul 27 13:50:23 2016 -0700
Committer: Internal Jenkins <cl...@gerrit.cloudera.org>
Committed: Thu Aug 11 01:31:37 2016 +0000
----------------------------------------------------------------------
be/src/runtime/buffered-tuple-stream-test.cc | 18 ++++++++++++++++++
be/src/runtime/buffered-tuple-stream.cc | 5 +++++
be/src/runtime/buffered-tuple-stream.h | 3 ++-
3 files changed, 25 insertions(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/e6702c67/be/src/runtime/buffered-tuple-stream-test.cc
----------------------------------------------------------------------
diff --git a/be/src/runtime/buffered-tuple-stream-test.cc b/be/src/runtime/buffered-tuple-stream-test.cc
index 7d28e69..558b617 100644
--- a/be/src/runtime/buffered-tuple-stream-test.cc
+++ b/be/src/runtime/buffered-tuple-stream-test.cc
@@ -750,6 +750,24 @@ TEST_F(SimpleTupleStreamTest, BigRow) {
nullable_stream.Close();
}
+// Test for IMPALA-3923: overflow of 32-bit int in GetRows().
+TEST_F(SimpleTupleStreamTest, TestGetRowsOverflow) {
+ InitBlockMgr(-1, 8 * 1024 * 1024);
+ BufferedTupleStream stream(runtime_state_, *int_desc_, runtime_state_->block_mgr(),
+ client_, false, false);
+ ASSERT_OK(stream.Init(-1, NULL, true));
+
+ Status status;
+ // Add more rows than can be fit in a RowBatch (limited by its 32-bit row count).
+ // Actually adding the rows would take a very long time, so just set num_rows_.
+ // This puts the stream in an inconsistent state, but exercises the right code path.
+ stream.num_rows_ = 1L << 33;
+ bool got_rows;
+ scoped_ptr<RowBatch> overflow_batch;
+ ASSERT_FALSE(stream.GetRows(&overflow_batch, &got_rows).ok());
+ stream.Close();
+}
+
// Basic API test. No data should be going to disk.
TEST_F(SimpleNullStreamTest, Basic) {
InitBlockMgr(-1, IO_BLOCK_SIZE);
http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/e6702c67/be/src/runtime/buffered-tuple-stream.cc
----------------------------------------------------------------------
diff --git a/be/src/runtime/buffered-tuple-stream.cc b/be/src/runtime/buffered-tuple-stream.cc
index 263071e..978a221 100644
--- a/be/src/runtime/buffered-tuple-stream.cc
+++ b/be/src/runtime/buffered-tuple-stream.cc
@@ -503,6 +503,11 @@ int BufferedTupleStream::ComputeNumNullIndicatorBytes(int block_size) const {
}
Status BufferedTupleStream::GetRows(scoped_ptr<RowBatch>* batch, bool* got_rows) {
+ if (num_rows() > numeric_limits<int>::max()) {
+ // RowBatch::num_rows_ is a 32-bit int, avoid an overflow.
+ return Status(Substitute("Trying to read $0 rows into in-memory batch failed. Limit "
+ "is $1", num_rows(), numeric_limits<int>::max()));
+ }
RETURN_IF_ERROR(PinStream(false, got_rows));
if (!*got_rows) return Status::OK();
bool got_read_buffer;
http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/e6702c67/be/src/runtime/buffered-tuple-stream.h
----------------------------------------------------------------------
diff --git a/be/src/runtime/buffered-tuple-stream.h b/be/src/runtime/buffered-tuple-stream.h
index c5d9793..e14f16c 100644
--- a/be/src/runtime/buffered-tuple-stream.h
+++ b/be/src/runtime/buffered-tuple-stream.h
@@ -317,9 +317,10 @@ class BufferedTupleStream {
std::string DebugString() const;
private:
- friend class MultiNullableTupleStreamTest_TestComputeRowSize_Test;
friend class ArrayTupleStreamTest_TestArrayDeepCopy_Test;
friend class ArrayTupleStreamTest_TestComputeRowSize_Test;
+ friend class MultiNullableTupleStreamTest_TestComputeRowSize_Test;
+ friend class SimpleTupleStreamTest_TestGetRowsOverflow_Test;
/// If true, this stream is still using small buffers.
bool use_small_buffers_;
[4/4] incubator-impala git commit: IMPALA-3957: Test failure in S3
build: TestLoadData.test_load
Posted by sa...@apache.org.
IMPALA-3957: Test failure in S3 build: TestLoadData.test_load
The test_load() test failed with an error which said that the number of
of files in the destination was wrong. This could probably be because
the filesystem_client.copy() (for S3) in the setup_method() silently
failed without copying one of the files as a one off error.
I'm not sure why S3 failed to do the copy, but this patch adds an
assert after the copy to make sure that if it the s3_client.copy()
fails to do the copy, it will assert instead of continuing with the
rest of the tests.
Change-Id: I966a469e94099d3d971e470ae6e992386070c5e9
Reviewed-on: http://gerrit.cloudera.org:8080/3881
Reviewed-by: Sailesh Mukil <sa...@cloudera.com>
Tested-by: Internal Jenkins
Project: http://git-wip-us.apache.org/repos/asf/incubator-impala/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-impala/commit/ac4f22b1
Tree: http://git-wip-us.apache.org/repos/asf/incubator-impala/tree/ac4f22b1
Diff: http://git-wip-us.apache.org/repos/asf/incubator-impala/diff/ac4f22b1
Branch: refs/heads/master
Commit: ac4f22b1b0a68600884358d373fffbcd817f22b8
Parents: e6702c6
Author: Sailesh Mukil <sa...@cloudera.com>
Authored: Tue Aug 9 22:10:54 2016 -0700
Committer: Internal Jenkins <cl...@gerrit.cloudera.org>
Committed: Thu Aug 11 03:06:09 2016 +0000
----------------------------------------------------------------------
tests/util/s3_util.py | 2 ++
1 file changed, 2 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/ac4f22b1/tests/util/s3_util.py
----------------------------------------------------------------------
diff --git a/tests/util/s3_util.py b/tests/util/s3_util.py
index 549a5c9..ec5cfdf 100644
--- a/tests/util/s3_util.py
+++ b/tests/util/s3_util.py
@@ -45,6 +45,8 @@ class S3Client(BaseFilesystem):
def copy(self, src, dst):
self.s3client.copy_object(Bucket=self.bucketname,
CopySource={'Bucket':self.bucketname, 'Key':src}, Key=dst)
+ assert self.exists(dst), \
+ 'S3 copy failed: Destination file {dst} does not exist'.format(dst=dst)
# Since S3 is a key-value store, it does not have a command like 'ls' for a directory
# structured filesystem. It lists everything under a path recursively.