You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@impala.apache.org by ar...@apache.org on 2019/07/25 17:45:18 UTC
[impala] branch master updated (b2136c3 -> 6d4cde7)
This is an automated email from the ASF dual-hosted git repository.
arodoni pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/impala.git.
from b2136c3 IMPALA-8615: [DOCS] Document the scalable admission control parameters
new e8b2282 [DOCS] Specified the client flags should be on coordinator
new 1c18a46 IMPALA-5031: Fix undefined behavior: ptr overflow
new 33b5c3d IMPALA-8772: Import Testcase failed for SQL without table refs
new 6d4cde7 IMPALA-8641: Document compression codec zstd in Parquet
The 4 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
be/src/exec/parquet/hdfs-parquet-scanner.cc | 11 ++--
docs/topics/impala_client.xml | 3 +-
docs/topics/impala_compression_codec.xml | 23 +++++++-
docs/topics/impala_parquet.xml | 8 +--
.../apache/impala/service/CatalogOpExecutor.java | 66 ++++++++++++----------
tests/metadata/test_testcase_builder.py | 57 +++++++++++++++++++
6 files changed, 124 insertions(+), 44 deletions(-)
create mode 100644 tests/metadata/test_testcase_builder.py
[impala] 04/04: IMPALA-8641: Document compression codec zstd in
Parquet
Posted by ar...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
arodoni pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/impala.git
commit 6d4cde7c926e0c002f976697d8437165cca4c247
Author: Abhishek <ar...@cloudera.com>
AuthorDate: Sat Jun 8 09:06:20 2019 -0700
IMPALA-8641: Document compression codec zstd in Parquet
Updated the documentation for query_option compression_codec to also
include zstd.
Change-Id: Id55dbc3297ec1560b04e4da2c93cc1aeb1e6fb2f
Reviewed-on: http://gerrit.cloudera.org:8080/13910
Tested-by: Impala Public Jenkins <im...@cloudera.com>
Reviewed-by: Alex Rodoni <ar...@cloudera.com>
---
docs/topics/impala_compression_codec.xml | 23 ++++++++++++++++++++---
docs/topics/impala_parquet.xml | 8 ++++----
2 files changed, 24 insertions(+), 7 deletions(-)
diff --git a/docs/topics/impala_compression_codec.xml b/docs/topics/impala_compression_codec.xml
index 07caad3..8808bb7 100644
--- a/docs/topics/impala_compression_codec.xml
+++ b/docs/topics/impala_compression_codec.xml
@@ -31,6 +31,7 @@ under the License.
<data name="Category" value="Parquet"/>
<data name="Category" value="Snappy"/>
<data name="Category" value="Gzip"/>
+ <data name="Category" value="Zstd"/>
<data name="Category" value="Developers"/>
<data name="Category" value="Data Analysts"/>
</metadata>
@@ -56,11 +57,20 @@ under the License.
<p conref="../shared/impala_common.xml#common/syntax_blurb"/>
-<codeblock>SET COMPRESSION_CODEC=<varname>codec_name</varname>;</codeblock>
+<codeblock>SET COMPRESSION_CODEC=<varname>codec_name</varname>; // Supported for all codecs.
+SET COMPRESSION_CODEC=<varname>codec_name</varname>:<varname>compression_level</varname>; // Only supported for ZSTD.
+</codeblock>
<p>
The allowed values for this query option are <codeph>SNAPPY</codeph> (the default), <codeph>GZIP</codeph>,
- and <codeph>NONE</codeph>.
+ <codeph>ZSTD</codeph>, and <codeph>NONE</codeph>.
+ </p>
+
+ <p>
+ <codeph>ZSTD</codeph> also supports setting a compression level. The lower the level, the faster the speed at
+ the cost of compression ratio. Compression levels from 1 up to 22 are supported for <codeph>ZSTD</codeph>.
+ The default compression level 3 is used, if one is not passed using the <codeph>compression_codec</codeph>
+ query option.
</p>
<note>
@@ -92,7 +102,14 @@ under the License.
<p conref="../shared/impala_common.xml#common/example_blurb"/>
-<codeblock>set compression_codec=gzip;
+<codeblock>
+set compression_codec=zstd; // Default compression level 3.
+insert into parquet_table_zstd_default_compressed select * from t1;
+
+set compression_codec=zstd:12; // Compression level 12.
+insert into parquet_table_zstd_highly_compressed select * from t1;
+
+set compression_codec=gzip;
insert into parquet_table_highly_compressed select * from t1;
set compression_codec=snappy;
diff --git a/docs/topics/impala_parquet.xml b/docs/topics/impala_parquet.xml
index 9a6b01b..1a19841 100644
--- a/docs/topics/impala_parquet.xml
+++ b/docs/topics/impala_parquet.xml
@@ -449,10 +449,10 @@ under the License.
underlying compression is controlled by the <codeph>COMPRESSION_CODEC</codeph> query
option. (Prior to Impala 2.0, the query option name was
<codeph>PARQUET_COMPRESSION_CODEC</codeph>.) The allowed values for this query option
- are <codeph>snappy</codeph> (the default), <codeph>gzip</codeph>, and
- <codeph>none</codeph>. The option value is not case-sensitive. If the option is set to
- an unrecognized value, all kinds of queries will fail due to the invalid option setting,
- not just queries involving Parquet tables.
+ are <codeph>snappy</codeph> (the default), <codeph>gzip</codeph>, <codeph>zstd</codeph>
+ and <codeph>none</codeph>. The option value is not case-sensitive. If the option is set
+ to an unrecognized value, all kinds of queries will fail due to the invalid option
+ setting, not just queries involving Parquet tables.
</p>
</conbody>
[impala] 01/04: [DOCS] Specified the client flags should be on
coordinator
Posted by ar...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
arodoni pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/impala.git
commit e8b2282985320737e292778a01ffd46640f5bfc6
Author: Alex Rodoni <ar...@cloudera.com>
AuthorDate: Wed Jul 24 17:09:08 2019 -0700
[DOCS] Specified the client flags should be on coordinator
Change-Id: Ib48416a336eb7d9b226621e32e0b72e3ec31f778
Reviewed-on: http://gerrit.cloudera.org:8080/13914
Reviewed-by: Alex Rodoni <ar...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
docs/topics/impala_client.xml | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/docs/topics/impala_client.xml b/docs/topics/impala_client.xml
index 802ed88..099ba4a 100644
--- a/docs/topics/impala_client.xml
+++ b/docs/topics/impala_client.xml
@@ -76,7 +76,8 @@ under the License.
<title>Impala Startup Options for Client Connections</title>
<p>
- The following options control client connections to Impala.
+ Use the following flags when starting Impala Daemon coordinator to control client
+ connections to Impala.
</p>
<dl>
[impala] 03/04: IMPALA-8772: Import Testcase failed for SQL without
table refs
Posted by ar...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
arodoni pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/impala.git
commit 33b5c3dcbe4d0bd574122af5afd8af8a07ddc78a
Author: Jiawei Wang <ji...@cloudera.com>
AuthorDate: Mon Jul 22 14:28:49 2019 -0700
IMPALA-8772: Import Testcase failed for SQL without table refs
Description:
Query like this: select 5 * 4; can generate valid testcase, but
cannot be loaded because it does not involve any table and view
references.
Fix:
Add null check for CatalogOpExecutor when doing testcase data load
Tests:
tests/metadata/test_testcase_builder
Change-Id: I50d77d7de44bb140598a1e8db4da85a5ec87f31e
TODO: NEED TO DELETE GENERATED TESTCASE AFTER THIS IS DONE
Reviewed-on: http://gerrit.cloudera.org:8080/13893
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
.../apache/impala/service/CatalogOpExecutor.java | 66 ++++++++++++----------
tests/metadata/test_testcase_builder.py | 57 +++++++++++++++++++
2 files changed, 92 insertions(+), 31 deletions(-)
diff --git a/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java b/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
index 0a15915..0951349 100644
--- a/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
+++ b/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
@@ -445,42 +445,46 @@ public class CatalogOpExecutor {
// the state in-memory and do not flush it to HMS, the older state can be recovered
// by loading everything back from HMS. For ex: INVALIDATE METADATA.
int numDbsAdded = 0;
- for (TDatabase thriftDb: testCaseData.getDbs()) {
- Db db = Db.fromTDatabase(thriftDb);
- // Set a new version to force an overwrite if a Db already exists with the same
- // name.
- db.setCatalogVersion(catalog_.incrementAndGetCatalogVersion());
- Db ret = catalog_.addDb(db.getName(), db.getMetaStoreDb());
- if (ret != null) {
- ++numDbsAdded;
- response.result.addToUpdated_catalog_objects(db.toTCatalogObject());
+ if (testCaseData.getDbs() != null) {
+ for (TDatabase thriftDb : testCaseData.getDbs()) {
+ Db db = Db.fromTDatabase(thriftDb);
+ // Set a new version to force an overwrite if a Db already exists with the same
+ // name.
+ db.setCatalogVersion(catalog_.incrementAndGetCatalogVersion());
+ Db ret = catalog_.addDb(db.getName(), db.getMetaStoreDb());
+ if (ret != null) {
+ ++numDbsAdded;
+ response.result.addToUpdated_catalog_objects(db.toTCatalogObject());
+ }
}
}
int numTblsAdded = 0;
int numViewsAdded = 0;
- for(TTable tTable: testCaseData.tables_and_views) {
- Db db = catalog_.getDb(tTable.db_name);
- // Db should have been created by now.
- Preconditions.checkNotNull(db, String.format("Missing db %s", tTable.db_name));
- Table t = Table.fromThrift(db, tTable);
- // Set a new version to force an overwrite if a table already exists with the same
- // name.
- t.setCatalogVersion(catalog_.incrementAndGetCatalogVersion());
- catalog_.addTable(db, t);
- if (t instanceof View) {
- ++numViewsAdded;
- } else {
- ++numTblsAdded;
- }
- // The table lock is needed here since toTCatalogObject() calls Table#toThrift()
- // which expects the current thread to hold this lock. For more details refer
- // to IMPALA-4092.
- t.getLock().lock();
- try {
- response.result.addToUpdated_catalog_objects(t.toTCatalogObject());
- } finally {
- t.getLock().unlock();
+ if (testCaseData.getTables_and_views() != null) {
+ for (TTable tTable : testCaseData.tables_and_views) {
+ Db db = catalog_.getDb(tTable.db_name);
+ // Db should have been created by now.
+ Preconditions.checkNotNull(db, String.format("Missing db %s", tTable.db_name));
+ Table t = Table.fromThrift(db, tTable);
+ // Set a new version to force an overwrite if a table already exists with the same
+ // name.
+ t.setCatalogVersion(catalog_.incrementAndGetCatalogVersion());
+ catalog_.addTable(db, t);
+ if (t instanceof View) {
+ ++numViewsAdded;
+ } else {
+ ++numTblsAdded;
+ }
+ // The table lock is needed here since toTCatalogObject() calls Table#toThrift()
+ // which expects the current thread to hold this lock. For more details refer
+ // to IMPALA-4092.
+ t.getLock().lock();
+ try {
+ response.result.addToUpdated_catalog_objects(t.toTCatalogObject());
+ } finally {
+ t.getLock().unlock();
+ }
}
}
StringBuilder responseStr = new StringBuilder();
diff --git a/tests/metadata/test_testcase_builder.py b/tests/metadata/test_testcase_builder.py
new file mode 100644
index 0000000..242eae9
--- /dev/null
+++ b/tests/metadata/test_testcase_builder.py
@@ -0,0 +1,57 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from tests.common.impala_test_suite import ImpalaTestSuite
+from tests.common.test_dimensions import (
+ create_single_exec_option_dimension,
+ create_uncompressed_text_dimension)
+
+
+class TestTestcaseBuilder(ImpalaTestSuite):
+
+ @classmethod
+ def get_workload(self):
+ return 'functional-query'
+
+ @classmethod
+ def add_test_dimensions(cls):
+ super(TestTestcaseBuilder, cls).add_test_dimensions()
+ # This test only needs to be run once.
+ cls.ImpalaTestMatrix.add_dimension(create_single_exec_option_dimension())
+ cls.ImpalaTestMatrix.add_dimension(
+ create_uncompressed_text_dimension(cls.get_workload()))
+
+ def test_query_without_from(self):
+ # Generate Testcase Data for query without table reference
+ testcase_generate_query = """COPY TESTCASE TO 'hdfs:///tmp' SELECT 5 * 20"""
+ result = self.execute_query_expect_success(self.client, testcase_generate_query)
+ assert len(result.data) == 1, "Testcase builder wrong result: {0}".format(result.data)
+
+ # Check file exists
+ testcase_path = str(result.data)[1: -1]
+ index = testcase_path.index('/tmp')
+ hdfs_path = testcase_path[index:-1]
+ assert self.hdfs_client.exists(hdfs_path), "File not generated {0}".format(hdfs_path)
+
+ try:
+ # Test load testcase works
+ testcase_load_query = "COPY TESTCASE FROM {0}".format(testcase_path)
+ self.execute_query_expect_success(self.client, testcase_load_query)
+ finally:
+ # Delete testcase file from tmp
+ status = self.hdfs_client.delete_file_dir(hdfs_path)
+ assert status, "Delete generated testcase file failed with {0}".format(status)
[impala] 02/04: IMPALA-5031: Fix undefined behavior: ptr overflow
Posted by ar...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
arodoni pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/impala.git
commit 1c18a46c16484c596404f8d2661585ce0e53e187
Author: Jim Apple <jb...@apache.org>
AuthorDate: Sun Jul 21 17:23:42 2019 -0700
IMPALA-5031: Fix undefined behavior: ptr overflow
In expr.add, the standard says:
When an expression that has integral type is added to or
subtracted from a pointer, the result has the type of the pointer
operand. ... If both the pointer operand and the result point to
elements of the same array object, or one past the last element of
the array object, the evaluation shall not produce an overflow;
otherwise, the behavior is undefined.
This is triggered in the end-to-end tests.h The interesting part of
the backtrace is:
exec/parquet/hdfs-parquet-scanner.cc:1405:45: runtime error: pointer
index expression with base 0x00001300e0e9 overflowed to
0xffffffff1300e0ea
#0 HdfsParquetScanner::ProcessFooter()
exec/parquet/hdfs-parquet-scanner.cc:1405:45
#1 HdfsParquetScanner::Open(ScannerContext*)
exec/parquet/hdfs-parquet-scanner.cc:186:26
#2 HdfsScanNodeBase::CreateAndOpenScannerHelper(
HdfsPartitionDescriptor*, ScannerContext*,
scoped_ptr<HdfsScanner>*) exec/hdfs-scan-node-base.cc:721:59
#3 HdfsScanNodeMt::CreateAndOpenScanner(HdfsPartitionDescriptor*,
ScannerContext*, scoped_ptr<HdfsScanner>*)
exec/hdfs-scan-node-mt.cc:127:19
#4 HdfsScanNodeMt::GetNext(RuntimeState*, RowBatch*, bool*)
exec/hdfs-scan-node-mt.cc:97:21
Change-Id: I81c7db75b564045106edf3d46e2c4a62be77359f
Reviewed-on: http://gerrit.cloudera.org:8080/13889
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
be/src/exec/parquet/hdfs-parquet-scanner.cc | 11 ++++++-----
1 file changed, 6 insertions(+), 5 deletions(-)
diff --git a/be/src/exec/parquet/hdfs-parquet-scanner.cc b/be/src/exec/parquet/hdfs-parquet-scanner.cc
index fe6e724..35c2e70 100644
--- a/be/src/exec/parquet/hdfs-parquet-scanner.cc
+++ b/be/src/exec/parquet/hdfs-parquet-scanner.cc
@@ -1402,11 +1402,16 @@ Status HdfsParquetScanner::ProcessFooter() {
// the magic number
uint8_t* metadata_size_ptr = magic_number_ptr - sizeof(int32_t);
uint32_t metadata_size = *reinterpret_cast<uint32_t*>(metadata_size_ptr);
- uint8_t* metadata_ptr = metadata_size_ptr - metadata_size;
// The start of the metadata is:
// file_len - 4-byte footer length field - 4-byte version number field - metadata size
int64_t metadata_start = file_len - sizeof(int32_t) - sizeof(PARQUET_VERSION_NUMBER) -
metadata_size;
+ if (UNLIKELY(metadata_start < 0)) {
+ return Status(Substitute("File '$0' is invalid. Invalid metadata size in file "
+ "footer: $1 bytes. File size: $2 bytes.",
+ filename(), metadata_size, file_len));
+ }
+ uint8_t* metadata_ptr = metadata_size_ptr - metadata_size;
// If the metadata was too big, we need to read it into a contiguous buffer before
// deserializing it.
@@ -1420,10 +1425,6 @@ Status HdfsParquetScanner::ProcessFooter() {
int64_t partition_id = context_->partition_descriptor()->id();
const HdfsFileDesc* file_desc = scan_node_->GetFileDesc(partition_id, filename());
DCHECK_EQ(file_desc, stream_->file_desc());
- if (metadata_start < 0) {
- return Status(Substitute("File '$0' is invalid. Invalid metadata size in file "
- "footer: $1 bytes. File size: $2 bytes.", filename(), metadata_size, file_len));
- }
if (!metadata_buffer.TryAllocate(metadata_size)) {
string details = Substitute("Could not allocate buffer of $0 bytes for Parquet "