You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@impala.apache.org by st...@apache.org on 2022/03/02 02:01:30 UTC
[impala] 02/02: IMPALA-11120: Fix codec not set in generating ORC tables
This is an automated email from the ASF dual-hosted git repository.
stigahuang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/impala.git
commit b2e4b29f06141ad34eef2cbadfda259124792ac2
Author: stiga-huang <hu...@gmail.com>
AuthorDate: Mon Feb 14 11:07:12 2022 +0800
IMPALA-11120: Fix codec not set in generating ORC tables
We use 'mapred.output.compression.codec' to set the compression codec in
generating test files by Hive. However, it doesn't affect ORC files.
Instead, we need to set 'orc.compress' in tblproperties for each ORC
tables. The default value of 'orc.compress' is ZLIB which corresponds to
our 'def' codec. We only need to set it for non-def codecs.
This patch also fixes a bug in build_compression_codec_statement() that
would raise KeyError when loading lz4 non-avro tables.
Tests
- Loaded tpch data in orc/none/none, orc/def/block, orc/snap/block,
orc/lz4/block and verified there compression codecs.
Change-Id: I02bd5d9400864145133ff019a3d076a6cab36fcc
Reviewed-on: http://gerrit.cloudera.org:8080/18228
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
testdata/bin/generate-schema-statements.py | 10 +++++++++-
1 file changed, 9 insertions(+), 1 deletion(-)
diff --git a/testdata/bin/generate-schema-statements.py b/testdata/bin/generate-schema-statements.py
index 061601f..e9b52b4 100755
--- a/testdata/bin/generate-schema-statements.py
+++ b/testdata/bin/generate-schema-statements.py
@@ -426,7 +426,7 @@ def avro_schema(columns):
return json.dumps(record)
def build_compression_codec_statement(codec, compression_type, file_format):
- codec = AVRO_COMPRESSION_MAP[codec] if file_format == 'avro' else COMPRESSION_MAP[codec]
+ codec = (AVRO_COMPRESSION_MAP if file_format == 'avro' else COMPRESSION_MAP).get(codec)
if not codec:
return str()
return (AVRO_COMPRESSION_CODEC % codec) if file_format == 'avro' else (
@@ -688,6 +688,14 @@ def generate_statements(output_name, test_vectors, sections,
create_file_format == 'orc' and
'transactional' not in tblproperties):
tblproperties['transactional'] = 'true'
+ if create_file_format == 'orc' and create_codec != 'def':
+ # The default value of 'orc.compress' is ZLIB which corresponds to 'def'.
+ # We just need it for non-def codec.
+ # The original codec name can be used except snap.
+ if create_codec == 'snap':
+ tblproperties['orc.compress'] = 'SNAPPY'
+ else:
+ tblproperties['orc.compress'] = create_codec
hdfs_location = '{0}.{1}{2}'.format(db_name, table_name, db_suffix)
# hdfs file names for functional datasets are stored