You are viewing a plain text version of this content. The canonical link for it is here.

Posted to reviews@impala.apache.org by "Daniel Becker (Code Review)" <ge...@cloudera.org> on 2024/03/20 13:24:01 UTC

[Impala-ASF-CR] IMPALA-12926: Remove AuxColumnType and add 'is binary' field to ColumnType

Daniel Becker has uploaded this change for review. ( http://gerrit.cloudera.org:8080/21157


Change subject: IMPALA-12926: Remove AuxColumnType and add 'is_binary' field to ColumnType
......................................................................

IMPALA-12926: Remove AuxColumnType and add 'is_binary' field to ColumnType

Currently the STRING and BINARY types are not distinguished in most of
the backend. In contrast to the frontend, PrimitiveType::TYPE_BINARY is
not used there at all, TYPE_STRING being used instead. This is to ensure
that everything that works for STRING also works for BINARY. So far only
file readers and writers have had to handle them differently, and they
have access to ColumnDescriptors which contain AuxColumnType fields that
differentiate these two types.

However, only top-level columns have ColumnDescriptors. Adding support
for BINARYs within complex types (see IMPALA-11491 and IMPALA-12651)
necessitates adding type information about STRING vs BINARY to embedded
fields as well.

Using PrimitiveType::TYPE_BINARY would probably be the cleanest solution
but it would affect huge parts of the code as TYPE_BINARY would have to
be added to hundreds of switch statements and this would be error prone.

Instead, this change introduces a new field in ColumnType: 'is_binary',
which is true if the type is a BINARY and false otherwise. We keep using
TYPE_STRING as the PrimitiveType of the ColumnType for BINARYs. This way
full type information is present in ColumnType but code that does not
differentiate between STRING and BINARY will continue to work for
BINARY.

With this change, AuxColumnType is no longer needed and is removed.

Testing:
 - added an extra test in binary-type.test

Change-Id: Icedbad4e24a46e7731de11cc14218761d11fb86f
---
M be/src/exec/file-metadata-utils.cc
M be/src/exec/hbase/hbase-scan-node.cc
M be/src/exec/hbase/hbase-table-writer.cc
M be/src/exec/hdfs-scanner-ir.cc
M be/src/exec/hdfs-scanner.cc
M be/src/exec/hdfs-text-table-writer.cc
M be/src/exec/json/hdfs-json-scanner.cc
M be/src/exec/parquet/hdfs-parquet-table-writer.cc
M be/src/exec/parquet/parquet-metadata-utils.cc
M be/src/exec/parquet/parquet-metadata-utils.h
M be/src/exec/rcfile/hdfs-rcfile-scanner.cc
M be/src/exec/text-converter.cc
M be/src/exec/text-converter.h
M be/src/exec/text-converter.inline.h
M be/src/exec/text/hdfs-text-scanner.cc
M be/src/runtime/descriptors.cc
M be/src/runtime/descriptors.h
M be/src/runtime/types.cc
M be/src/runtime/types.h
M be/src/service/hs2-util.cc
M testdata/workloads/functional-query/queries/QueryTest/binary-type.test
21 files changed, 80 insertions(+), 115 deletions(-)



  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/57/21157/3
-- 
To view, visit http://gerrit.cloudera.org:8080/21157
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: Icedbad4e24a46e7731de11cc14218761d11fb86f
Gerrit-Change-Number: 21157
Gerrit-PatchSet: 3
Gerrit-Owner: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>