You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Mihaly Szjatinya (Code Review)" <ge...@cloudera.org> on 2024/11/10 18:17:50 UTC
[Impala-ASF-CR] IMPALA-10319: Support arbitrary encodings on Text/Sequence files
Mihaly Szjatinya has uploaded this change for review. ( http://gerrit.cloudera.org:8080/22049
Change subject: IMPALA-10319: Support arbitrary encodings on Text/Sequence files
......................................................................
IMPALA-10319: Support arbitrary encodings on Text/Sequence files
[Draft]
As proposed in Jira, this implements decoding of text buffers for
Impala/Hive text tables. Given a table with 'serialization.encoding'
property set, Hive is able to encode the inserted data into charset
specified, consequently saving it into a text file. Objective is to
perform the opposite operation upon reading data buffers from text files
by employing boost::locale::conv::to_utf() library capability.
Special handling is provided for different BOM scenarios, as well as for
line delimiters, which are not part of the encoding on the Hive side.
For example, given a standard line delimiter '\n', represented by a
single byte '0A' in UTF-8 and two bytes '0x0A 0x00' in UTF-16LE, Hive
inserts it as a single '0A'.
Note: although the described behavior allows keeping the range split
logic simpler, on the negative side it renders Hive files not readable
by regular decoders. Conversely, regular multi-byte files with
delimiters are not correctly read by Hive/Impala. This limitation has no
effect on single-byte charsets.
Design Notes:
1. Memory management. Current draft is a naive
<conv::to_utf() std::string segments -> concat into std::string -> pool>
approach. This can be improved in two steps:
a. Get rid of string for gathering segments and allocate directly on
pool.
b. Provide custom allocator for segment strings, returned by
conv::to_utf() to get those allocated and hopefully RVO'd directly on
pool as well. At the very least, those strings could be tracked via
TrackedString.
2. Decoding filled buffer as a whole. Currently decoding is performed
line by line upon the whole buffer in HdfsTextScanner::FillByteBuffer(),
after it has been read and, if needed, decompressed. This has the
definite advantage of having all the decoding logic in one place,
potentially extractable to a separate class.
Alternatives would be:
a. Decoding tuples /(lines upon their materialization on later stages of
HdfsTextScanner. This would employ the existing splitting logic, even
though decoding itself would be done similarly line by line.
b. Preliminarily transforming malformed line delimiters to secure a
smooth conv::to_utf() in one go.
3. An optimization may be done to avoid delimiters only for multi-byte
encodings.
TODO / Nice to Have:
1. Once design approved, this should be extracted into class (hierarchy)
and used for Sequence files. Potentially expandable for encoding by
writers.
2. Test / think through all possible delimiter (FIELD_DELIM, LINE_DELIM,
COLLECTION_DELIM, ESCAPE_CHAR) scenarios.
3. Test archived scenarios.
4. Juxtapose java vs boost encodings supported.
5. Tests all errors/warnings scenarious / corner cases.
Change-Id: I787cd01caa52a19d6645519a6cedabe0a5253a65
---
M be/src/exec/hdfs-scanner.h
M be/src/exec/text/hdfs-text-scanner.cc
M be/src/exec/text/hdfs-text-scanner.h
M be/src/runtime/descriptors.cc
M be/src/runtime/descriptors.h
M common/thrift/CatalogObjects.thrift
M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
A testdata/decoding/cp1251_names.txt
A testdata/decoding/gbk_names.txt
A testdata/decoding/koi8r_names.txt
A testdata/decoding/latin1_names.txt
A testdata/decoding/shift_jis_names.txt
A testdata/decoding/utf16_names.txt
A testdata/decoding/utf16be_names.txt
A testdata/decoding/utf16le_names.txt
A testdata/workloads/functional-query/queries/QueryTest/decoding.test
A tests/query_test/test_decoding.py
17 files changed, 550 insertions(+), 3 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/49/22049/1
--
To view, visit http://gerrit.cloudera.org:8080/22049
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I787cd01caa52a19d6645519a6cedabe0a5253a65
Gerrit-Change-Number: 22049
Gerrit-PatchSet: 1
Gerrit-Owner: Mihaly Szjatinya <ms...@pm.me>
[Impala-ASF-CR] WIP IMPALA-10319: Support arbitrary encodings on Text/Sequence files
Posted by "Mihaly Szjatinya (Code Review)" <ge...@cloudera.org>.
Mihaly Szjatinya has uploaded a new patch set (#8). ( http://gerrit.cloudera.org:8080/22049 )
Change subject: WIP IMPALA-10319: Support arbitrary encodings on Text/Sequence files
......................................................................
WIP IMPALA-10319: Support arbitrary encodings on Text/Sequence files
As proposed in Jira, this implements decoding of text buffers for
Impala/Hive text tables. Given a table with 'serialization.encoding'
property set, Hive is able to encode the inserted data into charset
specified, consequently saving it into a text file. Objective is to
perform the opposite operation upon reading data buffers from text files
by employing boost::locale::conv::to_utf() library capability.
Since Hive doesn't encode line delimiters, charsets that would have
delimiters stored differently from ASCII are not allowed.
One difference from Hive is that Impala impelents
'serialization.encoding' only as a per partition serdeproperty to avoid
confusion of allowing both serde and tbl properties.
TODO / Nice to Have:
1. Once design approved, this should be extracted into class (hierarchy)
and used for Sequence files. Potentially expandable for encoding by
writers.
2. Test / think through all possible delimiter (FIELD_DELIM, LINE_DELIM,
COLLECTION_DELIM, ESCAPE_CHAR) scenarios.
3. Test archived scenarios.
4. Test complex types
5. Test per-partition cases
6. Juxtapose java vs boost encodings supported.
7. Test all errors/warnings scenarious / corner cases.
Change-Id: I787cd01caa52a19d6645519a6cedabe0a5253a65
---
M be/src/exec/hdfs-scanner.h
M be/src/exec/hdfs-text-table-writer.cc
M be/src/exec/hdfs-text-table-writer.h
M be/src/exec/scanner-context.h
M be/src/exec/text/hdfs-text-scanner.cc
M be/src/exec/text/hdfs-text-scanner.h
M be/src/runtime/descriptors.cc
M be/src/runtime/descriptors.h
M be/src/runtime/row-batch.h
M be/src/util/CMakeLists.txt
A be/src/util/char-codec.cc
A be/src/util/char-codec.h
M common/thrift/CatalogObjects.thrift
M common/thrift/generate_error_codes.py
M fe/src/main/java/org/apache/impala/analysis/AlterTableSetTblProperties.java
M fe/src/main/java/org/apache/impala/catalog/HdfsStorageDescriptor.java
M fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java
A testdata/charcodec/cp1251_big_utf8.txt
A testdata/charcodec/cp1251_names.txt
A testdata/charcodec/cp1251_names_utf8.txt
A testdata/charcodec/gbk_big_utf8.txt
A testdata/charcodec/gbk_names.txt
A testdata/charcodec/gbk_names_utf8.txt
A testdata/charcodec/koi8r_big_utf8.txt
A testdata/charcodec/koi8r_names.txt
A testdata/charcodec/koi8r_names_utf8.txt
A testdata/charcodec/latin1_big_utf8.txt
A testdata/charcodec/latin1_names.txt
A testdata/charcodec/latin1_names_utf8.txt
A testdata/charcodec/shift_jis_big_utf8.txt
A testdata/charcodec/shift_jis_names.txt
A testdata/charcodec/shift_jis_names_utf8.txt
M testdata/datasets/functional/functional_schema_template.sql
M testdata/datasets/functional/schema_constraints.csv
A testdata/workloads/functional-query/queries/QueryTest/dec-alltypes.test
A testdata/workloads/functional-query/queries/QueryTest/dec-hive.test
A testdata/workloads/functional-query/queries/QueryTest/dec-small.test
A testdata/workloads/functional-query/queries/QueryTest/enc-dec-alltypes.test
A testdata/workloads/functional-query/queries/QueryTest/enc-dec-big.test
A testdata/workloads/functional-query/queries/QueryTest/enc-dec-gen.test
A testdata/workloads/functional-query/queries/QueryTest/enc-dec-small.test
A tests/query_test/test_charcodec.py
42 files changed, 51,246 insertions(+), 15 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/49/22049/8
--
To view, visit http://gerrit.cloudera.org:8080/22049
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I787cd01caa52a19d6645519a6cedabe0a5253a65
Gerrit-Change-Number: 22049
Gerrit-PatchSet: 8
Gerrit-Owner: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
[Impala-ASF-CR] WIP IMPALA-10319: Support arbitrary encodings on Text/Sequence files
Posted by "Mihaly Szjatinya (Code Review)" <ge...@cloudera.org>.
Mihaly Szjatinya has uploaded a new patch set (#15). ( http://gerrit.cloudera.org:8080/22049 )
Change subject: WIP IMPALA-10319: Support arbitrary encodings on Text/Sequence files
......................................................................
WIP IMPALA-10319: Support arbitrary encodings on Text/Sequence files
As proposed in Jira, this implements decoding and encoding of text
buffers for Impala/Hive text tables. Given a table with
'serialization.encoding' property set, similarly to Hive, Impala should
be able to encode the inserted data into charset specified, consequently
saving it into a text file. The opposite decoding operation should be
performed upon reading data buffers from text files. Both operations
employ boost::locale::conv library.
Since Hive doesn't encode line delimiters, charsets that would have
delimiters stored differently from ASCII are not allowed.
One difference from Hive is that Impala implements
'serialization.encoding' only as a per partition serdeproperty to avoid
confusion of allowing both serde and tbl properties. (See related
IMPALA-13748)
Change-Id: I787cd01caa52a19d6645519a6cedabe0a5253a65
---
M be/src/exec/hdfs-scanner.h
M be/src/exec/hdfs-text-table-writer.cc
M be/src/exec/hdfs-text-table-writer.h
M be/src/exec/scanner-context.h
M be/src/exec/text/hdfs-text-scanner.cc
M be/src/exec/text/hdfs-text-scanner.h
M be/src/runtime/descriptors.cc
M be/src/runtime/descriptors.h
M be/src/runtime/mem-tracker.cc
M be/src/runtime/mem-tracker.h
M be/src/runtime/row-batch.h
M be/src/util/CMakeLists.txt
A be/src/util/char-codec.cc
A be/src/util/char-codec.h
M common/thrift/CatalogObjects.thrift
M common/thrift/generate_error_codes.py
M fe/src/main/java/org/apache/impala/analysis/AlterTableSetTblProperties.java
M fe/src/main/java/org/apache/impala/catalog/HdfsStorageDescriptor.java
M fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java
A tests/query_test/test_charcodec.py
20 files changed, 832 insertions(+), 16 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/49/22049/15
--
To view, visit http://gerrit.cloudera.org:8080/22049
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I787cd01caa52a19d6645519a6cedabe0a5253a65
Gerrit-Change-Number: 22049
Gerrit-PatchSet: 15
Gerrit-Owner: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
[Impala-ASF-CR] WIP IMPALA-10319: Support arbitrary encodings on Text/Sequence files
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22049 )
Change subject: WIP IMPALA-10319: Support arbitrary encodings on Text/Sequence files
......................................................................
Patch Set 15:
(1 comment)
gerrit-auto-critic failed. You can reproduce it locally using command:
python3 bin/jenkins/critique-gerrit-review.py --dryrun
To run it, you might need a virtual env with Python3's venv installed.
http://gerrit.cloudera.org:8080/#/c/22049/15/tests/query_test/test_charcodec.py
File tests/query_test/test_charcodec.py:
http://gerrit.cloudera.org:8080/#/c/22049/15/tests/query_test/test_charcodec.py@44
PS15, Line 44: u
flake8: F821 undefined name 'unichr'
--
To view, visit http://gerrit.cloudera.org:8080/22049
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I787cd01caa52a19d6645519a6cedabe0a5253a65
Gerrit-Change-Number: 22049
Gerrit-PatchSet: 15
Gerrit-Owner: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Wed, 26 Mar 2025 13:39:25 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] WIP IMPALA-10319: Support arbitrary encodings on Text/Sequence files
Posted by "Mihaly Szjatinya (Code Review)" <ge...@cloudera.org>.
Mihaly Szjatinya has uploaded a new patch set (#10). ( http://gerrit.cloudera.org:8080/22049 )
Change subject: WIP IMPALA-10319: Support arbitrary encodings on Text/Sequence files
......................................................................
WIP IMPALA-10319: Support arbitrary encodings on Text/Sequence files
As proposed in Jira, this implements decoding and encoding of text
buffers for Impala/Hive text tables. Given a table with
'serialization.encoding' property set, similarly to Hive, Impala should
be able to encode the inserted data into charset specified, consequently
saving it into a text file. The opposite decoding operation should be
performed upon reading data buffers from text files. Both operations
employ boost::locale::conv library.
Since Hive doesn't encode line delimiters, charsets that would have
delimiters stored differently from ASCII are not allowed.
One difference from Hive is that Impala implements
'serialization.encoding' only as a per partition serdeproperty to avoid
confusion of allowing both serde and tbl properties.
Change-Id: I787cd01caa52a19d6645519a6cedabe0a5253a65
---
M be/src/exec/hdfs-scanner.h
M be/src/exec/hdfs-text-table-writer.cc
M be/src/exec/hdfs-text-table-writer.h
M be/src/exec/scanner-context.h
M be/src/exec/text/hdfs-text-scanner.cc
M be/src/exec/text/hdfs-text-scanner.h
M be/src/runtime/descriptors.cc
M be/src/runtime/descriptors.h
M be/src/runtime/row-batch.h
M be/src/util/CMakeLists.txt
A be/src/util/char-codec.cc
A be/src/util/char-codec.h
M common/thrift/CatalogObjects.thrift
M common/thrift/generate_error_codes.py
M fe/src/main/java/org/apache/impala/analysis/AlterTableSetTblProperties.java
M fe/src/main/java/org/apache/impala/catalog/HdfsStorageDescriptor.java
M fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java
A tests/query_test/test_charcodec.py
18 files changed, 735 insertions(+), 15 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/49/22049/10
--
To view, visit http://gerrit.cloudera.org:8080/22049
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I787cd01caa52a19d6645519a6cedabe0a5253a65
Gerrit-Change-Number: 22049
Gerrit-PatchSet: 10
Gerrit-Owner: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
[Impala-ASF-CR] IMPALA-10319: Support arbitrary encodings on Text/Sequence files
Posted by "Peter Rozsa (Code Review)" <ge...@cloudera.org>.
Peter Rozsa has posted comments on this change. ( http://gerrit.cloudera.org:8080/22049 )
Change subject: IMPALA-10319: Support arbitrary encodings on Text/Sequence files
......................................................................
Patch Set 21:
(19 comments)
It looks good. I left mostly minor comments.
http://gerrit.cloudera.org:8080/#/c/22049/21/be/src/exec/hdfs-scanner.h
File be/src/exec/hdfs-scanner.h:
http://gerrit.cloudera.org:8080/#/c/22049/21/be/src/exec/hdfs-scanner.h@326
PS21, Line 326: decocoded
typo: decoded
http://gerrit.cloudera.org:8080/#/c/22049/21/be/src/exec/hdfs-text-table-writer.cc
File be/src/exec/hdfs-text-table-writer.cc:
http://gerrit.cloudera.org:8080/#/c/22049/21/be/src/exec/hdfs-text-table-writer.cc@56
PS21, Line 56: HdfsTextTableWriter::~HdfsTextTableWriter() {
nit: can fit in one line
http://gerrit.cloudera.org:8080/#/c/22049/21/be/src/util/char-codec.cc
File be/src/util/char-codec.cc:
http://gerrit.cloudera.org:8080/#/c/22049/21/be/src/util/char-codec.cc@61
PS21, Line 61: scopedMemTracker
nit: could be scoped_mem_tracker
http://gerrit.cloudera.org:8080/#/c/22049/21/be/src/util/char-codec.cc@103
PS21, Line 103: prefix
Is it possible to use partial_symbol to build up the prefix? That way, we could avoid an additional vector and copying.
http://gerrit.cloudera.org:8080/#/c/22049/21/be/src/util/char-codec.cc@113
PS21, Line 113: (char*)
nit: reinterpret_cast instead of c-style casts
http://gerrit.cloudera.org:8080/#/c/22049/21/be/src/util/char-codec.cc@159
PS21, Line 159: (
nit: reinterpret_cast instead of c-style casts
http://gerrit.cloudera.org:8080/#/c/22049/21/fe/src/main/java/org/apache/impala/analysis/AlterTableSetTblProperties.java
File fe/src/main/java/org/apache/impala/analysis/AlterTableSetTblProperties.java:
http://gerrit.cloudera.org:8080/#/c/22049/21/fe/src/main/java/org/apache/impala/analysis/AlterTableSetTblProperties.java@289
PS21, Line 289: !=
It should be .equals()
http://gerrit.cloudera.org:8080/#/c/22049/21/fe/src/main/java/org/apache/impala/analysis/AlterTableSetTblProperties.java@294
PS21, Line 294: SERIALIZATION.ENCODING
nit: could be lowercase with apostrophe: 'serialization.encoding'.
http://gerrit.cloudera.org:8080/#/c/22049/21/fe/src/main/java/org/apache/impala/analysis/AlterTableSetTblProperties.java@295
PS21, Line 295: %1$s
nit: could be simplified as %s
http://gerrit.cloudera.org:8080/#/c/22049/21/fe/src/main/java/org/apache/impala/analysis/AlterTableSetTblProperties.java@303
PS21, Line 303: %1$s / %2$s
nit: could be simplified as %s / %s
http://gerrit.cloudera.org:8080/#/c/22049/21/fe/src/main/java/org/apache/impala/analysis/AlterTableSetTblProperties.java@308
PS21, Line 308: FeFsTable
nit: unnecessary cast
http://gerrit.cloudera.org:8080/#/c/22049/21/fe/src/main/java/org/apache/impala/analysis/AlterTableSetTblProperties.java@314
PS21, Line 314: %1$s / %2$s
nit: could be simplified as %s / %s
http://gerrit.cloudera.org:8080/#/c/22049/21/fe/src/main/java/org/apache/impala/analysis/AlterTableSetTblProperties.java@320
PS21, Line 320: Unsupported encoding: " + encoding + "."
nit: it could use a format-string similarly to the previous exception messages.
http://gerrit.cloudera.org:8080/#/c/22049/21/fe/src/main/java/org/apache/impala/analysis/AlterTableSetTblProperties.java@318
PS21, Line 318: String encoding = tblProperties.get(serdeConstants.SERIALIZATION_ENCODING);
: if (!Charset.isSupported(encoding)) {
: throw new AnalysisException("Unsupported encoding: " + encoding + ".");
: }
nit: it could be moved after the first property check at L291
http://gerrit.cloudera.org:8080/#/c/22049/21/fe/src/main/java/org/apache/impala/analysis/AlterTableSetTblProperties.java@331
PS21, Line 331: %1$s
nit: could be simplified as %s
http://gerrit.cloudera.org:8080/#/c/22049/21/fe/src/main/java/org/apache/impala/analysis/AlterTableSetTblProperties.java@332
PS21, Line 332: Please refer to Jira ticket IMPALA-10319 for more info.
nit: I'd keep it simpler by skipping "Jira ticket": Please refer to IMPALA-10319 for more info. See L350, and the similar error message in CreateTableStmt
http://gerrit.cloudera.org:8080/#/c/22049/21/fe/src/main/java/org/apache/impala/analysis/AlterTableSetTblProperties.java@337
PS21, Line 337: FeFsTable
nit: unneccessary cast
http://gerrit.cloudera.org:8080/#/c/22049/21/fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java
File fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java:
http://gerrit.cloudera.org:8080/#/c/22049/21/fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java@21
PS21, Line 21: import java.nio.charset.StandardCharsets;
nit: not used
http://gerrit.cloudera.org:8080/#/c/22049/21/tests/query_test/test_charcodec.py
File tests/query_test/test_charcodec.py:
http://gerrit.cloudera.org:8080/#/c/22049/21/tests/query_test/test_charcodec.py@87
PS21, Line 87: data_file_path = os.path.join(os.environ['IMPALA_HOME'], "testdata",
It could use tempfile instead, no extra cleanup needed with it
--
To view, visit http://gerrit.cloudera.org:8080/22049
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I787cd01caa52a19d6645519a6cedabe0a5253a65
Gerrit-Change-Number: 22049
Gerrit-PatchSet: 21
Gerrit-Owner: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Peter Rozsa <pr...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Thu, 08 May 2025 12:53:15 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10319: Support arbitrary encodings on Text files
Posted by "Mihaly Szjatinya (Code Review)" <ge...@cloudera.org>.
Mihaly Szjatinya has uploaded a new patch set (#24). ( http://gerrit.cloudera.org:8080/22049 )
Change subject: IMPALA-10319: Support arbitrary encodings on Text files
......................................................................
IMPALA-10319: Support arbitrary encodings on Text files
As proposed in Jira, this implements decoding and encoding of text
buffers for Impala/Hive text tables. Given a table with
'serialization.encoding' property set, similarly to Hive, Impala should
be able to encode the inserted data into charset specified, consequently
saving it into a text file. The opposite decoding operation should be
performed upon reading data buffers from text files. Both operations
employ boost::locale::conv library.
Since Hive doesn't encode line delimiters, charsets that would have
delimiters stored differently from ASCII are not allowed.
One difference from Hive is that Impala implements
'serialization.encoding' only as a per partition serdeproperty to avoid
confusion of allowing both serde and tbl properties. (See related
IMPALA-13748)
Note: Due to precreated non-UTF-8 files present in the patch
'gerrit-code-review-checks' was performed locally. (See IMPALA-14100)
Change-Id: I787cd01caa52a19d6645519a6cedabe0a5253a65
---
M be/src/exec/hdfs-scanner.h
M be/src/exec/hdfs-text-table-writer.cc
M be/src/exec/hdfs-text-table-writer.h
M be/src/exec/scanner-context.h
M be/src/exec/text/hdfs-text-scanner.cc
M be/src/exec/text/hdfs-text-scanner.h
M be/src/runtime/descriptors.cc
M be/src/runtime/descriptors.h
M be/src/runtime/mem-tracker.cc
M be/src/runtime/mem-tracker.h
M be/src/runtime/row-batch.h
M be/src/util/CMakeLists.txt
A be/src/util/char-codec.cc
A be/src/util/char-codec.h
M common/thrift/CatalogObjects.thrift
M common/thrift/generate_error_codes.py
M fe/src/main/java/org/apache/impala/analysis/AlterTableSetTblProperties.java
M fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java
M fe/src/main/java/org/apache/impala/catalog/HdfsStorageDescriptor.java
M fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java
A testdata/charcodec/cp1251_names.txt
A testdata/charcodec/cp1251_names_utf8.txt
A testdata/charcodec/gbk_names.txt
A testdata/charcodec/gbk_names_error.txt
A testdata/charcodec/gbk_names_utf8.txt
A testdata/charcodec/koi8r_names.txt
A testdata/charcodec/koi8r_names_utf8.txt
A testdata/charcodec/latin1_names.txt
A testdata/charcodec/latin1_names_utf8.txt
A testdata/charcodec/shift_jis_names.txt
A testdata/charcodec/shift_jis_names_error.txt
A testdata/charcodec/shift_jis_names_utf8.txt
A tests/query_test/test_charcodec.py
33 files changed, 1,070 insertions(+), 20 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/49/22049/24
--
To view, visit http://gerrit.cloudera.org:8080/22049
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I787cd01caa52a19d6645519a6cedabe0a5253a65
Gerrit-Change-Number: 22049
Gerrit-PatchSet: 24
Gerrit-Owner: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Peter Rozsa <pr...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
[Impala-ASF-CR] IMPALA-10319: Support arbitrary encodings on Text files
Posted by "Mihaly Szjatinya (Code Review)" <ge...@cloudera.org>.
Mihaly Szjatinya has posted comments on this change. ( http://gerrit.cloudera.org:8080/22049 )
Change subject: IMPALA-10319: Support arbitrary encodings on Text files
......................................................................
Patch Set 25:
(4 comments)
http://gerrit.cloudera.org:8080/#/c/22049/21/tests/query_test/test_charcodec.py
File tests/query_test/test_charcodec.py:
http://gerrit.cloudera.org:8080/#/c/22049/21/tests/query_test/test_charcodec.py@72
PS21, Line 72: class TestCharCodecGen(ImpalaTestSuite):
> I would be surprised if files sizes would really matter for Impala (maybe f
There's a couple tests that take a bit longer, but not one really stands out. We can move them to exhaustive mode, but I've tried it now and the non-exhaustive is only 2,5 min, which seems a reasonable time. Previous 4 min results were either my mistake, or my machine was slowed down for some reason.
http://gerrit.cloudera.org:8080/#/c/22049/23/tests/query_test/test_charcodec.py
File tests/query_test/test_charcodec.py:
http://gerrit.cloudera.org:8080/#/c/22049/23/tests/query_test/test_charcodec.py@379
PS23, Line 379:
> There are some Impala test environments where Hive is not available (e.g. s
Ok, I removed these from where Hive is not involved.
http://gerrit.cloudera.org:8080/#/c/22049/23/tests/query_test/test_charcodec.py@432
PS23, Line 432: self.execute_query(
: """insert overwrite {}.{} select * from {}.{}"""
: .format(db, encoded_table, db, utf8_table))
:
> The ordering comes after decoding.
Ack
http://gerrit.cloudera.org:8080/#/c/22049/24/tests/query_test/test_charcodec.py
File tests/query_test/test_charcodec.py:
http://gerrit.cloudera.org:8080/#/c/22049/24/tests/query_test/test_charcodec.py@426
PS24, Line 426:
> nit: extra whitespace
Ack
--
To view, visit http://gerrit.cloudera.org:8080/22049
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I787cd01caa52a19d6645519a6cedabe0a5253a65
Gerrit-Change-Number: 22049
Gerrit-PatchSet: 25
Gerrit-Owner: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Peter Rozsa <pr...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Fri, 30 May 2025 11:17:44 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10319: Support arbitrary encodings on Text files
Posted by "Peter Rozsa (Code Review)" <ge...@cloudera.org>.
Peter Rozsa has posted comments on this change. ( http://gerrit.cloudera.org:8080/22049 )
Change subject: IMPALA-10319: Support arbitrary encodings on Text files
......................................................................
Patch Set 23:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/22049/21/be/src/util/char-codec.cc
File be/src/util/char-codec.cc:
http://gerrit.cloudera.org:8080/#/c/22049/21/be/src/util/char-codec.cc@61
PS21, Line 61: scopedMemTracker
> let's leave it like e.g. MemTracker
I meant the variable name could follow the snake-case, as other variable names.
http://gerrit.cloudera.org:8080/#/c/22049/21/be/src/util/char-codec.cc@103
PS21, Line 103: prefix
> These are really different entities with different scope, but maybe I didn'
partial_symbol can be used to build up the symbol from the new buffer, as it's only appended by new characters and tries to convert to a valid one. I had the following idea, it's possible that I miss something, in this case, please correct me:
```Status CharCodec::HandlePrefix(uint8_t** buf_start, uint8_t* buf_end,
std::string* result_prefix) {
if (!partial_symbol_.empty()) {
DCHECK_LT(partial_symbol_.size(), MAX_SYMBOL);
while (partial_symbol_.size() < MAX_SYMBOL && *buf_start < buf_end) {
partial_symbol_.push_back(**buf_start);
++(*buf_start);
try {
*result_prefix = boost::locale::conv::to_utf<char>(
reinterpret_cast<const char*>(partial_symbol_.data()),
reinterpret_cast<const char*>(partial_symbol_.data() + partial_symbol_.size()),
encoding_, boost::locale::conv::stop);
partial_symbol_.clear();
return Status::OK();
} catch (boost::locale::conv::conversion_error&) {
continue;
} catch (const std::exception& e) {
return Status(TErrorCode::CHARSET_CONVERSION_ERROR, e.what());
}
}
return Status(TErrorCode::CHARSET_CONVERSION_ERROR, "Unable to decode buffer");
}```
--
To view, visit http://gerrit.cloudera.org:8080/22049
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I787cd01caa52a19d6645519a6cedabe0a5253a65
Gerrit-Change-Number: 22049
Gerrit-PatchSet: 23
Gerrit-Owner: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Peter Rozsa <pr...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Wed, 28 May 2025 15:14:22 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] WIP IMPALA-10319: Support arbitrary encodings on Text/Sequence files
Posted by "Mihaly Szjatinya (Code Review)" <ge...@cloudera.org>.
Mihaly Szjatinya has posted comments on this change. ( http://gerrit.cloudera.org:8080/22049 )
Change subject: WIP IMPALA-10319: Support arbitrary encodings on Text/Sequence files
......................................................................
Patch Set 17:
(18 comments)
http://gerrit.cloudera.org:8080/#/c/22049/17/be/src/exec/hdfs-scanner.h
File be/src/exec/hdfs-scanner.h:
http://gerrit.cloudera.org:8080/#/c/22049/17/be/src/exec/hdfs-scanner.h@324
PS17, Line 324:
> nit: extra space. Same goes for next line.
ack
http://gerrit.cloudera.org:8080/#/c/22049/17/be/src/exec/hdfs-text-table-writer.h
File be/src/exec/hdfs-text-table-writer.h:
http://gerrit.cloudera.org:8080/#/c/22049/17/be/src/exec/hdfs-text-table-writer.h@93
PS17, Line 93: boost::scoped_ptr
> We prefer std::unique_ptr over scoped_ptr
yeah, good idea
http://gerrit.cloudera.org:8080/#/c/22049/17/be/src/exec/hdfs-text-table-writer.cc
File be/src/exec/hdfs-text-table-writer.cc:
http://gerrit.cloudera.org:8080/#/c/22049/17/be/src/exec/hdfs-text-table-writer.cc@56
PS17, Line 56: HdfsTextTableWriter::~HdfsTextTableWriter() {
: }
> Why did you move the empty destructor to the .cc file?
After adding a scoped_ptr<CharCodec> to class, its destructor now requires a complete CharCodec. This persists for std::unique_ptr (but not for shared_ptr) and IIRC has to do with the implementation details of the unique_ptr's deleter.
http://gerrit.cloudera.org:8080/#/c/22049/17/be/src/exec/hdfs-text-table-writer.cc@62
PS17, Line 62: auto
> No need to copy the string, it could be const auto&
Yes, good point.
http://gerrit.cloudera.org:8080/#/c/22049/17/be/src/exec/hdfs-text-table-writer.cc@64
PS17, Line 64: NULL
> nit: we prefer nullptr over NULL
ack
http://gerrit.cloudera.org:8080/#/c/22049/17/be/src/exec/hdfs-text-table-writer.cc@161
PS17, Line 161: if (encoder_.get() != nullptr)
> This could be just 'if (encoder_)'
Agree, I just saw this pattern all over the code. However not in this particular file, so probably ok to change it here.
http://gerrit.cloudera.org:8080/#/c/22049/17/be/src/exec/text/hdfs-text-scanner.cc
File be/src/exec/text/hdfs-text-scanner.cc:
http://gerrit.cloudera.org:8080/#/c/22049/17/be/src/exec/text/hdfs-text-scanner.cc@253
PS17, Line 253: auto
> This could be const auto&
Ack
http://gerrit.cloudera.org:8080/#/c/22049/17/be/src/exec/text/hdfs-text-scanner.cc@545
PS17, Line 545: if (decoder_.get() != nullptr)
> nit: Could be 'if (decoder_)'
In this and related modules it is scoped_ptr with this pattern everywhere. Even in this very function there is 'if (decompressor_.get() == nullptr)' above.
What is the general approach in such cases, if any? To clarify all similar comments.
http://gerrit.cloudera.org:8080/#/c/22049/17/be/src/runtime/descriptors.h
File be/src/runtime/descriptors.h:
http://gerrit.cloudera.org:8080/#/c/22049/17/be/src/runtime/descriptors.h@402
PS17, Line 402: std::string
> Do you need to return a new object? If not, it could return const std::stri
Agree, fixed
http://gerrit.cloudera.org:8080/#/c/22049/17/be/src/util/char-codec.h
File be/src/util/char-codec.h:
http://gerrit.cloudera.org:8080/#/c/22049/17/be/src/util/char-codec.h@21
PS17, Line 21: #include <boost/scoped_ptr.hpp>
> Unused include
ack
http://gerrit.cloudera.org:8080/#/c/22049/17/be/src/util/char-codec.h@41
PS17, Line 41:
> nit: redundant space
ack
http://gerrit.cloudera.org:8080/#/c/22049/17/be/src/util/char-codec.cc
File be/src/util/char-codec.cc:
http://gerrit.cloudera.org:8080/#/c/22049/17/be/src/util/char-codec.cc@63
PS17, Line 63: buf_end - buf_start
> nit: *bytes_read
ack
http://gerrit.cloudera.org:8080/#/c/22049/17/be/src/util/char-codec.cc@86
PS17, Line 86: *buffer
> This should be 'out_buffer_'.
Yeah, my bad.
http://gerrit.cloudera.org:8080/#/c/22049/17/be/src/util/char-codec.cc@104
PS17, Line 104: std::vector<uint8_t> prefix = partial_symbol_;
> You could reserve memory after this line to decrease number of re-allocatio
Good idea, but I think it's even more simple:
prefix.reserve(MAX_SYMBOL);
http://gerrit.cloudera.org:8080/#/c/22049/17/fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java
File fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java:
http://gerrit.cloudera.org:8080/#/c/22049/17/fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java@940
PS17, Line 940:
> nit: indentation is off
ack
http://gerrit.cloudera.org:8080/#/c/22049/17/tests/query_test/test_charcodec.py
File tests/query_test/test_charcodec.py:
http://gerrit.cloudera.org:8080/#/c/22049/17/tests/query_test/test_charcodec.py@29
PS17, Line 29: def get_table_loc(db, tbl_name):
: return '%s/%s.db/%s/' % (WAREHOUSE, db, tbl_name,)
> We should probably move this to ImpalaTestSuite. We have a similar function
Actually there's a similar function already: 'ImpalaTestSuite::_get_table_location()', which is totally fine for this test. So I'll just use that one instead.
http://gerrit.cloudera.org:8080/#/c/22049/17/tests/query_test/test_charcodec.py@34
PS17, Line 34: problematic_symbols_hiragana = {0x3040, 0x3094, 0x3095, 0x3096, 0x3097, 0x3098, 0x3099,
> Please add comment why they are problematic.
boost::locale::conv() stambles on these and if I get it right strictly speaking they are not even part of the encoding. In any case it is not crucial for this test to use the exhaustive set of all possible symbols. I rewrote this part to avoid confusion.
http://gerrit.cloudera.org:8080/#/c/22049/17/tests/query_test/test_charcodec.py@44
PS17, Line 44: u
> flake8: F821 undefined name 'unichr'
@Zoltan, do you know how to fix this btw?
--
To view, visit http://gerrit.cloudera.org:8080/22049
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I787cd01caa52a19d6645519a6cedabe0a5253a65
Gerrit-Change-Number: 22049
Gerrit-PatchSet: 17
Gerrit-Owner: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Sun, 27 Apr 2025 11:57:39 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] WIP IMPALA-10319: Support arbitrary encodings on Text/Sequence files
Posted by "Mihaly Szjatinya (Code Review)" <ge...@cloudera.org>.
Mihaly Szjatinya has posted comments on this change. ( http://gerrit.cloudera.org:8080/22049 )
Change subject: WIP IMPALA-10319: Support arbitrary encodings on Text/Sequence files
......................................................................
Patch Set 18:
@Zoltan, thanks for review, left the answers.
--
To view, visit http://gerrit.cloudera.org:8080/22049
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I787cd01caa52a19d6645519a6cedabe0a5253a65
Gerrit-Change-Number: 22049
Gerrit-PatchSet: 18
Gerrit-Owner: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Sun, 27 Apr 2025 12:00:07 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] WIP IMPALA-10319: Support arbitrary encodings on Text/Sequence files
Posted by "Mihaly Szjatinya (Code Review)" <ge...@cloudera.org>.
Mihaly Szjatinya has uploaded a new patch set (#19). ( http://gerrit.cloudera.org:8080/22049 )
Change subject: WIP IMPALA-10319: Support arbitrary encodings on Text/Sequence files
......................................................................
WIP IMPALA-10319: Support arbitrary encodings on Text/Sequence files
As proposed in Jira, this implements decoding and encoding of text
buffers for Impala/Hive text tables. Given a table with
'serialization.encoding' property set, similarly to Hive, Impala should
be able to encode the inserted data into charset specified, consequently
saving it into a text file. The opposite decoding operation should be
performed upon reading data buffers from text files. Both operations
employ boost::locale::conv library.
Since Hive doesn't encode line delimiters, charsets that would have
delimiters stored differently from ASCII are not allowed.
One difference from Hive is that Impala implements
'serialization.encoding' only as a per partition serdeproperty to avoid
confusion of allowing both serde and tbl properties. (See related
IMPALA-13748)
Change-Id: I787cd01caa52a19d6645519a6cedabe0a5253a65
---
M be/src/exec/hdfs-scanner.h
M be/src/exec/hdfs-text-table-writer.cc
M be/src/exec/hdfs-text-table-writer.h
M be/src/exec/scanner-context.h
M be/src/exec/text/hdfs-text-scanner.cc
M be/src/exec/text/hdfs-text-scanner.h
M be/src/runtime/descriptors.cc
M be/src/runtime/descriptors.h
M be/src/runtime/mem-tracker.cc
M be/src/runtime/mem-tracker.h
M be/src/runtime/row-batch.h
M be/src/util/CMakeLists.txt
A be/src/util/char-codec.cc
A be/src/util/char-codec.h
M common/thrift/CatalogObjects.thrift
M common/thrift/generate_error_codes.py
M fe/src/main/java/org/apache/impala/analysis/AlterTableSetTblProperties.java
M fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java
M fe/src/main/java/org/apache/impala/catalog/HdfsStorageDescriptor.java
M fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java
A tests/query_test/test_charcodec.py
21 files changed, 905 insertions(+), 17 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/49/22049/19
--
To view, visit http://gerrit.cloudera.org:8080/22049
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I787cd01caa52a19d6645519a6cedabe0a5253a65
Gerrit-Change-Number: 22049
Gerrit-PatchSet: 19
Gerrit-Owner: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
[Impala-ASF-CR] WIP IMPALA-10319: Support arbitrary encodings on Text/Sequence files
Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/22049 )
Change subject: WIP IMPALA-10319: Support arbitrary encodings on Text/Sequence files
......................................................................
Patch Set 4:
(5 comments)
http://gerrit.cloudera.org:8080/#/c/22049/4/be/src/exec/text/hdfs-text-scanner.cc
File be/src/exec/text/hdfs-text-scanner.cc:
http://gerrit.cloudera.org:8080/#/c/22049/4/be/src/exec/text/hdfs-text-scanner.cc@545
PS4, Line 545: DecodeBuffer
Is it possible for the end of buffer to cut a multi-line character in two? In this case decoding the buffer would not succeed/give incorrect result.
I have two ideas on how to handle this (both rely on the line separator \n being single character):
1. find the last line separator in the buffer and save the remainder as a state of CharCodec and append it to the start of the next buffer
2. do the decoding per line instead of per buffer
http://gerrit.cloudera.org:8080/#/c/22049/4/be/src/util/decoder.cc
File be/src/util/decoder.cc:
http://gerrit.cloudera.org:8080/#/c/22049/4/be/src/util/decoder.cc@45
PS4, Line 45: encoding_ == "UTF-8"
hdfs-text-scanner.cc doesn't create a decoder if encoding is UTF-8, so I think that this won't get executed
http://gerrit.cloudera.org:8080/#/c/22049/4/be/src/util/decoder.cc@51
PS4, Line 51: to_utf
Based on https://beta.boost.org/doc/libs/1_85_0/libs/locale/doc/html/group__Charset.html#ga2f2e2da92b8ca7041241f7f1082d5cf8 this can throw exception, which would result in a crash in Impala. The possible exceptions should be caught and converted to error status.
http://gerrit.cloudera.org:8080/#/c/22049/4/be/src/util/decoder.cc@52
PS4, Line 52: (char *)(*buffer + start), (char *)(*buffer + *bytes_read), encoding_);
There is also a "method_type how" optional parameter that decides what to do on decode error. The default is skipping the character, which seems error prone to me. It may be better to return an error in this case.
http://gerrit.cloudera.org:8080/#/c/22049/4/fe/src/main/java/org/apache/impala/catalog/HdfsStorageDescriptor.java
File fe/src/main/java/org/apache/impala/catalog/HdfsStorageDescriptor.java:
http://gerrit.cloudera.org:8080/#/c/22049/4/fe/src/main/java/org/apache/impala/catalog/HdfsStorageDescriptor.java@233
PS4, Line 233: // Extract encoding
: String encoding = DEFAULT_ENCODING;
: String encodingValue = parameters.get(serdeConstants.SERIALIZATION_ENCODING);
: if (encodingValue != null) {
: encoding = encodingValue;
: }
It is not clear to me, does the patch support using this this property as table property, serde property, or both? I saw examples for both among the tests, e.g. stored as textfile tblproperties("serialization.encoding"="GBK"); vs ALTER TABLE {table_name} SET SERDEPROPERTIES('serialization.encoding'='GBK');
This decision should be also mentioned in the commit message, as it seems a difference compared to the Jira.
--
To view, visit http://gerrit.cloudera.org:8080/22049
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I787cd01caa52a19d6645519a6cedabe0a5253a65
Gerrit-Change-Number: 22049
Gerrit-PatchSet: 4
Gerrit-Owner: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Comment-Date: Mon, 16 Dec 2024 18:29:48 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] WIP IMPALA-10319: Support arbitrary encodings on Text/Sequence files
Posted by "Mihaly Szjatinya (Code Review)" <ge...@cloudera.org>.
Mihaly Szjatinya has posted comments on this change. ( http://gerrit.cloudera.org:8080/22049 )
Change subject: WIP IMPALA-10319: Support arbitrary encodings on Text/Sequence files
......................................................................
Patch Set 5:
(5 comments)
1. Fixed some of the comments.
2. First draft text encoding + encoding-decoding tests.
3. Restructured tests so far.
http://gerrit.cloudera.org:8080/#/c/22049/4/be/src/exec/text/hdfs-text-scanner.cc
File be/src/exec/text/hdfs-text-scanner.cc:
http://gerrit.cloudera.org:8080/#/c/22049/4/be/src/exec/text/hdfs-text-scanner.cc@545
PS4, Line 545: DecodeBuffer
> Is it possible for the end of buffer to cut a multi-line character in two?
Good point, although I'm not sure HdfsTextScanner doesn't handle this on a different level. Yet there is a special case for line.delim being two bytes: \r\n.
I'll try to fix this in the next patch.
http://gerrit.cloudera.org:8080/#/c/22049/4/be/src/util/decoder.cc
File be/src/util/decoder.cc:
http://gerrit.cloudera.org:8080/#/c/22049/4/be/src/util/decoder.cc@45
PS4, Line 45:
> hdfs-text-scanner.cc doesn't create a decoder if encoding is UTF-8, so I th
Removed.
http://gerrit.cloudera.org:8080/#/c/22049/4/be/src/util/decoder.cc@51
PS4, Line 51:
> Based on https://beta.boost.org/doc/libs/1_85_0/libs/locale/doc/html/group_
Agree, done.
http://gerrit.cloudera.org:8080/#/c/22049/4/be/src/util/decoder.cc@52
PS4, Line 52:
> There is also a "method_type how" optional parameter that decides what to d
Good point.
http://gerrit.cloudera.org:8080/#/c/22049/4/fe/src/main/java/org/apache/impala/catalog/HdfsStorageDescriptor.java
File fe/src/main/java/org/apache/impala/catalog/HdfsStorageDescriptor.java:
http://gerrit.cloudera.org:8080/#/c/22049/4/fe/src/main/java/org/apache/impala/catalog/HdfsStorageDescriptor.java@233
PS4, Line 233: // Extract encoding
: String encoding = DEFAULT_ENCODING;
: String encodingValue = parameters.get(serdeConstants.SERIALIZATION_ENCODING);
: if (encodingValue != null) {
: encoding = encodingValue;
: }
> I saw examples for both among the tests
I guess those were statements from HIVE_QUERY section, which support both. Updated to use only serdeproperties.
> This decision should be also mentioned in the commit message,
Agree, done.
--
To view, visit http://gerrit.cloudera.org:8080/22049
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I787cd01caa52a19d6645519a6cedabe0a5253a65
Gerrit-Change-Number: 22049
Gerrit-PatchSet: 5
Gerrit-Owner: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Comment-Date: Mon, 23 Dec 2024 00:08:55 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] WIP IMPALA-10319: Support arbitrary encodings on Text/Sequence files
Posted by "Mihaly Szjatinya (Code Review)" <ge...@cloudera.org>.
Mihaly Szjatinya has uploaded a new patch set (#4). ( http://gerrit.cloudera.org:8080/22049 )
Change subject: WIP IMPALA-10319: Support arbitrary encodings on Text/Sequence files
......................................................................
WIP IMPALA-10319: Support arbitrary encodings on Text/Sequence files
As proposed in Jira, this implements decoding of text buffers for
Impala/Hive text tables. Given a table with 'serialization.encoding'
property set, Hive is able to encode the inserted data into charset
specified, consequently saving it into a text file. Objective is to
perform the opposite operation upon reading data buffers from text files
by employing boost::locale::conv::to_utf() library capability.
Since Hive doesn't encode line delimiters, charsets that would have
delimiters stored differently from ASCII are not allowed.
TODO / Nice to Have:
1. Once design approved, this should be extracted into class (hierarchy)
and used for Sequence files. Potentially expandable for encoding by
writers.
2. Test / think through all possible delimiter (FIELD_DELIM, LINE_DELIM,
COLLECTION_DELIM, ESCAPE_CHAR) scenarios.
3. Test archived scenarios.
4. Test complex types
5. Test per-partition cases
6. Juxtapose java vs boost encodings supported.
7. Test all errors/warnings scenarious / corner cases.
Change-Id: I787cd01caa52a19d6645519a6cedabe0a5253a65
---
M be/src/exec/hdfs-scanner.h
M be/src/exec/text/hdfs-text-scanner.cc
M be/src/exec/text/hdfs-text-scanner.h
M be/src/runtime/descriptors.cc
M be/src/runtime/descriptors.h
M be/src/util/CMakeLists.txt
A be/src/util/decoder.cc
A be/src/util/decoder.h
M common/thrift/CatalogObjects.thrift
M fe/src/main/java/org/apache/impala/analysis/AlterTableSetTblProperties.java
M fe/src/main/java/org/apache/impala/catalog/HdfsStorageDescriptor.java
M fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java
M testdata/datasets/functional/functional_schema_template.sql
M testdata/datasets/functional/schema_constraints.csv
A testdata/decoding/cp1251_names.txt
A testdata/decoding/gbk_names.txt
A testdata/decoding/koi8r_names.txt
A testdata/decoding/latin1_names.txt
A testdata/decoding/shift_jis_names.txt
A testdata/decoding/utf16_names.txt
A testdata/decoding/utf16be_names.txt
A testdata/decoding/utf16le_names.txt
A testdata/workloads/functional-query/queries/QueryTest/decoding.test
A testdata/workloads/functional-query/queries/QueryTest/decoding_big.test
A tests/query_test/test_decoding.py
25 files changed, 537 insertions(+), 8 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/49/22049/4
--
To view, visit http://gerrit.cloudera.org:8080/22049
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I787cd01caa52a19d6645519a6cedabe0a5253a65
Gerrit-Change-Number: 22049
Gerrit-PatchSet: 4
Gerrit-Owner: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
[Impala-ASF-CR] WIP IMPALA-10319: Support arbitrary encodings on Text/Sequence files
Posted by "Mihaly Szjatinya (Code Review)" <ge...@cloudera.org>.
Mihaly Szjatinya has posted comments on this change. ( http://gerrit.cloudera.org:8080/22049 )
Change subject: WIP IMPALA-10319: Support arbitrary encodings on Text/Sequence files
......................................................................
Patch Set 7:
(2 comments)
Changes:
1. Fixed split symbols by storing partial symbol.
2. Added self generating tests for arbitrarily large volumes.
3. Improved 'alter table' analysis to check for current line.delim instead of just '\n'.
4. Changed encodingValue from required to optional
5. Bugfixing.
http://gerrit.cloudera.org:8080/#/c/22049/4/be/src/exec/text/hdfs-text-scanner.cc
File be/src/exec/text/hdfs-text-scanner.cc:
http://gerrit.cloudera.org:8080/#/c/22049/4/be/src/exec/text/hdfs-text-scanner.cc@545
PS4, Line 545: r) {
> Good point, although I'm not sure HdfsTextScanner doesn't handle this on a
Implemented the 1st option for this. Applied heuristic to find split symbol at the beginning and at the end of the buffer, to avoid copying.
http://gerrit.cloudera.org:8080/#/c/22049/6/common/thrift/CatalogObjects.thrift
File common/thrift/CatalogObjects.thrift:
http://gerrit.cloudera.org:8080/#/c/22049/6/common/thrift/CatalogObjects.thrift@359
PS6, Line 359: 9: optional string encodingValue
> Please use 'optional' instead of 'required'
Ack
--
To view, visit http://gerrit.cloudera.org:8080/22049
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I787cd01caa52a19d6645519a6cedabe0a5253a65
Gerrit-Change-Number: 22049
Gerrit-PatchSet: 7
Gerrit-Owner: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Sun, 26 Jan 2025 23:11:05 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] WIP IMPALA-10319: Support arbitrary encodings on Text/Sequence files
Posted by "Mihaly Szjatinya (Code Review)" <ge...@cloudera.org>.
Mihaly Szjatinya has uploaded a new patch set (#12). ( http://gerrit.cloudera.org:8080/22049 )
Change subject: WIP IMPALA-10319: Support arbitrary encodings on Text/Sequence files
......................................................................
WIP IMPALA-10319: Support arbitrary encodings on Text/Sequence files
As proposed in Jira, this implements decoding and encoding of text
buffers for Impala/Hive text tables. Given a table with
'serialization.encoding' property set, similarly to Hive, Impala should
be able to encode the inserted data into charset specified, consequently
saving it into a text file. The opposite decoding operation should be
performed upon reading data buffers from text files. Both operations
employ boost::locale::conv library.
Since Hive doesn't encode line delimiters, charsets that would have
delimiters stored differently from ASCII are not allowed.
One difference from Hive is that Impala implements
'serialization.encoding' only as a per partition serdeproperty to avoid
confusion of allowing both serde and tbl properties.
Change-Id: I787cd01caa52a19d6645519a6cedabe0a5253a65
---
M be/src/exec/hdfs-scanner.h
M be/src/exec/hdfs-text-table-writer.cc
M be/src/exec/hdfs-text-table-writer.h
M be/src/exec/scanner-context.h
M be/src/exec/text/hdfs-text-scanner.cc
M be/src/exec/text/hdfs-text-scanner.h
M be/src/runtime/descriptors.cc
M be/src/runtime/descriptors.h
M be/src/runtime/row-batch.h
M be/src/util/CMakeLists.txt
A be/src/util/char-codec.cc
A be/src/util/char-codec.h
M common/thrift/CatalogObjects.thrift
M common/thrift/generate_error_codes.py
M fe/src/main/java/org/apache/impala/analysis/AlterTableSetTblProperties.java
M fe/src/main/java/org/apache/impala/catalog/HdfsStorageDescriptor.java
M fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java
A tests/query_test/test_charcodec.py
18 files changed, 817 insertions(+), 16 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/49/22049/12
--
To view, visit http://gerrit.cloudera.org:8080/22049
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I787cd01caa52a19d6645519a6cedabe0a5253a65
Gerrit-Change-Number: 22049
Gerrit-PatchSet: 12
Gerrit-Owner: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
[Impala-ASF-CR] IMPALA-10319: Support arbitrary encodings on Text files
Posted by "Mihaly Szjatinya (Code Review)" <ge...@cloudera.org>.
Mihaly Szjatinya has uploaded a new patch set (#26). ( http://gerrit.cloudera.org:8080/22049 )
Change subject: IMPALA-10319: Support arbitrary encodings on Text files
......................................................................
IMPALA-10319: Support arbitrary encodings on Text files
As proposed in Jira, this implements decoding and encoding of text
buffers for Impala/Hive text tables. Given a table with
'serialization.encoding' property set, similarly to Hive, Impala should
be able to encode the inserted data into charset specified, consequently
saving it into a text file. The opposite decoding operation should be
performed upon reading data buffers from text files. Both operations
employ boost::locale::conv library.
Since Hive doesn't encode line delimiters, charsets that would have
delimiters stored differently from ASCII are not allowed.
One difference from Hive is that Impala implements
'serialization.encoding' only as a per partition serdeproperty to avoid
confusion of allowing both serde and tbl properties. (See related
IMPALA-13748)
Note: Due to precreated non-UTF-8 files present in the patch
'gerrit-code-review-checks' was performed locally. (See IMPALA-14100)
Change-Id: I787cd01caa52a19d6645519a6cedabe0a5253a65
---
M be/src/exec/hdfs-scanner.h
M be/src/exec/hdfs-text-table-writer.cc
M be/src/exec/hdfs-text-table-writer.h
M be/src/exec/scanner-context.h
M be/src/exec/text/hdfs-text-scanner.cc
M be/src/exec/text/hdfs-text-scanner.h
M be/src/runtime/descriptors.cc
M be/src/runtime/descriptors.h
M be/src/runtime/mem-tracker.cc
M be/src/runtime/mem-tracker.h
M be/src/runtime/row-batch.h
M be/src/util/CMakeLists.txt
A be/src/util/char-codec.cc
A be/src/util/char-codec.h
M common/thrift/CatalogObjects.thrift
M common/thrift/generate_error_codes.py
M fe/src/main/java/org/apache/impala/analysis/AlterTableSetTblProperties.java
M fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java
M fe/src/main/java/org/apache/impala/catalog/HdfsStorageDescriptor.java
M fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java
A testdata/charcodec/cp1251_names.txt
A testdata/charcodec/cp1251_names_utf8.txt
A testdata/charcodec/gbk_names.txt
A testdata/charcodec/gbk_names_error.txt
A testdata/charcodec/gbk_names_utf8.txt
A testdata/charcodec/koi8r_names.txt
A testdata/charcodec/koi8r_names_utf8.txt
A testdata/charcodec/latin1_names.txt
A testdata/charcodec/latin1_names_utf8.txt
A testdata/charcodec/shift_jis_names.txt
A testdata/charcodec/shift_jis_names_error.txt
A testdata/charcodec/shift_jis_names_utf8.txt
A tests/query_test/test_charcodec.py
33 files changed, 1,062 insertions(+), 20 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/49/22049/26
--
To view, visit http://gerrit.cloudera.org:8080/22049
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I787cd01caa52a19d6645519a6cedabe0a5253a65
Gerrit-Change-Number: 22049
Gerrit-PatchSet: 26
Gerrit-Owner: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Peter Rozsa <pr...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
[Impala-ASF-CR] IMPALA-10319: Support arbitrary encodings on Text files
Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/22049 )
Change subject: IMPALA-10319: Support arbitrary encodings on Text files
......................................................................
Patch Set 26: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/22049
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I787cd01caa52a19d6645519a6cedabe0a5253a65
Gerrit-Change-Number: 22049
Gerrit-PatchSet: 26
Gerrit-Owner: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Peter Rozsa <pr...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Fri, 30 May 2025 13:59:30 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10319: Support arbitrary encodings on Text files
Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/22049 )
Change subject: IMPALA-10319: Support arbitrary encodings on Text files
......................................................................
Patch Set 26: -Code-Review
there are two errors in the builds, one seems to be flaky test, the other is about linense checking - the new files in testdata/data should be added to
https://github.com/apache/impala/blob/master/bin/rat_exclude_files.txt (e.g. testdata/data/charcodec/* )
--
To view, visit http://gerrit.cloudera.org:8080/22049
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I787cd01caa52a19d6645519a6cedabe0a5253a65
Gerrit-Change-Number: 22049
Gerrit-PatchSet: 26
Gerrit-Owner: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Peter Rozsa <pr...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Sat, 31 May 2025 08:26:16 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10319: Support arbitrary encodings on Text files
Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/22049 )
Change subject: IMPALA-10319: Support arbitrary encodings on Text files
......................................................................
Patch Set 23:
(10 comments)
http://gerrit.cloudera.org:8080/#/c/22049/21/tests/query_test/test_charcodec.py
File tests/query_test/test_charcodec.py:
http://gerrit.cloudera.org:8080/#/c/22049/21/tests/query_test/test_charcodec.py@72
PS21, Line 72: class TestCharCodecGen(ImpalaTestSuite):
> The whole thing takes ~18m on my machine. So I've left only a single gbk en
It seems a bit slow, especially the 32 mins, we could check if there are some steps that could be faster or unneeded dimensions.
http://gerrit.cloudera.org:8080/#/c/22049/23/tests/query_test/test_charcodec.py
File tests/query_test/test_charcodec.py:
http://gerrit.cloudera.org:8080/#/c/22049/23/tests/query_test/test_charcodec.py@362
PS23, Line 362:
nit: trailing white space
http://gerrit.cloudera.org:8080/#/c/22049/23/tests/query_test/test_charcodec.py@372
PS23, Line 372: data_file_path = os.path.join(os.environ['IMPALA_HOME'], "testdata",
: "charcodec", datafile)
: self.filesystem_client.copy_from_local(data_file_path,
: self._get_table_location("{0}.{1}".format(db, tbl_name), vector))
: self.execute_query("""REFRESH {}.{}""".format(db, tbl_name))
there are a few function in file_utils.py that helps with such tasks, e.g. create_table_and_copy_files()
http://gerrit.cloudera.org:8080/#/c/22049/23/tests/query_test/test_charcodec.py@379
PS23, Line 379: @SkipIfFS.hive
Does this actually need Hive?
http://gerrit.cloudera.org:8080/#/c/22049/23/tests/query_test/test_charcodec.py@386
PS23, Line 386: utf8_table = self.prepare_test_table(vector, db, enc + '_names_utf8', enc + '_names_utf8.txt', None)
Too long line (why didn't jenkins warn about this?)
http://gerrit.cloudera.org:8080/#/c/22049/23/tests/query_test/test_charcodec.py@395
PS23, Line 395: result = self.client.execute("""select * from {}.{} except select * from {}.{}
: union all select * from {}.{} except select * from {}.{}"""
There could be a utility function like check_tables_equal that checks this (+ probably count(*) to verify that it is not empty)
http://gerrit.cloudera.org:8080/#/c/22049/23/tests/query_test/test_charcodec.py@400
PS23, Line 400: f
Does this actually need Hive?
http://gerrit.cloudera.org:8080/#/c/22049/23/tests/query_test/test_charcodec.py@410
PS23, Line 410: @SkipIfFS.hive
Does this actually need Hive?
http://gerrit.cloudera.org:8080/#/c/22049/23/tests/query_test/test_charcodec.py@417
PS23, Line 417: err = self.execute_query_expect_failure(self.client, """insert overwrite {}.{}
nit: trailing white space
http://gerrit.cloudera.org:8080/#/c/22049/23/tests/query_test/test_charcodec.py@432
PS23, Line 432: result_hive = self.run_stmt_in_hive("""select name from {}.{}""".format(db, encoded_table))
: result_impala = self.client.execute("""select name from {}.{}""".format(db, utf8_table))
: result_hive_list = result_hive.strip().split('\n')[1:]
: assert result_hive_list == result_impala.data
This may work for tables with a single file, but generally we cannot make assumptions about the order of rows in the result if there is no ORDER BY in the query.
--
To view, visit http://gerrit.cloudera.org:8080/22049
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I787cd01caa52a19d6645519a6cedabe0a5253a65
Gerrit-Change-Number: 22049
Gerrit-PatchSet: 23
Gerrit-Owner: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Peter Rozsa <pr...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Tue, 27 May 2025 14:58:43 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] WIP IMPALA-10319: Support arbitrary encodings on Text/Sequence files
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22049 )
Change subject: WIP IMPALA-10319: Support arbitrary encodings on Text/Sequence files
......................................................................
Patch Set 14:
(1 comment)
gerrit-auto-critic failed. You can reproduce it locally using command:
python3 bin/jenkins/critique-gerrit-review.py --dryrun
To run it, you might need a virtual env with Python3's venv installed.
http://gerrit.cloudera.org:8080/#/c/22049/14/tests/query_test/test_charcodec.py
File tests/query_test/test_charcodec.py:
http://gerrit.cloudera.org:8080/#/c/22049/14/tests/query_test/test_charcodec.py@44
PS14, Line 44: u
flake8: F821 undefined name 'unichr'
--
To view, visit http://gerrit.cloudera.org:8080/22049
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I787cd01caa52a19d6645519a6cedabe0a5253a65
Gerrit-Change-Number: 22049
Gerrit-PatchSet: 14
Gerrit-Owner: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Thu, 06 Mar 2025 10:42:22 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] WIP IMPALA-10319: Support arbitrary encodings on Text/Sequence files
Posted by "Mihaly Szjatinya (Code Review)" <ge...@cloudera.org>.
Mihaly Szjatinya has posted comments on this change. ( http://gerrit.cloudera.org:8080/22049 )
Change subject: WIP IMPALA-10319: Support arbitrary encodings on Text/Sequence files
......................................................................
Patch Set 14:
Minor fixes for jenkins tests
--
To view, visit http://gerrit.cloudera.org:8080/22049
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I787cd01caa52a19d6645519a6cedabe0a5253a65
Gerrit-Change-Number: 22049
Gerrit-PatchSet: 14
Gerrit-Owner: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Thu, 06 Mar 2025 10:43:05 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] WIP IMPALA-10319: Support arbitrary encodings on Text/Sequence files
Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/22049 )
Change subject: WIP IMPALA-10319: Support arbitrary encodings on Text/Sequence files
......................................................................
Patch Set 3:
(5 comments)
http://gerrit.cloudera.org:8080/#/c/22049/2//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/22049/2//COMMIT_MSG@7
PS2, Line 7: 19: Sup
> Frankly, at this point it's probably easier to simply add support for writi
I am ok with implementing writing too, it doesn't look hard and would make loading testdata easier (though it is still useful to test Hive interop + fixed text files).
>add a way to completely ignore serialization.encoding, both during reading and writing
This still seems useful to me, it would be simple to add a flag to ignore encoding.
http://gerrit.cloudera.org:8080/#/c/22049/2//COMMIT_MSG@23
PS2, Line 23: logic simpler, on the negative side it renders Hive files not readable
> I haven't found the concrete code for this in Hive yet. I was able to debug
We discussed this offline with Mihaly and the current consensus is to implement this only for encodings where the Hive behavior seems ok, so skip utf16/utf32.
I tried exporting to csv with openoffice and when the encoding is utf-16, it also encodes new line characters (and adds no BOM), so Hive won't read it correctly. AFAIK excel works the same way. So I am pretty certain that HIVE-14846 is a bug, but I would still prefer to avoid implementing a solution that is incompatible with Hive, so it seems better to just give an error for encodings where \n\r is different than in ASCII.
http://gerrit.cloudera.org:8080/#/c/22049/3/common/thrift/CatalogObjects.thrift
File common/thrift/CatalogObjects.thrift:
http://gerrit.cloudera.org:8080/#/c/22049/3/common/thrift/CatalogObjects.thrift@527
PS3, Line 527: encodingValue
This should be stored in THdfsStorageDescriptor with other SERDE properties: https://github.com/apache/impala/blob/2085edbe1cdeb6e01b14b83113b5bc5b9e70f86f/common/thrift/CatalogObjects.thrift#L350
This would also enable setting it per partition.
http://gerrit.cloudera.org:8080/#/c/22049/3/tests/query_test/test_decoding.py
File tests/query_test/test_decoding.py:
http://gerrit.cloudera.org:8080/#/c/22049/3/tests/query_test/test_decoding.py@36
PS3, Line 36: CREATE TABLE
It would be nice to also create a table during dataload:
https://github.com/apache/impala/blob/master/testdata/datasets/functional/functional_schema_template.sql
Section DEPENDENT_LOAD_HIVE can be used to run DMLs in Hive.
http://gerrit.cloudera.org:8080/#/c/22049/3/tests/query_test/test_decoding.py@49
PS3, Line 49: prepare_test_table('gbk_names', 'gbk_names.txt', 'GBK')
These small tables are useful to test encoding, but having a bigger table could provide more coverage. If all rows are returned in a single batch than some parts of memory handling may not get executed.
--
To view, visit http://gerrit.cloudera.org:8080/22049
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I787cd01caa52a19d6645519a6cedabe0a5253a65
Gerrit-Change-Number: 22049
Gerrit-PatchSet: 3
Gerrit-Owner: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Comment-Date: Tue, 19 Nov 2024 12:50:56 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10319: Support arbitrary encodings on Text files
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22049 )
Change subject: IMPALA-10319: Support arbitrary encodings on Text files
......................................................................
Patch Set 26:
Build Failed
https://jenkins.impala.io/job/gerrit-code-review-checks/19349/ : Initial code review checks failed. See linked job for details on the failure.
--
To view, visit http://gerrit.cloudera.org:8080/22049
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I787cd01caa52a19d6645519a6cedabe0a5253a65
Gerrit-Change-Number: 22049
Gerrit-PatchSet: 26
Gerrit-Owner: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Peter Rozsa <pr...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Fri, 30 May 2025 14:19:37 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10319: Support arbitrary encodings on Text files
Posted by "Mihaly Szjatinya (Code Review)" <ge...@cloudera.org>.
Mihaly Szjatinya has uploaded a new patch set (#28). ( http://gerrit.cloudera.org:8080/22049 )
Change subject: IMPALA-10319: Support arbitrary encodings on Text files
......................................................................
IMPALA-10319: Support arbitrary encodings on Text files
As proposed in Jira, this implements decoding and encoding of text
buffers for Impala/Hive text tables. Given a table with
'serialization.encoding' property set, similarly to Hive, Impala should
be able to encode the inserted data into charset specified, consequently
saving it into a text file. The opposite decoding operation should be
performed upon reading data buffers from text files. Both operations
employ boost::locale::conv library.
Since Hive doesn't encode line delimiters, charsets that would have
delimiters stored differently from ASCII are not allowed.
One difference from Hive is that Impala implements
'serialization.encoding' only as a per partition serdeproperty to avoid
confusion of allowing both serde and tbl properties. (See related
IMPALA-13748)
Note: Due to precreated non-UTF-8 files present in the patch
'gerrit-code-review-checks' was performed locally. (See IMPALA-14100)
Change-Id: I787cd01caa52a19d6645519a6cedabe0a5253a65
---
M be/src/exec/hdfs-scanner.h
M be/src/exec/hdfs-text-table-writer.cc
M be/src/exec/hdfs-text-table-writer.h
M be/src/exec/scanner-context.h
M be/src/exec/text/hdfs-text-scanner.cc
M be/src/exec/text/hdfs-text-scanner.h
M be/src/runtime/descriptors.cc
M be/src/runtime/descriptors.h
M be/src/runtime/mem-tracker.cc
M be/src/runtime/mem-tracker.h
M be/src/runtime/row-batch.h
M be/src/util/CMakeLists.txt
A be/src/util/char-codec.cc
A be/src/util/char-codec.h
M bin/rat_exclude_files.txt
M common/thrift/CatalogObjects.thrift
M common/thrift/generate_error_codes.py
M fe/src/main/java/org/apache/impala/analysis/AlterTableSetTblProperties.java
M fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java
M fe/src/main/java/org/apache/impala/catalog/HdfsStorageDescriptor.java
M fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java
A testdata/charcodec/cp1251_names.txt
A testdata/charcodec/cp1251_names_utf8.txt
A testdata/charcodec/gbk_names.txt
A testdata/charcodec/gbk_names_error.txt
A testdata/charcodec/gbk_names_utf8.txt
A testdata/charcodec/koi8r_names.txt
A testdata/charcodec/koi8r_names_utf8.txt
A testdata/charcodec/latin1_names.txt
A testdata/charcodec/latin1_names_utf8.txt
A testdata/charcodec/shift_jis_names.txt
A testdata/charcodec/shift_jis_names_error.txt
A testdata/charcodec/shift_jis_names_utf8.txt
A tests/query_test/test_charcodec.py
34 files changed, 1,063 insertions(+), 20 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/49/22049/28
--
To view, visit http://gerrit.cloudera.org:8080/22049
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I787cd01caa52a19d6645519a6cedabe0a5253a65
Gerrit-Change-Number: 22049
Gerrit-PatchSet: 28
Gerrit-Owner: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Peter Rozsa <pr...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
[Impala-ASF-CR] IMPALA-10319: Support arbitrary encodings on Text files
Posted by "Mihaly Szjatinya (Code Review)" <ge...@cloudera.org>.
Mihaly Szjatinya has posted comments on this change. ( http://gerrit.cloudera.org:8080/22049 )
Change subject: IMPALA-10319: Support arbitrary encodings on Text files
......................................................................
Patch Set 24:
(12 comments)
http://gerrit.cloudera.org:8080/#/c/22049/21/be/src/util/char-codec.cc
File be/src/util/char-codec.cc:
http://gerrit.cloudera.org:8080/#/c/22049/21/be/src/util/char-codec.cc@61
PS21, Line 61: scoped_mem_track
> I meant the variable name could follow the snake-case, as other variable na
Oh, ok, why not.
http://gerrit.cloudera.org:8080/#/c/22049/21/be/src/util/char-codec.cc@103
PS21, Line 103: prefix
> partial_symbol can be used to build up the symbol from the new buffer, as i
This would indeed save an extra vector and a few lines of code, but on the other side it would lower function's exception-safety guarantee from strong to basic, by:
1. chaning class-level 'partial_symbol_' in case of throw outside funcion.
2. more importantly, changing 'buf_start' in case of throw outside function, and thus invalidating it.
This may be functionally ok with the current code outside the function but generally it's a good design to keep throwing functions with strong guarantee. Besides, it's easier to debug this way.
We could probably still get rid of 'success' flag, but I would keep it for consistency with later 'HandleSuffix'.
http://gerrit.cloudera.org:8080/#/c/22049/21/tests/query_test/test_charcodec.py
File tests/query_test/test_charcodec.py:
http://gerrit.cloudera.org:8080/#/c/22049/21/tests/query_test/test_charcodec.py@72
PS21, Line 72: class TestCharCodecGen(ImpalaTestSuite):
> It seems a bit slow, especially the 32 mins, we could check if there are so
If it's not important to have 'disable_codegen' both True and False of exhaustive mode, I've constrained it out to cut time in half.
We might also generate smaller tables.
http://gerrit.cloudera.org:8080/#/c/22049/23/tests/query_test/test_charcodec.py
File tests/query_test/test_charcodec.py:
http://gerrit.cloudera.org:8080/#/c/22049/23/tests/query_test/test_charcodec.py@362
PS23, Line 362: # There is no reason to run these tests using all dimensions.
> nit: trailing white space
ack
http://gerrit.cloudera.org:8080/#/c/22049/23/tests/query_test/test_charcodec.py@372
PS23, Line 372: self.execute_query("""CREATE TABLE IF NOT EXISTS {}.{} (name STRING)
: STORED AS TEXTFILE""".format(db, tbl_name))
: if encoding:
: self.execute_query("""ALTER TABLE {}.{} SET
: SERDEPROPERTIES("serialization.encoding"="{}")"""
> there are a few function in file_utils.py that helps with such tasks, e.g.
I wasn't aware of that function, but unfortunately it copies the entire folder, while we have a folder with different files for each table.
And more generally perhaps it would be more consistent with the above tests where we do this manually for different reasons.
http://gerrit.cloudera.org:8080/#/c/22049/23/tests/query_test/test_charcodec.py@379
PS23, Line 379: "charcod
> Does this actually need Hive?
This comes down simply to 'IS_HDFS' check and I thought many tests use this just as a basic sanity check, not necessarily only with hive involved. Isn't that the case?
http://gerrit.cloudera.org:8080/#/c/22049/23/tests/query_test/test_charcodec.py@386
PS23, Line 386: def test_precreated_files(self, vector, unique_database):
> Too long line (why didn't jenkins warn about this?)
Sorry for this and similar, just hadn't updated small fixes yet.
http://gerrit.cloudera.org:8080/#/c/22049/23/tests/query_test/test_charcodec.py@395
PS23, Line 395: vector, db, enc + '_names_none', enc + '_names.txt', None)
: with pytest.raises(AssertionError) as exc_info:
> There could be a utility function like check_tables_equal that checks this
We can use '_compare_tables' from this file above. Good idea.
http://gerrit.cloudera.org:8080/#/c/22049/23/tests/query_test/test_charcodec.py@400
PS23, Line 400: t
> Does this actually need Hive?
Ack
http://gerrit.cloudera.org:8080/#/c/22049/23/tests/query_test/test_charcodec.py@410
PS23, Line 410: if enc not in ['gbk', 'shift_jis']: pytest.skip()
> Does this actually need Hive?
Ack
http://gerrit.cloudera.org:8080/#/c/22049/23/tests/query_test/test_charcodec.py@417
PS23, Line 417: @SkipIfFS.hive
> nit: trailing white space
Ack
http://gerrit.cloudera.org:8080/#/c/22049/23/tests/query_test/test_charcodec.py@432
PS23, Line 432: """Write table with Impala and read it back with Hive."""
: db = unique_database
: enc = vector.get_value('charset')
:
> This may work for tables with a single file, but generally we cannot make a
Interestingly enough. after adding 'order by name' to both queries the comparison doesn't work anymore. It's probably due to ordering taking place before decoding, which likely deserves a separate discussion.
In this test however we have tiny single files. Or if needed we can do an ordering in python here.
--
To view, visit http://gerrit.cloudera.org:8080/22049
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I787cd01caa52a19d6645519a6cedabe0a5253a65
Gerrit-Change-Number: 22049
Gerrit-PatchSet: 24
Gerrit-Owner: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Peter Rozsa <pr...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Wed, 28 May 2025 22:52:32 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] WIP IMPALA-10319: Support arbitrary encodings on Text/Sequence files
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22049 )
Change subject: WIP IMPALA-10319: Support arbitrary encodings on Text/Sequence files
......................................................................
Patch Set 9:
(17 comments)
gerrit-auto-critic failed. You can reproduce it locally using command:
python3 bin/jenkins/critique-gerrit-review.py --dryrun
To run it, you might need a virtual env with Python3's venv installed.
http://gerrit.cloudera.org:8080/#/c/22049/9/tests/query_test/test_charcodec.py
File tests/query_test/test_charcodec.py:
http://gerrit.cloudera.org:8080/#/c/22049/9/tests/query_test/test_charcodec.py@23
PS9, Line 23: from tests.common.test_result_verifier import verify_query_result_is_equal
flake8: F401 'tests.common.test_result_verifier.verify_query_result_is_equal' imported but unused
http://gerrit.cloudera.org:8080/#/c/22049/9/tests/query_test/test_charcodec.py@44
PS9, Line 44: u
flake8: F821 undefined name 'unichr'
http://gerrit.cloudera.org:8080/#/c/22049/9/tests/query_test/test_charcodec.py@45
PS9, Line 45: u
flake8: F821 undefined name 'unichr'
http://gerrit.cloudera.org:8080/#/c/22049/9/tests/query_test/test_charcodec.py@46
PS9, Line 46: u
flake8: F821 undefined name 'unichr'
http://gerrit.cloudera.org:8080/#/c/22049/9/tests/query_test/test_charcodec.py@48
PS9, Line 48: u
flake8: F821 undefined name 'unichr'
http://gerrit.cloudera.org:8080/#/c/22049/9/tests/query_test/test_charcodec.py@49
PS9, Line 49: u
flake8: F821 undefined name 'unichr'
http://gerrit.cloudera.org:8080/#/c/22049/9/tests/query_test/test_charcodec.py@59
PS9, Line 59: class TestCharCodecGen(ImpalaTestSuite):
flake8: E302 expected 2 blank lines, found 1
http://gerrit.cloudera.org:8080/#/c/22049/9/tests/query_test/test_charcodec.py@135
PS9, Line 135: v
flake8: U100 Unused argument 'vector'
http://gerrit.cloudera.org:8080/#/c/22049/9/tests/query_test/test_charcodec.py@147
PS9, Line 147: v
flake8: U100 Unused argument 'vector'
http://gerrit.cloudera.org:8080/#/c/22049/9/tests/query_test/test_charcodec.py@174
PS9, Line 174: v
flake8: U100 Unused argument 'vector'
http://gerrit.cloudera.org:8080/#/c/22049/9/tests/query_test/test_charcodec.py@203
PS9, Line 203:
flake8: W293 blank line contains whitespace
http://gerrit.cloudera.org:8080/#/c/22049/9/tests/query_test/test_charcodec.py@203
PS9, Line 203:
line has trailing whitespace
http://gerrit.cloudera.org:8080/#/c/22049/9/tests/query_test/test_charcodec.py@225
PS9, Line 225: v
flake8: U100 Unused argument 'vector'
http://gerrit.cloudera.org:8080/#/c/22049/9/tests/query_test/test_charcodec.py@233
PS9, Line 233: ,
flake8: E231 missing whitespace after ','
http://gerrit.cloudera.org:8080/#/c/22049/9/tests/query_test/test_charcodec.py@289
PS9, Line 289:
flake8: W293 blank line contains whitespace
http://gerrit.cloudera.org:8080/#/c/22049/9/tests/query_test/test_charcodec.py@289
PS9, Line 289:
line has trailing whitespace
http://gerrit.cloudera.org:8080/#/c/22049/9/tests/query_test/test_charcodec.py@291
PS9, Line 291: v
flake8: U100 Unused argument 'vector'
--
To view, visit http://gerrit.cloudera.org:8080/22049
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I787cd01caa52a19d6645519a6cedabe0a5253a65
Gerrit-Change-Number: 22049
Gerrit-PatchSet: 9
Gerrit-Owner: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Tue, 11 Feb 2025 21:48:09 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] WIP IMPALA-10319: Support arbitrary encodings on Text/Sequence files
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22049 )
Change subject: WIP IMPALA-10319: Support arbitrary encodings on Text/Sequence files
......................................................................
Patch Set 11:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/18263/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/22049
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I787cd01caa52a19d6645519a6cedabe0a5253a65
Gerrit-Change-Number: 22049
Gerrit-PatchSet: 11
Gerrit-Owner: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Tue, 11 Feb 2025 22:42:58 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] WIP IMPALA-10319: Support arbitrary encodings on Text/Sequence files
Posted by "Mihaly Szjatinya (Code Review)" <ge...@cloudera.org>.
Mihaly Szjatinya has posted comments on this change. ( http://gerrit.cloudera.org:8080/22049 )
Change subject: WIP IMPALA-10319: Support arbitrary encodings on Text/Sequence files
......................................................................
Patch Set 3:
(3 comments)
Thanks, should I proceed with expanding it for Sequence and writing?
http://gerrit.cloudera.org:8080/#/c/22049/2//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/22049/2//COMMIT_MSG@7
PS2, Line 7: 19: Sup
> The patch only adds read support, this could be highlighted in the title.
Frankly, at this point it's probably easier to simply add support for writing, mirroring what scanning does.
http://gerrit.cloudera.org:8080/#/c/22049/2//COMMIT_MSG@9
PS2, Line 9: As prop
> In Impala project the non-final status of a patch is usually marked in the
Done
http://gerrit.cloudera.org:8080/#/c/22049/2//COMMIT_MSG@23
PS2, Line 23: logic simpler, on the negative side it renders Hive files not readable
I haven't found the concrete code for this in Hive yet. I was able to debug some parts of code but not all of the code.
Technically in this scope we should just reflect the state of art Hive behavior whatever that is. But can we push this Jira issue you found to Hive folks to give an answer?
> but I see no valid reason behind including BOM in every line
yes, that's almost certainly a bug.
--
To view, visit http://gerrit.cloudera.org:8080/22049
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I787cd01caa52a19d6645519a6cedabe0a5253a65
Gerrit-Change-Number: 22049
Gerrit-PatchSet: 3
Gerrit-Owner: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Comment-Date: Sun, 17 Nov 2024 10:53:58 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] WIP IMPALA-10319: Support arbitrary encodings on Text/Sequence files
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22049 )
Change subject: WIP IMPALA-10319: Support arbitrary encodings on Text/Sequence files
......................................................................
Patch Set 18:
(7 comments)
gerrit-auto-critic failed. You can reproduce it locally using command:
python3 bin/jenkins/critique-gerrit-review.py --dryrun
To run it, you might need a virtual env with Python3's venv installed.
http://gerrit.cloudera.org:8080/#/c/22049/18/tests/query_test/test_charcodec.py
File tests/query_test/test_charcodec.py:
http://gerrit.cloudera.org:8080/#/c/22049/18/tests/query_test/test_charcodec.py@22
PS18, Line 22: from tests.util.filesystem_utils import WAREHOUSE
flake8: F401 'tests.util.filesystem_utils.WAREHOUSE' imported but unused
http://gerrit.cloudera.org:8080/#/c/22049/18/tests/query_test/test_charcodec.py@29
PS18, Line 29: u
flake8: F821 undefined name 'unichr'
http://gerrit.cloudera.org:8080/#/c/22049/18/tests/query_test/test_charcodec.py@48
PS18, Line 48: def _generate_random_word(charset, min_length=1, max_length=20):
flake8: E302 expected 2 blank lines, found 1
http://gerrit.cloudera.org:8080/#/c/22049/18/tests/query_test/test_charcodec.py@52
PS18, Line 52: def _compare_tables(selfobj, db, utf8_table, encoded_table):
flake8: E302 expected 2 blank lines, found 1
http://gerrit.cloudera.org:8080/#/c/22049/18/tests/query_test/test_charcodec.py@66
PS18, Line 66: class TestCharCodecGen(ImpalaTestSuite):
flake8: E302 expected 2 blank lines, found 1
http://gerrit.cloudera.org:8080/#/c/22049/18/tests/query_test/test_charcodec.py@251
PS18, Line 251:
flake8: W291 trailing whitespace
http://gerrit.cloudera.org:8080/#/c/22049/18/tests/query_test/test_charcodec.py@251
PS18, Line 251: ####################################################################
line has trailing whitespace
--
To view, visit http://gerrit.cloudera.org:8080/22049
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I787cd01caa52a19d6645519a6cedabe0a5253a65
Gerrit-Change-Number: 22049
Gerrit-PatchSet: 18
Gerrit-Owner: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Sun, 27 Apr 2025 12:00:25 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] WIP IMPALA-10319: Support arbitrary encodings on Text/Sequence files
Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/22049 )
Change subject: WIP IMPALA-10319: Support arbitrary encodings on Text/Sequence files
......................................................................
Patch Set 19: Code-Review+1
(10 comments)
Found a few other things, but otherwise looks good.
http://gerrit.cloudera.org:8080/#/c/22049/19//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/22049/19//COMMIT_MSG@7
PS19, Line 7: WIP
Is it still WIP?
http://gerrit.cloudera.org:8080/#/c/22049/19/be/src/exec/text/hdfs-text-scanner.h
File be/src/exec/text/hdfs-text-scanner.h:
http://gerrit.cloudera.org:8080/#/c/22049/19/be/src/exec/text/hdfs-text-scanner.h@272
PS19, Line 272: boost::scoped_ptr
nit: Can this be unique_ptr?
http://gerrit.cloudera.org:8080/#/c/22049/17/be/src/exec/text/hdfs-text-scanner.h
File be/src/exec/text/hdfs-text-scanner.h:
http://gerrit.cloudera.org:8080/#/c/22049/17/be/src/exec/text/hdfs-text-scanner.h@272
PS17, Line 272: boost::scoped_ptr
> We prefer std::unique_ptr over boost:scoped_ptr
This comment is unresolved.
http://gerrit.cloudera.org:8080/#/c/22049/17/be/src/exec/text/hdfs-text-scanner.cc
File be/src/exec/text/hdfs-text-scanner.cc:
http://gerrit.cloudera.org:8080/#/c/22049/17/be/src/exec/text/hdfs-text-scanner.cc@545
PS17, Line 545: if (decoder_.get() != nullptr)
> In this and related modules it is scoped_ptr with this pattern everywhere.
I think in new methods/classes, the less code is the better. In files where we already do 'decompressor_.get() == nullptr' many times I'm OK if we lean towards consistency.
http://gerrit.cloudera.org:8080/#/c/22049/19/be/src/util/char-codec.h
File be/src/util/char-codec.h:
http://gerrit.cloudera.org:8080/#/c/22049/19/be/src/util/char-codec.h@34
PS19, Line 34: // Output buffer size for streaming encoded file.
: static const int64_t STREAM_OUT_BUF_SIZE;
Seems to be unused.
http://gerrit.cloudera.org:8080/#/c/22049/19/be/src/util/char-codec.cc
File be/src/util/char-codec.cc:
http://gerrit.cloudera.org:8080/#/c/22049/19/be/src/util/char-codec.cc@37
PS19, Line 37: MAX_SYMBOL
To limit the scope of this constant, it could be static or put into an unnamed namespace.
http://gerrit.cloudera.org:8080/#/c/22049/19/be/src/util/char-codec.cc@119
PS19, Line 119: // continue
Just a thought: it might make sense to make the 'continue' explicit, so if later we add code after the try-catch (but still in the for-loop body), it won't be executed.
http://gerrit.cloudera.org:8080/#/c/22049/17/tests/query_test/test_charcodec.py
File tests/query_test/test_charcodec.py:
http://gerrit.cloudera.org:8080/#/c/22049/17/tests/query_test/test_charcodec.py@34
PS17, Line 34:
> boost::locale::conv() stambles on these and if I get it right strictly spea
I think it was better to list them explicitly under the name 'problematic_symbols_...', I was just curious about what makes them problematic, but I'm OK if we don't exactly know. I see that few symbols don't have a mapping based on wikipedia:
https://en.wikipedia.org/wiki/Hiragana_(Unicode_block)
http://gerrit.cloudera.org:8080/#/c/22049/17/tests/query_test/test_charcodec.py@44
PS17, Line 44: '
> @Zoltan, do you know how to fix this btw?
Is this self-assignment required at all?
http://gerrit.cloudera.org:8080/#/c/22049/19/tests/query_test/test_charcodec.py
File tests/query_test/test_charcodec.py:
http://gerrit.cloudera.org:8080/#/c/22049/19/tests/query_test/test_charcodec.py@48
PS19, Line 48: max_length=20):
I don't see tests with large words. Can we add a few?
--
To view, visit http://gerrit.cloudera.org:8080/22049
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I787cd01caa52a19d6645519a6cedabe0a5253a65
Gerrit-Change-Number: 22049
Gerrit-PatchSet: 19
Gerrit-Owner: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Mon, 28 Apr 2025 13:58:39 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10319: Support arbitrary encodings on Text/Sequence files
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22049 )
Change subject: IMPALA-10319: Support arbitrary encodings on Text/Sequence files
......................................................................
Patch Set 21:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/19051/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/22049
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I787cd01caa52a19d6645519a6cedabe0a5253a65
Gerrit-Change-Number: 22049
Gerrit-PatchSet: 21
Gerrit-Owner: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Tue, 29 Apr 2025 12:09:03 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] WIP IMPALA-10319: Support arbitrary encodings on Text/Sequence files
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22049 )
Change subject: WIP IMPALA-10319: Support arbitrary encodings on Text/Sequence files
......................................................................
Patch Set 17:
(1 comment)
gerrit-auto-critic failed. You can reproduce it locally using command:
python3 bin/jenkins/critique-gerrit-review.py --dryrun
To run it, you might need a virtual env with Python3's venv installed.
http://gerrit.cloudera.org:8080/#/c/22049/17/tests/query_test/test_charcodec.py
File tests/query_test/test_charcodec.py:
http://gerrit.cloudera.org:8080/#/c/22049/17/tests/query_test/test_charcodec.py@44
PS17, Line 44: u
flake8: F821 undefined name 'unichr'
--
To view, visit http://gerrit.cloudera.org:8080/22049
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I787cd01caa52a19d6645519a6cedabe0a5253a65
Gerrit-Change-Number: 22049
Gerrit-PatchSet: 17
Gerrit-Owner: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Tue, 15 Apr 2025 19:53:42 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] WIP IMPALA-10319: Support arbitrary encodings on Text/Sequence files
Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/22049 )
Change subject: WIP IMPALA-10319: Support arbitrary encodings on Text/Sequence files
......................................................................
Patch Set 17:
(19 comments)
Overall looks good. Found only minor issues.
http://gerrit.cloudera.org:8080/#/c/22049/17/be/src/exec/hdfs-scanner.h
File be/src/exec/hdfs-scanner.h:
http://gerrit.cloudera.org:8080/#/c/22049/17/be/src/exec/hdfs-scanner.h@324
PS17, Line 324:
nit: extra space. Same goes for next line.
http://gerrit.cloudera.org:8080/#/c/22049/17/be/src/exec/hdfs-text-table-writer.h
File be/src/exec/hdfs-text-table-writer.h:
http://gerrit.cloudera.org:8080/#/c/22049/17/be/src/exec/hdfs-text-table-writer.h@93
PS17, Line 93: boost::scoped_ptr
We prefer std::unique_ptr over scoped_ptr
http://gerrit.cloudera.org:8080/#/c/22049/17/be/src/exec/hdfs-text-table-writer.cc
File be/src/exec/hdfs-text-table-writer.cc:
http://gerrit.cloudera.org:8080/#/c/22049/17/be/src/exec/hdfs-text-table-writer.cc@56
PS17, Line 56: HdfsTextTableWriter::~HdfsTextTableWriter() {
: }
Why did you move the empty destructor to the .cc file?
http://gerrit.cloudera.org:8080/#/c/22049/17/be/src/exec/hdfs-text-table-writer.cc@62
PS17, Line 62: auto
No need to copy the string, it could be const auto&
http://gerrit.cloudera.org:8080/#/c/22049/17/be/src/exec/hdfs-text-table-writer.cc@64
PS17, Line 64: NULL
nit: we prefer nullptr over NULL
http://gerrit.cloudera.org:8080/#/c/22049/17/be/src/exec/hdfs-text-table-writer.cc@161
PS17, Line 161: if (encoder_.get() != nullptr)
This could be just 'if (encoder_)'
http://gerrit.cloudera.org:8080/#/c/22049/17/be/src/exec/text/hdfs-text-scanner.h
File be/src/exec/text/hdfs-text-scanner.h:
http://gerrit.cloudera.org:8080/#/c/22049/17/be/src/exec/text/hdfs-text-scanner.h@272
PS17, Line 272: boost::scoped_ptr
We prefer std::unique_ptr over boost:scoped_ptr
http://gerrit.cloudera.org:8080/#/c/22049/17/be/src/exec/text/hdfs-text-scanner.cc
File be/src/exec/text/hdfs-text-scanner.cc:
http://gerrit.cloudera.org:8080/#/c/22049/17/be/src/exec/text/hdfs-text-scanner.cc@253
PS17, Line 253: auto
This could be const auto&
http://gerrit.cloudera.org:8080/#/c/22049/17/be/src/exec/text/hdfs-text-scanner.cc@545
PS17, Line 545: if (decoder_.get() != nullptr)
nit: Could be 'if (decoder_)'
http://gerrit.cloudera.org:8080/#/c/22049/17/be/src/exec/text/hdfs-text-scanner.cc@548
PS17, Line 548: decompressor_.get() != nullptr
nit: 'decompressor_ == nullptr' should be enough. Same goos for L749.
http://gerrit.cloudera.org:8080/#/c/22049/17/be/src/runtime/descriptors.h
File be/src/runtime/descriptors.h:
http://gerrit.cloudera.org:8080/#/c/22049/17/be/src/runtime/descriptors.h@402
PS17, Line 402: std::string
Do you need to return a new object? If not, it could return const std::string&
http://gerrit.cloudera.org:8080/#/c/22049/17/be/src/util/char-codec.h
File be/src/util/char-codec.h:
http://gerrit.cloudera.org:8080/#/c/22049/17/be/src/util/char-codec.h@21
PS17, Line 21: #include <boost/scoped_ptr.hpp>
Unused include
http://gerrit.cloudera.org:8080/#/c/22049/17/be/src/util/char-codec.h@41
PS17, Line 41:
nit: redundant space
http://gerrit.cloudera.org:8080/#/c/22049/17/be/src/util/char-codec.cc
File be/src/util/char-codec.cc:
http://gerrit.cloudera.org:8080/#/c/22049/17/be/src/util/char-codec.cc@63
PS17, Line 63: buf_end - buf_start
nit: *bytes_read
http://gerrit.cloudera.org:8080/#/c/22049/17/be/src/util/char-codec.cc@86
PS17, Line 86: *buffer
This should be 'out_buffer_'.
http://gerrit.cloudera.org:8080/#/c/22049/17/be/src/util/char-codec.cc@104
PS17, Line 104: std::vector<uint8_t> prefix = partial_symbol_;
You could reserve memory after this line to decrease number of re-allocations and copyings:
std::vector<uint8_t> prefix;
prefix.reserve(partial_symbol_.size() + buf_end - *buf_start);
prefix.assign(partial_symbol_.begin(), partial_symbol_.end();
http://gerrit.cloudera.org:8080/#/c/22049/17/fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java
File fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java:
http://gerrit.cloudera.org:8080/#/c/22049/17/fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java@940
PS17, Line 940:
nit: indentation is off
http://gerrit.cloudera.org:8080/#/c/22049/17/tests/query_test/test_charcodec.py
File tests/query_test/test_charcodec.py:
http://gerrit.cloudera.org:8080/#/c/22049/17/tests/query_test/test_charcodec.py@29
PS17, Line 29: def get_table_loc(db, tbl_name):
: return '%s/%s.db/%s/' % (WAREHOUSE, db, tbl_name,)
We should probably move this to ImpalaTestSuite. We have a similar function in test_iceberg.py as well.
http://gerrit.cloudera.org:8080/#/c/22049/17/tests/query_test/test_charcodec.py@34
PS17, Line 34: problematic_symbols_hiragana = {0x3040, 0x3094, 0x3095, 0x3096, 0x3097, 0x3098, 0x3099,
Please add comment why they are problematic.
--
To view, visit http://gerrit.cloudera.org:8080/22049
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I787cd01caa52a19d6645519a6cedabe0a5253a65
Gerrit-Change-Number: 22049
Gerrit-PatchSet: 17
Gerrit-Owner: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Fri, 25 Apr 2025 12:56:05 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10319: Support arbitrary encodings on Text/Sequence files
Posted by "Mihaly Szjatinya (Code Review)" <ge...@cloudera.org>.
Mihaly Szjatinya has posted comments on this change. ( http://gerrit.cloudera.org:8080/22049 )
Change subject: IMPALA-10319: Support arbitrary encodings on Text/Sequence files
......................................................................
Patch Set 19:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/22049/19/be/src/util/char-codec.cc
File be/src/util/char-codec.cc:
http://gerrit.cloudera.org:8080/#/c/22049/19/be/src/util/char-codec.cc@37
PS19, Line 37: MAX_SYMBOL
> To limit the scope of this constant, it could be static or put into an unna
Agree, maybe even better include it into class.
http://gerrit.cloudera.org:8080/#/c/22049/17/tests/query_test/test_charcodec.py
File tests/query_test/test_charcodec.py:
http://gerrit.cloudera.org:8080/#/c/22049/17/tests/query_test/test_charcodec.py@34
PS17, Line 34:
> I think it was better to list them explicitly under the name 'problematic_s
Changed to something in between, pls. check how it's now
--
To view, visit http://gerrit.cloudera.org:8080/22049
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I787cd01caa52a19d6645519a6cedabe0a5253a65
Gerrit-Change-Number: 22049
Gerrit-PatchSet: 19
Gerrit-Owner: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Tue, 29 Apr 2025 11:38:11 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] WIP IMPALA-10319: Support arbitrary encodings on Text/Sequence files
Posted by "Mihaly Szjatinya (Code Review)" <ge...@cloudera.org>.
Mihaly Szjatinya has uploaded a new patch set (#16). ( http://gerrit.cloudera.org:8080/22049 )
Change subject: WIP IMPALA-10319: Support arbitrary encodings on Text/Sequence files
......................................................................
WIP IMPALA-10319: Support arbitrary encodings on Text/Sequence files
As proposed in Jira, this implements decoding and encoding of text
buffers for Impala/Hive text tables. Given a table with
'serialization.encoding' property set, similarly to Hive, Impala should
be able to encode the inserted data into charset specified, consequently
saving it into a text file. The opposite decoding operation should be
performed upon reading data buffers from text files. Both operations
employ boost::locale::conv library.
Since Hive doesn't encode line delimiters, charsets that would have
delimiters stored differently from ASCII are not allowed.
One difference from Hive is that Impala implements
'serialization.encoding' only as a per partition serdeproperty to avoid
confusion of allowing both serde and tbl properties. (See related
IMPALA-13748)
Change-Id: I787cd01caa52a19d6645519a6cedabe0a5253a65
---
M be/src/exec/hdfs-scanner.h
M be/src/exec/hdfs-text-table-writer.cc
M be/src/exec/hdfs-text-table-writer.h
M be/src/exec/scanner-context.h
M be/src/exec/text/hdfs-text-scanner.cc
M be/src/exec/text/hdfs-text-scanner.h
M be/src/runtime/descriptors.cc
M be/src/runtime/descriptors.h
M be/src/runtime/mem-tracker.cc
M be/src/runtime/mem-tracker.h
M be/src/runtime/row-batch.h
M be/src/util/CMakeLists.txt
A be/src/util/char-codec.cc
A be/src/util/char-codec.h
M common/thrift/CatalogObjects.thrift
M common/thrift/generate_error_codes.py
M fe/src/main/java/org/apache/impala/analysis/AlterTableSetTblProperties.java
M fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java
M fe/src/main/java/org/apache/impala/catalog/HdfsStorageDescriptor.java
M fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java
A tests/query_test/test_charcodec.py
21 files changed, 897 insertions(+), 16 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/49/22049/16
--
To view, visit http://gerrit.cloudera.org:8080/22049
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I787cd01caa52a19d6645519a6cedabe0a5253a65
Gerrit-Change-Number: 22049
Gerrit-PatchSet: 16
Gerrit-Owner: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
[Impala-ASF-CR] WIP IMPALA-10319: Support arbitrary encodings on Text/Sequence files
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22049 )
Change subject: WIP IMPALA-10319: Support arbitrary encodings on Text/Sequence files
......................................................................
Patch Set 16:
(2 comments)
gerrit-auto-critic failed. You can reproduce it locally using command:
python3 bin/jenkins/critique-gerrit-review.py --dryrun
To run it, you might need a virtual env with Python3's venv installed.
http://gerrit.cloudera.org:8080/#/c/22049/16/tests/query_test/test_charcodec.py
File tests/query_test/test_charcodec.py:
http://gerrit.cloudera.org:8080/#/c/22049/16/tests/query_test/test_charcodec.py@44
PS16, Line 44: u
flake8: F821 undefined name 'unichr'
http://gerrit.cloudera.org:8080/#/c/22049/16/tests/query_test/test_charcodec.py@48
PS16, Line 48: class TestCharCodecGen(ImpalaTestSuite):
flake8: E302 expected 2 blank lines, found 1
--
To view, visit http://gerrit.cloudera.org:8080/22049
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I787cd01caa52a19d6645519a6cedabe0a5253a65
Gerrit-Change-Number: 22049
Gerrit-PatchSet: 16
Gerrit-Owner: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Tue, 15 Apr 2025 19:50:12 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] WIP IMPALA-10319: Support arbitrary encodings on Text/Sequence files
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22049 )
Change subject: WIP IMPALA-10319: Support arbitrary encodings on Text/Sequence files
......................................................................
Patch Set 15:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/18711/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/22049
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I787cd01caa52a19d6645519a6cedabe0a5253a65
Gerrit-Change-Number: 22049
Gerrit-PatchSet: 15
Gerrit-Owner: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Wed, 26 Mar 2025 14:02:32 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] WIP IMPALA-10319: Support arbitrary encodings on Text/Sequence files
Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/22049 )
Change subject: WIP IMPALA-10319: Support arbitrary encodings on Text/Sequence files
......................................................................
Patch Set 6:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/22049/6/common/thrift/CatalogObjects.thrift
File common/thrift/CatalogObjects.thrift:
http://gerrit.cloudera.org:8080/#/c/22049/6/common/thrift/CatalogObjects.thrift@359
PS6, Line 359: 9: required string encodingValue
> Modifying/adding this required field might break the compatibility between
Please use 'optional' instead of 'required'
--
To view, visit http://gerrit.cloudera.org:8080/22049
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I787cd01caa52a19d6645519a6cedabe0a5253a65
Gerrit-Change-Number: 22049
Gerrit-PatchSet: 6
Gerrit-Owner: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Wed, 08 Jan 2025 13:48:07 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] WIP IMPALA-10319: Support arbitrary encodings on Text/Sequence files
Posted by "Mihaly Szjatinya (Code Review)" <ge...@cloudera.org>.
Mihaly Szjatinya has posted comments on this change. ( http://gerrit.cloudera.org:8080/22049 )
Change subject: WIP IMPALA-10319: Support arbitrary encodings on Text/Sequence files
......................................................................
Patch Set 9:
(9 comments)
- Answered everything except mem management (for next patch).
- Added more self-generating tests.
- Removed previous .tests.
http://gerrit.cloudera.org:8080/#/c/22049/8//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/22049/8//COMMIT_MSG@19
PS8, Line 19:
> typo
ack
http://gerrit.cloudera.org:8080/#/c/22049/8//COMMIT_MSG@33
PS8, Line 33:
> Complex types do not seem relevant here as those are only supported in Parq
ack
http://gerrit.cloudera.org:8080/#/c/22049/8//COMMIT_MSG@35
PS8, Line 35:
> It doesn't seem that critical to me to test per-partition different encodin
I've already implemented it locally itm.
http://gerrit.cloudera.org:8080/#/c/22049/8//COMMIT_MSG@37
PS8, Line 37:
> This could be done by exposing a c++ function to check compatibility which
ack
http://gerrit.cloudera.org:8080/#/c/22049/8/be/src/exec/hdfs-text-table-writer.cc
File be/src/exec/hdfs-text-table-writer.cc:
http://gerrit.cloudera.org:8080/#/c/22049/8/be/src/exec/hdfs-text-table-writer.cc@162
PS8, Line 162: RETURN_IF_ERROR(encoder_->EncodeBuffer(
: rowbatch_stringstream_.str(), rowbatch_string));
> We don't needb to care about splitting multibyte characters because flush o
Correct, added comment to the header, describing the function.
http://gerrit.cloudera.org:8080/#/c/22049/8/be/src/exec/text/hdfs-text-scanner.cc
File be/src/exec/text/hdfs-text-scanner.cc:
http://gerrit.cloudera.org:8080/#/c/22049/8/be/src/exec/text/hdfs-text-scanner.cc@546
PS8, Line 546: RETURN_IF_ERROR(decoder_->DecodeBuffer(reinterpret_cast<uint8_t**>(&byte_buffer_ptr_),
: &byte_buffer_read_size_));
> optional: for me it looks clearer to have +2 output parameters instead of u
This follows the pattern inside Decompress* functions
http://gerrit.cloudera.org:8080/#/c/22049/8/be/src/util/char-codec.h
File be/src/util/char-codec.h:
http://gerrit.cloudera.org:8080/#/c/22049/8/be/src/util/char-codec.h@47
PS8, Line 47: /// HdfsTextTableW
> This could be const IMO.
Agree.
> If line delimiter could change, the question of multi byte > characters would come up again, so char may not be enough > to store it.
There are custom delimiters but it is not allowed to have multi-byte delimiter. The only special case is '\n\r', but that only means that '\n\r' is respected while delimiter is still '\n'.
https://github.com/apache/Impala/blob/master/fe/src/main/java/org/apache/impala/analysis/TableDef.java#L929
http://gerrit.cloudera.org:8080/#/c/22049/8/be/src/util/char-codec.cc
File be/src/util/char-codec.cc:
http://gerrit.cloudera.org:8080/#/c/22049/8/be/src/util/char-codec.cc@36
PS8, Line 36: DecodeBuffer
> It would be very nice to have some comment that describes the steps of this
Added function description to the header.
Split into smaller functions.
http://gerrit.cloudera.org:8080/#/c/22049/8/be/src/util/char-codec.cc@47
PS8, Line 47:
> can you add DCHECK_LT(partial_symbol_.size(), MAX_SYMBOL?
Agree, added.
--
To view, visit http://gerrit.cloudera.org:8080/22049
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I787cd01caa52a19d6645519a6cedabe0a5253a65
Gerrit-Change-Number: 22049
Gerrit-PatchSet: 9
Gerrit-Owner: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Tue, 11 Feb 2025 21:47:03 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] WIP IMPALA-10319: Support arbitrary encodings on Text/Sequence files
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22049 )
Change subject: WIP IMPALA-10319: Support arbitrary encodings on Text/Sequence files
......................................................................
Patch Set 13:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/18438/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/22049
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I787cd01caa52a19d6645519a6cedabe0a5253a65
Gerrit-Change-Number: 22049
Gerrit-PatchSet: 13
Gerrit-Owner: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Sun, 02 Mar 2025 21:33:14 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10319: Support arbitrary encodings on Text files
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22049 )
Change subject: IMPALA-10319: Support arbitrary encodings on Text files
......................................................................
Patch Set 25:
Build Failed
https://jenkins.impala.io/job/gerrit-code-review-checks/19345/ : Initial code review checks failed. See linked job for details on the failure.
--
To view, visit http://gerrit.cloudera.org:8080/22049
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I787cd01caa52a19d6645519a6cedabe0a5253a65
Gerrit-Change-Number: 22049
Gerrit-PatchSet: 25
Gerrit-Owner: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Peter Rozsa <pr...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Fri, 30 May 2025 11:42:10 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10319: Support arbitrary encodings on Text files
Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/22049 )
Change subject: IMPALA-10319: Support arbitrary encodings on Text files
......................................................................
Patch Set 25: Code-Review+1
(1 comment)
LGTM, can upgrade to +2 if no one else has comments + the rebase is done.
http://gerrit.cloudera.org:8080/#/c/22049/21/tests/query_test/test_charcodec.py
File tests/query_test/test_charcodec.py:
http://gerrit.cloudera.org:8080/#/c/22049/21/tests/query_test/test_charcodec.py@72
PS21, Line 72: class TestCharCodecGen(ImpalaTestSuite):
> There's a couple tests that take a bit longer, but not one really stands ou
ack
--
To view, visit http://gerrit.cloudera.org:8080/22049
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I787cd01caa52a19d6645519a6cedabe0a5253a65
Gerrit-Change-Number: 22049
Gerrit-PatchSet: 25
Gerrit-Owner: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Peter Rozsa <pr...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Fri, 30 May 2025 12:40:43 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10319: Support arbitrary encodings on Text files
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/22049 )
Change subject: IMPALA-10319: Support arbitrary encodings on Text files
......................................................................
IMPALA-10319: Support arbitrary encodings on Text files
As proposed in Jira, this implements decoding and encoding of text
buffers for Impala/Hive text tables. Given a table with
'serialization.encoding' property set, similarly to Hive, Impala should
be able to encode the inserted data into charset specified, consequently
saving it into a text file. The opposite decoding operation should be
performed upon reading data buffers from text files. Both operations
employ boost::locale::conv library.
Since Hive doesn't encode line delimiters, charsets that would have
delimiters stored differently from ASCII are not allowed.
One difference from Hive is that Impala implements
'serialization.encoding' only as a per partition serdeproperty to avoid
confusion of allowing both serde and tbl properties. (See related
IMPALA-13748)
Note: Due to precreated non-UTF-8 files present in the patch
'gerrit-code-review-checks' was performed locally. (See IMPALA-14100)
Change-Id: I787cd01caa52a19d6645519a6cedabe0a5253a65
Reviewed-on: http://gerrit.cloudera.org:8080/22049
Reviewed-by: Csaba Ringhofer <cs...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
M be/src/exec/hdfs-scanner.h
M be/src/exec/hdfs-text-table-writer.cc
M be/src/exec/hdfs-text-table-writer.h
M be/src/exec/scanner-context.h
M be/src/exec/text/hdfs-text-scanner.cc
M be/src/exec/text/hdfs-text-scanner.h
M be/src/runtime/descriptors.cc
M be/src/runtime/descriptors.h
M be/src/runtime/mem-tracker.cc
M be/src/runtime/mem-tracker.h
M be/src/runtime/row-batch.h
M be/src/util/CMakeLists.txt
A be/src/util/char-codec.cc
A be/src/util/char-codec.h
M bin/rat_exclude_files.txt
M common/thrift/CatalogObjects.thrift
M common/thrift/generate_error_codes.py
M fe/src/main/java/org/apache/impala/analysis/AlterTableSetTblProperties.java
M fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java
M fe/src/main/java/org/apache/impala/catalog/HdfsStorageDescriptor.java
M fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java
A testdata/charcodec/cp1251_names.txt
A testdata/charcodec/cp1251_names_utf8.txt
A testdata/charcodec/gbk_names.txt
A testdata/charcodec/gbk_names_error.txt
A testdata/charcodec/gbk_names_utf8.txt
A testdata/charcodec/koi8r_names.txt
A testdata/charcodec/koi8r_names_utf8.txt
A testdata/charcodec/latin1_names.txt
A testdata/charcodec/latin1_names_utf8.txt
A testdata/charcodec/shift_jis_names.txt
A testdata/charcodec/shift_jis_names_error.txt
A testdata/charcodec/shift_jis_names_utf8.txt
A tests/query_test/test_charcodec.py
34 files changed, 1,063 insertions(+), 20 deletions(-)
Approvals:
Csaba Ringhofer: Looks good to me, approved
Impala Public Jenkins: Verified
--
To view, visit http://gerrit.cloudera.org:8080/22049
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I787cd01caa52a19d6645519a6cedabe0a5253a65
Gerrit-Change-Number: 22049
Gerrit-PatchSet: 29
Gerrit-Owner: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Peter Rozsa <pr...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
[Impala-ASF-CR] WIP IMPALA-10319: Support arbitrary encodings on Text/Sequence files
Posted by "Mihaly Szjatinya (Code Review)" <ge...@cloudera.org>.
Mihaly Szjatinya has uploaded a new patch set (#3). ( http://gerrit.cloudera.org:8080/22049 )
Change subject: WIP IMPALA-10319: Support arbitrary encodings on Text/Sequence files
......................................................................
WIP IMPALA-10319: Support arbitrary encodings on Text/Sequence files
As proposed in Jira, this implements decoding of text buffers for
Impala/Hive text tables. Given a table with 'serialization.encoding'
property set, Hive is able to encode the inserted data into charset
specified, consequently saving it into a text file. Objective is to
perform the opposite operation upon reading data buffers from text files
by employing boost::locale::conv::to_utf() library capability.
Special handling is provided for different BOM scenarios, as well as for
line delimiters, which are not part of the encoding on the Hive side.
For example, given a standard line delimiter '\n', represented by a
single byte '0A' in UTF-8 and two bytes '0x0A 0x00' in UTF-16LE, Hive
inserts it as a single '0A'.
Note: although the described behavior allows keeping the range split
logic simpler, on the negative side it renders Hive files not readable
by regular decoders. Conversely, regular multi-byte files with
delimiters are not correctly read by Hive/Impala. This limitation has no
effect on single-byte charsets.
Design Notes:
1. Memory management. Current draft is a naive
<conv::to_utf() std::string segments -> concat into std::string -> pool>
approach. This can be improved in two steps:
a. Get rid of string for gathering segments and allocate directly on
pool.
b. Provide custom allocator for segment strings, returned by
conv::to_utf() to get those allocated and hopefully RVO'd directly on
pool as well. At the very least, those strings could be tracked via
TrackedString.
2. Decoding filled buffer as a whole. Currently decoding is performed
line by line upon the whole buffer in HdfsTextScanner::FillByteBuffer(),
after it has been read and, if needed, decompressed. This has the
definite advantage of having all the decoding logic in one place,
potentially extractable to a separate class.
Alternatives would be:
a. Decoding tuples / lines upon their materialization on later stages of
HdfsTextScanner. This would employ the existing splitting logic, even
though decoding itself would be done similarly line by line.
b. Preliminarily transforming malformed line delimiters to secure a
smooth conv::to_utf() in one go.
3. An optimization may be done to avoid delimiters only for multi-byte
encodings.
TODO / Nice to Have:
1. Once design approved, this should be extracted into class (hierarchy)
and used for Sequence files. Potentially expandable for encoding by
writers.
2. Test / think through all possible delimiter (FIELD_DELIM, LINE_DELIM,
COLLECTION_DELIM, ESCAPE_CHAR) scenarios.
3. Test archived scenarios.
4. Juxtapose java vs boost encodings supported.
5. Tests all errors/warnings scenarious / corner cases.
Change-Id: I787cd01caa52a19d6645519a6cedabe0a5253a65
---
M be/src/exec/hdfs-scanner.h
M be/src/exec/text/hdfs-text-scanner.cc
M be/src/exec/text/hdfs-text-scanner.h
M be/src/runtime/descriptors.cc
M be/src/runtime/descriptors.h
M common/thrift/CatalogObjects.thrift
M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
A testdata/decoding/cp1251_names.txt
A testdata/decoding/gbk_names.txt
A testdata/decoding/koi8r_names.txt
A testdata/decoding/latin1_names.txt
A testdata/decoding/shift_jis_names.txt
A testdata/decoding/utf16_names.txt
A testdata/decoding/utf16be_names.txt
A testdata/decoding/utf16le_names.txt
A testdata/workloads/functional-query/queries/QueryTest/decoding.test
A tests/query_test/test_decoding.py
17 files changed, 550 insertions(+), 2 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/49/22049/3
--
To view, visit http://gerrit.cloudera.org:8080/22049
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I787cd01caa52a19d6645519a6cedabe0a5253a65
Gerrit-Change-Number: 22049
Gerrit-PatchSet: 3
Gerrit-Owner: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
[Impala-ASF-CR] WIP IMPALA-10319: Support arbitrary encodings on Text/Sequence files
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22049 )
Change subject: WIP IMPALA-10319: Support arbitrary encodings on Text/Sequence files
......................................................................
Patch Set 14:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/18490/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/22049
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I787cd01caa52a19d6645519a6cedabe0a5253a65
Gerrit-Change-Number: 22049
Gerrit-PatchSet: 14
Gerrit-Owner: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Thu, 06 Mar 2025 11:07:31 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] WIP IMPALA-10319: Support arbitrary encodings on Text/Sequence files
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22049 )
Change subject: WIP IMPALA-10319: Support arbitrary encodings on Text/Sequence files
......................................................................
Patch Set 4:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/17690/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/22049
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I787cd01caa52a19d6645519a6cedabe0a5253a65
Gerrit-Change-Number: 22049
Gerrit-PatchSet: 4
Gerrit-Owner: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Comment-Date: Mon, 16 Dec 2024 00:00:19 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10319: Support arbitrary encodings on Text files
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22049 )
Change subject: IMPALA-10319: Support arbitrary encodings on Text files
......................................................................
Patch Set 26:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/11954/ DRY_RUN=false
--
To view, visit http://gerrit.cloudera.org:8080/22049
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I787cd01caa52a19d6645519a6cedabe0a5253a65
Gerrit-Change-Number: 22049
Gerrit-PatchSet: 26
Gerrit-Owner: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Peter Rozsa <pr...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Fri, 30 May 2025 14:00:07 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10319: Support arbitrary encodings on Text files
Posted by "Mihaly Szjatinya (Code Review)" <ge...@cloudera.org>.
Mihaly Szjatinya has uploaded a new patch set (#22). ( http://gerrit.cloudera.org:8080/22049 )
Change subject: IMPALA-10319: Support arbitrary encodings on Text files
......................................................................
IMPALA-10319: Support arbitrary encodings on Text files
As proposed in Jira, this implements decoding and encoding of text
buffers for Impala/Hive text tables. Given a table with
'serialization.encoding' property set, similarly to Hive, Impala should
be able to encode the inserted data into charset specified, consequently
saving it into a text file. The opposite decoding operation should be
performed upon reading data buffers from text files. Both operations
employ boost::locale::conv library.
Since Hive doesn't encode line delimiters, charsets that would have
delimiters stored differently from ASCII are not allowed.
One difference from Hive is that Impala implements
'serialization.encoding' only as a per partition serdeproperty to avoid
confusion of allowing both serde and tbl properties. (See related
IMPALA-13748)
Change-Id: I787cd01caa52a19d6645519a6cedabe0a5253a65
---
M be/src/exec/hdfs-scanner.h
M be/src/exec/hdfs-text-table-writer.cc
M be/src/exec/hdfs-text-table-writer.h
M be/src/exec/scanner-context.h
M be/src/exec/text/hdfs-text-scanner.cc
M be/src/exec/text/hdfs-text-scanner.h
M be/src/runtime/descriptors.cc
M be/src/runtime/descriptors.h
M be/src/runtime/mem-tracker.cc
M be/src/runtime/mem-tracker.h
M be/src/runtime/row-batch.h
M be/src/util/CMakeLists.txt
A be/src/util/char-codec.cc
A be/src/util/char-codec.h
M common/thrift/CatalogObjects.thrift
M common/thrift/generate_error_codes.py
M fe/src/main/java/org/apache/impala/analysis/AlterTableSetTblProperties.java
M fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java
M fe/src/main/java/org/apache/impala/catalog/HdfsStorageDescriptor.java
M fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java
A testdata/charcodec/cp1251_names.txt
A testdata/charcodec/cp1251_names_utf8.txt
A testdata/charcodec/gbk_names.txt
A testdata/charcodec/gbk_names_error.txt
A testdata/charcodec/gbk_names_utf8.txt
A testdata/charcodec/koi8r_names.txt
A testdata/charcodec/koi8r_names_utf8.txt
A testdata/charcodec/latin1_names.txt
A testdata/charcodec/latin1_names_utf8.txt
A testdata/charcodec/shift_jis_names.txt
A testdata/charcodec/shift_jis_names_error.txt
A testdata/charcodec/shift_jis_names_utf8.txt
A testdata/tmpkwdpNX/charcodec_shift_jis_gen_utf8_0.txt
A testdata/tmpuTZPRB/charcodec_cp1251_gen_utf8_0.txt
A tests/query_test/test_charcodec.py
35 files changed, 21,052 insertions(+), 20 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/49/22049/22
--
To view, visit http://gerrit.cloudera.org:8080/22049
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I787cd01caa52a19d6645519a6cedabe0a5253a65
Gerrit-Change-Number: 22049
Gerrit-PatchSet: 22
Gerrit-Owner: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Peter Rozsa <pr...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
[Impala-ASF-CR] IMPALA-10319: Support arbitrary encodings on Text files
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22049 )
Change subject: IMPALA-10319: Support arbitrary encodings on Text files
......................................................................
Patch Set 22:
Build Failed
https://jenkins.impala.io/job/gerrit-code-review-checks/19266/ : Initial code review checks failed. See linked job for details on the failure.
--
To view, visit http://gerrit.cloudera.org:8080/22049
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I787cd01caa52a19d6645519a6cedabe0a5253a65
Gerrit-Change-Number: 22049
Gerrit-PatchSet: 22
Gerrit-Owner: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Peter Rozsa <pr...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Wed, 21 May 2025 13:50:19 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10319: Support arbitrary encodings on Text files
Posted by "Mihaly Szjatinya (Code Review)" <ge...@cloudera.org>.
Mihaly Szjatinya has uploaded a new patch set (#23). ( http://gerrit.cloudera.org:8080/22049 )
Change subject: IMPALA-10319: Support arbitrary encodings on Text files
......................................................................
IMPALA-10319: Support arbitrary encodings on Text files
As proposed in Jira, this implements decoding and encoding of text
buffers for Impala/Hive text tables. Given a table with
'serialization.encoding' property set, similarly to Hive, Impala should
be able to encode the inserted data into charset specified, consequently
saving it into a text file. The opposite decoding operation should be
performed upon reading data buffers from text files. Both operations
employ boost::locale::conv library.
Since Hive doesn't encode line delimiters, charsets that would have
delimiters stored differently from ASCII are not allowed.
One difference from Hive is that Impala implements
'serialization.encoding' only as a per partition serdeproperty to avoid
confusion of allowing both serde and tbl properties. (See related
IMPALA-13748)
Change-Id: I787cd01caa52a19d6645519a6cedabe0a5253a65
---
M be/src/exec/hdfs-scanner.h
M be/src/exec/hdfs-text-table-writer.cc
M be/src/exec/hdfs-text-table-writer.h
M be/src/exec/scanner-context.h
M be/src/exec/text/hdfs-text-scanner.cc
M be/src/exec/text/hdfs-text-scanner.h
M be/src/runtime/descriptors.cc
M be/src/runtime/descriptors.h
M be/src/runtime/mem-tracker.cc
M be/src/runtime/mem-tracker.h
M be/src/runtime/row-batch.h
M be/src/util/CMakeLists.txt
A be/src/util/char-codec.cc
A be/src/util/char-codec.h
M common/thrift/CatalogObjects.thrift
M common/thrift/generate_error_codes.py
M fe/src/main/java/org/apache/impala/analysis/AlterTableSetTblProperties.java
M fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java
M fe/src/main/java/org/apache/impala/catalog/HdfsStorageDescriptor.java
M fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java
A testdata/charcodec/cp1251_names.txt
A testdata/charcodec/cp1251_names_utf8.txt
A testdata/charcodec/gbk_names.txt
A testdata/charcodec/gbk_names_error.txt
A testdata/charcodec/gbk_names_utf8.txt
A testdata/charcodec/koi8r_names.txt
A testdata/charcodec/koi8r_names_utf8.txt
A testdata/charcodec/latin1_names.txt
A testdata/charcodec/latin1_names_utf8.txt
A testdata/charcodec/shift_jis_names.txt
A testdata/charcodec/shift_jis_names_error.txt
A testdata/charcodec/shift_jis_names_utf8.txt
A tests/query_test/test_charcodec.py
33 files changed, 1,052 insertions(+), 20 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/49/22049/23
--
To view, visit http://gerrit.cloudera.org:8080/22049
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I787cd01caa52a19d6645519a6cedabe0a5253a65
Gerrit-Change-Number: 22049
Gerrit-PatchSet: 23
Gerrit-Owner: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Peter Rozsa <pr...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
[Impala-ASF-CR] IMPALA-10319: Support arbitrary encodings on Text files
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22049 )
Change subject: IMPALA-10319: Support arbitrary encodings on Text files
......................................................................
Patch Set 23:
Build Failed
https://jenkins.impala.io/job/gerrit-code-review-checks/19308/ : Initial code review checks failed. See linked job for details on the failure.
--
To view, visit http://gerrit.cloudera.org:8080/22049
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I787cd01caa52a19d6645519a6cedabe0a5253a65
Gerrit-Change-Number: 22049
Gerrit-PatchSet: 23
Gerrit-Owner: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Peter Rozsa <pr...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Mon, 26 May 2025 10:59:20 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] WIP IMPALA-10319: Support arbitrary encodings on Text/Sequence files
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22049 )
Change subject: WIP IMPALA-10319: Support arbitrary encodings on Text/Sequence files
......................................................................
Patch Set 16:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/18924/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/22049
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I787cd01caa52a19d6645519a6cedabe0a5253a65
Gerrit-Change-Number: 22049
Gerrit-PatchSet: 16
Gerrit-Owner: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Tue, 15 Apr 2025 20:15:15 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] WIP IMPALA-10319: Support arbitrary encodings on Text/Sequence files
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22049 )
Change subject: WIP IMPALA-10319: Support arbitrary encodings on Text/Sequence files
......................................................................
Patch Set 10:
(5 comments)
gerrit-auto-critic failed. You can reproduce it locally using command:
python3 bin/jenkins/critique-gerrit-review.py --dryrun
To run it, you might need a virtual env with Python3's venv installed.
http://gerrit.cloudera.org:8080/#/c/22049/10/tests/query_test/test_charcodec.py
File tests/query_test/test_charcodec.py:
http://gerrit.cloudera.org:8080/#/c/22049/10/tests/query_test/test_charcodec.py@43
PS10, Line 43: u
flake8: F821 undefined name 'unichr'
http://gerrit.cloudera.org:8080/#/c/22049/10/tests/query_test/test_charcodec.py@44
PS10, Line 44: u
flake8: F821 undefined name 'unichr'
http://gerrit.cloudera.org:8080/#/c/22049/10/tests/query_test/test_charcodec.py@45
PS10, Line 45: u
flake8: F821 undefined name 'unichr'
http://gerrit.cloudera.org:8080/#/c/22049/10/tests/query_test/test_charcodec.py@47
PS10, Line 47: u
flake8: F821 undefined name 'unichr'
http://gerrit.cloudera.org:8080/#/c/22049/10/tests/query_test/test_charcodec.py@48
PS10, Line 48: u
flake8: F821 undefined name 'unichr'
--
To view, visit http://gerrit.cloudera.org:8080/22049
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I787cd01caa52a19d6645519a6cedabe0a5253a65
Gerrit-Change-Number: 22049
Gerrit-PatchSet: 10
Gerrit-Owner: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Tue, 11 Feb 2025 22:10:39 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] WIP IMPALA-10319: Support arbitrary encodings on Text/Sequence files
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22049 )
Change subject: WIP IMPALA-10319: Support arbitrary encodings on Text/Sequence files
......................................................................
Patch Set 11:
(1 comment)
gerrit-auto-critic failed. You can reproduce it locally using command:
python3 bin/jenkins/critique-gerrit-review.py --dryrun
To run it, you might need a virtual env with Python3's venv installed.
http://gerrit.cloudera.org:8080/#/c/22049/11/tests/query_test/test_charcodec.py
File tests/query_test/test_charcodec.py:
http://gerrit.cloudera.org:8080/#/c/22049/11/tests/query_test/test_charcodec.py@44
PS11, Line 44: u
flake8: F821 undefined name 'unichr'
--
To view, visit http://gerrit.cloudera.org:8080/22049
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I787cd01caa52a19d6645519a6cedabe0a5253a65
Gerrit-Change-Number: 22049
Gerrit-PatchSet: 11
Gerrit-Owner: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Tue, 11 Feb 2025 22:17:32 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] WIP IMPALA-10319: Support arbitrary encodings on Text/Sequence files
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22049 )
Change subject: WIP IMPALA-10319: Support arbitrary encodings on Text/Sequence files
......................................................................
Patch Set 10:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/18262/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/22049
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I787cd01caa52a19d6645519a6cedabe0a5253a65
Gerrit-Change-Number: 22049
Gerrit-PatchSet: 10
Gerrit-Owner: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Tue, 11 Feb 2025 22:32:49 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] WIP IMPALA-10319: Support arbitrary encodings on Text/Sequence files
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22049 )
Change subject: WIP IMPALA-10319: Support arbitrary encodings on Text/Sequence files
......................................................................
Patch Set 3:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/17489/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/22049
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I787cd01caa52a19d6645519a6cedabe0a5253a65
Gerrit-Change-Number: 22049
Gerrit-PatchSet: 3
Gerrit-Owner: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Comment-Date: Sun, 17 Nov 2024 11:15:04 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] WIP IMPALA-10319: Support arbitrary encodings on Text/Sequence files
Posted by "Mihaly Szjatinya (Code Review)" <ge...@cloudera.org>.
Mihaly Szjatinya has posted comments on this change. ( http://gerrit.cloudera.org:8080/22049 )
Change subject: WIP IMPALA-10319: Support arbitrary encodings on Text/Sequence files
......................................................................
Patch Set 16:
(4 comments)
- fixed issues in threads
- improved test to use charset dimensions instead of loops
http://gerrit.cloudera.org:8080/#/c/22049/15/be/src/runtime/mem-tracker.cc
File be/src/runtime/mem-tracker.cc:
http://gerrit.cloudera.org:8080/#/c/22049/15/be/src/runtime/mem-tracker.cc@587
PS15, Line 587:
: ScopedMemTracker::~ScopedMemTracker() {
: i
> We should also consider whether the allocation actually happened and was su
Good point, fixed.
http://gerrit.cloudera.org:8080/#/c/22049/14/fe/src/main/java/org/apache/impala/analysis/AlterTableSetTblProperties.java
File fe/src/main/java/org/apache/impala/analysis/AlterTableSetTblProperties.java:
http://gerrit.cloudera.org:8080/#/c/22049/14/fe/src/main/java/org/apache/impala/analysis/AlterTableSetTblProperties.java@297
PS14, Line 297:
> I am unsure about the detailed verification.
Ok, I took this from a similiar verification but probably not applicable here indeed. Removing this and the check above.
http://gerrit.cloudera.org:8080/#/c/22049/14/fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java
File fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java:
http://gerrit.cloudera.org:8080/#/c/22049/14/fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java@1068
PS14, Line 1068: "alter table
> CREATE TABLE ... SERDEPROPERTIES could be also tested. See https://github.c
You're right, I've overlooked 'CREATE TABLE WITH' syntax. Added necessary check to CreateTableStmt.java and tests.
http://gerrit.cloudera.org:8080/#/c/22049/14/tests/query_test/test_charcodec.py
File tests/query_test/test_charcodec.py:
http://gerrit.cloudera.org:8080/#/c/22049/14/tests/query_test/test_charcodec.py@77
PS14, Line 77: ###########################################################
> It is not clear to me how this patch handles Sequence files - AlterTableSet
You're right, that was added prematurely. Removing for now.
--
To view, visit http://gerrit.cloudera.org:8080/22049
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I787cd01caa52a19d6645519a6cedabe0a5253a65
Gerrit-Change-Number: 22049
Gerrit-PatchSet: 16
Gerrit-Owner: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Tue, 15 Apr 2025 19:49:40 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] WIP IMPALA-10319: Support arbitrary encodings on Text/Sequence files
Posted by "Mihaly Szjatinya (Code Review)" <ge...@cloudera.org>.
Mihaly Szjatinya has uploaded a new patch set (#18). ( http://gerrit.cloudera.org:8080/22049 )
Change subject: WIP IMPALA-10319: Support arbitrary encodings on Text/Sequence files
......................................................................
WIP IMPALA-10319: Support arbitrary encodings on Text/Sequence files
As proposed in Jira, this implements decoding and encoding of text
buffers for Impala/Hive text tables. Given a table with
'serialization.encoding' property set, similarly to Hive, Impala should
be able to encode the inserted data into charset specified, consequently
saving it into a text file. The opposite decoding operation should be
performed upon reading data buffers from text files. Both operations
employ boost::locale::conv library.
Since Hive doesn't encode line delimiters, charsets that would have
delimiters stored differently from ASCII are not allowed.
One difference from Hive is that Impala implements
'serialization.encoding' only as a per partition serdeproperty to avoid
confusion of allowing both serde and tbl properties. (See related
IMPALA-13748)
Change-Id: I787cd01caa52a19d6645519a6cedabe0a5253a65
---
M be/src/exec/hdfs-scanner.h
M be/src/exec/hdfs-text-table-writer.cc
M be/src/exec/hdfs-text-table-writer.h
M be/src/exec/scanner-context.h
M be/src/exec/text/hdfs-text-scanner.cc
M be/src/exec/text/hdfs-text-scanner.h
M be/src/runtime/descriptors.cc
M be/src/runtime/descriptors.h
M be/src/runtime/mem-tracker.cc
M be/src/runtime/mem-tracker.h
M be/src/runtime/row-batch.h
M be/src/util/CMakeLists.txt
A be/src/util/char-codec.cc
A be/src/util/char-codec.h
M common/thrift/CatalogObjects.thrift
M common/thrift/generate_error_codes.py
M fe/src/main/java/org/apache/impala/analysis/AlterTableSetTblProperties.java
M fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java
M fe/src/main/java/org/apache/impala/catalog/HdfsStorageDescriptor.java
M fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java
A tests/query_test/test_charcodec.py
21 files changed, 903 insertions(+), 17 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/49/22049/18
--
To view, visit http://gerrit.cloudera.org:8080/22049
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I787cd01caa52a19d6645519a6cedabe0a5253a65
Gerrit-Change-Number: 22049
Gerrit-PatchSet: 18
Gerrit-Owner: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
[Impala-ASF-CR] IMPALA-10319: Support arbitrary encodings on Text/Sequence files
Posted by "Mihaly Szjatinya (Code Review)" <ge...@cloudera.org>.
Mihaly Szjatinya has posted comments on this change. ( http://gerrit.cloudera.org:8080/22049 )
Change subject: IMPALA-10319: Support arbitrary encodings on Text/Sequence files
......................................................................
Patch Set 19:
(8 comments)
http://gerrit.cloudera.org:8080/#/c/22049/19//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/22049/19//COMMIT_MSG@7
PS19, Line 7: WIP
> Is it still WIP?
Not anymore, ack
http://gerrit.cloudera.org:8080/#/c/22049/19/be/src/exec/text/hdfs-text-scanner.h
File be/src/exec/text/hdfs-text-scanner.h:
http://gerrit.cloudera.org:8080/#/c/22049/19/be/src/exec/text/hdfs-text-scanner.h@272
PS19, Line 272: boost::scoped_ptr
> nit: Can this be unique_ptr?
Yes, changing also the others
http://gerrit.cloudera.org:8080/#/c/22049/17/be/src/exec/text/hdfs-text-scanner.cc
File be/src/exec/text/hdfs-text-scanner.cc:
http://gerrit.cloudera.org:8080/#/c/22049/17/be/src/exec/text/hdfs-text-scanner.cc@545
PS17, Line 545: if (decoder_.get() != nullptr)
> I think in new methods/classes, the less code is the better. In files where
ok, leaving this as is then
http://gerrit.cloudera.org:8080/#/c/22049/17/be/src/exec/text/hdfs-text-scanner.cc@548
PS17, Line 548: decompressor_.get() != nullptr
> nit: 'decompressor_ == nullptr' should be enough. Same goos for L749.
Leaving as is for consistency (see comment above)
http://gerrit.cloudera.org:8080/#/c/22049/19/be/src/util/char-codec.h
File be/src/util/char-codec.h:
http://gerrit.cloudera.org:8080/#/c/22049/19/be/src/util/char-codec.h@34
PS19, Line 34: // Output buffer size for streaming encoded file.
: static const int64_t STREAM_OUT_BUF_SIZE;
> Seems to be unused.
indeed, removing
http://gerrit.cloudera.org:8080/#/c/22049/19/be/src/util/char-codec.cc
File be/src/util/char-codec.cc:
http://gerrit.cloudera.org:8080/#/c/22049/19/be/src/util/char-codec.cc@119
PS19, Line 119: // continue
> Just a thought: it might make sense to make the 'continue' explicit, so if
Ok, why not.
http://gerrit.cloudera.org:8080/#/c/22049/17/tests/query_test/test_charcodec.py
File tests/query_test/test_charcodec.py:
http://gerrit.cloudera.org:8080/#/c/22049/17/tests/query_test/test_charcodec.py@44
PS17, Line 44: '
> Is this self-assignment required at all?
Not really, thanks
http://gerrit.cloudera.org:8080/#/c/22049/19/tests/query_test/test_charcodec.py
File tests/query_test/test_charcodec.py:
http://gerrit.cloudera.org:8080/#/c/22049/19/tests/query_test/test_charcodec.py@48
PS19, Line 48: max_length=20):
> I don't see tests with large words. Can we add a few?
sure
--
To view, visit http://gerrit.cloudera.org:8080/22049
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I787cd01caa52a19d6645519a6cedabe0a5253a65
Gerrit-Change-Number: 22049
Gerrit-PatchSet: 19
Gerrit-Owner: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Tue, 29 Apr 2025 11:33:42 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10319: Support arbitrary encodings on Text files
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22049 )
Change subject: IMPALA-10319: Support arbitrary encodings on Text files
......................................................................
Patch Set 28:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/11957/ DRY_RUN=false
--
To view, visit http://gerrit.cloudera.org:8080/22049
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I787cd01caa52a19d6645519a6cedabe0a5253a65
Gerrit-Change-Number: 22049
Gerrit-PatchSet: 28
Gerrit-Owner: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Peter Rozsa <pr...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Sun, 01 Jun 2025 15:27:37 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10319: Support arbitrary encodings on Text files
Posted by "Mihaly Szjatinya (Code Review)" <ge...@cloudera.org>.
Mihaly Szjatinya has posted comments on this change. ( http://gerrit.cloudera.org:8080/22049 )
Change subject: IMPALA-10319: Support arbitrary encodings on Text files
......................................................................
Patch Set 26:
Thank you, rebased.
--
To view, visit http://gerrit.cloudera.org:8080/22049
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I787cd01caa52a19d6645519a6cedabe0a5253a65
Gerrit-Change-Number: 22049
Gerrit-PatchSet: 26
Gerrit-Owner: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Peter Rozsa <pr...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Fri, 30 May 2025 13:53:46 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] WIP IMPALA-10319: Support arbitrary encodings on Text/Sequence files
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22049 )
Change subject: WIP IMPALA-10319: Support arbitrary encodings on Text/Sequence files
......................................................................
Patch Set 7:
Build Failed
https://jenkins.impala.io/job/gerrit-code-review-checks/18091/ : Initial code review checks failed. See linked job for details on the failure.
--
To view, visit http://gerrit.cloudera.org:8080/22049
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I787cd01caa52a19d6645519a6cedabe0a5253a65
Gerrit-Change-Number: 22049
Gerrit-PatchSet: 7
Gerrit-Owner: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Sun, 26 Jan 2025 23:36:56 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] WIP IMPALA-10319: Support arbitrary encodings on Text/Sequence files
Posted by "Mihaly Szjatinya (Code Review)" <ge...@cloudera.org>.
Mihaly Szjatinya has uploaded a new patch set (#5). ( http://gerrit.cloudera.org:8080/22049 )
Change subject: WIP IMPALA-10319: Support arbitrary encodings on Text/Sequence files
......................................................................
WIP IMPALA-10319: Support arbitrary encodings on Text/Sequence files
As proposed in Jira, this implements decoding of text buffers for
Impala/Hive text tables. Given a table with 'serialization.encoding'
property set, Hive is able to encode the inserted data into charset
specified, consequently saving it into a text file. Objective is to
perform the opposite operation upon reading data buffers from text files
by employing boost::locale::conv::to_utf() library capability.
Since Hive doesn't encode line delimiters, charsets that would have
delimiters stored differently from ASCII are not allowed.
One difference from Hive is that Impala impelents
'serialization.encoding' only as a per partition serdeproperty to avoid
confusion of allowing both serde and tbl properties.
TODO / Nice to Have:
1. Once design approved, this should be extracted into class (hierarchy)
and used for Sequence files. Potentially expandable for encoding by
writers.
2. Test / think through all possible delimiter (FIELD_DELIM, LINE_DELIM,
COLLECTION_DELIM, ESCAPE_CHAR) scenarios.
3. Test archived scenarios.
4. Test complex types
5. Test per-partition cases
6. Juxtapose java vs boost encodings supported.
7. Test all errors/warnings scenarious / corner cases.
Change-Id: I787cd01caa52a19d6645519a6cedabe0a5253a65
---
M be/src/exec/hdfs-scanner.h
M be/src/exec/hdfs-text-table-writer.cc
M be/src/exec/hdfs-text-table-writer.h
M be/src/exec/text/hdfs-text-scanner.cc
M be/src/exec/text/hdfs-text-scanner.h
M be/src/runtime/descriptors.cc
M be/src/runtime/descriptors.h
M be/src/util/CMakeLists.txt
A be/src/util/char-codec.cc
A be/src/util/char-codec.h
M common/thrift/CatalogObjects.thrift
M common/thrift/generate_error_codes.py
M fe/src/main/java/org/apache/impala/analysis/AlterTableSetTblProperties.java
M fe/src/main/java/org/apache/impala/catalog/HdfsStorageDescriptor.java
M fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java
A testdata/charcodec/cp1251_big_utf8.txt
A testdata/charcodec/cp1251_names.txt
A testdata/charcodec/cp1251_names_utf8.txt
A testdata/charcodec/gbk_big_utf8.txt
A testdata/charcodec/gbk_names.txt
A testdata/charcodec/gbk_names_utf8.txt
A testdata/charcodec/koi8r_big_utf8.txt
A testdata/charcodec/koi8r_names.txt
A testdata/charcodec/koi8r_names_utf8.txt
A testdata/charcodec/latin1_big_utf8.txt
A testdata/charcodec/latin1_names.txt
A testdata/charcodec/latin1_names_utf8.txt
A testdata/charcodec/shift_jis_big_utf8.txt
A testdata/charcodec/shift_jis_names.txt
A testdata/charcodec/shift_jis_names_utf8.txt
M testdata/datasets/functional/functional_schema_template.sql
M testdata/datasets/functional/schema_constraints.csv
A testdata/workloads/functional-query/queries/QueryTest/decoding-small.test
A testdata/workloads/functional-query/queries/QueryTest/decoding_alltypes.test
A testdata/workloads/functional-query/queries/QueryTest/encoding-decoding-alltypes.test
A testdata/workloads/functional-query/queries/QueryTest/encoding-decoding-big.test
A testdata/workloads/functional-query/queries/QueryTest/encoding-decoding-small.test
A tests/query_test/test_charcodec.py
38 files changed, 50,919 insertions(+), 11 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/49/22049/5
--
To view, visit http://gerrit.cloudera.org:8080/22049
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I787cd01caa52a19d6645519a6cedabe0a5253a65
Gerrit-Change-Number: 22049
Gerrit-PatchSet: 5
Gerrit-Owner: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
[Impala-ASF-CR] IMPALA-10319: Support arbitrary encodings on Text/Sequence files
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22049 )
Change subject: IMPALA-10319: Support arbitrary encodings on Text/Sequence files
......................................................................
Patch Set 1:
(8 comments)
gerrit-auto-critic failed. You can reproduce it locally using command:
python2 bin/jenkins/critique-gerrit-review.py --dryrun
To run it, you might need a virtual env with virtualenv installed.
http://gerrit.cloudera.org:8080/#/c/22049/1/be/src/exec/text/hdfs-text-scanner.h
File be/src/exec/text/hdfs-text-scanner.h:
http://gerrit.cloudera.org:8080/#/c/22049/1/be/src/exec/text/hdfs-text-scanner.h@183
PS1, Line 183: /// variables is brittle.
line has trailing whitespace
http://gerrit.cloudera.org:8080/#/c/22049/1/be/src/exec/text/hdfs-text-scanner.cc
File be/src/exec/text/hdfs-text-scanner.cc:
http://gerrit.cloudera.org:8080/#/c/22049/1/be/src/exec/text/hdfs-text-scanner.cc@100
PS1, Line 100: }
line has trailing whitespace
http://gerrit.cloudera.org:8080/#/c/22049/1/be/src/exec/text/hdfs-text-scanner.cc@569
PS1, Line 569: while (i < size && !((i + 1 < size &&
line has trailing whitespace
http://gerrit.cloudera.org:8080/#/c/22049/1/be/src/exec/text/hdfs-text-scanner.cc@592
PS1, Line 592: return temp_decoding_pool_->mem_tracker()->MemLimitExceeded(nullptr, details,
line has trailing whitespace
http://gerrit.cloudera.org:8080/#/c/22049/1/be/src/runtime/descriptors.cc
File be/src/runtime/descriptors.cc:
http://gerrit.cloudera.org:8080/#/c/22049/1/be/src/runtime/descriptors.cc@253
PS1, Line 253: encoding_value_ = tdesc.hdfsTable.__isset.encodingValue
line has trailing whitespace
http://gerrit.cloudera.org:8080/#/c/22049/1/tests/query_test/test_decoding.py
File tests/query_test/test_decoding.py:
http://gerrit.cloudera.org:8080/#/c/22049/1/tests/query_test/test_decoding.py@21
PS1, Line 21: from tests.util.filesystem_utils import get_fs_path, IS_HDFS, WAREHOUSE
flake8: F401 'tests.util.filesystem_utils.IS_HDFS' imported but unused
http://gerrit.cloudera.org:8080/#/c/22049/1/tests/query_test/test_decoding.py@21
PS1, Line 21: from tests.util.filesystem_utils import get_fs_path, IS_HDFS, WAREHOUSE
flake8: F401 'tests.util.filesystem_utils.get_fs_path' imported but unused
http://gerrit.cloudera.org:8080/#/c/22049/1/tests/query_test/test_decoding.py@24
PS1, Line 24: class TestDecoding(ImpalaTestSuite):
flake8: E302 expected 2 blank lines, found 1
--
To view, visit http://gerrit.cloudera.org:8080/22049
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I787cd01caa52a19d6645519a6cedabe0a5253a65
Gerrit-Change-Number: 22049
Gerrit-PatchSet: 1
Gerrit-Owner: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Sun, 10 Nov 2024 18:18:58 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10319: Support arbitrary encodings on Text/Sequence files
Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/22049 )
Change subject: IMPALA-10319: Support arbitrary encodings on Text/Sequence files
......................................................................
Patch Set 2:
(3 comments)
Added some high level comments. The code looks good, but there are some decisions to be made about the feature that will affect the implementation.
http://gerrit.cloudera.org:8080/#/c/22049/2//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/22049/2//COMMIT_MSG@7
PS2, Line 7: Support
The patch only adds read support, this could be highlighted in the title.
It is also a question what to do when someone tries to insert to a table where serialization.encoding is set - currently Impala won't respect the encoding. My suggestion is the following:
- by default give an error when trying to write a table where encoding is set
- add a way to completely ignore serialization.encoding, both during reading and writing - not sure which granularity would be the best (flag, query option, table property), at first a flag would be the easiest to implement
http://gerrit.cloudera.org:8080/#/c/22049/2//COMMIT_MSG@9
PS2, Line 9: [Draft]
In Impala project the non-final status of a patch is usually marked in the title, e.g. to make it clear that it is work in progress: WIP MPALA-10319: Support ...
http://gerrit.cloudera.org:8080/#/c/22049/2//COMMIT_MSG@23
PS2, Line 23: Note: although the described behavior allows keeping the range split
Can you add some pointers to the encoder logic in Hive? Are there comments in Hive code that explain why it was implemented this way?
It is not clear to me whether this is intentional behavior in Hive or they had some specific encodings in mind and forgot about multi byte encodings. My guess is that this is an oversight - the single character new line MAY make sense for splitting, but I see no valid reason behind including BOM in every line.
Found a Hive ticket that treats this as a bug: https://issues.apache.org/jira/browse/HIVE-14846 (there is no confirmation though from Hive people on whether that consider this a bug). Probably it would make sense to aim for encodings where the new line character is the same as in ascii.
--
To view, visit http://gerrit.cloudera.org:8080/22049
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I787cd01caa52a19d6645519a6cedabe0a5253a65
Gerrit-Change-Number: 22049
Gerrit-PatchSet: 2
Gerrit-Owner: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Tue, 12 Nov 2024 09:26:46 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] WIP IMPALA-10319: Support arbitrary encodings on Text/Sequence files
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22049 )
Change subject: WIP IMPALA-10319: Support arbitrary encodings on Text/Sequence files
......................................................................
Patch Set 12:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/18437/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/22049
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I787cd01caa52a19d6645519a6cedabe0a5253a65
Gerrit-Change-Number: 22049
Gerrit-PatchSet: 12
Gerrit-Owner: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Sun, 02 Mar 2025 18:23:15 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] WIP IMPALA-10319: Support arbitrary encodings on Text/Sequence files
Posted by "Mihaly Szjatinya (Code Review)" <ge...@cloudera.org>.
Mihaly Szjatinya has posted comments on this change. ( http://gerrit.cloudera.org:8080/22049 )
Change subject: WIP IMPALA-10319: Support arbitrary encodings on Text/Sequence files
......................................................................
Patch Set 13:
(3 comments)
Improved memory management. Added 'DecodingTime' counter.
http://gerrit.cloudera.org:8080/#/c/22049/8/be/src/exec/text/hdfs-text-scanner.cc
File be/src/exec/text/hdfs-text-scanner.cc:
http://gerrit.cloudera.org:8080/#/c/22049/8/be/src/exec/text/hdfs-text-scanner.cc@532
PS8, Line 532: *eosr = stream_->eosr();
> What will happen with decoder_ if it is end of stream, but the last charact
Good point, I had left this for subsequent patches as less important. Added the check.
http://gerrit.cloudera.org:8080/#/c/22049/8/be/src/util/char-codec.cc
File be/src/util/char-codec.cc:
http://gerrit.cloudera.org:8080/#/c/22049/8/be/src/util/char-codec.cc@77
PS8, Line 77: return memory_pool_->mem_tracker()->MemLimitExceeded(nullptr, details, *bytes_read);
: }
> This can do a quite large untracked memory allocation. MemTracker's Consume
Agree, added TryConsume() within a helper class to secure Release() for every return path. Perhaps we may think of more sophisticated estimates though, if that's important.
http://gerrit.cloudera.org:8080/#/c/22049/8/be/src/util/char-codec.cc@113
PS8, Line 113: partial_symbol_.clear();
> We could release the decompressed buffer's memory at this point if we would
Agree, but instead of releasing here, I employed the mechanism from DecompressStreamToBuffer(), which moves previous buffer onto row_batch's pool on every subsequent call, which later gets released either still on ScannerThread, or on ScanNode's side in case it was sent to queue. As testing shows this secures stable memory usage throughout reading large files.
Potential problem is a significant exceeding of row_batch's limit (set to either 1024 rows or 8MB of attached memory), particularity in cases when both decompression and decoding are in conjunction.
--
To view, visit http://gerrit.cloudera.org:8080/22049
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I787cd01caa52a19d6645519a6cedabe0a5253a65
Gerrit-Change-Number: 22049
Gerrit-PatchSet: 13
Gerrit-Owner: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Sun, 02 Mar 2025 21:08:07 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] WIP IMPALA-10319: Support arbitrary encodings on Text/Sequence files
Posted by "Mihaly Szjatinya (Code Review)" <ge...@cloudera.org>.
Mihaly Szjatinya has posted comments on this change. ( http://gerrit.cloudera.org:8080/22049 )
Change subject: WIP IMPALA-10319: Support arbitrary encodings on Text/Sequence files
......................................................................
Patch Set 4:
(3 comments)
Made following changes:
1. Extracted decoder into separate class to be used for writing text files and reading Sequence files. Memory / pool management still to be improved.
2. Moved 'serialization.encoding' property to THdfsStorageDescriptor as per partition SerdeProperty.
3. Prohibited multibyte charsets.
4. Adjusted tests. Added test draft for dataload.
What's left:
1. Add encoding for Text files.
2. Add decoding for Sequence files.
3. Improve memory management.
4. Add property ignore option.
5. Extend tests.
http://gerrit.cloudera.org:8080/#/c/22049/2//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/22049/2//COMMIT_MSG@23
PS2, Line 23:
> We discussed this offline with Mihaly and the current consensus is to imple
Added a check for whether '\n' is compatible with ASCII on analysis stage. We should think of whether and how we want to check for an arbitrary 'line.delimiter':
1. 'line.delimiter' already set.
2. 'line.delimiter' being set together with the 'serialization.encoding' within the same 'alter table' query.
http://gerrit.cloudera.org:8080/#/c/22049/3/tests/query_test/test_decoding.py
File tests/query_test/test_decoding.py:
http://gerrit.cloudera.org:8080/#/c/22049/3/tests/query_test/test_decoding.py@36
PS3, Line 36: text_dimensio
> It would be nice to also create a table during dataload:
Added dataload from 'funcaional.alltypes' as a draft. Since we filter out multibyte charsets, and all data in 'functional.alltypes' is ASCII compatible, this effectively only checks that nothing is broken (which is also useful though).
To better check decoding itself we need to generate large tables with plenty of local / special symbols. I've seen multiple designs for that in tests, like using generator, pre-created files, copying from existing tables, etc. What would be the best option here?
To check with partitions, we need to implement writing first, since hive has a bug inserting partitions.
http://gerrit.cloudera.org:8080/#/c/22049/3/tests/query_test/test_decoding.py@49
PS3, Line 49:
> These small tables are useful to test encoding, but having a bigger table c
Ack
--
To view, visit http://gerrit.cloudera.org:8080/22049
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I787cd01caa52a19d6645519a6cedabe0a5253a65
Gerrit-Change-Number: 22049
Gerrit-PatchSet: 4
Gerrit-Owner: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Comment-Date: Sun, 15 Dec 2024 23:35:52 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] WIP IMPALA-10319: Support arbitrary encodings on Text/Sequence files
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22049 )
Change subject: WIP IMPALA-10319: Support arbitrary encodings on Text/Sequence files
......................................................................
Patch Set 5:
Build Failed
https://jenkins.impala.io/job/gerrit-code-review-checks/17760/ : Initial code review checks failed. See linked job for details on the failure.
--
To view, visit http://gerrit.cloudera.org:8080/22049
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I787cd01caa52a19d6645519a6cedabe0a5253a65
Gerrit-Change-Number: 22049
Gerrit-PatchSet: 5
Gerrit-Owner: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Comment-Date: Mon, 23 Dec 2024 00:21:33 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10319: Support arbitrary encodings on Text files
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22049 )
Change subject: IMPALA-10319: Support arbitrary encodings on Text files
......................................................................
Patch Set 26: Verified-1
Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/11954/
--
To view, visit http://gerrit.cloudera.org:8080/22049
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I787cd01caa52a19d6645519a6cedabe0a5253a65
Gerrit-Change-Number: 22049
Gerrit-PatchSet: 26
Gerrit-Owner: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Peter Rozsa <pr...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Fri, 30 May 2025 20:12:14 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10319: Support arbitrary encodings on Text files
Posted by "Mihaly Szjatinya (Code Review)" <ge...@cloudera.org>.
Mihaly Szjatinya has posted comments on this change. ( http://gerrit.cloudera.org:8080/22049 )
Change subject: IMPALA-10319: Support arbitrary encodings on Text files
......................................................................
Patch Set 28:
Updated 'rat_exclude_files.txt'.
--
To view, visit http://gerrit.cloudera.org:8080/22049
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I787cd01caa52a19d6645519a6cedabe0a5253a65
Gerrit-Change-Number: 22049
Gerrit-PatchSet: 28
Gerrit-Owner: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Peter Rozsa <pr...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Sun, 01 Jun 2025 13:42:17 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10319: Support arbitrary encodings on Text files
Posted by "Mihaly Szjatinya (Code Review)" <ge...@cloudera.org>.
Mihaly Szjatinya has uploaded a new patch set (#25). ( http://gerrit.cloudera.org:8080/22049 )
Change subject: IMPALA-10319: Support arbitrary encodings on Text files
......................................................................
IMPALA-10319: Support arbitrary encodings on Text files
As proposed in Jira, this implements decoding and encoding of text
buffers for Impala/Hive text tables. Given a table with
'serialization.encoding' property set, similarly to Hive, Impala should
be able to encode the inserted data into charset specified, consequently
saving it into a text file. The opposite decoding operation should be
performed upon reading data buffers from text files. Both operations
employ boost::locale::conv library.
Since Hive doesn't encode line delimiters, charsets that would have
delimiters stored differently from ASCII are not allowed.
One difference from Hive is that Impala implements
'serialization.encoding' only as a per partition serdeproperty to avoid
confusion of allowing both serde and tbl properties. (See related
IMPALA-13748)
Note: Due to precreated non-UTF-8 files present in the patch
'gerrit-code-review-checks' was performed locally. (See IMPALA-14100)
Change-Id: I787cd01caa52a19d6645519a6cedabe0a5253a65
---
M be/src/exec/hdfs-scanner.h
M be/src/exec/hdfs-text-table-writer.cc
M be/src/exec/hdfs-text-table-writer.h
M be/src/exec/scanner-context.h
M be/src/exec/text/hdfs-text-scanner.cc
M be/src/exec/text/hdfs-text-scanner.h
M be/src/runtime/descriptors.cc
M be/src/runtime/descriptors.h
M be/src/runtime/mem-tracker.cc
M be/src/runtime/mem-tracker.h
M be/src/runtime/row-batch.h
M be/src/util/CMakeLists.txt
A be/src/util/char-codec.cc
A be/src/util/char-codec.h
M common/thrift/CatalogObjects.thrift
M common/thrift/generate_error_codes.py
M fe/src/main/java/org/apache/impala/analysis/AlterTableSetTblProperties.java
M fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java
M fe/src/main/java/org/apache/impala/catalog/HdfsStorageDescriptor.java
M fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java
A testdata/charcodec/cp1251_names.txt
A testdata/charcodec/cp1251_names_utf8.txt
A testdata/charcodec/gbk_names.txt
A testdata/charcodec/gbk_names_error.txt
A testdata/charcodec/gbk_names_utf8.txt
A testdata/charcodec/koi8r_names.txt
A testdata/charcodec/koi8r_names_utf8.txt
A testdata/charcodec/latin1_names.txt
A testdata/charcodec/latin1_names_utf8.txt
A testdata/charcodec/shift_jis_names.txt
A testdata/charcodec/shift_jis_names_error.txt
A testdata/charcodec/shift_jis_names_utf8.txt
A tests/query_test/test_charcodec.py
33 files changed, 1,062 insertions(+), 20 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/49/22049/25
--
To view, visit http://gerrit.cloudera.org:8080/22049
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I787cd01caa52a19d6645519a6cedabe0a5253a65
Gerrit-Change-Number: 22049
Gerrit-PatchSet: 25
Gerrit-Owner: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Peter Rozsa <pr...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
[Impala-ASF-CR] IMPALA-10319: Support arbitrary encodings on Text files
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22049 )
Change subject: IMPALA-10319: Support arbitrary encodings on Text files
......................................................................
Patch Set 28: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/22049
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I787cd01caa52a19d6645519a6cedabe0a5253a65
Gerrit-Change-Number: 22049
Gerrit-PatchSet: 28
Gerrit-Owner: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Peter Rozsa <pr...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Sun, 01 Jun 2025 21:30:59 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] WIP IMPALA-10319: Support arbitrary encodings on Text/Sequence files
Posted by "Mihaly Szjatinya (Code Review)" <ge...@cloudera.org>.
Mihaly Szjatinya has uploaded a new patch set (#17). ( http://gerrit.cloudera.org:8080/22049 )
Change subject: WIP IMPALA-10319: Support arbitrary encodings on Text/Sequence files
......................................................................
WIP IMPALA-10319: Support arbitrary encodings on Text/Sequence files
As proposed in Jira, this implements decoding and encoding of text
buffers for Impala/Hive text tables. Given a table with
'serialization.encoding' property set, similarly to Hive, Impala should
be able to encode the inserted data into charset specified, consequently
saving it into a text file. The opposite decoding operation should be
performed upon reading data buffers from text files. Both operations
employ boost::locale::conv library.
Since Hive doesn't encode line delimiters, charsets that would have
delimiters stored differently from ASCII are not allowed.
One difference from Hive is that Impala implements
'serialization.encoding' only as a per partition serdeproperty to avoid
confusion of allowing both serde and tbl properties. (See related
IMPALA-13748)
Change-Id: I787cd01caa52a19d6645519a6cedabe0a5253a65
---
M be/src/exec/hdfs-scanner.h
M be/src/exec/hdfs-text-table-writer.cc
M be/src/exec/hdfs-text-table-writer.h
M be/src/exec/scanner-context.h
M be/src/exec/text/hdfs-text-scanner.cc
M be/src/exec/text/hdfs-text-scanner.h
M be/src/runtime/descriptors.cc
M be/src/runtime/descriptors.h
M be/src/runtime/mem-tracker.cc
M be/src/runtime/mem-tracker.h
M be/src/runtime/row-batch.h
M be/src/util/CMakeLists.txt
A be/src/util/char-codec.cc
A be/src/util/char-codec.h
M common/thrift/CatalogObjects.thrift
M common/thrift/generate_error_codes.py
M fe/src/main/java/org/apache/impala/analysis/AlterTableSetTblProperties.java
M fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java
M fe/src/main/java/org/apache/impala/catalog/HdfsStorageDescriptor.java
M fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java
A tests/query_test/test_charcodec.py
21 files changed, 898 insertions(+), 16 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/49/22049/17
--
To view, visit http://gerrit.cloudera.org:8080/22049
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I787cd01caa52a19d6645519a6cedabe0a5253a65
Gerrit-Change-Number: 22049
Gerrit-PatchSet: 17
Gerrit-Owner: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
[Impala-ASF-CR] IMPALA-10319: Support arbitrary encodings on Text files
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22049 )
Change subject: IMPALA-10319: Support arbitrary encodings on Text files
......................................................................
Patch Set 24:
Build Failed
https://jenkins.impala.io/job/gerrit-code-review-checks/19336/ : Initial code review checks failed. See linked job for details on the failure.
--
To view, visit http://gerrit.cloudera.org:8080/22049
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I787cd01caa52a19d6645519a6cedabe0a5253a65
Gerrit-Change-Number: 22049
Gerrit-PatchSet: 24
Gerrit-Owner: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Peter Rozsa <pr...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Wed, 28 May 2025 23:16:53 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10319: Support arbitrary encodings on Text/Sequence files
Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/22049 )
Change subject: IMPALA-10319: Support arbitrary encodings on Text/Sequence files
......................................................................
Patch Set 21: Code-Review+1
LGTM!
--
To view, visit http://gerrit.cloudera.org:8080/22049
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I787cd01caa52a19d6645519a6cedabe0a5253a65
Gerrit-Change-Number: 22049
Gerrit-PatchSet: 21
Gerrit-Owner: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Wed, 07 May 2025 15:05:25 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] WIP IMPALA-10319: Support arbitrary encodings on Text/Sequence files
Posted by "Mihaly Szjatinya (Code Review)" <ge...@cloudera.org>.
Mihaly Szjatinya has uploaded a new patch set (#6). ( http://gerrit.cloudera.org:8080/22049 )
Change subject: WIP IMPALA-10319: Support arbitrary encodings on Text/Sequence files
......................................................................
WIP IMPALA-10319: Support arbitrary encodings on Text/Sequence files
As proposed in Jira, this implements decoding of text buffers for
Impala/Hive text tables. Given a table with 'serialization.encoding'
property set, Hive is able to encode the inserted data into charset
specified, consequently saving it into a text file. Objective is to
perform the opposite operation upon reading data buffers from text files
by employing boost::locale::conv::to_utf() library capability.
Since Hive doesn't encode line delimiters, charsets that would have
delimiters stored differently from ASCII are not allowed.
One difference from Hive is that Impala impelents
'serialization.encoding' only as a per partition serdeproperty to avoid
confusion of allowing both serde and tbl properties.
TODO / Nice to Have:
1. Once design approved, this should be extracted into class (hierarchy)
and used for Sequence files. Potentially expandable for encoding by
writers.
2. Test / think through all possible delimiter (FIELD_DELIM, LINE_DELIM,
COLLECTION_DELIM, ESCAPE_CHAR) scenarios.
3. Test archived scenarios.
4. Test complex types
5. Test per-partition cases
6. Juxtapose java vs boost encodings supported.
7. Test all errors/warnings scenarious / corner cases.
Change-Id: I787cd01caa52a19d6645519a6cedabe0a5253a65
---
M be/src/exec/hdfs-scanner.h
M be/src/exec/hdfs-text-table-writer.cc
M be/src/exec/hdfs-text-table-writer.h
M be/src/exec/text/hdfs-text-scanner.cc
M be/src/exec/text/hdfs-text-scanner.h
M be/src/runtime/descriptors.cc
M be/src/runtime/descriptors.h
M be/src/util/CMakeLists.txt
A be/src/util/char-codec.cc
A be/src/util/char-codec.h
M common/thrift/CatalogObjects.thrift
M common/thrift/generate_error_codes.py
M fe/src/main/java/org/apache/impala/analysis/AlterTableSetTblProperties.java
M fe/src/main/java/org/apache/impala/catalog/HdfsStorageDescriptor.java
M fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java
A testdata/charcodec/cp1251_big_utf8.txt
A testdata/charcodec/cp1251_names.txt
A testdata/charcodec/cp1251_names_utf8.txt
A testdata/charcodec/gbk_big_utf8.txt
A testdata/charcodec/gbk_names.txt
A testdata/charcodec/gbk_names_utf8.txt
A testdata/charcodec/koi8r_big_utf8.txt
A testdata/charcodec/koi8r_names.txt
A testdata/charcodec/koi8r_names_utf8.txt
A testdata/charcodec/latin1_big_utf8.txt
A testdata/charcodec/latin1_names.txt
A testdata/charcodec/latin1_names_utf8.txt
A testdata/charcodec/shift_jis_big_utf8.txt
A testdata/charcodec/shift_jis_names.txt
A testdata/charcodec/shift_jis_names_utf8.txt
M testdata/datasets/functional/functional_schema_template.sql
M testdata/datasets/functional/schema_constraints.csv
A testdata/workloads/functional-query/queries/QueryTest/decoding-alltypes.test
A testdata/workloads/functional-query/queries/QueryTest/decoding-small.test
A testdata/workloads/functional-query/queries/QueryTest/encoding-decoding-alltypes.test
A testdata/workloads/functional-query/queries/QueryTest/encoding-decoding-big.test
A testdata/workloads/functional-query/queries/QueryTest/encoding-decoding-small.test
A tests/query_test/test_charcodec.py
38 files changed, 50,972 insertions(+), 11 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/49/22049/6
--
To view, visit http://gerrit.cloudera.org:8080/22049
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I787cd01caa52a19d6645519a6cedabe0a5253a65
Gerrit-Change-Number: 22049
Gerrit-PatchSet: 6
Gerrit-Owner: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
[Impala-ASF-CR] WIP IMPALA-10319: Support arbitrary encodings on Text/Sequence files
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22049 )
Change subject: WIP IMPALA-10319: Support arbitrary encodings on Text/Sequence files
......................................................................
Patch Set 6:
Build Failed
https://jenkins.impala.io/job/gerrit-code-review-checks/17903/ : Initial code review checks failed. See linked job for details on the failure.
--
To view, visit http://gerrit.cloudera.org:8080/22049
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I787cd01caa52a19d6645519a6cedabe0a5253a65
Gerrit-Change-Number: 22049
Gerrit-PatchSet: 6
Gerrit-Owner: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Comment-Date: Wed, 08 Jan 2025 13:28:02 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] WIP IMPALA-10319: Support arbitrary encodings on Text/Sequence files
Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/22049 )
Change subject: WIP IMPALA-10319: Support arbitrary encodings on Text/Sequence files
......................................................................
Patch Set 8:
(12 comments)
The decoding logic looks good to me, my new comments focus mainly on memory management.
http://gerrit.cloudera.org:8080/#/c/22049/8//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/22049/8//COMMIT_MSG@19
PS8, Line 19: impelents
typo
http://gerrit.cloudera.org:8080/#/c/22049/8//COMMIT_MSG@33
PS8, Line 33: 4. Test complex types
Complex types do not seem relevant here as those are only supported in Parquet/ORC.
http://gerrit.cloudera.org:8080/#/c/22049/8//COMMIT_MSG@35
PS8, Line 35: 5. Test per-partition cases
It doesn't seem that critical to me to test per-partition different encoding - generally we should test that per-partition serde properties are respected, but it is not necessary to do that for each serde property.
http://gerrit.cloudera.org:8080/#/c/22049/8//COMMIT_MSG@37
PS8, Line 37: 6. Juxtapose java vs boost encodings supported.
This could be done by exposing a c++ function to check compatibility which encodes some string using the given encoding. In practice I don't think that anyone would use an encoding that is not supported at any side, so I am ok with skipping this one. In case someone uses an encoding that gets through Java but is rejected in c++ then a proper error will be returned by the current code.
http://gerrit.cloudera.org:8080/#/c/22049/8/be/src/exec/hdfs-text-table-writer.cc
File be/src/exec/hdfs-text-table-writer.cc:
http://gerrit.cloudera.org:8080/#/c/22049/8/be/src/exec/hdfs-text-table-writer.cc@162
PS8, Line 162: RETURN_IF_ERROR(encoder_->EncodeBuffer(
: rowbatch_stringstream_.str(), rowbatch_string));
We don't needb to care about splitting multibyte characters because flush only happens after full lines, right? A comment could be added about this.
http://gerrit.cloudera.org:8080/#/c/22049/8/be/src/exec/text/hdfs-text-scanner.cc
File be/src/exec/text/hdfs-text-scanner.cc:
http://gerrit.cloudera.org:8080/#/c/22049/8/be/src/exec/text/hdfs-text-scanner.cc@532
PS8, Line 532: *eosr = stream_->eosr();
What will happen with decoder_ if it is end of stream, but the last characters could not be decoded? If I understand things correctly then the remaining bytes will remain in partial_symbol_ and be ignored. I think that we should return decode error in this case.
http://gerrit.cloudera.org:8080/#/c/22049/8/be/src/exec/text/hdfs-text-scanner.cc@546
PS8, Line 546: RETURN_IF_ERROR(decoder_->DecodeBuffer(reinterpret_cast<uint8_t**>(&byte_buffer_ptr_),
: &byte_buffer_read_size_));
optional: for me it looks clearer to have +2 output parameters instead of using the same ptr+int for input and output
http://gerrit.cloudera.org:8080/#/c/22049/8/be/src/util/char-codec.h
File be/src/util/char-codec.h:
http://gerrit.cloudera.org:8080/#/c/22049/8/be/src/util/char-codec.h@47
PS8, Line 47: char tuple_delim_;
This could be const IMO. If line delimiter could change, the question of multi byte characters would come up again, so char may not be enough to store it.
http://gerrit.cloudera.org:8080/#/c/22049/8/be/src/util/char-codec.cc
File be/src/util/char-codec.cc:
http://gerrit.cloudera.org:8080/#/c/22049/8/be/src/util/char-codec.cc@36
PS8, Line 36: DecodeBuffer
It would be very nice to have some comment that describes the steps of this function as it is quite complicated.
I would also consider extracting a few smaller functions, though I don't have exact ideas for the interfaces.
http://gerrit.cloudera.org:8080/#/c/22049/8/be/src/util/char-codec.cc@47
PS8, Line 47:
can you add DCHECK_LT(partial_symbol_.size(), MAX_SYMBOL?
http://gerrit.cloudera.org:8080/#/c/22049/8/be/src/util/char-codec.cc@77
PS8, Line 77: result_core = boost::locale::conv::to_utf<char>(
: (char*)buf_start, (char*)last_delim + 1, encoding_, boost::locale::conv::stop);
This can do a quite large untracked memory allocation. MemTracker's Consume/TryConsume/Release can be used to notify Impala's memory management about memory usage by library's that do not use Impala's memory functions. While we do not know the size needed for the utf8 buffer, a generally safe estimate could work, e.g. 2*encode_buffer_size
http://gerrit.cloudera.org:8080/#/c/22049/8/be/src/util/char-codec.cc@113
PS8, Line 113:
We could release the decompressed buffer's memory at this point if we wouldn't allocate the decoded buffer to the same pool. This mainly matters if the decompression buffer is large, for example a 256MB decompressed buffer will lead to 768 MB peak memory usage per scanner and 512MB after result_core is released. Impala can hold on to this memory for a long time as it won't be released until the last row batch of 1024 is extracted from it.
As this feature is probably not used that often I don't stick to optimizing it, but a follow up jira could be created to streamline memory management.
Note that I don't know if compressions where Impala doesn't support streaming (snappy, lz4 ...) are actually used for compressed text files. So far I met only gzip, where streaming is supported, so the decoded blocks won't be that large.
--
To view, visit http://gerrit.cloudera.org:8080/22049
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I787cd01caa52a19d6645519a6cedabe0a5253a65
Gerrit-Change-Number: 22049
Gerrit-PatchSet: 8
Gerrit-Owner: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Tue, 04 Feb 2025 16:07:39 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] WIP IMPALA-10319: Support arbitrary encodings on Text/Sequence files
Posted by "Mihaly Szjatinya (Code Review)" <ge...@cloudera.org>.
Mihaly Szjatinya has posted comments on this change. ( http://gerrit.cloudera.org:8080/22049 )
Change subject: WIP IMPALA-10319: Support arbitrary encodings on Text/Sequence files
......................................................................
Patch Set 15:
(13 comments)
http://gerrit.cloudera.org:8080/#/c/22049/14//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/22049/14//COMMIT_MSG@22
PS14, Line 22: confusion of allowing both serde and tbl properties. (See related
> Can you mention IMPALA-13748?
Ack
http://gerrit.cloudera.org:8080/#/c/22049/14/be/src/runtime/descriptors.h
File be/src/runtime/descriptors.h:
http://gerrit.cloudera.org:8080/#/c/22049/14/be/src/runtime/descriptors.h@452
PS14, Line 452: const std::string& avro_schema() const { return avro_schema_; }
> nit: extra line
Ack
http://gerrit.cloudera.org:8080/#/c/22049/14/be/src/util/char-codec.h
File be/src/util/char-codec.h:
http://gerrit.cloudera.org:8080/#/c/22049/14/be/src/util/char-codec.h@40
PS14, Line 40:
> nit: extra ,
Ack
http://gerrit.cloudera.org:8080/#/c/22049/14/be/src/util/char-codec.h@47
PS14, Line 47: elimiter fou
> nit: progressively
Ack
http://gerrit.cloudera.org:8080/#/c/22049/14/be/src/util/char-codec.h@57
PS14, Line 57: EncodeBuffer(co
> this could be mover to mem_tracker.h, e.g. as MemTracker::ScopedAllocation
Ack
http://gerrit.cloudera.org:8080/#/c/22049/14/be/src/util/char-codec.h@80
PS14, Line 80:
: /// Buffer to hold transformed data.
: uint8_t* out_buffer_ = nullptr;
:
: /// Length of the output buffer.
: int64_t buffer_length_ = 0;
: };
: }
:
:
:
:
:
:
:
:
:
> These are only used by DecodeBuffer - can you add a comment about that, e.g
Ack
http://gerrit.cloudera.org:8080/#/c/22049/8/be/src/util/char-codec.cc
File be/src/util/char-codec.cc:
http://gerrit.cloudera.org:8080/#/c/22049/8/be/src/util/char-codec.cc@77
PS8, Line 77: memory_pool_->FreeAll();
: } else if (eosr) {
> I think that the current estimates are good enough.
Ack
http://gerrit.cloudera.org:8080/#/c/22049/8/be/src/util/char-codec.cc@113
PS8, Line 113: (char*)prefix.data() + prefix.size(), encoding_, boost::locale::conv::stop);
> I think that we are still suboptimal here - for example if I understand cor
You're right. In case of decompression the decompressed buffer can be released. This is also where std::string for decoded data comes handy, allowing to free up the pool prior to allocating memory for decoded data and thus to get away without an additional pool.
http://gerrit.cloudera.org:8080/#/c/22049/14/be/src/util/char-codec.cc
File be/src/util/char-codec.cc:
http://gerrit.cloudera.org:8080/#/c/22049/14/be/src/util/char-codec.cc@25
PS14, Line 25: #include "runtime/mem-pool.h"
> nit: usually includes to Impala headers are sorted alphabetically
Ack
http://gerrit.cloudera.org:8080/#/c/22049/14/be/src/util/char-codec.cc@50
PS14, Line 50: if (pool != nullptr) {
> Both branches mean that out_buffer_ is no longer usable, right? I think tha
Agree.
http://gerrit.cloudera.org:8080/#/c/22049/14/be/src/util/char-codec.cc@57
PS14, Line 57:
> Can you add a comment about reason for this size?
Ack
http://gerrit.cloudera.org:8080/#/c/22049/14/be/src/util/char-codec.cc@71
PS14, Line 71: return Status(TErrorCode::CHARSET_CONVERSION_ERROR,
: "End of stream reached with partial symbol.");
> Isn't it possible for out_buffer_ to be set but not large enough? For examp
You are right, logic here doesn't make sense, I've probably just written smth to start with and forgot to get back to it later. My bad.
Approach you describe seems reasonable.
http://gerrit.cloudera.org:8080/#/c/22049/14/tests/query_test/test_charcodec.py
File tests/query_test/test_charcodec.py:
http://gerrit.cloudera.org:8080/#/c/22049/14/tests/query_test/test_charcodec.py@77
PS14, Line 77: lambda v: v.get_value('table_format').file_format == 'text'
> Is there any test for SequenceFile?
IIRC we have previously agreed to move SequenceFiles decoding to a separate jira, once this one is finalized. Then we'll also expand tests.
--
To view, visit http://gerrit.cloudera.org:8080/22049
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I787cd01caa52a19d6645519a6cedabe0a5253a65
Gerrit-Change-Number: 22049
Gerrit-PatchSet: 15
Gerrit-Owner: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Wed, 26 Mar 2025 13:37:17 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10319: Support arbitrary encodings on Text files
Posted by "Mihaly Szjatinya (Code Review)" <ge...@cloudera.org>.
Mihaly Szjatinya has posted comments on this change. ( http://gerrit.cloudera.org:8080/22049 )
Change subject: IMPALA-10319: Support arbitrary encodings on Text files
......................................................................
Patch Set 22:
(30 comments)
Thanks for review!
http://gerrit.cloudera.org:8080/#/c/22049/21//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/22049/21//COMMIT_MSG@7
PS21, Line 7: files
> What's the situation now when serialization.encoding is set for a Sequence
I've proposed earlier to do Sequence files in a separate issue if that's ok, since even implementation for Text got pretty bulky. Edited this ticket and created a follow-up for Sequence files.
http://gerrit.cloudera.org:8080/#/c/22049/21/be/src/exec/hdfs-scanner.h
File be/src/exec/hdfs-scanner.h:
http://gerrit.cloudera.org:8080/#/c/22049/21/be/src/exec/hdfs-scanner.h@326
PS21, Line 326:
> typo: decoded
Ack
http://gerrit.cloudera.org:8080/#/c/22049/21/be/src/exec/hdfs-text-table-writer.cc
File be/src/exec/hdfs-text-table-writer.cc:
http://gerrit.cloudera.org:8080/#/c/22049/21/be/src/exec/hdfs-text-table-writer.cc@56
PS21, Line 56: HdfsTextTableWriter::~HdfsTextTableWriter() {
> nit: can fit in one line
Ack
http://gerrit.cloudera.org:8080/#/c/22049/21/be/src/util/char-codec.h
File be/src/util/char-codec.h:
http://gerrit.cloudera.org:8080/#/c/22049/21/be/src/util/char-codec.h@54
PS21, Line 54: std::string* result
> here and in the Handle* functions: Impala has the convention of using point
Ack
http://gerrit.cloudera.org:8080/#/c/22049/21/be/src/util/char-codec.cc
File be/src/util/char-codec.cc:
http://gerrit.cloudera.org:8080/#/c/22049/21/be/src/util/char-codec.cc@61
PS21, Line 61: scopedMemTracker
> nit: could be scoped_mem_tracker
let's leave it like e.g. MemTracker
http://gerrit.cloudera.org:8080/#/c/22049/21/be/src/util/char-codec.cc@103
PS21, Line 103: prefix
> Is it possible to use partial_symbol to build up the prefix? That way, we c
These are really different entities with different scope, but maybe I didn't get what you mean.
http://gerrit.cloudera.org:8080/#/c/22049/21/be/src/util/char-codec.cc@113
PS21, Line 113: boost::
> nit: reinterpret_cast instead of c-style casts
Ack
http://gerrit.cloudera.org:8080/#/c/22049/21/be/src/util/char-codec.cc@159
PS21, Line 159: u
> nit: reinterpret_cast instead of c-style casts
Ack
http://gerrit.cloudera.org:8080/#/c/22049/21/fe/src/main/java/org/apache/impala/analysis/AlterTableSetTblProperties.java
File fe/src/main/java/org/apache/impala/analysis/AlterTableSetTblProperties.java:
http://gerrit.cloudera.org:8080/#/c/22049/21/fe/src/main/java/org/apache/impala/analysis/AlterTableSetTblProperties.java@289
PS21, Line 289: .e
> It should be .equals()
Yes, my bad.
http://gerrit.cloudera.org:8080/#/c/22049/21/fe/src/main/java/org/apache/impala/analysis/AlterTableSetTblProperties.java@294
PS21, Line 294: 'serialization.encodin
> nit: could be lowercase with apostrophe: 'serialization.encoding'.
Ack
http://gerrit.cloudera.org:8080/#/c/22049/21/fe/src/main/java/org/apache/impala/analysis/AlterTableSetTblProperties.java@295
PS21, Line 295: %s",
> nit: could be simplified as %s
Ack
http://gerrit.cloudera.org:8080/#/c/22049/21/fe/src/main/java/org/apache/impala/analysis/AlterTableSetTblProperties.java@303
PS21, Line 303: %s %s",
> nit: could be simplified as %s / %s
Ack
http://gerrit.cloudera.org:8080/#/c/22049/21/fe/src/main/java/org/apache/impala/analysis/AlterTableSetTblProperties.java@308
PS21, Line 308: l.getMeta
> nit: unnecessary cast
Ack
http://gerrit.cloudera.org:8080/#/c/22049/21/fe/src/main/java/org/apache/impala/analysis/AlterTableSetTblProperties.java@314
PS21, Line 314: %s %s", tbl
> nit: could be simplified as %s / %s
Ack
http://gerrit.cloudera.org:8080/#/c/22049/21/fe/src/main/java/org/apache/impala/analysis/AlterTableSetTblProperties.java@320
PS21, Line 320: tring.format("Unsupported encoding: %s."
> nit: it could use a format-string similarly to the previous exception messa
Ack
http://gerrit.cloudera.org:8080/#/c/22049/21/fe/src/main/java/org/apache/impala/analysis/AlterTableSetTblProperties.java@318
PS21, Line 318: String encoding = tblProperties.get(serdeConstants.SERIALIZATION_ENCODING);
: if (!Charset.isSupported(encoding)) {
: throw new AnalysisException(String.format("Unsupported encoding: %s.", encoding));
: }
> nit: it could be moved after the first property check at L291
Hmm, it kind of goes from general to more specific.
http://gerrit.cloudera.org:8080/#/c/22049/21/fe/src/main/java/org/apache/impala/analysis/AlterTableSetTblProperties.java@331
PS21, Line 331: %s.
> nit: could be simplified as %s
Ack
http://gerrit.cloudera.org:8080/#/c/22049/21/fe/src/main/java/org/apache/impala/analysis/AlterTableSetTblProperties.java@332
PS21, Line 332: Please refer to IMPALA-10319 for more info.",
> nit: I'd keep it simpler by skipping "Jira ticket": Please refer to IMPALA-
Ack
http://gerrit.cloudera.org:8080/#/c/22049/21/fe/src/main/java/org/apache/impala/analysis/AlterTableSetTblProperties.java@337
PS21, Line 337: l.getMeta
> nit: unneccessary cast
Ack
http://gerrit.cloudera.org:8080/#/c/22049/21/fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java
File fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java:
http://gerrit.cloudera.org:8080/#/c/22049/21/fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java@21
PS21, Line 21: import java.util.ArrayList;
> nit: not used
Ack
http://gerrit.cloudera.org:8080/#/c/22049/21/tests/query_test/test_charcodec.py
File tests/query_test/test_charcodec.py:
http://gerrit.cloudera.org:8080/#/c/22049/21/tests/query_test/test_charcodec.py@59
PS21, Line 59: .format(db, utf8_table))
> it would be nice to also pass the expected row count to the function to ens
Agree, done
http://gerrit.cloudera.org:8080/#/c/22049/21/tests/query_test/test_charcodec.py@68
PS21, Line 68: sert result.data
> general notes on tests: the coverage seems great in general, but I am conce
Some of these were already added in earlier patches and then removed. Added all in a separate class.
http://gerrit.cloudera.org:8080/#/c/22049/21/tests/query_test/test_charcodec.py@74
PS21, Line 74: def add_test_dimensions(cls):
> A ticket could be mentioned that deals with supporting encoding for other f
Ack
http://gerrit.cloudera.org:8080/#/c/22049/21/tests/query_test/test_charcodec.py@79
PS21, Line 79: encodings = [enc for enc in encodings if enc == 'gbk']
> what is the reason for this line? some comment could be added like "Basic t
Ack
http://gerrit.cloudera.org:8080/#/c/22049/21/tests/query_test/test_charcodec.py@87
PS21, Line 87:
> It could use tempfile instead, no extra cleanup needed with it
Ack, this was also addressed by Csaba.
http://gerrit.cloudera.org:8080/#/c/22049/21/tests/query_test/test_charcodec.py@109
PS21, Line 109: encoding_name_tbl = encoding_name.replace('-', '')
> a comment like '# remove REFRESH when IMPALA-13749 is fixed" would be usefu
Ack
http://gerrit.cloudera.org:8080/#/c/22049/21/tests/query_test/test_charcodec.py@127
PS21, Line 127: STORED AS TEXTFILE""".format(db, encoded_table))
> Some basic function comment would be nice, e.g. """Write encoded table with
Ok, added some comments.
http://gerrit.cloudera.org:8080/#/c/22049/21/tests/query_test/test_charcodec.py@134
PS21, Line 134: return encoded_table
> I think that it would be better to create a temporary directory that is cle
It looks like auto-deleting 'tempfile.TemporaryDirectory' is only available for python 3. 'tempfile.mkdtemp' still requires shutil.rmtree(tmp_dir).
We can make a simple context manager class and use it everywhere, if we don't plan to fully switch to python3.
http://gerrit.cloudera.org:8080/#/c/22049/21/tests/query_test/test_charcodec.py@195
PS21, Line 195:
> The reason for choosing the codecs is that Snappy doesn't support streaming
Ack
http://gerrit.cloudera.org:8080/#/c/22049/21/tests/query_test/test_charcodec.py@303
PS21, Line 303: PARTITIONED BY (part STRING)
> looks duplicated (also see my comment at line 134)
Ack, but this particular folder is not temp but table's path.
--
To view, visit http://gerrit.cloudera.org:8080/22049
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I787cd01caa52a19d6645519a6cedabe0a5253a65
Gerrit-Change-Number: 22049
Gerrit-PatchSet: 22
Gerrit-Owner: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Peter Rozsa <pr...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Wed, 21 May 2025 13:21:29 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10319: Support arbitrary encodings on Text files
Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/22049 )
Change subject: IMPALA-10319: Support arbitrary encodings on Text files
......................................................................
Patch Set 24: Code-Review+1
(4 comments)
http://gerrit.cloudera.org:8080/#/c/22049/21/tests/query_test/test_charcodec.py
File tests/query_test/test_charcodec.py:
http://gerrit.cloudera.org:8080/#/c/22049/21/tests/query_test/test_charcodec.py@72
PS21, Line 72: class TestCharCodecGen(ImpalaTestSuite):
> If it's not important to have 'disable_codegen' both True and False of exha
I would be surprised if files sizes would really matter for Impala (maybe for the file generation it does matter). The current speed seems acceptable to me, but it could be still interesting to see what takes time, probably some slow tests could be moved to exhaustive only.
http://gerrit.cloudera.org:8080/#/c/22049/23/tests/query_test/test_charcodec.py
File tests/query_test/test_charcodec.py:
http://gerrit.cloudera.org:8080/#/c/22049/23/tests/query_test/test_charcodec.py@379
PS23, Line 379: "charcod
> This comes down simply to 'IS_HDFS' check and I thought many tests use this
There are some Impala test environments where Hive is not available (e.g. s3, mainly for historical reasons), so tests that use Hive are skipped.
http://gerrit.cloudera.org:8080/#/c/22049/23/tests/query_test/test_charcodec.py@432
PS23, Line 432: """Write table with Impala and read it back with Hive."""
: db = unique_database
: enc = vector.get_value('charset')
:
> Interestingly enough. after adding 'order by name' to both queries the comp
The ordering comes after decoding.
My guess is that the difference comes from different ordering for unicode characters in Hive vs Impala. This is actually something interesting to know, even if we don't want to "fix" it.
http://gerrit.cloudera.org:8080/#/c/22049/24/tests/query_test/test_charcodec.py
File tests/query_test/test_charcodec.py:
http://gerrit.cloudera.org:8080/#/c/22049/24/tests/query_test/test_charcodec.py@426
PS24, Line 426:
nit: extra whitespace
--
To view, visit http://gerrit.cloudera.org:8080/22049
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I787cd01caa52a19d6645519a6cedabe0a5253a65
Gerrit-Change-Number: 22049
Gerrit-PatchSet: 24
Gerrit-Owner: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Peter Rozsa <pr...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Thu, 29 May 2025 15:12:35 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10319: Support arbitrary encodings on Text files
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22049 )
Change subject: IMPALA-10319: Support arbitrary encodings on Text files
......................................................................
Patch Set 28:
Build Failed
https://jenkins.impala.io/job/gerrit-code-review-checks/19361/ : Initial code review checks failed. See linked job for details on the failure.
--
To view, visit http://gerrit.cloudera.org:8080/22049
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I787cd01caa52a19d6645519a6cedabe0a5253a65
Gerrit-Change-Number: 22049
Gerrit-PatchSet: 28
Gerrit-Owner: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Peter Rozsa <pr...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Sun, 01 Jun 2025 14:04:33 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10319: Support arbitrary encodings on Text files
Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/22049 )
Change subject: IMPALA-10319: Support arbitrary encodings on Text files
......................................................................
Patch Set 28: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/22049
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I787cd01caa52a19d6645519a6cedabe0a5253a65
Gerrit-Change-Number: 22049
Gerrit-PatchSet: 28
Gerrit-Owner: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Peter Rozsa <pr...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Sun, 01 Jun 2025 15:27:30 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] WIP IMPALA-10319: Support arbitrary encodings on Text/Sequence files
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22049 )
Change subject: WIP IMPALA-10319: Support arbitrary encodings on Text/Sequence files
......................................................................
Patch Set 19:
(1 comment)
gerrit-auto-critic failed. You can reproduce it locally using command:
python3 bin/jenkins/critique-gerrit-review.py --dryrun
To run it, you might need a virtual env with Python3's venv installed.
http://gerrit.cloudera.org:8080/#/c/22049/19/tests/query_test/test_charcodec.py
File tests/query_test/test_charcodec.py:
http://gerrit.cloudera.org:8080/#/c/22049/19/tests/query_test/test_charcodec.py@28
PS19, Line 28: u
flake8: F821 undefined name 'unichr'
--
To view, visit http://gerrit.cloudera.org:8080/22049
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I787cd01caa52a19d6645519a6cedabe0a5253a65
Gerrit-Change-Number: 22049
Gerrit-PatchSet: 19
Gerrit-Owner: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Sun, 27 Apr 2025 21:16:53 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] WIP IMPALA-10319: Support arbitrary encodings on Text/Sequence files
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22049 )
Change subject: WIP IMPALA-10319: Support arbitrary encodings on Text/Sequence files
......................................................................
Patch Set 19:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/19035/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/22049
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I787cd01caa52a19d6645519a6cedabe0a5253a65
Gerrit-Change-Number: 22049
Gerrit-PatchSet: 19
Gerrit-Owner: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Sun, 27 Apr 2025 21:42:12 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10319: Support arbitrary encodings on Text/Sequence files
Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/22049 )
Change subject: IMPALA-10319: Support arbitrary encodings on Text/Sequence files
......................................................................
Patch Set 21:
(12 comments)
The implementations looks good (with the exception of a coding style comment), finally found time to process the tests.
http://gerrit.cloudera.org:8080/#/c/22049/21//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/22049/21//COMMIT_MSG@7
PS21, Line 7: Sequence
What's the situation now when serialization.encoding is set for a Sequence file?
http://gerrit.cloudera.org:8080/#/c/22049/21/be/src/util/char-codec.h
File be/src/util/char-codec.h:
http://gerrit.cloudera.org:8080/#/c/22049/21/be/src/util/char-codec.h@54
PS21, Line 54: std::string& result
here and in the Handle* functions: Impala has the convention of using pointer in out parameters instead of non-const refs
http://gerrit.cloudera.org:8080/#/c/22049/21/tests/query_test/test_charcodec.py
File tests/query_test/test_charcodec.py:
http://gerrit.cloudera.org:8080/#/c/22049/21/tests/query_test/test_charcodec.py@59
PS21, Line 59: assert int(count_utf8.get_data()) == int(count_encoded.get_data())
it would be nice to also pass the expected row count to the function to ensure that both tables are in the expected state - currently the tests could pass even if the tables are empty due to some error during table loading
http://gerrit.cloudera.org:8080/#/c/22049/21/tests/query_test/test_charcodec.py@68
PS21, Line 68: TestCharCodecGen
general notes on tests: the coverage seems great in general, but I am concerned that most tests would pass even if the encoding was ignore both during reading and writing
- a test would be nice where we check if Hive reads correctly an encoded table written by Impala
- a test would be nice where a pre-created file (e.g. something minimal added in testdata/data) is read and the result is checked with and without the encoding property
Also, encoding error code paths are not tested:
- a pre-created file could be tested with encoding error to test the error path in the reader
- a string could be written that is not valid utf8 - for example casting the binary col in functional.binary_tbl and writing it to a table with encoding set should result in an error
http://gerrit.cloudera.org:8080/#/c/22049/21/tests/query_test/test_charcodec.py@72
PS21, Line 72: cls.ImpalaTestMatrix.add_dimension(ImpalaTestDimension(
Did you check how much time does the test takes? If it is slow, then it would make sense to only run some encodings in exhaustive jobs.
http://gerrit.cloudera.org:8080/#/c/22049/21/tests/query_test/test_charcodec.py@74
PS21, Line 74: # There is no reason to run these tests using all dimensions.
A ticket could be mentioned that deals with supporting encoding for other file formats.
http://gerrit.cloudera.org:8080/#/c/22049/21/tests/query_test/test_charcodec.py@79
PS21, Line 79: ####################################################################
what is the reason for this line? some comment could be added like "Basic tests"
http://gerrit.cloudera.org:8080/#/c/22049/21/tests/query_test/test_charcodec.py@109
PS21, Line 109: self.execute_query("""REFRESH {}.{}""".format(db, tbl_name))
a comment like '# remove REFRESH when IMPALA-13749 is fixed" would be useful to clean these up later
http://gerrit.cloudera.org:8080/#/c/22049/21/tests/query_test/test_charcodec.py@127
PS21, Line 127: def test_enc_dec_gen(self, vector, unique_database):
Some basic function comment would be nice, e.g. """Write encoded table with Impala and read it back""". It took me some time to understand which read/write paths in Hive/Impala are covered by different tests.
http://gerrit.cloudera.org:8080/#/c/22049/21/tests/query_test/test_charcodec.py@134
PS21, Line 134: [os.remove(file_path) for file_path in file_paths]
I think that it would be better to create a temporary directory that is cleaned up automatically - the current cleanup code would not run in case there is an exception and could leave testdata dir in a dirty state.
See https://docs.python.org/3/library/tempfile.html / "import tempfile" in the code on how to do this.
http://gerrit.cloudera.org:8080/#/c/22049/21/tests/query_test/test_charcodec.py@195
PS21, Line 195: "Snappy", "ZStandard"]
The reason for choosing the codecs is that Snappy doesn't support streaming while ZStandard does, right? This could be mentioned in a comment.
http://gerrit.cloudera.org:8080/#/c/22049/21/tests/query_test/test_charcodec.py@303
PS21, Line 303: self.filesystem_client.make_dir(part_dir)
looks duplicated (also see my comment at line 134)
--
To view, visit http://gerrit.cloudera.org:8080/22049
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I787cd01caa52a19d6645519a6cedabe0a5253a65
Gerrit-Change-Number: 22049
Gerrit-PatchSet: 21
Gerrit-Owner: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Thu, 08 May 2025 10:37:42 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] WIP IMPALA-10319: Support arbitrary encodings on Text/Sequence files
Posted by "Mihaly Szjatinya (Code Review)" <ge...@cloudera.org>.
Mihaly Szjatinya has uploaded a new patch set (#9). ( http://gerrit.cloudera.org:8080/22049 )
Change subject: WIP IMPALA-10319: Support arbitrary encodings on Text/Sequence files
......................................................................
WIP IMPALA-10319: Support arbitrary encodings on Text/Sequence files
As proposed in Jira, this implements decoding and encoding of text
buffers for Impala/Hive text tables. Given a table with
'serialization.encoding' property set, similarly to Hive, Impala should
be able to encode the inserted data into charset specified, consequently
saving it into a text file. The opposite decoding operation should be
performed upon reading data buffers from text files. Both operations
employ boost::locale::conv library.
Since Hive doesn't encode line delimiters, charsets that would have
delimiters stored differently from ASCII are not allowed.
One difference from Hive is that Impala implements
'serialization.encoding' only as a per partition serdeproperty to avoid
confusion of allowing both serde and tbl properties.
Change-Id: I787cd01caa52a19d6645519a6cedabe0a5253a65
---
M be/src/exec/hdfs-scanner.h
M be/src/exec/hdfs-text-table-writer.cc
M be/src/exec/hdfs-text-table-writer.h
M be/src/exec/scanner-context.h
M be/src/exec/text/hdfs-text-scanner.cc
M be/src/exec/text/hdfs-text-scanner.h
M be/src/runtime/descriptors.cc
M be/src/runtime/descriptors.h
M be/src/runtime/row-batch.h
M be/src/util/CMakeLists.txt
A be/src/util/char-codec.cc
A be/src/util/char-codec.h
M common/thrift/CatalogObjects.thrift
M common/thrift/generate_error_codes.py
M fe/src/main/java/org/apache/impala/analysis/AlterTableSetTblProperties.java
M fe/src/main/java/org/apache/impala/catalog/HdfsStorageDescriptor.java
M fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java
A tests/query_test/test_charcodec.py
18 files changed, 735 insertions(+), 15 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/49/22049/9
--
To view, visit http://gerrit.cloudera.org:8080/22049
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I787cd01caa52a19d6645519a6cedabe0a5253a65
Gerrit-Change-Number: 22049
Gerrit-PatchSet: 9
Gerrit-Owner: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
[Impala-ASF-CR] WIP IMPALA-10319: Support arbitrary encodings on Text/Sequence files
Posted by "Mihaly Szjatinya (Code Review)" <ge...@cloudera.org>.
Mihaly Szjatinya has uploaded a new patch set (#7). ( http://gerrit.cloudera.org:8080/22049 )
Change subject: WIP IMPALA-10319: Support arbitrary encodings on Text/Sequence files
......................................................................
WIP IMPALA-10319: Support arbitrary encodings on Text/Sequence files
As proposed in Jira, this implements decoding of text buffers for
Impala/Hive text tables. Given a table with 'serialization.encoding'
property set, Hive is able to encode the inserted data into charset
specified, consequently saving it into a text file. Objective is to
perform the opposite operation upon reading data buffers from text files
by employing boost::locale::conv::to_utf() library capability.
Since Hive doesn't encode line delimiters, charsets that would have
delimiters stored differently from ASCII are not allowed.
One difference from Hive is that Impala impelents
'serialization.encoding' only as a per partition serdeproperty to avoid
confusion of allowing both serde and tbl properties.
TODO / Nice to Have:
1. Once design approved, this should be extracted into class (hierarchy)
and used for Sequence files. Potentially expandable for encoding by
writers.
2. Test / think through all possible delimiter (FIELD_DELIM, LINE_DELIM,
COLLECTION_DELIM, ESCAPE_CHAR) scenarios.
3. Test archived scenarios.
4. Test complex types
5. Test per-partition cases
6. Juxtapose java vs boost encodings supported.
7. Test all errors/warnings scenarious / corner cases.
Change-Id: I787cd01caa52a19d6645519a6cedabe0a5253a65
---
M be/src/exec/hdfs-scanner.h
M be/src/exec/hdfs-text-table-writer.cc
M be/src/exec/hdfs-text-table-writer.h
M be/src/exec/scanner-context.h
M be/src/exec/text/hdfs-text-scanner.cc
M be/src/exec/text/hdfs-text-scanner.h
M be/src/runtime/descriptors.cc
M be/src/runtime/descriptors.h
M be/src/runtime/row-batch.h
M be/src/util/CMakeLists.txt
A be/src/util/char-codec.cc
A be/src/util/char-codec.h
M common/thrift/CatalogObjects.thrift
M common/thrift/generate_error_codes.py
M fe/src/main/java/org/apache/impala/analysis/AlterTableSetTblProperties.java
M fe/src/main/java/org/apache/impala/catalog/HdfsStorageDescriptor.java
M fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java
A testdata/charcodec/cp1251_big_utf8.txt
A testdata/charcodec/cp1251_names.txt
A testdata/charcodec/cp1251_names_utf8.txt
A testdata/charcodec/gbk_big_utf8.txt
A testdata/charcodec/gbk_names.txt
A testdata/charcodec/gbk_names_utf8.txt
A testdata/charcodec/koi8r_big_utf8.txt
A testdata/charcodec/koi8r_names.txt
A testdata/charcodec/koi8r_names_utf8.txt
A testdata/charcodec/latin1_big_utf8.txt
A testdata/charcodec/latin1_names.txt
A testdata/charcodec/latin1_names_utf8.txt
A testdata/charcodec/shift_jis_big_utf8.txt
A testdata/charcodec/shift_jis_names.txt
A testdata/charcodec/shift_jis_names_utf8.txt
M testdata/datasets/functional/functional_schema_template.sql
M testdata/datasets/functional/schema_constraints.csv
A testdata/workloads/functional-query/queries/QueryTest/dec-alltypes.test
A testdata/workloads/functional-query/queries/QueryTest/dec-hive.test
A testdata/workloads/functional-query/queries/QueryTest/dec-small.test
A testdata/workloads/functional-query/queries/QueryTest/enc-dec-alltypes.test
A testdata/workloads/functional-query/queries/QueryTest/enc-dec-big.test
A testdata/workloads/functional-query/queries/QueryTest/enc-dec-gen.test
A testdata/workloads/functional-query/queries/QueryTest/enc-dec-small.test
A tests/query_test/test_charcodec.py
42 files changed, 51,246 insertions(+), 15 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/49/22049/7
--
To view, visit http://gerrit.cloudera.org:8080/22049
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I787cd01caa52a19d6645519a6cedabe0a5253a65
Gerrit-Change-Number: 22049
Gerrit-PatchSet: 7
Gerrit-Owner: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
[Impala-ASF-CR] WIP IMPALA-10319: Support arbitrary encodings on Text/Sequence files
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22049 )
Change subject: WIP IMPALA-10319: Support arbitrary encodings on Text/Sequence files
......................................................................
Patch Set 12:
(3 comments)
gerrit-auto-critic failed. You can reproduce it locally using command:
python3 bin/jenkins/critique-gerrit-review.py --dryrun
To run it, you might need a virtual env with Python3's venv installed.
http://gerrit.cloudera.org:8080/#/c/22049/12/tests/query_test/test_charcodec.py
File tests/query_test/test_charcodec.py:
http://gerrit.cloudera.org:8080/#/c/22049/12/tests/query_test/test_charcodec.py@44
PS12, Line 44: u
flake8: F821 undefined name 'unichr'
http://gerrit.cloudera.org:8080/#/c/22049/12/tests/query_test/test_charcodec.py@178
PS12, Line 178:
flake8: W291 trailing whitespace
http://gerrit.cloudera.org:8080/#/c/22049/12/tests/query_test/test_charcodec.py@178
PS12, Line 178: """.format("false" if codec == "None" else "true",
line has trailing whitespace
--
To view, visit http://gerrit.cloudera.org:8080/22049
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I787cd01caa52a19d6645519a6cedabe0a5253a65
Gerrit-Change-Number: 22049
Gerrit-PatchSet: 12
Gerrit-Owner: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Sun, 02 Mar 2025 17:59:30 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] WIP IMPALA-10319: Support arbitrary encodings on Text/Sequence files
Posted by "Mihaly Szjatinya (Code Review)" <ge...@cloudera.org>.
Mihaly Szjatinya has uploaded a new patch set (#13). ( http://gerrit.cloudera.org:8080/22049 )
Change subject: WIP IMPALA-10319: Support arbitrary encodings on Text/Sequence files
......................................................................
WIP IMPALA-10319: Support arbitrary encodings on Text/Sequence files
As proposed in Jira, this implements decoding and encoding of text
buffers for Impala/Hive text tables. Given a table with
'serialization.encoding' property set, similarly to Hive, Impala should
be able to encode the inserted data into charset specified, consequently
saving it into a text file. The opposite decoding operation should be
performed upon reading data buffers from text files. Both operations
employ boost::locale::conv library.
Since Hive doesn't encode line delimiters, charsets that would have
delimiters stored differently from ASCII are not allowed.
One difference from Hive is that Impala implements
'serialization.encoding' only as a per partition serdeproperty to avoid
confusion of allowing both serde and tbl properties.
Change-Id: I787cd01caa52a19d6645519a6cedabe0a5253a65
---
M be/src/exec/hdfs-scanner.h
M be/src/exec/hdfs-text-table-writer.cc
M be/src/exec/hdfs-text-table-writer.h
M be/src/exec/scanner-context.h
M be/src/exec/text/hdfs-text-scanner.cc
M be/src/exec/text/hdfs-text-scanner.h
M be/src/runtime/descriptors.cc
M be/src/runtime/descriptors.h
M be/src/runtime/row-batch.h
M be/src/util/CMakeLists.txt
A be/src/util/char-codec.cc
A be/src/util/char-codec.h
M common/thrift/CatalogObjects.thrift
M common/thrift/generate_error_codes.py
M fe/src/main/java/org/apache/impala/analysis/AlterTableSetTblProperties.java
M fe/src/main/java/org/apache/impala/catalog/HdfsStorageDescriptor.java
M fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java
A tests/query_test/test_charcodec.py
18 files changed, 817 insertions(+), 16 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/49/22049/13
--
To view, visit http://gerrit.cloudera.org:8080/22049
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I787cd01caa52a19d6645519a6cedabe0a5253a65
Gerrit-Change-Number: 22049
Gerrit-PatchSet: 13
Gerrit-Owner: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
[Impala-ASF-CR] IMPALA-10319: Support arbitrary encodings on Text/Sequence files
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22049 )
Change subject: IMPALA-10319: Support arbitrary encodings on Text/Sequence files
......................................................................
Patch Set 20:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/19050/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/22049
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I787cd01caa52a19d6645519a6cedabe0a5253a65
Gerrit-Change-Number: 22049
Gerrit-PatchSet: 20
Gerrit-Owner: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Tue, 29 Apr 2025 12:05:14 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] WIP IMPALA-10319: Support arbitrary encodings on Text/Sequence files
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22049 )
Change subject: WIP IMPALA-10319: Support arbitrary encodings on Text/Sequence files
......................................................................
Patch Set 17:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/18925/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/22049
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I787cd01caa52a19d6645519a6cedabe0a5253a65
Gerrit-Change-Number: 22049
Gerrit-PatchSet: 17
Gerrit-Owner: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Tue, 15 Apr 2025 20:19:56 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] WIP IMPALA-10319: Support arbitrary encodings on Text/Sequence files
Posted by "Mihaly Szjatinya (Code Review)" <ge...@cloudera.org>.
Mihaly Szjatinya has uploaded a new patch set (#14). ( http://gerrit.cloudera.org:8080/22049 )
Change subject: WIP IMPALA-10319: Support arbitrary encodings on Text/Sequence files
......................................................................
WIP IMPALA-10319: Support arbitrary encodings on Text/Sequence files
As proposed in Jira, this implements decoding and encoding of text
buffers for Impala/Hive text tables. Given a table with
'serialization.encoding' property set, similarly to Hive, Impala should
be able to encode the inserted data into charset specified, consequently
saving it into a text file. The opposite decoding operation should be
performed upon reading data buffers from text files. Both operations
employ boost::locale::conv library.
Since Hive doesn't encode line delimiters, charsets that would have
delimiters stored differently from ASCII are not allowed.
One difference from Hive is that Impala implements
'serialization.encoding' only as a per partition serdeproperty to avoid
confusion of allowing both serde and tbl properties.
Change-Id: I787cd01caa52a19d6645519a6cedabe0a5253a65
---
M be/src/exec/hdfs-scanner.h
M be/src/exec/hdfs-text-table-writer.cc
M be/src/exec/hdfs-text-table-writer.h
M be/src/exec/scanner-context.h
M be/src/exec/text/hdfs-text-scanner.cc
M be/src/exec/text/hdfs-text-scanner.h
M be/src/runtime/descriptors.cc
M be/src/runtime/descriptors.h
M be/src/runtime/row-batch.h
M be/src/util/CMakeLists.txt
A be/src/util/char-codec.cc
A be/src/util/char-codec.h
M common/thrift/CatalogObjects.thrift
M common/thrift/generate_error_codes.py
M fe/src/main/java/org/apache/impala/analysis/AlterTableSetTblProperties.java
M fe/src/main/java/org/apache/impala/catalog/HdfsStorageDescriptor.java
M fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java
A tests/query_test/test_charcodec.py
18 files changed, 820 insertions(+), 16 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/49/22049/14
--
To view, visit http://gerrit.cloudera.org:8080/22049
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I787cd01caa52a19d6645519a6cedabe0a5253a65
Gerrit-Change-Number: 22049
Gerrit-PatchSet: 14
Gerrit-Owner: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
[Impala-ASF-CR] WIP IMPALA-10319: Support arbitrary encodings on Text/Sequence files
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22049 )
Change subject: WIP IMPALA-10319: Support arbitrary encodings on Text/Sequence files
......................................................................
Patch Set 18:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/19034/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/22049
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I787cd01caa52a19d6645519a6cedabe0a5253a65
Gerrit-Change-Number: 22049
Gerrit-PatchSet: 18
Gerrit-Owner: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Sun, 27 Apr 2025 12:24:36 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10319: Support arbitrary encodings on Text/Sequence files
Posted by "Mihaly Szjatinya (Code Review)" <ge...@cloudera.org>.
Mihaly Szjatinya has uploaded a new patch set (#20). ( http://gerrit.cloudera.org:8080/22049 )
Change subject: IMPALA-10319: Support arbitrary encodings on Text/Sequence files
......................................................................
IMPALA-10319: Support arbitrary encodings on Text/Sequence files
As proposed in Jira, this implements decoding and encoding of text
buffers for Impala/Hive text tables. Given a table with
'serialization.encoding' property set, similarly to Hive, Impala should
be able to encode the inserted data into charset specified, consequently
saving it into a text file. The opposite decoding operation should be
performed upon reading data buffers from text files. Both operations
employ boost::locale::conv library.
Since Hive doesn't encode line delimiters, charsets that would have
delimiters stored differently from ASCII are not allowed.
One difference from Hive is that Impala implements
'serialization.encoding' only as a per partition serdeproperty to avoid
confusion of allowing both serde and tbl properties. (See related
IMPALA-13748)
Change-Id: I787cd01caa52a19d6645519a6cedabe0a5253a65
---
M be/src/exec/hdfs-scanner.h
M be/src/exec/hdfs-text-table-writer.cc
M be/src/exec/hdfs-text-table-writer.h
M be/src/exec/scanner-context.h
M be/src/exec/text/hdfs-text-scanner.cc
M be/src/exec/text/hdfs-text-scanner.h
M be/src/runtime/descriptors.cc
M be/src/runtime/descriptors.h
M be/src/runtime/mem-tracker.cc
M be/src/runtime/mem-tracker.h
M be/src/runtime/row-batch.h
M be/src/util/CMakeLists.txt
A be/src/util/char-codec.cc
A be/src/util/char-codec.h
M common/thrift/CatalogObjects.thrift
M common/thrift/generate_error_codes.py
M fe/src/main/java/org/apache/impala/analysis/AlterTableSetTblProperties.java
M fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java
M fe/src/main/java/org/apache/impala/catalog/HdfsStorageDescriptor.java
M fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java
A tests/query_test/test_charcodec.py
21 files changed, 918 insertions(+), 17 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/49/22049/20
--
To view, visit http://gerrit.cloudera.org:8080/22049
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I787cd01caa52a19d6645519a6cedabe0a5253a65
Gerrit-Change-Number: 22049
Gerrit-PatchSet: 20
Gerrit-Owner: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
[Impala-ASF-CR] IMPALA-10319: Support arbitrary encodings on Text/Sequence files
Posted by "Mihaly Szjatinya (Code Review)" <ge...@cloudera.org>.
Mihaly Szjatinya has uploaded a new patch set (#21). ( http://gerrit.cloudera.org:8080/22049 )
Change subject: IMPALA-10319: Support arbitrary encodings on Text/Sequence files
......................................................................
IMPALA-10319: Support arbitrary encodings on Text/Sequence files
As proposed in Jira, this implements decoding and encoding of text
buffers for Impala/Hive text tables. Given a table with
'serialization.encoding' property set, similarly to Hive, Impala should
be able to encode the inserted data into charset specified, consequently
saving it into a text file. The opposite decoding operation should be
performed upon reading data buffers from text files. Both operations
employ boost::locale::conv library.
Since Hive doesn't encode line delimiters, charsets that would have
delimiters stored differently from ASCII are not allowed.
One difference from Hive is that Impala implements
'serialization.encoding' only as a per partition serdeproperty to avoid
confusion of allowing both serde and tbl properties. (See related
IMPALA-13748)
Change-Id: I787cd01caa52a19d6645519a6cedabe0a5253a65
---
M be/src/exec/hdfs-scanner.h
M be/src/exec/hdfs-text-table-writer.cc
M be/src/exec/hdfs-text-table-writer.h
M be/src/exec/scanner-context.h
M be/src/exec/text/hdfs-text-scanner.cc
M be/src/exec/text/hdfs-text-scanner.h
M be/src/runtime/descriptors.cc
M be/src/runtime/descriptors.h
M be/src/runtime/mem-tracker.cc
M be/src/runtime/mem-tracker.h
M be/src/runtime/row-batch.h
M be/src/util/CMakeLists.txt
A be/src/util/char-codec.cc
A be/src/util/char-codec.h
M common/thrift/CatalogObjects.thrift
M common/thrift/generate_error_codes.py
M fe/src/main/java/org/apache/impala/analysis/AlterTableSetTblProperties.java
M fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java
M fe/src/main/java/org/apache/impala/catalog/HdfsStorageDescriptor.java
M fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java
A tests/query_test/test_charcodec.py
21 files changed, 920 insertions(+), 19 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/49/22049/21
--
To view, visit http://gerrit.cloudera.org:8080/22049
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I787cd01caa52a19d6645519a6cedabe0a5253a65
Gerrit-Change-Number: 22049
Gerrit-PatchSet: 21
Gerrit-Owner: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
[Impala-ASF-CR] IMPALA-10319: Support arbitrary encodings on Text files
Posted by "Mihaly Szjatinya (Code Review)" <ge...@cloudera.org>.
Mihaly Szjatinya has posted comments on this change. ( http://gerrit.cloudera.org:8080/22049 )
Change subject: IMPALA-10319: Support arbitrary encodings on Text files
......................................................................
Patch Set 23:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/22049/21/tests/query_test/test_charcodec.py
File tests/query_test/test_charcodec.py:
http://gerrit.cloudera.org:8080/#/c/22049/21/tests/query_test/test_charcodec.py@72
PS21, Line 72: class TestCharCodecGen(ImpalaTestSuite):
> Did you check how much time does the test takes? If it is slow, then it wou
The whole thing takes ~18m on my machine. So I've left only a single gbk encoding for non-exhaustive mode, now it's 4m. And the exhaustive mode now takes 32, since by itself it adds some more dimensions.
--
To view, visit http://gerrit.cloudera.org:8080/22049
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I787cd01caa52a19d6645519a6cedabe0a5253a65
Gerrit-Change-Number: 22049
Gerrit-PatchSet: 23
Gerrit-Owner: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Peter Rozsa <pr...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Mon, 26 May 2025 23:09:18 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] WIP IMPALA-10319: Support arbitrary encodings on Text/Sequence files
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22049 )
Change subject: WIP IMPALA-10319: Support arbitrary encodings on Text/Sequence files
......................................................................
Patch Set 9:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/18261/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/22049
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I787cd01caa52a19d6645519a6cedabe0a5253a65
Gerrit-Change-Number: 22049
Gerrit-PatchSet: 9
Gerrit-Owner: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Tue, 11 Feb 2025 22:12:48 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] WIP IMPALA-10319: Support arbitrary encodings on Text/Sequence files
Posted by "Mihaly Szjatinya (Code Review)" <ge...@cloudera.org>.
Mihaly Szjatinya has uploaded a new patch set (#11). ( http://gerrit.cloudera.org:8080/22049 )
Change subject: WIP IMPALA-10319: Support arbitrary encodings on Text/Sequence files
......................................................................
WIP IMPALA-10319: Support arbitrary encodings on Text/Sequence files
As proposed in Jira, this implements decoding and encoding of text
buffers for Impala/Hive text tables. Given a table with
'serialization.encoding' property set, similarly to Hive, Impala should
be able to encode the inserted data into charset specified, consequently
saving it into a text file. The opposite decoding operation should be
performed upon reading data buffers from text files. Both operations
employ boost::locale::conv library.
Since Hive doesn't encode line delimiters, charsets that would have
delimiters stored differently from ASCII are not allowed.
One difference from Hive is that Impala implements
'serialization.encoding' only as a per partition serdeproperty to avoid
confusion of allowing both serde and tbl properties.
Change-Id: I787cd01caa52a19d6645519a6cedabe0a5253a65
---
M be/src/exec/hdfs-scanner.h
M be/src/exec/hdfs-text-table-writer.cc
M be/src/exec/hdfs-text-table-writer.h
M be/src/exec/scanner-context.h
M be/src/exec/text/hdfs-text-scanner.cc
M be/src/exec/text/hdfs-text-scanner.h
M be/src/runtime/descriptors.cc
M be/src/runtime/descriptors.h
M be/src/runtime/row-batch.h
M be/src/util/CMakeLists.txt
A be/src/util/char-codec.cc
A be/src/util/char-codec.h
M common/thrift/CatalogObjects.thrift
M common/thrift/generate_error_codes.py
M fe/src/main/java/org/apache/impala/analysis/AlterTableSetTblProperties.java
M fe/src/main/java/org/apache/impala/catalog/HdfsStorageDescriptor.java
M fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java
A tests/query_test/test_charcodec.py
18 files changed, 741 insertions(+), 15 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/49/22049/11
--
To view, visit http://gerrit.cloudera.org:8080/22049
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I787cd01caa52a19d6645519a6cedabe0a5253a65
Gerrit-Change-Number: 22049
Gerrit-PatchSet: 11
Gerrit-Owner: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
[Impala-ASF-CR] WIP IMPALA-10319: Support arbitrary encodings on Text/Sequence files
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22049 )
Change subject: WIP IMPALA-10319: Support arbitrary encodings on Text/Sequence files
......................................................................
Patch Set 8:
Build Failed
https://jenkins.impala.io/job/gerrit-code-review-checks/18093/ : Initial code review checks failed. See linked job for details on the failure.
--
To view, visit http://gerrit.cloudera.org:8080/22049
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I787cd01caa52a19d6645519a6cedabe0a5253a65
Gerrit-Change-Number: 22049
Gerrit-PatchSet: 8
Gerrit-Owner: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Mon, 27 Jan 2025 11:40:30 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] WIP IMPALA-10319: Support arbitrary encodings on Text/Sequence files
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22049 )
Change subject: WIP IMPALA-10319: Support arbitrary encodings on Text/Sequence files
......................................................................
Patch Set 13:
(1 comment)
gerrit-auto-critic failed. You can reproduce it locally using command:
python3 bin/jenkins/critique-gerrit-review.py --dryrun
To run it, you might need a virtual env with Python3's venv installed.
http://gerrit.cloudera.org:8080/#/c/22049/13/tests/query_test/test_charcodec.py
File tests/query_test/test_charcodec.py:
http://gerrit.cloudera.org:8080/#/c/22049/13/tests/query_test/test_charcodec.py@44
PS13, Line 44: u
flake8: F821 undefined name 'unichr'
--
To view, visit http://gerrit.cloudera.org:8080/22049
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I787cd01caa52a19d6645519a6cedabe0a5253a65
Gerrit-Change-Number: 22049
Gerrit-PatchSet: 13
Gerrit-Owner: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Sun, 02 Mar 2025 21:08:15 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] WIP IMPALA-10319: Support arbitrary encodings on Text/Sequence files
Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/22049 )
Change subject: WIP IMPALA-10319: Support arbitrary encodings on Text/Sequence files
......................................................................
Patch Set 15:
(4 comments)
http://gerrit.cloudera.org:8080/#/c/22049/15/be/src/runtime/mem-tracker.cc
File be/src/runtime/mem-tracker.cc:
http://gerrit.cloudera.org:8080/#/c/22049/15/be/src/runtime/mem-tracker.cc@587
PS15, Line 587: if (tracker_ != nullptr) {
: tracker_->Release(size_);
: }
We should also consider whether the allocation actually happened and was successful.
My suggestion is to set size_ to 0 in the constructor and increase it in TryConsume based on a parameter if it was successful. This would allow calling TryConsume multiple times.
http://gerrit.cloudera.org:8080/#/c/22049/14/fe/src/main/java/org/apache/impala/analysis/AlterTableSetTblProperties.java
File fe/src/main/java/org/apache/impala/analysis/AlterTableSetTblProperties.java:
http://gerrit.cloudera.org:8080/#/c/22049/14/fe/src/main/java/org/apache/impala/analysis/AlterTableSetTblProperties.java@297
PS14, Line 297: if (tbl instanceof FeIcebergTable) {
I am unsure about the detailed verification.
- I am not sure if FeIcebergTable should support any serde property in general
- this strictness also disallows unsetting the property if it incorrectly set in another engine
- the same verification should be done for create table too
http://gerrit.cloudera.org:8080/#/c/22049/14/fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java
File fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java:
http://gerrit.cloudera.org:8080/#/c/22049/14/fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java@1068
PS14, Line 1068: "alter table
CREATE TABLE ... SERDEPROPERTIES could be also tested. See https://github.com/apache/impala/blob/b121a40d20107bad6c04732ba580f26639acd43a/docker/quickstart_client/load_tpcds_parquet.sql#L78 for example.
Meanwhile I am not sure about the serde prop verification, see my comment in AlterTableSetTblProperties.java
http://gerrit.cloudera.org:8080/#/c/22049/14/tests/query_test/test_charcodec.py
File tests/query_test/test_charcodec.py:
http://gerrit.cloudera.org:8080/#/c/22049/14/tests/query_test/test_charcodec.py@77
PS14, Line 77: lambda v: v.get_value('table_format').file_format == 'text'
> IIRC we have previously agreed to move SequenceFiles decoding to a separate
It is not clear to me how this patch handles Sequence files - AlterTableSetTblProperties.java line 319 suggests that it is supported.
--
To view, visit http://gerrit.cloudera.org:8080/22049
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I787cd01caa52a19d6645519a6cedabe0a5253a65
Gerrit-Change-Number: 22049
Gerrit-PatchSet: 15
Gerrit-Owner: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Mon, 31 Mar 2025 12:19:40 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] WIP IMPALA-10319: Support arbitrary encodings on Text/Sequence files
Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/22049 )
Change subject: WIP IMPALA-10319: Support arbitrary encodings on Text/Sequence files
......................................................................
Patch Set 14:
(13 comments)
http://gerrit.cloudera.org:8080/#/c/22049/14//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/22049/14//COMMIT_MSG@22
PS14, Line 22: confusion of allowing both serde and tbl properties.
Can you mention IMPALA-13748?
http://gerrit.cloudera.org:8080/#/c/22049/14/be/src/runtime/descriptors.h
File be/src/runtime/descriptors.h:
http://gerrit.cloudera.org:8080/#/c/22049/14/be/src/runtime/descriptors.h@452
PS14, Line 452:
nit: extra line
http://gerrit.cloudera.org:8080/#/c/22049/14/be/src/util/char-codec.h
File be/src/util/char-codec.h:
http://gerrit.cloudera.org:8080/#/c/22049/14/be/src/util/char-codec.h@40
PS14, Line 40: ,
nit: extra ,
http://gerrit.cloudera.org:8080/#/c/22049/14/be/src/util/char-codec.h@47
PS14, Line 47: progressivly
nit: progressively
http://gerrit.cloudera.org:8080/#/c/22049/14/be/src/util/char-codec.h@57
PS14, Line 57: MemTrackerHelper
this could be mover to mem_tracker.h, e.g. as MemTracker::ScopedAllocation
http://gerrit.cloudera.org:8080/#/c/22049/14/be/src/util/char-codec.h@80
PS14, Line 80: /// Delimiter used to separate tuples.
: const char tuple_delim_;
:
: /// Buffer to hold the partial symbol that could not be decoded in the previous call to
: /// DecodeBuffer.
: std::vector<uint8_t> partial_symbol_;
:
: /// Can we reuse the output buffer or do we need to allocate on each call?
: bool reuse_buffer_;
:
: /// Buffer to hold transformed data.
: /// Either passed from the caller or allocated from memory_pool_.
: uint8_t* out_buffer_ = nullptr;
:
: /// Length of the output buffer.
: int64_t buffer_length_ = 0;
: };
These are only used by DecodeBuffer - can you add a comment about that, e.g. the following member are only used by decoding, not encoding?
http://gerrit.cloudera.org:8080/#/c/22049/8/be/src/util/char-codec.cc
File be/src/util/char-codec.cc:
http://gerrit.cloudera.org:8080/#/c/22049/8/be/src/util/char-codec.cc@77
PS8, Line 77: return memory_pool_->mem_tracker()->MemLimitExceeded(nullptr, details, *bytes_read);
: }
> Agree, added TryConsume() within a helper class to secure Release() for eve
I think that the current estimates are good enough.
http://gerrit.cloudera.org:8080/#/c/22049/8/be/src/util/char-codec.cc@113
PS8, Line 113: partial_symbol_.clear();
> Agree, but instead of releasing here, I employed the mechanism from Decompr
I think that we are still suboptimal here - for example if I understand correctly in case of stream compression + !reuse_buffer we'll attach both the decompressed buffer and the decoded buffer to the pool, while the the decompressed buffer is no longer used. Probably the best would be to clean this up in a follow up commit.
>Potential problem is a significant exceeding of row_batch's limit
My understanding is that we can't be much smarter here - for example if decompression leads to a very large uncompressed block we need to attach it to the row batch's pool.
A way to solve it would be to copy string from the buffer in case the buffer is very large.
http://gerrit.cloudera.org:8080/#/c/22049/14/be/src/util/char-codec.cc
File be/src/util/char-codec.cc:
http://gerrit.cloudera.org:8080/#/c/22049/14/be/src/util/char-codec.cc@25
PS14, Line 25: #include "gutil/strings/substitute.h"
nit: usually includes to Impala headers are sorted alphabetically
http://gerrit.cloudera.org:8080/#/c/22049/14/be/src/util/char-codec.cc@50
PS14, Line 50: }
Both branches mean that out_buffer_ is no longer usable, right? I think that it would be clearer to set it to nullptr and check only for out_buffer_ == nullptr in line 71.
http://gerrit.cloudera.org:8080/#/c/22049/14/be/src/util/char-codec.cc@57
PS14, Line 57: (buf_end - buf_start) * 2
Can you add a comment about reason for this size?
http://gerrit.cloudera.org:8080/#/c/22049/14/be/src/util/char-codec.cc@71
PS14, Line 71: if (!reuse_buffer_ || out_buffer_ == nullptr) {
: buffer_length_ = std::max(STREAM_OUT_BUF_SIZE, *bytes_read);
Isn't it possible for out_buffer_ to be set but not large enough? For example if there are two compressed blocks and the uncompressed size of the second is smaller than the first's.
I think that we should check for out_buffer_ == nullptr || buffer_length_ < *bytes_read and free the old buffer it is not large enough (see also my comment at line 50).
I also think that std::max(STREAM_OUT_BUF_SIZE, *bytes_read); can lead to allocating too much memory, for example in case of small files or when using stream compression - if reuse_buffer_ is false, than simply allocating *bytes_read seems the best. If reuse_buffer_ is true, then it may make sense to overallocate to avoid multiple allocations when *bytes_read is growing, but using *bytes_read looks also good and makes the code simpler.
http://gerrit.cloudera.org:8080/#/c/22049/14/tests/query_test/test_charcodec.py
File tests/query_test/test_charcodec.py:
http://gerrit.cloudera.org:8080/#/c/22049/14/tests/query_test/test_charcodec.py@77
PS14, Line 77: lambda v: v.get_value('table_format').file_format == 'text'
Is there any test for SequenceFile?
--
To view, visit http://gerrit.cloudera.org:8080/22049
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I787cd01caa52a19d6645519a6cedabe0a5253a65
Gerrit-Change-Number: 22049
Gerrit-PatchSet: 14
Gerrit-Owner: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Thu, 20 Mar 2025 08:27:48 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10319: Support arbitrary encodings on Text/Sequence files
Posted by "Mihaly Szjatinya (Code Review)" <ge...@cloudera.org>.
Mihaly Szjatinya has uploaded a new patch set (#2). ( http://gerrit.cloudera.org:8080/22049 )
Change subject: IMPALA-10319: Support arbitrary encodings on Text/Sequence files
......................................................................
IMPALA-10319: Support arbitrary encodings on Text/Sequence files
[Draft]
As proposed in Jira, this implements decoding of text buffers for
Impala/Hive text tables. Given a table with 'serialization.encoding'
property set, Hive is able to encode the inserted data into charset
specified, consequently saving it into a text file. Objective is to
perform the opposite operation upon reading data buffers from text files
by employing boost::locale::conv::to_utf() library capability.
Special handling is provided for different BOM scenarios, as well as for
line delimiters, which are not part of the encoding on the Hive side.
For example, given a standard line delimiter '\n', represented by a
single byte '0A' in UTF-8 and two bytes '0x0A 0x00' in UTF-16LE, Hive
inserts it as a single '0A'.
Note: although the described behavior allows keeping the range split
logic simpler, on the negative side it renders Hive files not readable
by regular decoders. Conversely, regular multi-byte files with
delimiters are not correctly read by Hive/Impala. This limitation has no
effect on single-byte charsets.
Design Notes:
1. Memory management. Current draft is a naive
<conv::to_utf() std::string segments -> concat into std::string -> pool>
approach. This can be improved in two steps:
a. Get rid of string for gathering segments and allocate directly on
pool.
b. Provide custom allocator for segment strings, returned by
conv::to_utf() to get those allocated and hopefully RVO'd directly on
pool as well. At the very least, those strings could be tracked via
TrackedString.
2. Decoding filled buffer as a whole. Currently decoding is performed
line by line upon the whole buffer in HdfsTextScanner::FillByteBuffer(),
after it has been read and, if needed, decompressed. This has the
definite advantage of having all the decoding logic in one place,
potentially extractable to a separate class.
Alternatives would be:
a. Decoding tuples /(lines upon their materialization on later stages of
HdfsTextScanner. This would employ the existing splitting logic, even
though decoding itself would be done similarly line by line.
b. Preliminarily transforming malformed line delimiters to secure a
smooth conv::to_utf() in one go.
3. An optimization may be done to avoid delimiters only for multi-byte
encodings.
TODO / Nice to Have:
1. Once design approved, this should be extracted into class (hierarchy)
and used for Sequence files. Potentially expandable for encoding by
writers.
2. Test / think through all possible delimiter (FIELD_DELIM, LINE_DELIM,
COLLECTION_DELIM, ESCAPE_CHAR) scenarios.
3. Test archived scenarios.
4. Juxtapose java vs boost encodings supported.
5. Tests all errors/warnings scenarious / corner cases.
Change-Id: I787cd01caa52a19d6645519a6cedabe0a5253a65
---
M be/src/exec/hdfs-scanner.h
M be/src/exec/text/hdfs-text-scanner.cc
M be/src/exec/text/hdfs-text-scanner.h
M be/src/runtime/descriptors.cc
M be/src/runtime/descriptors.h
M common/thrift/CatalogObjects.thrift
M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
A testdata/decoding/cp1251_names.txt
A testdata/decoding/gbk_names.txt
A testdata/decoding/koi8r_names.txt
A testdata/decoding/latin1_names.txt
A testdata/decoding/shift_jis_names.txt
A testdata/decoding/utf16_names.txt
A testdata/decoding/utf16be_names.txt
A testdata/decoding/utf16le_names.txt
A testdata/workloads/functional-query/queries/QueryTest/decoding.test
A tests/query_test/test_decoding.py
17 files changed, 550 insertions(+), 2 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/49/22049/2
--
To view, visit http://gerrit.cloudera.org:8080/22049
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I787cd01caa52a19d6645519a6cedabe0a5253a65
Gerrit-Change-Number: 22049
Gerrit-PatchSet: 2
Gerrit-Owner: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
[Impala-ASF-CR] IMPALA-10319: Support arbitrary encodings on Text/Sequence files
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22049 )
Change subject: IMPALA-10319: Support arbitrary encodings on Text/Sequence files
......................................................................
Patch Set 2:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/17430/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/22049
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I787cd01caa52a19d6645519a6cedabe0a5253a65
Gerrit-Change-Number: 22049
Gerrit-PatchSet: 2
Gerrit-Owner: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Sun, 10 Nov 2024 19:02:46 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10319: Support arbitrary encodings on Text/Sequence files
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22049 )
Change subject: IMPALA-10319: Support arbitrary encodings on Text/Sequence files
......................................................................
Patch Set 1:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/17429/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/22049
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I787cd01caa52a19d6645519a6cedabe0a5253a65
Gerrit-Change-Number: 22049
Gerrit-PatchSet: 1
Gerrit-Owner: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Sun, 10 Nov 2024 18:44:33 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] WIP IMPALA-10319: Support arbitrary encodings on Text/Sequence files
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22049 )
Change subject: WIP IMPALA-10319: Support arbitrary encodings on Text/Sequence files
......................................................................
Patch Set 6:
(2 comments)
gerrit-auto-critic failed. You can reproduce it locally using command:
python2 bin/jenkins/critique-gerrit-review.py --dryrun
To run it, you might need a virtual env with virtualenv installed.
http://gerrit.cloudera.org:8080/#/c/22049/6/common/thrift/CatalogObjects.thrift
File common/thrift/CatalogObjects.thrift:
PS6:
This file is used in communication between impalad and catalogd/statestore. Please make sure impalads can still work with new/old versions of catalogd and statestore. Basically only new optional fields can be added.
http://gerrit.cloudera.org:8080/#/c/22049/6/common/thrift/CatalogObjects.thrift@359
PS6, Line 359: 9: required string encodingValue
Modifying/adding this required field might break the compatibility between impalad and catalogd/statestore during upgrade
--
To view, visit http://gerrit.cloudera.org:8080/22049
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I787cd01caa52a19d6645519a6cedabe0a5253a65
Gerrit-Change-Number: 22049
Gerrit-PatchSet: 6
Gerrit-Owner: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Comment-Date: Wed, 08 Jan 2025 13:03:32 +0000
Gerrit-HasComments: Yes