You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Daniel Vanko (Code Review)" <ge...@cloudera.org> on 2026/07/15 15:05:10 UTC

[Impala-ASF-CR] IMPALA-5961: Convert generated TPC-DS test data to UTF-8

Daniel Vanko has uploaded this change for review. ( http://gerrit.cloudera.org:8080/24585


Change subject: IMPALA-5961: Convert generated TPC-DS test data to UTF-8
......................................................................

IMPALA-5961: Convert generated TPC-DS test data to UTF-8

The TPC-DS data generator (dsdgen) emits some string columns in
ISO-8859-1 (Latin-1) rather than UTF-8. For example,
customer.c_birth_country contains a 0xD4 byte (o-circumflex in
Latin-1) that is not valid Unicode, which will cause data-load failure
if PARQUET_ANNOTATE_STRINGS_UTF8 is turned on (IMPALA-12675).

Convert the generated data files to UTF-8 in
testdata/datasets/tpcds/preload (via iconv) right after generation,
so the loaded TPC-DS test data is valid Unicode.

Change-Id: Ic72dd4ad641083c22e8759160251b620e9da605c
Assisted-by: Claude Opus 4.8 (Claude Code)
---
M testdata/datasets/tpcds/preload
1 file changed, 9 insertions(+), 0 deletions(-)



  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/85/24585/1
-- 
To view, visit http://gerrit.cloudera.org:8080/24585
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: Ic72dd4ad641083c22e8759160251b620e9da605c
Gerrit-Change-Number: 24585
Gerrit-PatchSet: 1
Gerrit-Owner: Daniel Vanko <dv...@cloudera.com>

[Impala-ASF-CR] IMPALA-5961: Convert generated TPC-DS test data to UTF-8

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24585 )

Change subject: IMPALA-5961: Convert generated TPC-DS test data to UTF-8
......................................................................


Patch Set 1:

Build Failed 

https://jenkins.impala.io/job/gerrit-code-review-checks/23248/ : Initial code review checks failed. See linked job for details on the failure.


-- 
To view, visit http://gerrit.cloudera.org:8080/24585
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic72dd4ad641083c22e8759160251b620e9da605c
Gerrit-Change-Number: 24585
Gerrit-PatchSet: 1
Gerrit-Owner: Daniel Vanko <dv...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Daniel Vanko <dv...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Comment-Date: Wed, 15 Jul 2026 15:25:03 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-5961: Read generated TPC-DS test data as ISO-8859-1

Posted by "Joe McDonnell (Code Review)" <ge...@cloudera.org>.
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/24585 )

Change subject: IMPALA-5961: Read generated TPC-DS test data as ISO-8859-1
......................................................................


Patch Set 3: Code-Review+1

This seems like the right way to do it, as the original data is encoded in ISO-8859-1. Specifying it on the table level is reasonable.


-- 
To view, visit http://gerrit.cloudera.org:8080/24585
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic72dd4ad641083c22e8759160251b620e9da605c
Gerrit-Change-Number: 24585
Gerrit-PatchSet: 3
Gerrit-Owner: Daniel Vanko <dv...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Daniel Vanko <dv...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Comment-Date: Fri, 17 Jul 2026 18:57:03 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-5961: Read generated TPC-DS test data as ISO-8859-1

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24585 )

Change subject: IMPALA-5961: Read generated TPC-DS test data as ISO-8859-1
......................................................................


Patch Set 2:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/23269/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/24585
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic72dd4ad641083c22e8759160251b620e9da605c
Gerrit-Change-Number: 24585
Gerrit-PatchSet: 2
Gerrit-Owner: Daniel Vanko <dv...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Daniel Vanko <dv...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Comment-Date: Thu, 16 Jul 2026 16:13:48 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-5961: Read generated TPC-DS test data as ISO-8859-1

Posted by "Daniel Vanko (Code Review)" <ge...@cloudera.org>.
Hello Joe McDonnell, Csaba Ringhofer, Noemi Pap-Takacs, Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/24585

to look at the new patch set (#5).

Change subject: IMPALA-5961: Read generated TPC-DS test data as ISO-8859-1
......................................................................

IMPALA-5961: Read generated TPC-DS test data as ISO-8859-1

The TPC-DS data generator (dsdgen) emits some string columns in
ISO-8859-1 (Latin-1) rather than UTF-8. For example,
customer.c_birth_country holds country names whose accented letters
are single Latin-1 bytes (e.g. 0xD4 for O-circumflex in "COTE
D'IVOIRE") that are not valid UTF-8, which will cause data-load failure
if PARQUET_ANNOTATE_STRINGS_UTF8 is turned on (IMPALA-12675).

Tag the TPC-DS text tables with the serde property
'serialization.encoding'='ISO-8859-1' so Impala decodes the raw data
files to UTF-8 on read -- including when the derived Parquet (and
other format) tables are populated from the text tables via
INSERT ... SELECT. The raw dsdgen output is kept as-is on disk, and
this exercises the serialization.encoding feature (IMPALA-10319) in
the standard data load.

The property is applied via ALTER TABLE ... SET SERDEPROPERTIES in the
LOAD section rather than at CREATE: Impala's CREATE does not accept a
custom SERDE, and 'serialization.encoding' is honored only from serde
properties (not TBLPROPERTIES). The LOAD section is the text-only load
phase, so the ALTER runs before the derived tables read the data.

Change-Id: Ic72dd4ad641083c22e8759160251b620e9da605c
Assisted-by: Claude Opus 4.8 (Claude Code)
---
M testdata/datasets/tpcds/tpcds_schema_template.sql
M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q04.test
M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q11.test
3 files changed, 28 insertions(+), 4 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/85/24585/5
-- 
To view, visit http://gerrit.cloudera.org:8080/24585
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ic72dd4ad641083c22e8759160251b620e9da605c
Gerrit-Change-Number: 24585
Gerrit-PatchSet: 5
Gerrit-Owner: Daniel Vanko <dv...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Daniel Vanko <dv...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>

[Impala-ASF-CR] IMPALA-5961: Read generated TPC-DS test data as ISO-8859-1

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24585 )

Change subject: IMPALA-5961: Read generated TPC-DS test data as ISO-8859-1
......................................................................


Patch Set 5:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/23364/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/24585
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic72dd4ad641083c22e8759160251b620e9da605c
Gerrit-Change-Number: 24585
Gerrit-PatchSet: 5
Gerrit-Owner: Daniel Vanko <dv...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Daniel Vanko <dv...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Comment-Date: Thu, 23 Jul 2026 09:27:42 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-5961: Read generated TPC-DS test data as ISO-8859-1

Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/24585 )

Change subject: IMPALA-5961: Read generated TPC-DS test data as ISO-8859-1
......................................................................


Patch Set 3: Code-Review+2

(1 comment)

http://gerrit.cloudera.org:8080/#/c/24585/3/testdata/datasets/tpcds/tpcds_schema_template.sql
File testdata/datasets/tpcds/tpcds_schema_template.sql:

http://gerrit.cloudera.org:8080/#/c/24585/3/testdata/datasets/tpcds/tpcds_schema_template.sql@44
PS3, Line 44: ALTER TABLE {db_name}{db_suffix}.{table_name} SET SERDEPROPERTIES('serialization.encoding'='ISO-8859-1');
It would be nicer to have a ---- SERDE_PROPERTIES section, but there are many things to clean up about these schema template files, this looks like just a minor improvement.



-- 
To view, visit http://gerrit.cloudera.org:8080/24585
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic72dd4ad641083c22e8759160251b620e9da605c
Gerrit-Change-Number: 24585
Gerrit-PatchSet: 3
Gerrit-Owner: Daniel Vanko <dv...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Daniel Vanko <dv...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Comment-Date: Wed, 22 Jul 2026 14:34:10 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-5961: Read generated TPC-DS test data as ISO-8859-1

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24585 )

Change subject: IMPALA-5961: Read generated TPC-DS test data as ISO-8859-1
......................................................................


Patch Set 4: Verified-1

Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/13776/


-- 
To view, visit http://gerrit.cloudera.org:8080/24585
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic72dd4ad641083c22e8759160251b620e9da605c
Gerrit-Change-Number: 24585
Gerrit-PatchSet: 4
Gerrit-Owner: Daniel Vanko <dv...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Daniel Vanko <dv...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Comment-Date: Wed, 22 Jul 2026 19:40:44 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-5961: Read generated TPC-DS test data as ISO-8859-1

Posted by "Daniel Vanko (Code Review)" <ge...@cloudera.org>.
Daniel Vanko has posted comments on this change. ( http://gerrit.cloudera.org:8080/24585 )

Change subject: IMPALA-5961: Read generated TPC-DS test data as ISO-8859-1
......................................................................


Patch Set 2:

(2 comments)

> Patch Set 2: Code-Review+1
> 
> lgtm, but I am curious about other reviewer's opinion about re-encoding during dataload vs setting it for tables

Thanks, although I'm not so sure about if the ALTER TABLE ... SET SERDEPROPERTIES('serialization.encoding'='ISO-8859-1') statements being on the beginning of LOAD section is semantically correct. README mentions an ALTER session as well, I'm checking if that can be used.

http://gerrit.cloudera.org:8080/#/c/24585/1/testdata/datasets/tpcds/preload
File testdata/datasets/tpcds/preload:

http://gerrit.cloudera.org:8080/#/c/24585/1/testdata/datasets/tpcds/preload@116
PS1, Line 116: 
> Did you check how much time this step takes?
On my desktop it took about 2-3 seconds (3-5% of the whole preload script).


http://gerrit.cloudera.org:8080/#/c/24585/1/testdata/datasets/tpcds/preload@117
PS1, Line 117: tables=("call_center" "catalog_page" "catalog_returns" "catalog_sales" "customer"
             :         "customer_address" "customer_demographics" "date_dim" "dbgen_version"
             :         "household_demographics" "income_band" "inventory" "item" "promotion"
             :         "reason" "ship_mode" "store" "store_returns" "store_sales" "time_dim"
             :         "warehouse" "web_page" "web_returns" "web_sales" "web_site")
             : 
             : # Verify that all the expected files exist. Impala expects each table to be in its
             : # ow
> An alternative would be do the conversion when reading with Impala.
Done



-- 
To view, visit http://gerrit.cloudera.org:8080/24585
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic72dd4ad641083c22e8759160251b620e9da605c
Gerrit-Change-Number: 24585
Gerrit-PatchSet: 2
Gerrit-Owner: Daniel Vanko <dv...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Daniel Vanko <dv...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Comment-Date: Thu, 16 Jul 2026 16:15:20 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-5961: Read generated TPC-DS test data as ISO-8859-1

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24585 )

Change subject: IMPALA-5961: Read generated TPC-DS test data as ISO-8859-1
......................................................................


Patch Set 3:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/23280/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/24585
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic72dd4ad641083c22e8759160251b620e9da605c
Gerrit-Change-Number: 24585
Gerrit-PatchSet: 3
Gerrit-Owner: Daniel Vanko <dv...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Daniel Vanko <dv...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Comment-Date: Fri, 17 Jul 2026 09:14:07 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-5961: Read generated TPC-DS test data as ISO-8859-1

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/24585 )

Change subject: IMPALA-5961: Read generated TPC-DS test data as ISO-8859-1
......................................................................

IMPALA-5961: Read generated TPC-DS test data as ISO-8859-1

The TPC-DS data generator (dsdgen) emits some string columns in
ISO-8859-1 (Latin-1) rather than UTF-8. For example,
customer.c_birth_country holds country names whose accented letters
are single Latin-1 bytes (e.g. 0xD4 for O-circumflex in "COTE
D'IVOIRE") that are not valid UTF-8, which will cause data-load failure
if PARQUET_ANNOTATE_STRINGS_UTF8 is turned on (IMPALA-12675).

Tag the TPC-DS text tables with the serde property
'serialization.encoding'='ISO-8859-1' so Impala decodes the raw data
files to UTF-8 on read -- including when the derived Parquet (and
other format) tables are populated from the text tables via
INSERT ... SELECT. The raw dsdgen output is kept as-is on disk, and
this exercises the serialization.encoding feature (IMPALA-10319) in
the standard data load.

The property is applied via ALTER TABLE ... SET SERDEPROPERTIES in the
LOAD section rather than at CREATE: Impala's CREATE does not accept a
custom SERDE, and 'serialization.encoding' is honored only from serde
properties (not TBLPROPERTIES). The LOAD section is the text-only load
phase, so the ALTER runs before the derived tables read the data.

Change-Id: Ic72dd4ad641083c22e8759160251b620e9da605c
Assisted-by: Claude Opus 4.8 (Claude Code)
Reviewed-on: http://gerrit.cloudera.org:8080/24585
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
M testdata/datasets/tpcds/tpcds_schema_template.sql
M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q04.test
M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q11.test
3 files changed, 28 insertions(+), 4 deletions(-)

Approvals:
  Impala Public Jenkins: Looks good to me, approved; Verified

-- 
To view, visit http://gerrit.cloudera.org:8080/24585
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: Ic72dd4ad641083c22e8759160251b620e9da605c
Gerrit-Change-Number: 24585
Gerrit-PatchSet: 7
Gerrit-Owner: Daniel Vanko <dv...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Daniel Vanko <dv...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>

[Impala-ASF-CR] IMPALA-5961: Read generated TPC-DS test data as ISO-8859-1

Posted by "Daniel Vanko (Code Review)" <ge...@cloudera.org>.
Daniel Vanko has uploaded a new patch set (#2). ( http://gerrit.cloudera.org:8080/24585 )

Change subject: IMPALA-5961: Read generated TPC-DS test data as ISO-8859-1
......................................................................

IMPALA-5961: Read generated TPC-DS test data as ISO-8859-1

The TPC-DS data generator (dsdgen) emits some string columns in
ISO-8859-1 (Latin-1) rather than UTF-8. For example,
customer.c_birth_country holds country names whose accented letters
are single Latin-1 bytes (e.g. 0xD4 for O-circumflex in "COTE
D'IVOIRE") that are not valid UTF-8, which has caused query failures
(e.g. tpcds-q30).

Tag the TPC-DS text tables with the serde property
'serialization.encoding'='ISO-8859-1' so Impala decodes the raw data
files to UTF-8 on read -- including when the derived Parquet (and
other format) tables are populated from the text tables via
INSERT ... SELECT. The raw dsdgen output is kept as-is on disk, and
this exercises the serialization.encoding feature (IMPALA-10319) in
the standard data load.

Change-Id: Ic72dd4ad641083c22e8759160251b620e9da605c
Assisted-by: Claude Opus 4.8 (Claude Code)
---
M testdata/datasets/tpcds/tpcds_schema_template.sql
1 file changed, 24 insertions(+), 0 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/85/24585/2
-- 
To view, visit http://gerrit.cloudera.org:8080/24585
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ic72dd4ad641083c22e8759160251b620e9da605c
Gerrit-Change-Number: 24585
Gerrit-PatchSet: 2
Gerrit-Owner: Daniel Vanko <dv...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Daniel Vanko <dv...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>

[Impala-ASF-CR] IMPALA-5961: Read generated TPC-DS test data as ISO-8859-1

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24585 )

Change subject: IMPALA-5961: Read generated TPC-DS test data as ISO-8859-1
......................................................................


Patch Set 6:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/13783/ DRY_RUN=false


-- 
To view, visit http://gerrit.cloudera.org:8080/24585
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic72dd4ad641083c22e8759160251b620e9da605c
Gerrit-Change-Number: 24585
Gerrit-PatchSet: 6
Gerrit-Owner: Daniel Vanko <dv...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Daniel Vanko <dv...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Comment-Date: Thu, 23 Jul 2026 09:06:22 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-5961: Read generated TPC-DS test data as ISO-8859-1

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24585 )

Change subject: IMPALA-5961: Read generated TPC-DS test data as ISO-8859-1
......................................................................


Patch Set 4:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/13776/ DRY_RUN=false


-- 
To view, visit http://gerrit.cloudera.org:8080/24585
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic72dd4ad641083c22e8759160251b620e9da605c
Gerrit-Change-Number: 24585
Gerrit-PatchSet: 4
Gerrit-Owner: Daniel Vanko <dv...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Daniel Vanko <dv...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Comment-Date: Wed, 22 Jul 2026 14:35:18 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-5961: Read generated TPC-DS test data as ISO-8859-1

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24585 )

Change subject: IMPALA-5961: Read generated TPC-DS test data as ISO-8859-1
......................................................................


Patch Set 4: Code-Review+2


-- 
To view, visit http://gerrit.cloudera.org:8080/24585
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic72dd4ad641083c22e8759160251b620e9da605c
Gerrit-Change-Number: 24585
Gerrit-PatchSet: 4
Gerrit-Owner: Daniel Vanko <dv...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Daniel Vanko <dv...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Comment-Date: Wed, 22 Jul 2026 14:35:17 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-5961: Read generated TPC-DS test data as ISO-8859-1

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24585 )

Change subject: IMPALA-5961: Read generated TPC-DS test data as ISO-8859-1
......................................................................


Patch Set 6: Verified+1


-- 
To view, visit http://gerrit.cloudera.org:8080/24585
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic72dd4ad641083c22e8759160251b620e9da605c
Gerrit-Change-Number: 24585
Gerrit-PatchSet: 6
Gerrit-Owner: Daniel Vanko <dv...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Daniel Vanko <dv...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Comment-Date: Thu, 23 Jul 2026 14:06:09 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-5961: Read generated TPC-DS test data as ISO-8859-1

Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/24585 )

Change subject: IMPALA-5961: Read generated TPC-DS test data as ISO-8859-1
......................................................................


Patch Set 5: Code-Review+2


-- 
To view, visit http://gerrit.cloudera.org:8080/24585
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic72dd4ad641083c22e8759160251b620e9da605c
Gerrit-Change-Number: 24585
Gerrit-PatchSet: 5
Gerrit-Owner: Daniel Vanko <dv...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Daniel Vanko <dv...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Comment-Date: Thu, 23 Jul 2026 09:05:59 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-5961: Read generated TPC-DS test data as ISO-8859-1

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24585 )

Change subject: IMPALA-5961: Read generated TPC-DS test data as ISO-8859-1
......................................................................


Patch Set 6: Code-Review+2


-- 
To view, visit http://gerrit.cloudera.org:8080/24585
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic72dd4ad641083c22e8759160251b620e9da605c
Gerrit-Change-Number: 24585
Gerrit-PatchSet: 6
Gerrit-Owner: Daniel Vanko <dv...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Daniel Vanko <dv...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Comment-Date: Thu, 23 Jul 2026 09:06:22 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-5961: Convert generated TPC-DS test data to UTF-8

Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/24585 )

Change subject: IMPALA-5961: Convert generated TPC-DS test data to UTF-8
......................................................................


Patch Set 1:

(2 comments)

http://gerrit.cloudera.org:8080/#/c/24585/1/testdata/datasets/tpcds/preload
File testdata/datasets/tpcds/preload:

http://gerrit.cloudera.org:8080/#/c/24585/1/testdata/datasets/tpcds/preload@116
PS1, Line 116: 
Did you check how much time this step takes?


http://gerrit.cloudera.org:8080/#/c/24585/1/testdata/datasets/tpcds/preload@117
PS1, Line 117: # dsdgen emits string data in ISO-8859-1 (Latin-1), which can contain bytes that are
             : # not valid UTF-8 (e.g. the 0xD4 in customer.c_birth_country). Convert the generated
             : # files to UTF-8 so the loaded TPC-DS data is valid Unicode (IMPALA-5961).
             : echo "Converting generated data files from ISO-8859-1 to UTF-8"
             : for data_file in *.dat; do
             :   iconv -f ISO-8859-1 -t UTF-8 "${data_file}" -o "${data_file}.utf8"
             :   mv "${data_file}.utf8" "${data_file}"
             : done
An alternative would be do the conversion when reading with Impala.
This can be done by setting TBLPROPERTIES("serialization.encoding"="ISO-8859-1") for the text tables.
I would prefer this approach because it would give more testing for this relatively new encoding feature.



-- 
To view, visit http://gerrit.cloudera.org:8080/24585
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic72dd4ad641083c22e8759160251b620e9da605c
Gerrit-Change-Number: 24585
Gerrit-PatchSet: 1
Gerrit-Owner: Daniel Vanko <dv...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Daniel Vanko <dv...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Comment-Date: Wed, 15 Jul 2026 15:56:09 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-5961: Read generated TPC-DS test data as ISO-8859-1

Posted by "Daniel Vanko (Code Review)" <ge...@cloudera.org>.
Daniel Vanko has uploaded a new patch set (#3). ( http://gerrit.cloudera.org:8080/24585 )

Change subject: IMPALA-5961: Read generated TPC-DS test data as ISO-8859-1
......................................................................

IMPALA-5961: Read generated TPC-DS test data as ISO-8859-1

The TPC-DS data generator (dsdgen) emits some string columns in
ISO-8859-1 (Latin-1) rather than UTF-8. For example,
customer.c_birth_country holds country names whose accented letters
are single Latin-1 bytes (e.g. 0xD4 for O-circumflex in "COTE
D'IVOIRE") that are not valid UTF-8, which will cause data-load failure
if PARQUET_ANNOTATE_STRINGS_UTF8 is turned on (IMPALA-12675).

Tag the TPC-DS text tables with the serde property
'serialization.encoding'='ISO-8859-1' so Impala decodes the raw data
files to UTF-8 on read -- including when the derived Parquet (and
other format) tables are populated from the text tables via
INSERT ... SELECT. The raw dsdgen output is kept as-is on disk, and
this exercises the serialization.encoding feature (IMPALA-10319) in
the standard data load.

The property is applied via ALTER TABLE ... SET SERDEPROPERTIES in the
LOAD section rather than at CREATE: Impala's CREATE does not accept a
custom SERDE, and 'serialization.encoding' is honored only from serde
properties (not TBLPROPERTIES). The LOAD section is the text-only load
phase, so the ALTER runs before the derived tables read the data.

Change-Id: Ic72dd4ad641083c22e8759160251b620e9da605c
Assisted-by: Claude Opus 4.8 (Claude Code)
---
M testdata/datasets/tpcds/tpcds_schema_template.sql
1 file changed, 24 insertions(+), 0 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/85/24585/3
-- 
To view, visit http://gerrit.cloudera.org:8080/24585
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ic72dd4ad641083c22e8759160251b620e9da605c
Gerrit-Change-Number: 24585
Gerrit-PatchSet: 3
Gerrit-Owner: Daniel Vanko <dv...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Daniel Vanko <dv...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>