You are viewing a plain text version of this content. The canonical link for it is here.
Posted to gitbox@hive.apache.org by "cyanzheng2926 (via GitHub)" <gi...@apache.org> on 2026/06/23 08:16:32 UTC
[PR] Hive-28728: fix STR_TO_MAP() returning garbled utf-8 characters when vectorization is enabled [hive]
cyanzheng2926 opened a new pull request, #6559:
URL: https://github.com/apache/hive/pull/6559
<!--
Thanks for sending a pull request! Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://cwiki.apache.org/confluence/display/Hive/HowToContribute
2. Ensure that you have created an issue on the Hive project JIRA: https://issues.apache.org/jira/projects/HIVE/summary
3. Ensure you have added or run the appropriate tests for your PR:
4. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP]HIVE-XXXXX: Your PR title ...'.
5. Be sure to keep the PR description updated to reflect all changes.
6. Please write your PR title to summarize what this PR proposes.
7. If possible, provide a concise example to reproduce the issue for a faster review.
-->
### What changes were proposed in this pull request?
<!--
Please clarify what changes you are proposing. The purpose of this section is to outline the changes and how this PR fixes the issue.
If possible, please consider writing useful notes for better and faster reviews in your PR. See the examples below.
1. If you refactor some codes with changing classes, showing the class hierarchy will help reviewers.
2. If you fix some SQL features, you can provide some references of other DBMSes.
3. If there is design documentation, please add the link.
4. If there is a discussion in the mailing list, please add the link.
-->
This PR is proposed for HIVE-28728, when under non-UTF8 default charset, STR_TO_MAP() returns garbled characters with vectorization enabled.
### Why are the changes needed?
<!--
Please clarify why the changes are needed. For instance,
1. If you propose a new API, clarify the use case for a new API.
2. If you fix a bug, you can clarify why it is a bug.
-->
Using default non-UTF8 charset should not give different behaviours (garbled characters vs correct characters) when vectorization is on/off.
This also aligns with the Jira HIVE-28544.
### Does this PR introduce _any_ user-facing change?
<!--
Note that it means *any* user-facing change including all aspects such as the documentation fix.
If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description, screenshot and/or a reproducable example to show the behavior difference if possible.
If possible, please also clarify if this is a user-facing change compared to the released Hive versions or within the unreleased branches such as master.
If no, write 'No'.
-->
No.
### How was this patch tested?
<!--
If tests were added, say they were added here. Please make sure to add some test cases that check the changes thoroughly including negative and positive cases if possible.
If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future.
If tests were not added, please describe why they were not added and/or why it was difficult to add.
-->
Created str_to_map_utf8.q and tested with TestMiniTezCliDriver TestMiniLlapCliDriver with corresponding settings
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28728: fix STR_TO_MAP() returning garbled utf-8 characters when vectorization is enabled [hive]
Posted by "cyanzheng2926 (via GitHub)" <gi...@apache.org>.
cyanzheng2926 commented on code in PR #6559:
URL: https://github.com/apache/hive/pull/6559#discussion_r3463956860
##########
itests/src/test/resources/testconfiguration.properties:
##########
@@ -180,6 +181,7 @@ minillap.query.files=\
skip_header_footer_aggr.q,\
skip_header_footer_proj.q,\
str_to_map.q,\
+ str_to_map_utf8.q,\
Review Comment:
Hi @ayushtkn thanks for the review! I’ll drop the Tez golden and remove str_to_map_utf8.q from minitez.query.files, keeping only the LLAP .q.out to match str_to_map.q.
One note on coverage: I've added SET commands for tez java configs to Dfile.encoding=US-ASCII which should launch tez containers with US_ASCII encoding without a POM change, whereas the LLAP driver will need -Dmaven.test.jvm.args="-Dfile.encoding=US-ASCII" for the same. With LLAP-only, precommit still validates vectorized STR_TO_MAP behavior, but the charset-specific guard is mainly for local/manual runs only since unless we add that JVM arg to CI later.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28728: fix STR_TO_MAP() returning garbled utf-8 characters when vectorization is enabled [hive]
Posted by "cyanzheng2926 (via GitHub)" <gi...@apache.org>.
cyanzheng2926 commented on code in PR #6559:
URL: https://github.com/apache/hive/pull/6559#discussion_r3464045505
##########
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorAssignRow.java:
##########
@@ -21,6 +21,7 @@
import java.util.ArrayList;
import java.util.List;
import java.util.Map;
+import java.nio.charset.StandardCharsets;
Review Comment:
Updated
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28728: fix STR_TO_MAP() returning garbled utf-8 characters when vectorization is enabled [hive]
Posted by "ayushtkn (via GitHub)" <gi...@apache.org>.
ayushtkn merged PR #6559:
URL: https://github.com/apache/hive/pull/6559
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28728: fix STR_TO_MAP() returning garbled utf-8 characters when vectorization is enabled [hive]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #6559:
URL: https://github.com/apache/hive/pull/6559#discussion_r3463816230
##########
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorAssignRow.java:
##########
@@ -21,6 +21,7 @@
import java.util.ArrayList;
import java.util.List;
import java.util.Map;
+import java.nio.charset.StandardCharsets;
Review Comment:
Java imports are out of order: `StandardCharsets` should be grouped with other `java.*` imports (typically before `java.util.*`) to match the ordering used elsewhere in this package and avoid checkstyle churn.
##########
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorAssignRow.java:
##########
@@ -465,7 +466,7 @@ private void assignRowColumn(
{
if (object instanceof String) {
String string = (String) object;
- byte[] bytes = string.getBytes();
+ byte[] bytes = string.getBytes(StandardCharsets.UTF_8);
((BytesColumnVector) columnVector).setVal(
batchIndex, bytes, 0, bytes.length);
Review Comment:
This change fixes `STRING` by forcing UTF-8 bytes, but the same method still uses platform-default `String#getBytes()` for `VARCHAR`/`CHAR` (and the analogous code path later in this class). With a non-UTF-8 default charset those branches can still emit garbled UTF-8; please switch those conversions to `getBytes(StandardCharsets.UTF_8)` as well (e.g., current `HiveVarchar.getValue().getBytes()` / `HiveChar.getStrippedValue().getBytes()`).
##########
ql/src/test/queries/clientpositive/str_to_map_utf8.q:
##########
@@ -0,0 +1,45 @@
+-- HIVE-28728: STR_TO_MAP() must preserve UTF-8 in vectorized execution when JVM default charset is not UTF-8.
+-- Tez container opts below force US-ASCII in Tez tasks
+-- Use driver-level mimic for testing with llap: -Dmaven.test.jvm.args="-Dfile.encoding=US-ASCII"
+
+SET tez.am.launch.cmd-opts=-Dfile.encoding=US-ASCII;
+SET hive.tez.java.opts=-Dfile.encoding=US-ASCII;
+SET hive.vectorized.execution.enabled=true;
+SET hive.fetch.task.conversion=none;
+
+CREATE TABLE hive28728_src (id string, name string, multi string) STORED AS ORC;
+INSERT INTO hive28728_src VALUES
+ ('100','hive', 'en:1'),
Review Comment:
This test creates several persistent tables (`hive28728_src`, `hive28728_result`, `hive28728_multi`, `hive28728_result_novec`) but never drops them. Since the suite cleanup scripts don’t remove these tables, reruns (or other tests that might reuse names) can fail on `CREATE TABLE`; please add `DROP TABLE IF EXISTS ...` before creation and/or drop the tables at the end (and update the expected `.q.out` files accordingly).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28728: fix STR_TO_MAP() returning garbled utf-8 characters when vectorization is enabled [hive]
Posted by "cyanzheng2926 (via GitHub)" <gi...@apache.org>.
cyanzheng2926 commented on code in PR #6559:
URL: https://github.com/apache/hive/pull/6559#discussion_r3463972543
##########
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorAssignRow.java:
##########
@@ -465,7 +466,7 @@ private void assignRowColumn(
{
if (object instanceof String) {
String string = (String) object;
- byte[] bytes = string.getBytes();
+ byte[] bytes = string.getBytes(StandardCharsets.UTF_8);
((BytesColumnVector) columnVector).setVal(
batchIndex, bytes, 0, bytes.length);
Review Comment:
For HIVE-28728 I kept scope to the STRING/String branch without touching VARCHAR/CHAR since GenericUDFStringToMap returns map<string,string> with javaStringObjectInspector, and vectorized STR_TO_MAP() should only hit case STRING. Updates to VARCHAR/CHAR paths can be followed up in a separate Jira
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28728: fix STR_TO_MAP() returning garbled utf-8 characters when vectorization is enabled [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #6559:
URL: https://github.com/apache/hive/pull/6559#issuecomment-4778678566
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6559) **Quality Gate passed**
Issues
 [1 New issue](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6559&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6559&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=6559&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6559&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6559&metric=new_duplicated_lines_density&view=list)
<!-- sqra-placement-anchor -->
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6559)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28728: fix STR_TO_MAP() returning garbled utf-8 characters when vectorization is enabled [hive]
Posted by "ayushtkn (via GitHub)" <gi...@apache.org>.
ayushtkn commented on code in PR #6559:
URL: https://github.com/apache/hive/pull/6559#discussion_r3463911351
##########
ql/src/test/queries/clientpositive/str_to_map_utf8.q:
##########
@@ -0,0 +1,45 @@
+-- HIVE-28728: STR_TO_MAP() must preserve UTF-8 in vectorized execution when JVM default charset is not UTF-8.
+-- Tez container opts below force US-ASCII in Tez tasks
+-- Use driver-level mimic for testing with llap: -Dmaven.test.jvm.args="-Dfile.encoding=US-ASCII"
+
+SET tez.am.launch.cmd-opts=-Dfile.encoding=US-ASCII;
+SET hive.tez.java.opts=-Dfile.encoding=US-ASCII;
+SET hive.vectorized.execution.enabled=true;
+SET hive.fetch.task.conversion=none;
+
+CREATE TABLE hive28728_src (id string, name string, multi string) STORED AS ORC;
+INSERT INTO hive28728_src VALUES
+ ('100','hive', 'en:1'),
Review Comment:
Not required, The framework takes care of afaik
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28728: fix STR_TO_MAP() returning garbled utf-8 characters when vectorization is enabled [hive]
Posted by "ayushtkn (via GitHub)" <gi...@apache.org>.
ayushtkn commented on code in PR #6559:
URL: https://github.com/apache/hive/pull/6559#discussion_r3463909429
##########
itests/src/test/resources/testconfiguration.properties:
##########
@@ -180,6 +181,7 @@ minillap.query.files=\
skip_header_footer_aggr.q,\
skip_header_footer_proj.q,\
str_to_map.q,\
+ str_to_map_utf8.q,\
Review Comment:
I don't think you need to output two .q.out, the LLAP one should suffice
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28728: fix STR_TO_MAP() returning garbled utf-8 characters when vectorization is enabled [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #6559:
URL: https://github.com/apache/hive/pull/6559#issuecomment-4785444082
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6559) **Quality Gate passed**
Issues
 [1 New issue](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6559&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6559&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=6559&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6559&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6559&metric=new_duplicated_lines_density&view=list)
<!-- sqra-placement-anchor -->
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6559)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org