You are viewing a plain text version of this content. The canonical link for it is here.
Posted to gitbox@hive.apache.org by GitBox <gi...@apache.org> on 2020/09/11 14:11:26 UTC
[GitHub] [hive] szlta opened a new pull request #1490: HIVE-24151
szlta opened a new pull request #1490:
URL: https://github.com/apache/hive/pull/1490
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] szlta commented on pull request #1490: HIVE-24151
Posted by GitBox <gi...@apache.org>.
szlta commented on pull request #1490:
URL: https://github.com/apache/hive/pull/1490#issuecomment-691122063
Since this is a partial revert I'm placing the diff of LazyStruct.java for: before HIVE-22360 vs my current commit:
szita@szita-MBP16:~/shadow/CDH/hive$ git diff 463dae9ee8f694002af492e7d05924423aeaed09:serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyStruct.java 5de36f990d89fcd5c3d7d2344a28e16e4c1f8c24:serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyStruct.java
diff --git a/serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyStruct.java b/serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyStruct.java
index f066aaa3bf5..66b15374dda 100644
--- a/serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyStruct.java
+++ b/serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyStruct.java
@@ -22,6 +22,8 @@
import java.util.List;
import com.google.common.primitives.Bytes;
+
+import org.apache.hadoop.hive.serde2.MultiDelimitSerDe;
import org.apache.hadoop.hive.serde2.SerDeException;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -294,10 +296,10 @@ public void parseMultiDelimit(byte[] rawRow, byte[] fieldDelimit) {
}
// the indexes of the delimiters
int[] delimitIndexes = findIndexes(rawRow, fieldDelimit);
- int diff = fieldDelimit.length - 1;
+ int diff = fieldDelimit.length - MultiDelimitSerDe.REPLACEMENT_DELIM_LENGTH;
// first field always starts from 0, even when missing
startPosition[0] = 0;
- for (int i = 1; i < fields.length; i++) {
+ for (int i = 1; i <= fields.length; i++) {
if (delimitIndexes[i - 1] != -1) {
int start = delimitIndexes[i - 1] + fieldDelimit.length;
startPosition[i] = start - i * diff;
@@ -305,7 +307,6 @@ public void parseMultiDelimit(byte[] rawRow, byte[] fieldDelimit) {
startPosition[i] = length + 1;
}
}
- startPosition[fields.length] = length + 1;
Arrays.fill(fieldInited, false);
parsed = true;
}
@@ -315,7 +316,7 @@ public void parseMultiDelimit(byte[] rawRow, byte[] fieldDelimit) {
if (fields.length <= 1) {
return new int[0];
}
- int[] indexes = new int[fields.length - 1];
+ int[] indexes = new int[fields.length];
Arrays.fill(indexes, -1);
indexes[0] = Bytes.indexOf(array, target);
if (indexes[0] == -1) {
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] szlta commented on pull request #1490: HIVE-24151
Posted by GitBox <gi...@apache.org>.
szlta commented on pull request #1490:
URL: https://github.com/apache/hive/pull/1490#issuecomment-691122063
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] szlta edited a comment on pull request #1490: HIVE-24151
Posted by GitBox <gi...@apache.org>.
szlta edited a comment on pull request #1490:
URL: https://github.com/apache/hive/pull/1490#issuecomment-691122063
Since this is a partial revert I'm placing the diff of LazyStruct.java for: before HIVE-22360 vs my current commit: https://issues.apache.org/jira/browse/HIVE-24151?focusedCommentId=17194290&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17194290 (github looks to be worse at displaying diffs in PR comments than jira is :) )
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] szlta edited a comment on pull request #1490: HIVE-24151
Posted by GitBox <gi...@apache.org>.
szlta edited a comment on pull request #1490:
URL: https://github.com/apache/hive/pull/1490#issuecomment-691122063
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] szlta edited a comment on pull request #1490: HIVE-24151
Posted by GitBox <gi...@apache.org>.
szlta edited a comment on pull request #1490:
URL: https://github.com/apache/hive/pull/1490#issuecomment-691122063
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] szlta commented on pull request #1490: HIVE-24151
Posted by GitBox <gi...@apache.org>.
szlta commented on pull request #1490:
URL: https://github.com/apache/hive/pull/1490#issuecomment-691122063
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] szlta commented on pull request #1490: HIVE-24151
Posted by GitBox <gi...@apache.org>.
szlta commented on pull request #1490:
URL: https://github.com/apache/hive/pull/1490#issuecomment-691122063
Since this is a partial revert I'm placing the diff of LazyStruct.java for: before HIVE-22360 vs my current commit:
szita@szita-MBP16:~/shadow/CDH/hive$ git diff 463dae9ee8f694002af492e7d05924423aeaed09:serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyStruct.java 5de36f990d89fcd5c3d7d2344a28e16e4c1f8c24:serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyStruct.java
diff --git a/serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyStruct.java b/serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyStruct.java
index f066aaa3bf5..66b15374dda 100644
--- a/serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyStruct.java
+++ b/serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyStruct.java
@@ -22,6 +22,8 @@
import java.util.List;
import com.google.common.primitives.Bytes;
+
+import org.apache.hadoop.hive.serde2.MultiDelimitSerDe;
import org.apache.hadoop.hive.serde2.SerDeException;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -294,10 +296,10 @@ public void parseMultiDelimit(byte[] rawRow, byte[] fieldDelimit) {
}
// the indexes of the delimiters
int[] delimitIndexes = findIndexes(rawRow, fieldDelimit);
- int diff = fieldDelimit.length - 1;
+ int diff = fieldDelimit.length - MultiDelimitSerDe.REPLACEMENT_DELIM_LENGTH;
// first field always starts from 0, even when missing
startPosition[0] = 0;
- for (int i = 1; i < fields.length; i++) {
+ for (int i = 1; i <= fields.length; i++) {
if (delimitIndexes[i - 1] != -1) {
int start = delimitIndexes[i - 1] + fieldDelimit.length;
startPosition[i] = start - i * diff;
@@ -305,7 +307,6 @@ public void parseMultiDelimit(byte[] rawRow, byte[] fieldDelimit) {
startPosition[i] = length + 1;
}
}
- startPosition[fields.length] = length + 1;
Arrays.fill(fieldInited, false);
parsed = true;
}
@@ -315,7 +316,7 @@ public void parseMultiDelimit(byte[] rawRow, byte[] fieldDelimit) {
if (fields.length <= 1) {
return new int[0];
}
- int[] indexes = new int[fields.length - 1];
+ int[] indexes = new int[fields.length];
Arrays.fill(indexes, -1);
indexes[0] = Bytes.indexOf(array, target);
if (indexes[0] == -1) {
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] szlta merged pull request #1490: HIVE-24151
Posted by GitBox <gi...@apache.org>.
szlta merged pull request #1490:
URL: https://github.com/apache/hive/pull/1490
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org