You are viewing a plain text version of this content. The canonical link for it is here.
Posted to gitbox@hive.apache.org by GitBox <gi...@apache.org> on 2021/05/14 09:16:33 UTC
[GitHub] [hive] adrian-wang opened a new pull request #2276: HIVE-21200: Vectorization: date column throwing java.lang.Unsupported…
adrian-wang opened a new pull request #2276:
URL: https://github.com/apache/hive/pull/2276
…OperationException for parquet
### What changes were proposed in this pull request?
The date type should be treated as int instead of long.
### Why are the changes needed?
otherwise exception will be thrown when read parquet files with date column when vectorization is enabled.
### Does this PR introduce _any_ user-facing change?
no.
### How was this patch tested?
add a unit test for date.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] sunchao commented on pull request #2276: HIVE-21200: Vectorization: date column throwing java.lang.UnsupportedOperationException for parquet
Posted by GitBox <gi...@apache.org>.
sunchao commented on pull request #2276:
URL: https://github.com/apache/hive/pull/2276#issuecomment-841695662
Merged. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] sunchao commented on a change in pull request #2276: HIVE-21200: Vectorization: date column throwing java.lang.Unsupported…
Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #2276:
URL: https://github.com/apache/hive/pull/2276#discussion_r632796159
##########
File path: ql/src/test/org/apache/hadoop/hive/ql/io/parquet/VectorizedColumnReaderTestBase.java
##########
@@ -372,6 +374,35 @@ protected void longRead(boolean isDictionaryEncoding) throws Exception {
}
}
+ protected void dateRead(boolean isDictionaryEncoding) throws InterruptedException, HiveException, IOException {
+ Configuration conf = new Configuration();
+ conf.set(IOConstants.COLUMNS,"date_field");
+ conf.set(IOConstants.COLUMNS_TYPES,"date");
+ conf.setBoolean(ColumnProjectionUtils.READ_ALL_COLUMNS, false);
+ conf.set(ColumnProjectionUtils.READ_COLUMN_IDS_CONF_STR, "0");
+ VectorizedParquetRecordReader reader =
+ createParquetReader("message test { required int date_field;}", conf);
+ VectorizedRowBatch previous = reader.createValue();
+ try {
+ int c = 0;
+ while (reader.next(NullWritable.get(), previous)) {
+ LongColumnVector vector = (LongColumnVector) previous.cols[0];
+ assertTrue(vector.noNulls);
+ for (int i = 0; i < vector.vector.length; i++) {
+ if(c == nElements){
Review comment:
nit: space after `if`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] sunchao merged pull request #2276: HIVE-21200: Vectorization: date column throwing java.lang.UnsupportedOperationException for parquet
Posted by GitBox <gi...@apache.org>.
sunchao merged pull request #2276:
URL: https://github.com/apache/hive/pull/2276
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] adrian-wang commented on pull request #2276: HIVE-21200: Vectorization: date column throwing java.lang.Unsupported…
Posted by GitBox <gi...@apache.org>.
adrian-wang commented on pull request #2276:
URL: https://github.com/apache/hive/pull/2276#issuecomment-841123252
@sunchao Could you please help review this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org