You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "Zouxxyy (via GitHub)" <gi...@apache.org> on 2023/06/06 02:29:15 UTC
[GitHub] [hudi] Zouxxyy commented on a diff in pull request #8888: [MINOR] Reformat HoodieAvroParquetReader
Zouxxyy commented on code in PR #8888:
URL: https://github.com/apache/hudi/pull/8888#discussion_r1218792317
##########
hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/avro/HoodieAvroParquetReader.java:
##########
@@ -90,7 +83,7 @@ public Void getCurrentKey() throws IOException, InterruptedException {
@Override
public ArrayWritable getCurrentValue() throws IOException, InterruptedException {
GenericRecord record = parquetRecordReader.getCurrentValue();
- return (ArrayWritable) HoodieRealtimeRecordReaderUtils.avroToArrayWritable(record, baseSchema, true);
+ return (ArrayWritable) HoodieRealtimeRecordReaderUtils.avroToArrayWritable(record, record.getSchema(), true);
Review Comment:
yes, I just tested it and it works fine, hive 3.1.3
```text
hive> select * from 0605_mc_hive_schema_fix;
OK
20230601185040816 20230601185040816_0_1 a_int:3 486591f0-3e2e-457a-9f3a-d8305d5850fd-0_0-11-11_20230601185040816.parquet 1 2 34 0.5 1.5 12345678.09 v1 c1 s1 1998-02-15 15:59:21.943 001 true
Time taken: 2.043 seconds, Fetched: 1 row(s)
hive> select a_tinyint, a_timestamp from 0605_mc_hive_schema_fix;
OK
1 1998-02-15 15:59:21.943
Time taken: 0.397 seconds, Fetched: 1 row(s)
hive> select a_tinyint from 0605_mc_hive_schema_fix;
OK
1
Time taken: 0.21 seconds, Fetched: 1 row(s)
hive> select a_timestamp from 0605_mc_hive_schema_fix;
select a_timestamp from 0605_mc_hive_schema_fix
OK
1998-02-15 15:59:21.943
Time taken: 0.867 seconds, Fetched: 1 row(s)
```
##########
hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/avro/HoodieAvroParquetReader.java:
##########
@@ -90,7 +83,7 @@ public Void getCurrentKey() throws IOException, InterruptedException {
@Override
public ArrayWritable getCurrentValue() throws IOException, InterruptedException {
GenericRecord record = parquetRecordReader.getCurrentValue();
- return (ArrayWritable) HoodieRealtimeRecordReaderUtils.avroToArrayWritable(record, baseSchema, true);
+ return (ArrayWritable) HoodieRealtimeRecordReaderUtils.avroToArrayWritable(record, record.getSchema(), true);
Review Comment:
@xicm yes, I just tested it and it works fine, hive 3.1.3
```text
hive> select * from 0605_mc_hive_schema_fix;
OK
20230601185040816 20230601185040816_0_1 a_int:3 486591f0-3e2e-457a-9f3a-d8305d5850fd-0_0-11-11_20230601185040816.parquet 1 2 34 0.5 1.5 12345678.09 v1 c1 s1 1998-02-15 15:59:21.943 001 true
Time taken: 2.043 seconds, Fetched: 1 row(s)
hive> select a_tinyint, a_timestamp from 0605_mc_hive_schema_fix;
OK
1 1998-02-15 15:59:21.943
Time taken: 0.397 seconds, Fetched: 1 row(s)
hive> select a_tinyint from 0605_mc_hive_schema_fix;
OK
1
Time taken: 0.21 seconds, Fetched: 1 row(s)
hive> select a_timestamp from 0605_mc_hive_schema_fix;
select a_timestamp from 0605_mc_hive_schema_fix
OK
1998-02-15 15:59:21.943
Time taken: 0.867 seconds, Fetched: 1 row(s)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org