You are viewing a plain text version of this content. The canonical link for it is here.
Posted to gitbox@hive.apache.org by GitBox <gi...@apache.org> on 2022/07/06 18:22:44 UTC
[GitHub] [hive] zabetak commented on a diff in pull request #3418: HIVE-26373: ClassCastException while inserting Avro data into Hbase table for nested struct with Timestamp
zabetak commented on code in PR #3418:
URL: https://github.com/apache/hive/pull/3418#discussion_r915124542
##########
hbase-handler/src/test/results/positive/hbase_avro_nested_timestamp.q.out:
##########
@@ -0,0 +1,45 @@
+PREHOOK: query: CREATE EXTERNAL TABLE tbl(
+`key` string COMMENT '',
+`data_frv4` struct<`id`:string, `dischargedate`:struct<`value`:timestamp>>)
+ROW FORMAT SERDE
+ 'org.apache.hadoop.hive.hbase.HBaseSerDe'
+STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
+WITH SERDEPROPERTIES (
+'serialization.format'='1',
+'hbase.columns.mapping' = ':key,data:frV4',
+'data.frV4.serialization.type'='avro',
+#### A masked pattern was here ####
+)
+TBLPROPERTIES (
+'hbase.table.name' = 'HiveAvroTable',
+'hbase.struct.autogenerate'='true')
+PREHOOK: type: CREATETABLE
+PREHOOK: Output: database:default
+PREHOOK: Output: default@tbl
+POSTHOOK: query: CREATE EXTERNAL TABLE tbl(
+`key` string COMMENT '',
+`data_frv4` struct<`id`:string, `dischargedate`:struct<`value`:timestamp>>)
+ROW FORMAT SERDE
+ 'org.apache.hadoop.hive.hbase.HBaseSerDe'
+STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
+WITH SERDEPROPERTIES (
+'serialization.format'='1',
+'hbase.columns.mapping' = ':key,data:frV4',
+'data.frV4.serialization.type'='avro',
+#### A masked pattern was here ####
+)
+TBLPROPERTIES (
+'hbase.table.name' = 'HiveAvroTable',
+'hbase.struct.autogenerate'='true')
+POSTHOOK: type: CREATETABLE
+POSTHOOK: Output: database:default
+POSTHOOK: Output: default@tbl
+PREHOOK: query: select data_frV4.dischargedate.value from tbl
+PREHOOK: type: QUERY
+PREHOOK: Input: default@tbl
+#### A masked pattern was here ####
+POSTHOOK: query: select data_frV4.dischargedate.value from tbl
+POSTHOOK: type: QUERY
+POSTHOOK: Input: default@tbl
+#### A masked pattern was here ####
+1970-01-19 20:16:19.2
Review Comment:
The result does not appear to be correct. I was expecting to see something like `2022-07-05 00:00:00`.
Are we doing something wrong when inserting the data? Are we doing something wrong while fetching the data?
##########
itests/util/src/main/java/org/apache/hadoop/hive/hbase/HBaseTestSetup.java:
##########
@@ -158,6 +170,41 @@ private void createHBaseTable() throws IOException {
}
}
+ private byte[] createAvroRecordWithNestedTimestamp() throws IOException {
+ String dataDir = System.getProperty("test.data.dir");
+ Schema schema = new Schema.Parser().parse(new File(dataDir+"/nested_ts.avsc"));
Review Comment:
This path concatenation may not be portable to different OS.
##########
itests/util/src/main/java/org/apache/hadoop/hive/hbase/HBaseTestSetup.java:
##########
@@ -158,6 +170,41 @@ private void createHBaseTable() throws IOException {
}
}
+ private byte[] createAvroRecordWithNestedTimestamp() throws IOException {
+ String dataDir = System.getProperty("test.data.dir");
+ Schema schema = new Schema.Parser().parse(new File(dataDir+"/nested_ts.avsc"));
Review Comment:
Make method static.
##########
itests/util/src/main/java/org/apache/hadoop/hive/hbase/HBaseTestSetup.java:
##########
@@ -158,6 +170,41 @@ private void createHBaseTable() throws IOException {
}
}
+ private byte[] createAvroRecordWithNestedTimestamp() throws IOException {
+ String dataDir = System.getProperty("test.data.dir");
+ Schema schema = new Schema.Parser().parse(new File(dataDir+"/nested_ts.avsc"));
+ GenericData.Record rootRecord = new GenericData.Record(schema);
+ rootRecord.put("id", "X338092");
+ GenericData.Record dateRecord = new GenericData.Record(schema.getField("dischargedate").schema());
+ dateRecord.put("value", LocalDate.of(2022,7,5).atStartOfDay().toEpochSecond(ZoneOffset.UTC));
+ rootRecord.put("dischargedate", dateRecord);
+
+ try (ByteArrayOutputStream out = new ByteArrayOutputStream()) {
+ try (DataFileWriter<GenericRecord> dataFileWriter
+ = new DataFileWriter<GenericRecord>(new GenericDatumWriter<>(schema))) {
+ dataFileWriter.create(schema, out);
+ dataFileWriter.append(rootRecord);
+ }
+ return out.toByteArray();
+ }
+ }
+
+ private void createAvroTable() throws IOException {
+ final String HBASE_TABLE_NAME = "HiveAvroTable";
+ HTableDescriptor htableDesc = new HTableDescriptor(TableName.valueOf(HBASE_TABLE_NAME));
+ htableDesc.addFamily(new HColumnDescriptor("data".getBytes()));
+
+ try (Admin hbaseAdmin = hbaseConn.getAdmin()) {
+ hbaseAdmin.createTable(htableDesc);
+ try (Table table = hbaseConn.getTable(TableName.valueOf(HBASE_TABLE_NAME))) {
Review Comment:
Extract `TableName.valueOf("HiveAvroTable")` to local variable and replace occurrences.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org