[jira] [Commented] (HIVE-6147) Support avro data stored in HBase columns

["Vamsi Subhash Achanta (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HIVE-6147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16588536#comment-16588536 ] 

Vamsi Subhash Achanta commented on HIVE-6147:
---------------------------------------------

[~swarnim]

Hi,

Is there any work currently going on to make the serialization work at a column level? If I have 2 columns with different avro schema for each column, the current code is not working. Ex:

{{CREATE EXTERNAL TABLE txn_store.transactions_single3_1}}
{{ROW FORMAT SERDE "org.apache.hadoop.hive.hbase.HBaseSerDe"}}
{{STORED BY "org.apache.hadoop.hive.hbase.HBaseStorageHandler"}}
{{WITH SERDEPROPERTIES (}}
{{"hbase.columns.mapping" = ":key,nsp:scr_p_1,nsp:scr_m_1,nsp:scr_a_1",}}{{"nsp.scr_p_1.serialization.type" = "avro",}}
{{"avro.schema.retriever" = "com.phonepe.hive.schema.PaymentNamespaceAvroSchemaRetriever",}}
{{"nsp.scr_p_1.avro.schema.url" = "hdfs://namenode:8020/user/admin/schemas/payment_namespace.avsc",}}{{"nsp.scr_m_1.serialization.type" = "avro",}}
{{"avro.schema.retriever" = "com.phonepe.hive.schema.MerchantNamespaceAvroSchemaRetriever",}}
{{"nsp.scr_m_1.avro.schema.url" = "hdfs://namenode.nm1:8020/user/admin/schemas/merchant_namespace.avsc",}}{{"nsp.scr_a_1.serialization.type" = "avro",}}
{{"avro.schema.retriever" = "com.phonepe.hive.schema.AccountingNamespaceAvroSchemaRetriever",}}
{{"nsp.scr_a_1.avro.schema.url" = "hdfs://namenode:8020/user/admin/schemas/accounting_namespace.avsc"}}
{{)}}
{{TBLPROPERTIES (}}
{{"hbase.table.name"="txn_store:transactions_single3_cf",}}
{{"hbase.table.default.storage.type"="binary",}}
{{"hbase.mapred.output.outputtable"="txn_store:transactions_single3_cf",}}
{{"hbase.struct.autogenerate"="true");}}

When executing the select query, it fails with below exception:
Caused by: org.apache.hive.service.cli.HiveSQLException: java.io.IOException: org.apache.hadoop.hive.ql.metadata.HiveException: Error evaluating nsp_scra1
 

> Support avro data stored in HBase columns
> -----------------------------------------
>
>                 Key: HIVE-6147
>                 URL: https://issues.apache.org/jira/browse/HIVE-6147
>             Project: Hive
>          Issue Type: Improvement
>          Components: HBase Handler
>    Affects Versions: 0.12.0, 0.13.0
>            Reporter: Swarnim Kulkarni
>            Assignee: Swarnim Kulkarni
>            Priority: Major
>             Fix For: 0.14.0
>
>         Attachments: HIVE-6147.1.patch.txt, HIVE-6147.2.patch.txt, HIVE-6147.3.patch.txt, HIVE-6147.3.patch.txt, HIVE-6147.4.patch.txt, HIVE-6147.5.patch.txt, HIVE-6147.6.patch.txt
>
>
> Presently, the HBase Hive integration supports querying only primitive data types in columns. It would be nice to be able to store and query Avro objects in HBase columns by making them visible as structs to Hive. This will allow Hive to perform ad hoc analysis of HBase data which can be deeply structured.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)