[jira] [Created] (HAWQ-53) Avro Union(complex) type containing nullable type and using null as default failed

["Goden Yao (JIRA)" <ji...@apache.org>]

Goden Yao created HAWQ-53:
-----------------------------

             Summary: Avro Union(complex) type containing nullable type and using null as default failed
                 Key: HAWQ-53
                 URL: https://issues.apache.org/jira/browse/HAWQ-53
             Project: Apache HAWQ
          Issue Type: Bug
          Components: External Tables & PXF
            Reporter: Goden Yao
            Assignee: Shivram Mani


>From code review [AvroResolver.java|https://github.com/apache/incubator-hawq/blob/master/pxf/pxf-hdfs/src/main/java/com/pivotal/pxf/plugins/hdfs/AvroResolver.java], we didn't specify null situation for complex types:
- Array
- Map
- Record

when using union type , which contains a nullable type(union) and use null as default - I think complex type with nullable type in value schema and with null as default is the problem.

{code:json}
...
"name" : "meta__kvpairs",
    "type" : [ "null", {
      "type" : "map",
      "values" : [ "null", "string" ],
      "default" : null
    } ],
    "default" : null
  },
...
{code}

Create external table from pxf to read it:

{code:sql}
"CREATE EXTERNAL TABLE meetmeevent_hdfs(
sourcetimestamp bigint,
sourceagent varchar(200),
sourceip_address varchar(20),
meetmeuser_id numeric(15,0),
meetmeinterested_uid numeric(15,0),
meetmevote varchar(4),
meetmeis_match numeric(3,0),
meetmenetwork_score numeric (10,0),
meetmeresponsiveness_score numeric (10,0),
meetmesession_id varchar(32),
meetmefriends numeric(1,0),
meetmeprevious_view varchar(32), 
meetmemodel numeric(15,0),
meetmescore numeric(15,0),
meetmemethod numeric(1,0),
meetmecontributions varchar(120),
meetmeclicksource varchar(20),
meetmeclickaction varchar(20),
meetmeprofileview_ts bigint,
meetmeplatform varchar(20),
metatopic_name varchar(100),
metarequest_user_agent varchar(200),
metarequest_session_id varchar(50),
metarequest_id bigint,
metakvpairs varchar(1000),
meta_handlers varchar(1000),
dt varchar(6),
hour varchar(2)
)
LOCATION ('pxf://dahdp2nn01.tag-dev.com:50070/data/ramblas/event_log/meetme/20150824/12/s_meetme.20.0.3492572.11600054147.1440442800000.avro?PROFILE=Avro&DATA-SCHEMA=/data/ramblas/schema/meetme')
FORMAT 'CUSTOM' (formatter='pxfwritable_import');
{code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)