[jira] [Created] (HIVE-15316) CTAS STORED AS AVRO: AvroTypeException Found default.record_0, expecting union

["David Maughan (JIRA)" <ji...@apache.org>]

David Maughan created HIVE-15316:
------------------------------------

             Summary: CTAS STORED AS AVRO: AvroTypeException Found default.record_0, expecting union
                 Key: HIVE-15316
                 URL: https://issues.apache.org/jira/browse/HIVE-15316
             Project: Hive
          Issue Type: Bug
          Components: Hive
    Affects Versions: 2.1.0
            Reporter: David Maughan
            Priority: Minor


There's an issue when querying a table that has been created as Avro via CTAS when the target struct is at least 2 struct-levels deep. It can be replicated with the following steps:

{code}
CREATE TABLE a
  STORED AS AVRO
  AS
SELECT named_struct('c', named_struct('d', 1)) as b;

SELECT b FROM a;

org.apache.avro.AvroTypeException: Found default.record_0, expecting union
{code}

The reason for this is that during table creation, the Avro schema is generated from the Hive columns in {{AvroSerDe}} and then passed through the Avro Schema Parser: {{new Schema.Parser().parse(schema.toString())}}. For the above example, this creates the below schema in the Avro file. Note that the lowest level struct, {{record_0}} has {{"namespace": "default"}}.

{code}
{
  "type": "record",
  "name": "a",
  "namespace": "default",
  "fields": [
    {
      "name": "b",
      "type": [
        "null",
        {
          "type": "record",
          "name": "record_1",
          "namespace": "",
          "doc": "struct<c:struct<d:int>>",
          "fields": [
            {
              "name": "c",
              "type": [
                "null",
                {
                  "type": "record",
                  "name": "record_0",
                  "namespace": "default",
                  "doc": "struct<d:int>",
                  "fields": [
                    {
                      "name": "d",
                      "type": [ "null", "int" ],
                      "doc": "int",
                      "default": null
                    }
                  ]
                }
              ],
              "doc": "struct<d:int>",
              "default": null
            }
          ]
        }
      ],
      "default": null
    }
  ]
}
{code}

On a subsequent select query, the Avro schema is again generated from the Hive columns. However, this time it is not passed through the Avro Schema Parser and the {{namespace}} attribute is not present in {{record_0}}. The actual Error message _"Found default.record_0, expecting union"_ is slightly misleading. Although it is a expected a union, it is specifically expected a null or a record named {{record_0}} but it finds {{default.record_0}}.

I believe this is a bug in Avro. I'm not sure whether correct behaviour is to cascade the namespace down or not but it is definitely an inconsistency between creating a schema via the builders and parser. I've created [AVRO-1965|https://issues.apache.org/jira/browse/AVRO-1965] for this. However, I believe that defensively passing the schema through the Avro Schema Parser on a select query would fix this issue in Hive without an Avro fix and version bump in Hive.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)