[jira] [Updated] (PIG-2579) Support for multiple input schemas in AvroStorage

["Cheolsoo Park (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/PIG-2579?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Cheolsoo Park updated PIG-2579:
-------------------------------

    Attachment: PIG-2579.patch
                PIG-2579-avro_test_files.tar.gz

I updated the original Stan's patch re-basing it to trunk. While I kept the core logic unchanged, I made some modifications as follows:
# Removed glob pattern related code as it's resolved in PIG-2492.
# Added an option 'multiple_schema' to AvroStorage. By default, AvroStorage assumes that all the input files have the same schema, but if 'multiple_schema' is passed to load function, it tries to merge every input schema.
# Allows multiple schemas with the same name. I use paths to identify schemas instead of their names.
# Refactored code.
# Added unit tests.

I think that the most arguable part is how to merge two different schemas into one. In shorts, the rules are as follows:
# Different primitive types can be merged if certain conditions are met. Please see AvroStorageUtils.mergeType() for more details.
# Only the same kind of complex types can be merged. e.g. record + record => ok, but record + array => error.
# For records, the union of fields is returned.
# For arrays/maps, their element types/value types are merged.
# For unions, the union of unions is returned.
# For fixeds, only the same size of fixeds can be merged.

It's easy to see in a unit test (TestAvroStorageUtils) what's expected when two schemas are merged.

Please let me know if you have any questions/concerns.

Thanks!
                
> Support for multiple input schemas in AvroStorage
> -------------------------------------------------
>
>                 Key: PIG-2579
>                 URL: https://issues.apache.org/jira/browse/PIG-2579
>             Project: Pig
>          Issue Type: New Feature
>          Components: piggybank
>    Affects Versions: 0.9.2, 0.11
>            Reporter: Stan Rosenberg
>            Assignee: Cheolsoo Park
>            Priority: Minor
>         Attachments: avro_storage_union_schema.patch, avro_storage_union_schema_test.tar.gz, PIG-2579-avro_test_files.tar.gz, PIG-2579.patch
>
>
> This is a barebones patch for AvroStorage which enables support of multiple input schemas.  The assumption is that the input consists of avro files having different schemas that can be unioned, e.g., flat records.  
> A simple illustrative example is attached (avro_storage_union_schema_test.tar.gz): run create_avro1.pig, followed by create_avro2.pig, followed by read_avro.pig.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira