[jira] [Created] (DRILL-6418) Handle Schema change in Unnest And Lateral for unnest field / non-unnest field

["Sorabh Hamirwasia (JIRA)" <ji...@apache.org>]

Sorabh Hamirwasia created DRILL-6418:
----------------------------------------

             Summary: Handle Schema change in Unnest And Lateral for unnest field / non-unnest field
                 Key: DRILL-6418
                 URL: https://issues.apache.org/jira/browse/DRILL-6418
             Project: Apache Drill
          Issue Type: Improvement
            Reporter: Sorabh Hamirwasia
            Assignee: Sorabh Hamirwasia
             Fix For: 1.14.0


Handling the schema change scenarios for LATERAL and UNNEST when schema change is observed for non-unnest field and unnest field. It should also handle the scenario when UNNEST field is Repeated Map Type and schema change happens only in the children field of Map component. Following issues were found:

1) There were issues found in how scan treats that kind of data where it scans 2 files, one has Map (let say cutomer_order) children type (custKey) as integer and other has custKey as string type. Then Scan just replaces the ValueVector from its output container for custKey but in the schema it has 2 fields with same name but different types.

2) Unnest and Lateral check for schema change across batches based on the MaterializedField and BatchSchema reference they store. But since it's a reference any change in pointed value changes their reference as well and hence comparison always returns true. So for Unnest it actually keeps a clone of the Materialized Field instead of reference and use that to determine for schema change. For LATERAL any OK_NEW_SCHEMA from upstream is treated as actual schema change and setup is done again.

3) Since the MaterializedField is mutable the isEquivalent method checks for object equality which is removed as that is not correct for mutable objects.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)