[jira] [Commented] (DRILL-6418) Handle Schema change in Unnest And Lateral for unnest field / non-unnest field

["ASF GitHub Bot (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/DRILL-6418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16479943#comment-16479943 ] 

ASF GitHub Bot commented on DRILL-6418:
---------------------------------------

sohami opened a new pull request #1271: DRILL-6418: Handle Schema change in Unnest And Lateral for unnest fie…
URL: https://github.com/apache/drill/pull/1271
 
 
   …ld / non-unnest field
   
               Note: Changed Lateral to handle non-empty right batch with OK_NEW_SCHEMA

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


> Handle Schema change in Unnest And Lateral for unnest field / non-unnest field
> ------------------------------------------------------------------------------
>
>                 Key: DRILL-6418
>                 URL: https://issues.apache.org/jira/browse/DRILL-6418
>             Project: Apache Drill
>          Issue Type: Improvement
>            Reporter: Sorabh Hamirwasia
>            Assignee: Sorabh Hamirwasia
>            Priority: Major
>             Fix For: 1.14.0
>
>
> Handling the schema change scenarios for LATERAL and UNNEST when schema change is observed for non-unnest field and unnest field. It should also handle the scenario when UNNEST field is Repeated Map Type and schema change happens only in the children field of Map component. Following issues were found:
> 1) There were issues found in how scan treats that kind of data where it scans 2 files, one has Map (let say cutomer_order) children type (custKey) as integer and other has custKey as string type. Then Scan just replaces the ValueVector from its output container for custKey but in the schema it has 2 fields with same name but different types.
> 2) Unnest and Lateral check for schema change across batches based on the MaterializedField and BatchSchema reference they store. But since it's a reference any change in pointed value changes their reference as well and hence comparison always returns true. So for Unnest it actually keeps a clone of the Materialized Field instead of reference and use that to determine for schema change. For LATERAL any OK_NEW_SCHEMA from upstream is treated as actual schema change and setup is done again.
> 3) Since the MaterializedField is mutable the isEquivalent method checks for object equality which is removed as that is not correct for mutable objects.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)