[jira] [Commented] (DRILL-5828) RecordBatchLoader permutes column order

["Paul Rogers (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/DRILL-5828?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16186618#comment-16186618 ] 

Paul Rogers commented on DRILL-5828:
------------------------------------

The fix for DRILL-5826 will provide a unit test that demonstrates this issue:

{code}
    final RecordBatchLoader batchLoader = new RecordBatchLoader(allocator);
    BatchSchema schema1 = new SchemaBuilder()
        .add("a", MinorType.INT)
        .add("b", MinorType.VARCHAR)
        .build();
    {
      // Prime the loader with the above schema
      assertTrue(loadBatch(allocator, batchLoader, schema1));
      assertTrue(schema1.isEquivalent(batchLoader.getSchema()));
      batchLoader.getContainer().zeroVectors();
    }
    {
      // Next batch has a permutated schema
      BatchSchema schema = new SchemaBuilder()
          .add("b", MinorType.VARCHAR)
          .add("a", MinorType.INT)
          .build();
      // Load the batch. Returns false, indicating no schema change
      assertFalse(loadBatch(allocator, batchLoader, schema));
      // But, the actual schema has changed (order): it matches the second schema
      assertTrue(schema.isEquivalent(batchLoader.getSchema()));
      batchLoader.getContainer().zeroVectors();
    }
{code}


> RecordBatchLoader permutes column order
> ---------------------------------------
>
>                 Key: DRILL-5828
>                 URL: https://issues.apache.org/jira/browse/DRILL-5828
>             Project: Apache Drill
>          Issue Type: Bug
>    Affects Versions: 1.11.0
>            Reporter: Paul Rogers
>            Priority: Minor
>
> The {{RecordBatchLoader}} class deserializes batches and checks for schema changes. As part of investigating DRILL-5826, it seems that {{RecordBatchLoader}} detects schema changes as follows:
> * If two batches have the same column in the same order, no schema change occurs. (Fine)
> * If batch A has schema (a, b) while batch B has (b, a), then no schema change occurs. (Fine)
> But, in the case of permutated columns (second case above), the {{RecordBatchLoader}} returns the column order of the second batch, though it says that no schema change has occurred.
> That is, {{RecordBatchLoader}} says that the schema has not changed, but the actual schema has changed (column order changed.)
> This is a potential problem: if a downstream batch counts on the same column order, then that assumption is violated by the behavior described above.
> Correct behavior would be to coerce the second batch to match the schema of the first batch, if the {{RecordBatchLoader}} indicates that no schema change occurred.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)