[jira] [Created] (DRILL-7434) TopNBatch constructs Union vector incorrectly

["Paul Rogers (Jira)" <ji...@apache.org>]

Paul Rogers created DRILL-7434:
----------------------------------

             Summary: TopNBatch constructs Union vector incorrectly
                 Key: DRILL-7434
                 URL: https://issues.apache.org/jira/browse/DRILL-7434
             Project: Apache Drill
          Issue Type: Bug
            Reporter: Paul Rogers


The Union type is an "experimental" type that has never been completed. Yet, we use it as if it works.

Consider the test {{TestTopNSchemaChanges.testMissingColumn()}}. Run this with the new batch validator enabled. This test creates a union vector. Here is how the schema looks:

{noformat}
(UNION:OPTIONAL), subtypes=([FLOAT8, INT]),
  children=([`internal` (MAP:REQUIRED), children=([`types` (UINT1:REQUIRED)])])
{noformat}

This is very hard to follow because the Union vector structure is complex (and has many issues.) Let's work though it.

We are looking at the {{MaterializedField}} for the union vector. It tells us that this Union has two types: {{FLOAT8}} and {{INT}}. All good.

The Union has a vector per type, stored in an "internal map".' That map shows up as child, it is there on the {{children}} list as {{internal}}. However, the metadata claims that only one vector exists in that map: the {{types}} vector (the one that tells us what type to use for each row.)  The vectors for {{FLOAT8}} and {{INT}} are missing.

If, however, we use our debugger and inspect the actual contents of the {{internal}} map, we get the following:

{noformat}
[`internal` (MAP:REQUIRED), children=([`types` (UINT1:REQUIRED)], [`float8` (FLOAT8:OPTIONAL)], [`int` (INT:OPTIONAL)])]
{noformat}

That is, the internal map has the correct schema, but the Union vector itself has the wrong (incomplete) schema.

This is an inherent design flaw with Union vector: it requires two copies of the schema to be in sync. Further {{MaterializedField}} was designed to be immutable, but the map and Union types require mutation. If the Union simply points to the actual Map vector {{MaterializedField}}, it will drift out of date since the map vector creates a new schema each time we add fields; the Union vector ends up pointing to the old one.

This is not a simple bug to fix, but the result of the bug is that the vectors end up corrupted, as detected by the Batch Validator. In fact, the bug itself is subtle.

The TopNBatch does pass vector validation. However, because of the incorrect metadata, the downstream {{RemovingRecordBatch}} creates the derived Union vector incorrectly: it fails to set the value count for the {{INT}} type.

{noformat}
Found one or more vector errors from RemovingRecordBatch
kl-type-INT - NullableIntVector: Row count = 3, but value count = 0
{noformat}

Where {{kl-type-INT}} is an ad-hoc way of saying we are checking the {{INT}} type vector for a Union named {{kl}}.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)