[jira] [Created] (DRILL-6048) ListVector is incomplete and broken, RepeatedListVector works

["Paul Rogers (JIRA)" <ji...@apache.org>]

Paul Rogers created DRILL-6048:
----------------------------------

             Summary: ListVector is incomplete and broken, RepeatedListVector works
                 Key: DRILL-6048
                 URL: https://issues.apache.org/jira/browse/DRILL-6048
             Project: Apache Drill
          Issue Type: Bug
    Affects Versions: 1.10.0
            Reporter: Paul Rogers


Drill provides two kinds of "list vectors": {{ListVector}} and {{RepeatedListVector}}. I attempted to use the {{ListVector}} to implement lists in JSON. While some parts work, others are broken and JIRA tickets were filed.

Once things worked well enough to run a query, it turned out that the Project operator failed. Digging into the cause, it appears that the {{ListVector}} is incomplete and not used. Its implementation of {{makeTransferPair()}} was clearly never tested. A list has contents, but when this method attempts to create the contents of the target vector, it fails to create the list contents.

Elsewhere, we saw that the constructor did correctly create the vector, and that the {{promoteToUnion()}} had holes. The sheer number of bugs leads to the conclusion that this class is not, in fact, used or usable.

Looking more carefully at the JSON and older writer code, it appears that the ListVector was *not* used for JSON, and that JSON has the limitations of a repeated vector (it cannot support lists with null elements.)

This implies that the JSON reader itself is broken as it does not support fully JSON semantics because it does not use the {{ListVector}} that was intended for this purpose.

So, the conclusion is that JSON uses:

* Repeated vectors for single-dimensional arrays (without null support)
* {{RepeatedListVector}} for two-dimensional arrays

This triggers the question: what do we do for three-dimensional arrays?



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)