[jira] [Commented] (CALCITE-2000) UNNEST a collection that has a field with nested data generates an Exception

["Ruben Q L (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/CALCITE-2000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17287151#comment-17287151 ] 

Ruben Q L commented on CALCITE-2000:
------------------------------------

I have been able to provide a fix for this issue, I admit it is a bit hacky, but I do not see any other way to solve this situation.

Basically the idea is using in {{SqlFunctions$ProductComparableListEnumerator}} an {{Object[]}} instead of {{E[]}}. Even if we need to return a {{FlatLists.ComparableList}} we can exploit the fact that indeed that (from ComparableList javadoc)
{noformat}
You can create an instance whose type does not extend {@link Comparable}, but you will get a {@link ClassCastException} at runtime when you call {@link #compareTo(Object)} if the elements of the list do not implement {@code Comparable}."
{noformat}

This is precisely the case of the scenario presented here (or in CALCITE-4064), where we will have an item (array or list) that is not a Comparable. But in these cases it is not a big problem, since we just need to create the UNNEST result, we do not require to compare it. So with the change proposed in the PR, we are avoiding the {{ArrayStoreException}} in the best case (e.g. in new unit test {{simpleUnnestArrayOfRows4}}), or delaying the problem until {{compareTo}} execution, in the worst case (if in fact a comparison-scenario arrives).

BTW, a similar Comparable-to-Object change was required in {{SqlFunction#flatList}} introduced in 1.24 via CALCITE-4063, since we can also reach the same situation were the item is not a Comparable, but e.g. an array (see new unit test {{simpleUnnestArrayOfRows5}}).

> UNNEST a collection that has a field with nested data generates an Exception
> ----------------------------------------------------------------------------
>
>                 Key: CALCITE-2000
>                 URL: https://issues.apache.org/jira/browse/CALCITE-2000
>             Project: Calcite
>          Issue Type: Bug
>            Reporter: Luis Fernando Kauer
>            Assignee: Ruben Q L
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: 1.27.0
>
>          Time Spent: 10m
>  Remaining Estimate: 0h
>
> When unnesting a column that has another level of nesting it generates the following exception:
> {noformat}
> Caused by: java.lang.ArrayStoreException
> 	at java.lang.System.arraycopy(Native Method)
> 	at org.apache.calcite.runtime.SqlFunctions$ProductComparableListEnumerator.current(SqlFunctions.java:2312)
> 	at org.apache.calcite.runtime.SqlFunctions$ProductComparableListEnumerator.current(SqlFunctions.java:1)
> 	at org.apache.calcite.linq4j.EnumerableDefaults$17$1.current(EnumerableDefaults.java:1968)
> 	at org.apache.calcite.linq4j.EnumerableDefaults$11$1.moveNext(EnumerableDefaults.java:1225)
> 	at Baz$4$1.moveNext(Unknown Source)
> 	at org.apache.calcite.linq4j.Linq4j$EnumeratorIterator.<init>(Linq4j.java:680)
> 	at org.apache.calcite.linq4j.Linq4j.enumeratorIterator(Linq4j.java:98)
> 	at org.apache.calcite.linq4j.AbstractEnumerable.iterator(AbstractEnumerable.java:33)
> 	at org.apache.calcite.avatica.MetaImpl.createCursor(MetaImpl.java:90)
> 	at org.apache.calcite.avatica.AvaticaResultSet.execute(AvaticaResultSet.java:206)
> 	at org.apache.calcite.jdbc.CalciteResultSet.execute(CalciteResultSet.java:67)
> 	at org.apache.calcite.jdbc.CalciteResultSet.execute(CalciteResultSet.java:1)
> 	at org.apache.calcite.avatica.AvaticaConnection$1.execute(AvaticaConnection.java:630)
> 	at org.apache.calcite.jdbc.CalciteMetaImpl.prepareAndExecute(CalciteMetaImpl.java:607)
> 	at org.apache.calcite.avatica.AvaticaConnection.prepareAndExecuteInternal(AvaticaConnection.java:638)
> 	at org.apache.calcite.avatica.AvaticaStatement.executeInternal(AvaticaStatement.java:149)
> 	... 29 more
> {noformat}
> The problem is that Calcite uses FlatLists internally that expect items to implement Comparable.  
> However, a column with a nested collection will have an array or a List as the column value, which do not implement Comparable, so it generates the Exception.
> To test this in JdbcTest, I created a field in Employee with a collection.
> For simplicity, I created a field called dependents in Employee class:
> {code:java}
> public static class HrSchema {
>     @Override public String toString() {
>       return "HrSchema";
>     }
>     public final Dependent[] dependents = {
>         new Dependent(10, "Michael"),
>         new Dependent(10, "Jane"),
>     };
>     public final Employee[] emps = {
>       new Employee(100, 10, "Bill", 10000, 1000, Arrays.asList(dependents)),
>       new Employee(200, 20, "Eric", 8000, 500,  Collections.<Dependent>emptyList()),
>       new Employee(150, 10, "Sebastian", 7000, null,  Collections.<Dependent>emptyList()),
>       new Employee(110, 10, "Theodore", 11500, 250,  Collections.<Dependent>emptyList()),
>     };
>     public final Department[] depts = {
>       new Department(10, "Sales", Arrays.asList(emps[0], emps[2]),
>           new Location(-122, 38)),
>       new Department(30, "Marketing", Collections.<Employee>emptyList(),
>           new Location(0, 52)),
>       new Department(40, "HR", Collections.singletonList(emps[1]), null),
>     };
>     public final Dependent[] locations = {
>       new Dependent(10, "San Francisco"),
>       new Dependent(20, "San Diego"),
>     };
>     public QueryableTable foo(int count) {
>       return Smalls.generateStrings(count);
>     }
>     public TranslatableTable view(String s) {
>       return Smalls.view(s);
>     }
>   }
>   public static class Employee {
>     public final int empid;
>     public final int deptno;
>     public final String name;
>     public final float salary;
>     public final Integer commission;
>     public final List<Dependent> dependents;
>     public Employee(int empid, int deptno, String name, float salary,
>         Integer commission) {
>       this(empid, deptno, name, salary, commission, Collections.<Dependent>emptyList());
>     }
>     public Employee(int empid, int deptno, String name, float salary,
>         Integer commission, List<Dependent> dependents) {
>       this.empid = empid;
>       this.deptno = deptno;
>       this.name = name;
>       this.salary = salary;
>       this.commission = commission;
>       this.dependents = dependents;
>     }
>     @Override public String toString() {
>       return "Employee [empid: " + empid + ", deptno: " + deptno
>           + ", name: " + name + "]";
>     }
>     @Override public boolean equals(Object obj) {
>       return obj == this
>           || obj instanceof Employee
>           && empid == ((Employee) obj).empid;
>     }
>   }
> {code}
> Just running the test case JdbcTest.testUnnestArrayColumn generates the exception.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)