You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@calcite.apache.org by "Ruben Q L (Jira)" <ji...@apache.org> on 2021/02/19 16:13:00 UTC
[jira] [Comment Edited] (CALCITE-2000) UNNEST a collection that has
a field with nested data generates an Exception
[ https://issues.apache.org/jira/browse/CALCITE-2000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17287151#comment-17287151 ]
Ruben Q L edited comment on CALCITE-2000 at 2/19/21, 4:12 PM:
--------------------------------------------------------------
I have been able to provide a fix for this issue, I admit it is a bit hacky, but I do not see any other way to solve this situation.
Basically the idea is using in {{SqlFunctions$ProductComparableListEnumerator}} an {{Object[]}} instead of {{E[]}}. Even if we need to return a {{FlatLists.ComparableList}} we can exploit the fact that indeed (see ComparableList javadoc):
{noformat}
You can create an instance whose type does not extend {@link Comparable}, but you will get a
{@link ClassCastException} at runtime when you call
{@link #compareTo(Object)} if the elements of the list do not implement
{@code Comparable}.
{noformat}
This is precisely the case of the scenario presented here (or in CALCITE-4064), where we will have an item (array or list) that is not a Comparable. But in these cases, it is not a big problem, since we just need to create the UNNEST result, we do not require to compare it. So with the change proposed in the PR, we are avoiding the {{ArrayStoreException}} in the best case (e.g. in new unit test {{simpleUnnestArrayOfRows4}}), or delaying the problem until {{compareTo}} execution, in the worst case (if in fact a comparison-scenario arrives).
BTW, a similar Comparable-to-Object change was required in {{SqlFunction#flatList}} introduced in 1.24 via CALCITE-4063, since we can also reach the same situation were the item is not a Comparable, but e.g. an array (see new unit test {{simpleUnnestArrayOfRows5}}).
was (Author: rubenql):
I have been able to provide a fix for this issue, I admit it is a bit hacky, but I do not see any other way to solve this situation.
Basically the idea is using in {{SqlFunctions$ProductComparableListEnumerator}} an {{Object[]}} instead of {{E[]}}. Even if we need to return a {{FlatLists.ComparableList}} we can exploit the fact that indeed that (from ComparableList javadoc)
{noformat}
You can create an instance whose type does not extend {@link Comparable}, but you will get a {@link ClassCastException} at runtime when you call {@link #compareTo(Object)} if the elements of the list do not implement {@code Comparable}."
{noformat}
This is precisely the case of the scenario presented here (or in CALCITE-4064), where we will have an item (array or list) that is not a Comparable. But in these cases it is not a big problem, since we just need to create the UNNEST result, we do not require to compare it. So with the change proposed in the PR, we are avoiding the {{ArrayStoreException}} in the best case (e.g. in new unit test {{simpleUnnestArrayOfRows4}}), or delaying the problem until {{compareTo}} execution, in the worst case (if in fact a comparison-scenario arrives).
BTW, a similar Comparable-to-Object change was required in {{SqlFunction#flatList}} introduced in 1.24 via CALCITE-4063, since we can also reach the same situation were the item is not a Comparable, but e.g. an array (see new unit test {{simpleUnnestArrayOfRows5}}).
> UNNEST a collection that has a field with nested data generates an Exception
> ----------------------------------------------------------------------------
>
> Key: CALCITE-2000
> URL: https://issues.apache.org/jira/browse/CALCITE-2000
> Project: Calcite
> Issue Type: Bug
> Reporter: Luis Fernando Kauer
> Assignee: Ruben Q L
> Priority: Major
> Labels: pull-request-available
> Fix For: 1.27.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> When unnesting a column that has another level of nesting it generates the following exception:
> {noformat}
> Caused by: java.lang.ArrayStoreException
> at java.lang.System.arraycopy(Native Method)
> at org.apache.calcite.runtime.SqlFunctions$ProductComparableListEnumerator.current(SqlFunctions.java:2312)
> at org.apache.calcite.runtime.SqlFunctions$ProductComparableListEnumerator.current(SqlFunctions.java:1)
> at org.apache.calcite.linq4j.EnumerableDefaults$17$1.current(EnumerableDefaults.java:1968)
> at org.apache.calcite.linq4j.EnumerableDefaults$11$1.moveNext(EnumerableDefaults.java:1225)
> at Baz$4$1.moveNext(Unknown Source)
> at org.apache.calcite.linq4j.Linq4j$EnumeratorIterator.<init>(Linq4j.java:680)
> at org.apache.calcite.linq4j.Linq4j.enumeratorIterator(Linq4j.java:98)
> at org.apache.calcite.linq4j.AbstractEnumerable.iterator(AbstractEnumerable.java:33)
> at org.apache.calcite.avatica.MetaImpl.createCursor(MetaImpl.java:90)
> at org.apache.calcite.avatica.AvaticaResultSet.execute(AvaticaResultSet.java:206)
> at org.apache.calcite.jdbc.CalciteResultSet.execute(CalciteResultSet.java:67)
> at org.apache.calcite.jdbc.CalciteResultSet.execute(CalciteResultSet.java:1)
> at org.apache.calcite.avatica.AvaticaConnection$1.execute(AvaticaConnection.java:630)
> at org.apache.calcite.jdbc.CalciteMetaImpl.prepareAndExecute(CalciteMetaImpl.java:607)
> at org.apache.calcite.avatica.AvaticaConnection.prepareAndExecuteInternal(AvaticaConnection.java:638)
> at org.apache.calcite.avatica.AvaticaStatement.executeInternal(AvaticaStatement.java:149)
> ... 29 more
> {noformat}
> The problem is that Calcite uses FlatLists internally that expect items to implement Comparable.
> However, a column with a nested collection will have an array or a List as the column value, which do not implement Comparable, so it generates the Exception.
> To test this in JdbcTest, I created a field in Employee with a collection.
> For simplicity, I created a field called dependents in Employee class:
> {code:java}
> public static class HrSchema {
> @Override public String toString() {
> return "HrSchema";
> }
> public final Dependent[] dependents = {
> new Dependent(10, "Michael"),
> new Dependent(10, "Jane"),
> };
> public final Employee[] emps = {
> new Employee(100, 10, "Bill", 10000, 1000, Arrays.asList(dependents)),
> new Employee(200, 20, "Eric", 8000, 500, Collections.<Dependent>emptyList()),
> new Employee(150, 10, "Sebastian", 7000, null, Collections.<Dependent>emptyList()),
> new Employee(110, 10, "Theodore", 11500, 250, Collections.<Dependent>emptyList()),
> };
> public final Department[] depts = {
> new Department(10, "Sales", Arrays.asList(emps[0], emps[2]),
> new Location(-122, 38)),
> new Department(30, "Marketing", Collections.<Employee>emptyList(),
> new Location(0, 52)),
> new Department(40, "HR", Collections.singletonList(emps[1]), null),
> };
> public final Dependent[] locations = {
> new Dependent(10, "San Francisco"),
> new Dependent(20, "San Diego"),
> };
> public QueryableTable foo(int count) {
> return Smalls.generateStrings(count);
> }
> public TranslatableTable view(String s) {
> return Smalls.view(s);
> }
> }
> public static class Employee {
> public final int empid;
> public final int deptno;
> public final String name;
> public final float salary;
> public final Integer commission;
> public final List<Dependent> dependents;
> public Employee(int empid, int deptno, String name, float salary,
> Integer commission) {
> this(empid, deptno, name, salary, commission, Collections.<Dependent>emptyList());
> }
> public Employee(int empid, int deptno, String name, float salary,
> Integer commission, List<Dependent> dependents) {
> this.empid = empid;
> this.deptno = deptno;
> this.name = name;
> this.salary = salary;
> this.commission = commission;
> this.dependents = dependents;
> }
> @Override public String toString() {
> return "Employee [empid: " + empid + ", deptno: " + deptno
> + ", name: " + name + "]";
> }
> @Override public boolean equals(Object obj) {
> return obj == this
> || obj instanceof Employee
> && empid == ((Employee) obj).empid;
> }
> }
> {code}
> Just running the test case JdbcTest.testUnnestArrayColumn generates the exception.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)