[jira] [Commented] (DRILL-4175) calcite parse sql error

["Jinfeng Ni (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/DRILL-4175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15378669#comment-15378669 ] 

Jinfeng Ni commented on DRILL-4175:
-----------------------------------

I also run into this IOBE in RexProgramBuilder intermittently when queries are submitted concurrently. Looking at the log, I realized the issue is in SelfPopulatingList, which is not thread-safe in Drill's forked Calcite (still on Calcite 1.4.0).  After comparing Calcite master branch, I found the fix of CALCITE-1009, and this DRILL JIRA. 

Here is the log that I got in RelProgramBuilder.  Apparently, $t30 - $t35 appeared again after $t59, which would lead incorrectly treating two expressions as identical. 

{code}
DEBUG o.a.calcite.rex.RexProgramBuilder - RexProgramBuilder -  registered localRefList :[$t0, $t1, $t2, $t3, $t4, $t5, $t6, $t7, $t8, $t9, $t10, $t11, $t12, $t13, $t14, $t15, $t16, $t17, $t18, $t19, $t20, $t21, $t22, $t23, $t24, $t25, $t26, $t27, $t28, $t29, $t30, $t31, $t32, $t33, $t34, $t35, $t36, $t37, $t38, $t39, $t40, $t41, $t42, $t43, $t44, $t45, $t46, $t47, $t48, $t49, $t50, $t51, $t52, $t53, $t54, $t55, $t56, $t57, $t58, $t59, $t30, $t31, $t32, $t33, $t34, $t35]
{code}

Although we have made efforts to rebase Drill on top of Calcite master (DRILL-3993), it's still not completely ready. Given that,  quick solution is to port CALCITE-1009 back to Drill's fork Calcite. 



> calcite parse sql error
> -----------------------
>
>                 Key: DRILL-4175
>                 URL: https://issues.apache.org/jira/browse/DRILL-4175
>             Project: Apache Drill
>          Issue Type: Bug
>         Environment: distribution
>            Reporter: huntersjm
>
> I queryed a sql just like `selelct v from table limit 1`,I get a error:
> org.apache.drill.common.exceptions.UserException: SYSTEM ERROR: IndexOutOfBoundsException: Index: 68, Size: 67
> After debug, I found there is a bug in calcite parse:
> first we look line 72 in org.apache.calcite.rex.RexProgramBuilder
> {noformat}
>    registerInternal(RexInputRef.of(i, fields), false);
> {noformat}
> there we get RexInputRef from RexInputRef.of, and it has a method named createName(int idex), here NAMES is SelfPopulatingList.class. SelfPopulatingList.class describe  as Thread-safe list, but in fact it is Thread-unsafe. when NAMES.get(index) is called distributed, it gets a error. We hope SelfPopulatingList.class to be {$0 $1 $2 ....$n}, but when it called distributed, it may be {$0,$1...$29,$30...$59,$30,$31...$59...}.
> We see method registerInternal
> {noformat}
> private RexLocalRef registerInternal(RexNode expr, boolean force) {
> expr = simplify(expr);
>     RexLocalRef ref;
>     final Pair<String, String> key;
>     if (expr instanceof RexLocalRef) {
>       key = null;
>       ref = (RexLocalRef) expr;
>     } else {
>       key = RexUtil.makeKey(expr);
>       ref = exprMap.get(key);
>     }
>     if (ref == null) {
>       if (validating) {
>         validate(
>             expr,
>             exprList.size());
>       }
> {noformat}
> Here makeKey(expr) hope to get different key, however it get same key, so addExpr(expr) called less, in this method
> {noformat}
> RexLocalRef ref;
>     final int index = exprList.size();
>     exprList.add(expr);
>     ref =
>         new RexLocalRef(
>             index,
>             expr.getType());
>     localRefList.add(ref);
>     return ref;
> {noformat}
> localRefList get error size, so in line 939,
> {noformat}
> final RexLocalRef ref = localRefList.get(index);
> {noformat}
> throw IndexOutOfBoundsException
> bugfix:
> We can't change origin code of calcite before they fix this bug, so we can init NAMEs in RexLocalRef on start. Just add 
> {noformat}
> RexInputRef.createName(2048);
> {noformat}
> on Bootstrap.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)