You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@drill.apache.org by Muhammad Gelbana <m....@gmail.com> on 2017/06/19 21:19:18 UTC
FindHardDistributionScans throws a NPE while visiting a TableScan
Everyone,
I made a copy of the Jdbc plugin and made modifications to it by adding a
few rules. None of the modification I made or the rules I wrote should have
anything extra to do with handling the following SQL query
SELECT * FROM incorta.SALES.SALES SALES WHERE 1 = 2 LIMIT 1
I know the query is useless, but I need to to know how to fix the following
error thrown while handling this query. This is the final query plan:
DEBUG o.a.d.e.p.s.h.DefaultSqlHandler - HEP_BOTTOM_UP:Convert SUM to $SUM0
> (0ms):
> DrillLimitRel(*fetch=[1]*): rowcount = 1.0, cumulative cost = {201.0
> rows, 205.0 cpu, 0.0 io, 0.0 network, 0.0 memory}, id = 2653
> DrillLimitRel(*offset=[0], fetch=[0]*): rowcount = 1.0, cumulative cost
> = {200.0 rows, 201.0 cpu, 0.0 io, 0.0 network, 0.0 memory}, id = 2651
> GelbanaJdbcDrel: rowcount = 100.0, cumulative cost = {200.0 rows,
> 201.0 cpu, 0.0 io, 0.0 network, 0.0 memory}, id = 2649
> JdbcTableScan(table=[[gelbana, SALES, SALES]]): rowcount = 100.0,
> cumulative cost = {100.0 rows, 101.0 cpu, 0.0 io, 0.0 network, 0.0 memory},
> id = 2572
This is the throw error stacktrace
[Error Id: 83ea094a-db24-4d6d-bf0d-271db26db933 on 640fb7ebbd1a:31010]
at
org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:543)
~[drill-common-1.9.0.jar:1.9.0]
at
org.apache.drill.exec.work.foreman.Foreman$ForemanResult.close(Foreman.java:825)
[drill-java-exec-1.9.0.jar:1.9.0]
at org.apache.drill.exec.work.foreman.Foreman.moveToState(Foreman.java:935)
[drill-java-exec-1.9.0.jar:1.9.0]
at org.apache.drill.exec.work.foreman.Foreman.run(Foreman.java:281)
[drill-java-exec-1.9.0.jar:1.9.0]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[na:1.8.0_131]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[na:1.8.0_131]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_131]
Caused by: org.apache.drill.exec.work.foreman.ForemanException: Unexpected
exception during fragment initialization: null
... 4 common frames omitted
*Caused by: java.lang.NullPointerException: null*
at
org.apache.drill.exec.planner.sql.handlers.FindHardDistributionScans.visit(FindHardDistributionScans.java:55)
~[drill-java-exec-1.9.0.jar:1.9.0]
at org.apache.calcite.rel.core.TableScan.accept(TableScan.java:166)
~[calcite-core-1.4.0-drill-r19.jar:1.4.0-drill-r19]
at org.apache.calcite.rel.RelShuttleImpl.visitChild(RelShuttleImpl.java:53)
~[calcite-core-1.4.0-drill-r19.jar:1.4.0-drill-r19]
at
org.apache.calcite.rel.RelShuttleImpl.visitChildren(RelShuttleImpl.java:68)
~[calcite-core-1.4.0-drill-r19.jar:1.4.0-drill-r19]
at org.apache.calcite.rel.RelShuttleImpl.visit(RelShuttleImpl.java:126)
~[calcite-core-1.4.0-drill-r19.jar:1.4.0-drill-r19]
at org.apache.calcite.rel.AbstractRelNode.accept(AbstractRelNode.java:256)
~[calcite-core-1.4.0-drill-r19.jar:1.4.0-drill-r19]
at org.apache.calcite.rel.RelShuttleImpl.visitChild(RelShuttleImpl.java:53)
~[calcite-core-1.4.0-drill-r19.jar:1.4.0-drill-r19]
at
org.apache.calcite.rel.RelShuttleImpl.visitChildren(RelShuttleImpl.java:68)
~[calcite-core-1.4.0-drill-r19.jar:1.4.0-drill-r19]
at org.apache.calcite.rel.RelShuttleImpl.visit(RelShuttleImpl.java:126)
~[calcite-core-1.4.0-drill-r19.jar:1.4.0-drill-r19]
at org.apache.calcite.rel.AbstractRelNode.accept(AbstractRelNode.java:256)
~[calcite-core-1.4.0-drill-r19.jar:1.4.0-drill-r19]
at org.apache.calcite.rel.RelShuttleImpl.visitChild(RelShuttleImpl.java:53)
~[calcite-core-1.4.0-drill-r19.jar:1.4.0-drill-r19]
at
org.apache.calcite.rel.RelShuttleImpl.visitChildren(RelShuttleImpl.java:68)
~[calcite-core-1.4.0-drill-r19.jar:1.4.0-drill-r19]
at org.apache.calcite.rel.RelShuttleImpl.visit(RelShuttleImpl.java:126)
~[calcite-core-1.4.0-drill-r19.jar:1.4.0-drill-r19]
at org.apache.calcite.rel.AbstractRelNode.accept(AbstractRelNode.java:256)
~[calcite-core-1.4.0-drill-r19.jar:1.4.0-drill-r19]
at
org.apache.drill.exec.planner.sql.handlers.FindHardDistributionScans.canForceSingleMode(FindHardDistributionScans.java:45)
~[drill-java-exec-1.9.0.jar:1.9.0]
at
org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.convertToDrel(DefaultSqlHandler.java:262)
~[drill-java-exec-1.9.0.jar:1.9.0]
at
org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.convertToDrel(DefaultSqlHandler.java:290)
~[drill-java-exec-1.9.0.jar:1.9.0]
at
org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.getPlan(DefaultSqlHandler.java:168)
~[drill-java-exec-1.9.0.jar:1.9.0]
at
org.apache.drill.exec.planner.sql.DrillSqlWorker.getPhysicalPlan(DrillSqlWorker.java:123)
~[drill-java-exec-1.9.0.jar:1.9.0]
at
org.apache.drill.exec.planner.sql.DrillSqlWorker.getPlan(DrillSqlWorker.java:97)
~[drill-java-exec-1.9.0.jar:1.9.0]
at org.apache.drill.exec.work.foreman.Foreman.runSQL(Foreman.java:1008)
[drill-java-exec-1.9.0.jar:1.9.0]
at org.apache.drill.exec.work.foreman.Foreman.run(Foreman.java:264)
[drill-java-exec-1.9.0.jar:1.9.0]
... 3 common frames omitted
Whic is because this statement
unwrap =
> scan.getTable().unwrap(DrillTranslatableTable.class).getDrillTable();
In *FindHardDistributionScans.java:55* evaluates
*scan.getTable().unwrap(DrillTranslatableTable.class)* to null
Would someone please explain to me what is Drill trying to do and what did
I do wrong ?
*---------------------*
*Muhammad Gelbana*
http://www.linkedin.com/in/mgelbana
Re: FindHardDistributionScans throws a NPE while visiting a TableScan
Posted by Muhammad Gelbana <m....@gmail.com>.
With pleasure. I can't successfully run all test cases
<https://issues.apache.org/jira/browse/DRILL-5606> ATM. When I overcome
that, I'll push the fix.
*---------------------*
*Muhammad Gelbana*
http://www.linkedin.com/in/mgelbana
On Wed, Jun 21, 2017 at 1:47 PM, Khurram Faraaz <kf...@mapr.com> wrote:
> Muhammad, please create a pull request and someone will review you code,
> ensure that existing unit tests don't fail due to your changes.
>
>
> Thanks,
>
> Khurram
>
> ________________________________
> From: Muhammad Gelbana <m....@gmail.com>
> Sent: Wednesday, June 21, 2017 4:11:41 PM
> To: dev@drill.apache.org
> Subject: Re: FindHardDistributionScans throws a NPE while visiting a
> TableScan
>
> This has been bugging me for sometime, and I've only solved it after
> starting this thread !
>
> I solved this by overriding the
> *org.apache.calcite.rel.AbstractRelNode.accept(RelShuttle)* method for the
> relational node(s) containing *JdbcTableScan* to avoid this.
>
> @Override
> > public RelNode accept(RelShuttle shuttle) {
> >
> > if(shuttle.getClass().getName().equals("org.apache.drill.
> exec.planner.sql.handlers.FindHardDistributionScans")){
> > return this;
> > }
> > return super.accept(shuttle);
> > }
>
>
> If someone finds this introducing another bug, please tell me about it.
>
>
> *---------------------*
> *Muhammad Gelbana*
> http://www.linkedin.com/in/mgelbana
>
> On Tue, Jun 20, 2017 at 2:13 AM, Jinfeng Ni <jn...@apache.org> wrote:
>
> > unwrap() essentially is doing a cast. If it returns null for
> > unwrap(DrillTranslatableTable.class) or unwrap(DrillTable.class), it
> means
> > the table associate with this TableScan does not implement either
> > interface. My suspicion is JDBC storage plugin returns JdbcTable [1],
> > unlikely other storage plugin which returns an instance implementing
> > DrillTable.
> >
> > This seems to indicate FindHardDistributionScans could not be used to
> > non-DrillTable. I'm not sure if that's the intention of that code,
> though.
> >
> > 1.
> > https://github.com/apache/calcite/blob/master/core/src/
> > main/java/org/apache/calcite/adapter/jdbc/JdbcSchema.java#L233-L234
> >
> > On Mon, Jun 19, 2017 at 2:19 PM, Muhammad Gelbana <m....@gmail.com>
> > wrote:
> >
> > > Everyone,
> > >
> > > I made a copy of the Jdbc plugin and made modifications to it by
> adding a
> > > few rules. None of the modification I made or the rules I wrote should
> > have
> > > anything extra to do with handling the following SQL query
> > >
> > > SELECT * FROM incorta.SALES.SALES SALES WHERE 1 = 2 LIMIT 1
> > >
> > >
> > > I know the query is useless, but I need to to know how to fix the
> > following
> > > error thrown while handling this query. This is the final query plan:
> > >
> > > DEBUG o.a.d.e.p.s.h.DefaultSqlHandler - HEP_BOTTOM_UP:Convert SUM to
> > $SUM0
> > > > (0ms):
> > > > DrillLimitRel(*fetch=[1]*): rowcount = 1.0, cumulative cost = {201.0
> > > > rows, 205.0 cpu, 0.0 io, 0.0 network, 0.0 memory}, id = 2653
> > > > DrillLimitRel(*offset=[0], fetch=[0]*): rowcount = 1.0, cumulative
> > cost
> > > > = {200.0 rows, 201.0 cpu, 0.0 io, 0.0 network, 0.0 memory}, id = 2651
> > > > GelbanaJdbcDrel: rowcount = 100.0, cumulative cost = {200.0 rows,
> > > > 201.0 cpu, 0.0 io, 0.0 network, 0.0 memory}, id = 2649
> > > > JdbcTableScan(table=[[gelbana, SALES, SALES]]): rowcount =
> 100.0,
> > > > cumulative cost = {100.0 rows, 101.0 cpu, 0.0 io, 0.0 network, 0.0
> > > memory},
> > > > id = 2572
> > >
> > >
> > > This is the throw error stacktrace
> > >
> > > [Error Id: 83ea094a-db24-4d6d-bf0d-271db26db933 on 640fb7ebbd1a:31010]
> > > at
> > > org.apache.drill.common.exceptions.UserException$
> > > Builder.build(UserException.java:543)
> > > ~[drill-common-1.9.0.jar:1.9.0]
> > > at
> > > org.apache.drill.exec.work.foreman.Foreman$ForemanResult.
> > > close(Foreman.java:825)
> > > [drill-java-exec-1.9.0.jar:1.9.0]
> > > at org.apache.drill.exec.work.foreman.Foreman.moveToState(
> > > Foreman.java:935)
> > > [drill-java-exec-1.9.0.jar:1.9.0]
> > > at org.apache.drill.exec.work.foreman.Foreman.run(Foreman.java:281)
> > > [drill-java-exec-1.9.0.jar:1.9.0]
> > > at
> > > java.util.concurrent.ThreadPoolExecutor.runWorker(
> > > ThreadPoolExecutor.java:1142)
> > > [na:1.8.0_131]
> > > at
> > > java.util.concurrent.ThreadPoolExecutor$Worker.run(
> > > ThreadPoolExecutor.java:617)
> > > [na:1.8.0_131]
> > > at java.lang.Thread.run(Thread.java:748) [na:1.8.0_131]
> > > Caused by: org.apache.drill.exec.work.foreman.ForemanException:
> > Unexpected
> > > exception during fragment initialization: null
> > > ... 4 common frames omitted
> > > *Caused by: java.lang.NullPointerException: null*
> > > at
> > > org.apache.drill.exec.planner.sql.handlers.FindHardDistributionScans.
> > > visit(FindHardDistributionScans.java:55)
> > > ~[drill-java-exec-1.9.0.jar:1.9.0]
> > > at org.apache.calcite.rel.core.TableScan.accept(TableScan.java:166)
> > > ~[calcite-core-1.4.0-drill-r19.jar:1.4.0-drill-r19]
> > > at org.apache.calcite.rel.RelShuttleImpl.visitChild(
> > > RelShuttleImpl.java:53)
> > > ~[calcite-core-1.4.0-drill-r19.jar:1.4.0-drill-r19]
> > > at
> > > org.apache.calcite.rel.RelShuttleImpl.visitChildren(
> > > RelShuttleImpl.java:68)
> > > ~[calcite-core-1.4.0-drill-r19.jar:1.4.0-drill-r19]
> > > at org.apache.calcite.rel.RelShuttleImpl.visit(
> RelShuttleImpl.java:126)
> > > ~[calcite-core-1.4.0-drill-r19.jar:1.4.0-drill-r19]
> > > at org.apache.calcite.rel.AbstractRelNode.accept(
> > AbstractRelNode.java:256)
> > > ~[calcite-core-1.4.0-drill-r19.jar:1.4.0-drill-r19]
> > > at org.apache.calcite.rel.RelShuttleImpl.visitChild(
> > > RelShuttleImpl.java:53)
> > > ~[calcite-core-1.4.0-drill-r19.jar:1.4.0-drill-r19]
> > > at
> > > org.apache.calcite.rel.RelShuttleImpl.visitChildren(
> > > RelShuttleImpl.java:68)
> > > ~[calcite-core-1.4.0-drill-r19.jar:1.4.0-drill-r19]
> > > at org.apache.calcite.rel.RelShuttleImpl.visit(
> RelShuttleImpl.java:126)
> > > ~[calcite-core-1.4.0-drill-r19.jar:1.4.0-drill-r19]
> > > at org.apache.calcite.rel.AbstractRelNode.accept(
> > AbstractRelNode.java:256)
> > > ~[calcite-core-1.4.0-drill-r19.jar:1.4.0-drill-r19]
> > > at org.apache.calcite.rel.RelShuttleImpl.visitChild(
> > > RelShuttleImpl.java:53)
> > > ~[calcite-core-1.4.0-drill-r19.jar:1.4.0-drill-r19]
> > > at
> > > org.apache.calcite.rel.RelShuttleImpl.visitChildren(
> > > RelShuttleImpl.java:68)
> > > ~[calcite-core-1.4.0-drill-r19.jar:1.4.0-drill-r19]
> > > at org.apache.calcite.rel.RelShuttleImpl.visit(
> RelShuttleImpl.java:126)
> > > ~[calcite-core-1.4.0-drill-r19.jar:1.4.0-drill-r19]
> > > at org.apache.calcite.rel.AbstractRelNode.accept(
> > AbstractRelNode.java:256)
> > > ~[calcite-core-1.4.0-drill-r19.jar:1.4.0-drill-r19]
> > > at
> > > org.apache.drill.exec.planner.sql.handlers.FindHardDistributionScans.
> > > canForceSingleMode(FindHardDistributionScans.java:45)
> > > ~[drill-java-exec-1.9.0.jar:1.9.0]
> > > at
> > > org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.
> > > convertToDrel(DefaultSqlHandler.java:262)
> > > ~[drill-java-exec-1.9.0.jar:1.9.0]
> > > at
> > > org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.
> > > convertToDrel(DefaultSqlHandler.java:290)
> > > ~[drill-java-exec-1.9.0.jar:1.9.0]
> > > at
> > > org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.getPlan(
> > > DefaultSqlHandler.java:168)
> > > ~[drill-java-exec-1.9.0.jar:1.9.0]
> > > at
> > > org.apache.drill.exec.planner.sql.DrillSqlWorker.getPhysicalPlan(
> > > DrillSqlWorker.java:123)
> > > ~[drill-java-exec-1.9.0.jar:1.9.0]
> > > at
> > > org.apache.drill.exec.planner.sql.DrillSqlWorker.getPlan(
> > > DrillSqlWorker.java:97)
> > > ~[drill-java-exec-1.9.0.jar:1.9.0]
> > > at org.apache.drill.exec.work.foreman.Foreman.runSQL(
> Foreman.java:1008)
> > > [drill-java-exec-1.9.0.jar:1.9.0]
> > > at org.apache.drill.exec.work.foreman.Foreman.run(Foreman.java:264)
> > > [drill-java-exec-1.9.0.jar:1.9.0]
> > > ... 3 common frames omitted
> > >
> > > Whic is because this statement
> > >
> > > unwrap =
> > > > scan.getTable().unwrap(DrillTranslatableTable.class).
> getDrillTable();
> > >
> > >
> > > In *FindHardDistributionScans.java:55* evaluates
> > > *scan.getTable().unwrap(DrillTranslatableTable.class)* to null
> > >
> > > Would someone please explain to me what is Drill trying to do and what
> > did
> > > I do wrong ?
> > >
> > > *---------------------*
> > > *Muhammad Gelbana*
> > > http://www.linkedin.com/in/mgelbana
> > >
> >
>
Re: FindHardDistributionScans throws a NPE while visiting a TableScan
Posted by Khurram Faraaz <kf...@mapr.com>.
Muhammad, please create a pull request and someone will review you code, ensure that existing unit tests don't fail due to your changes.
Thanks,
Khurram
________________________________
From: Muhammad Gelbana <m....@gmail.com>
Sent: Wednesday, June 21, 2017 4:11:41 PM
To: dev@drill.apache.org
Subject: Re: FindHardDistributionScans throws a NPE while visiting a TableScan
This has been bugging me for sometime, and I've only solved it after
starting this thread !
I solved this by overriding the
*org.apache.calcite.rel.AbstractRelNode.accept(RelShuttle)* method for the
relational node(s) containing *JdbcTableScan* to avoid this.
@Override
> public RelNode accept(RelShuttle shuttle) {
>
> if(shuttle.getClass().getName().equals("org.apache.drill.exec.planner.sql.handlers.FindHardDistributionScans")){
> return this;
> }
> return super.accept(shuttle);
> }
If someone finds this introducing another bug, please tell me about it.
*---------------------*
*Muhammad Gelbana*
http://www.linkedin.com/in/mgelbana
On Tue, Jun 20, 2017 at 2:13 AM, Jinfeng Ni <jn...@apache.org> wrote:
> unwrap() essentially is doing a cast. If it returns null for
> unwrap(DrillTranslatableTable.class) or unwrap(DrillTable.class), it means
> the table associate with this TableScan does not implement either
> interface. My suspicion is JDBC storage plugin returns JdbcTable [1],
> unlikely other storage plugin which returns an instance implementing
> DrillTable.
>
> This seems to indicate FindHardDistributionScans could not be used to
> non-DrillTable. I'm not sure if that's the intention of that code, though.
>
> 1.
> https://github.com/apache/calcite/blob/master/core/src/
> main/java/org/apache/calcite/adapter/jdbc/JdbcSchema.java#L233-L234
>
> On Mon, Jun 19, 2017 at 2:19 PM, Muhammad Gelbana <m....@gmail.com>
> wrote:
>
> > Everyone,
> >
> > I made a copy of the Jdbc plugin and made modifications to it by adding a
> > few rules. None of the modification I made or the rules I wrote should
> have
> > anything extra to do with handling the following SQL query
> >
> > SELECT * FROM incorta.SALES.SALES SALES WHERE 1 = 2 LIMIT 1
> >
> >
> > I know the query is useless, but I need to to know how to fix the
> following
> > error thrown while handling this query. This is the final query plan:
> >
> > DEBUG o.a.d.e.p.s.h.DefaultSqlHandler - HEP_BOTTOM_UP:Convert SUM to
> $SUM0
> > > (0ms):
> > > DrillLimitRel(*fetch=[1]*): rowcount = 1.0, cumulative cost = {201.0
> > > rows, 205.0 cpu, 0.0 io, 0.0 network, 0.0 memory}, id = 2653
> > > DrillLimitRel(*offset=[0], fetch=[0]*): rowcount = 1.0, cumulative
> cost
> > > = {200.0 rows, 201.0 cpu, 0.0 io, 0.0 network, 0.0 memory}, id = 2651
> > > GelbanaJdbcDrel: rowcount = 100.0, cumulative cost = {200.0 rows,
> > > 201.0 cpu, 0.0 io, 0.0 network, 0.0 memory}, id = 2649
> > > JdbcTableScan(table=[[gelbana, SALES, SALES]]): rowcount = 100.0,
> > > cumulative cost = {100.0 rows, 101.0 cpu, 0.0 io, 0.0 network, 0.0
> > memory},
> > > id = 2572
> >
> >
> > This is the throw error stacktrace
> >
> > [Error Id: 83ea094a-db24-4d6d-bf0d-271db26db933 on 640fb7ebbd1a:31010]
> > at
> > org.apache.drill.common.exceptions.UserException$
> > Builder.build(UserException.java:543)
> > ~[drill-common-1.9.0.jar:1.9.0]
> > at
> > org.apache.drill.exec.work.foreman.Foreman$ForemanResult.
> > close(Foreman.java:825)
> > [drill-java-exec-1.9.0.jar:1.9.0]
> > at org.apache.drill.exec.work.foreman.Foreman.moveToState(
> > Foreman.java:935)
> > [drill-java-exec-1.9.0.jar:1.9.0]
> > at org.apache.drill.exec.work.foreman.Foreman.run(Foreman.java:281)
> > [drill-java-exec-1.9.0.jar:1.9.0]
> > at
> > java.util.concurrent.ThreadPoolExecutor.runWorker(
> > ThreadPoolExecutor.java:1142)
> > [na:1.8.0_131]
> > at
> > java.util.concurrent.ThreadPoolExecutor$Worker.run(
> > ThreadPoolExecutor.java:617)
> > [na:1.8.0_131]
> > at java.lang.Thread.run(Thread.java:748) [na:1.8.0_131]
> > Caused by: org.apache.drill.exec.work.foreman.ForemanException:
> Unexpected
> > exception during fragment initialization: null
> > ... 4 common frames omitted
> > *Caused by: java.lang.NullPointerException: null*
> > at
> > org.apache.drill.exec.planner.sql.handlers.FindHardDistributionScans.
> > visit(FindHardDistributionScans.java:55)
> > ~[drill-java-exec-1.9.0.jar:1.9.0]
> > at org.apache.calcite.rel.core.TableScan.accept(TableScan.java:166)
> > ~[calcite-core-1.4.0-drill-r19.jar:1.4.0-drill-r19]
> > at org.apache.calcite.rel.RelShuttleImpl.visitChild(
> > RelShuttleImpl.java:53)
> > ~[calcite-core-1.4.0-drill-r19.jar:1.4.0-drill-r19]
> > at
> > org.apache.calcite.rel.RelShuttleImpl.visitChildren(
> > RelShuttleImpl.java:68)
> > ~[calcite-core-1.4.0-drill-r19.jar:1.4.0-drill-r19]
> > at org.apache.calcite.rel.RelShuttleImpl.visit(RelShuttleImpl.java:126)
> > ~[calcite-core-1.4.0-drill-r19.jar:1.4.0-drill-r19]
> > at org.apache.calcite.rel.AbstractRelNode.accept(
> AbstractRelNode.java:256)
> > ~[calcite-core-1.4.0-drill-r19.jar:1.4.0-drill-r19]
> > at org.apache.calcite.rel.RelShuttleImpl.visitChild(
> > RelShuttleImpl.java:53)
> > ~[calcite-core-1.4.0-drill-r19.jar:1.4.0-drill-r19]
> > at
> > org.apache.calcite.rel.RelShuttleImpl.visitChildren(
> > RelShuttleImpl.java:68)
> > ~[calcite-core-1.4.0-drill-r19.jar:1.4.0-drill-r19]
> > at org.apache.calcite.rel.RelShuttleImpl.visit(RelShuttleImpl.java:126)
> > ~[calcite-core-1.4.0-drill-r19.jar:1.4.0-drill-r19]
> > at org.apache.calcite.rel.AbstractRelNode.accept(
> AbstractRelNode.java:256)
> > ~[calcite-core-1.4.0-drill-r19.jar:1.4.0-drill-r19]
> > at org.apache.calcite.rel.RelShuttleImpl.visitChild(
> > RelShuttleImpl.java:53)
> > ~[calcite-core-1.4.0-drill-r19.jar:1.4.0-drill-r19]
> > at
> > org.apache.calcite.rel.RelShuttleImpl.visitChildren(
> > RelShuttleImpl.java:68)
> > ~[calcite-core-1.4.0-drill-r19.jar:1.4.0-drill-r19]
> > at org.apache.calcite.rel.RelShuttleImpl.visit(RelShuttleImpl.java:126)
> > ~[calcite-core-1.4.0-drill-r19.jar:1.4.0-drill-r19]
> > at org.apache.calcite.rel.AbstractRelNode.accept(
> AbstractRelNode.java:256)
> > ~[calcite-core-1.4.0-drill-r19.jar:1.4.0-drill-r19]
> > at
> > org.apache.drill.exec.planner.sql.handlers.FindHardDistributionScans.
> > canForceSingleMode(FindHardDistributionScans.java:45)
> > ~[drill-java-exec-1.9.0.jar:1.9.0]
> > at
> > org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.
> > convertToDrel(DefaultSqlHandler.java:262)
> > ~[drill-java-exec-1.9.0.jar:1.9.0]
> > at
> > org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.
> > convertToDrel(DefaultSqlHandler.java:290)
> > ~[drill-java-exec-1.9.0.jar:1.9.0]
> > at
> > org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.getPlan(
> > DefaultSqlHandler.java:168)
> > ~[drill-java-exec-1.9.0.jar:1.9.0]
> > at
> > org.apache.drill.exec.planner.sql.DrillSqlWorker.getPhysicalPlan(
> > DrillSqlWorker.java:123)
> > ~[drill-java-exec-1.9.0.jar:1.9.0]
> > at
> > org.apache.drill.exec.planner.sql.DrillSqlWorker.getPlan(
> > DrillSqlWorker.java:97)
> > ~[drill-java-exec-1.9.0.jar:1.9.0]
> > at org.apache.drill.exec.work.foreman.Foreman.runSQL(Foreman.java:1008)
> > [drill-java-exec-1.9.0.jar:1.9.0]
> > at org.apache.drill.exec.work.foreman.Foreman.run(Foreman.java:264)
> > [drill-java-exec-1.9.0.jar:1.9.0]
> > ... 3 common frames omitted
> >
> > Whic is because this statement
> >
> > unwrap =
> > > scan.getTable().unwrap(DrillTranslatableTable.class).getDrillTable();
> >
> >
> > In *FindHardDistributionScans.java:55* evaluates
> > *scan.getTable().unwrap(DrillTranslatableTable.class)* to null
> >
> > Would someone please explain to me what is Drill trying to do and what
> did
> > I do wrong ?
> >
> > *---------------------*
> > *Muhammad Gelbana*
> > http://www.linkedin.com/in/mgelbana
> >
>
Re: FindHardDistributionScans throws a NPE while visiting a TableScan
Posted by Muhammad Gelbana <m....@gmail.com>.
This has been bugging me for sometime, and I've only solved it after
starting this thread !
I solved this by overriding the
*org.apache.calcite.rel.AbstractRelNode.accept(RelShuttle)* method for the
relational node(s) containing *JdbcTableScan* to avoid this.
@Override
> public RelNode accept(RelShuttle shuttle) {
>
> if(shuttle.getClass().getName().equals("org.apache.drill.exec.planner.sql.handlers.FindHardDistributionScans")){
> return this;
> }
> return super.accept(shuttle);
> }
If someone finds this introducing another bug, please tell me about it.
*---------------------*
*Muhammad Gelbana*
http://www.linkedin.com/in/mgelbana
On Tue, Jun 20, 2017 at 2:13 AM, Jinfeng Ni <jn...@apache.org> wrote:
> unwrap() essentially is doing a cast. If it returns null for
> unwrap(DrillTranslatableTable.class) or unwrap(DrillTable.class), it means
> the table associate with this TableScan does not implement either
> interface. My suspicion is JDBC storage plugin returns JdbcTable [1],
> unlikely other storage plugin which returns an instance implementing
> DrillTable.
>
> This seems to indicate FindHardDistributionScans could not be used to
> non-DrillTable. I'm not sure if that's the intention of that code, though.
>
> 1.
> https://github.com/apache/calcite/blob/master/core/src/
> main/java/org/apache/calcite/adapter/jdbc/JdbcSchema.java#L233-L234
>
> On Mon, Jun 19, 2017 at 2:19 PM, Muhammad Gelbana <m....@gmail.com>
> wrote:
>
> > Everyone,
> >
> > I made a copy of the Jdbc plugin and made modifications to it by adding a
> > few rules. None of the modification I made or the rules I wrote should
> have
> > anything extra to do with handling the following SQL query
> >
> > SELECT * FROM incorta.SALES.SALES SALES WHERE 1 = 2 LIMIT 1
> >
> >
> > I know the query is useless, but I need to to know how to fix the
> following
> > error thrown while handling this query. This is the final query plan:
> >
> > DEBUG o.a.d.e.p.s.h.DefaultSqlHandler - HEP_BOTTOM_UP:Convert SUM to
> $SUM0
> > > (0ms):
> > > DrillLimitRel(*fetch=[1]*): rowcount = 1.0, cumulative cost = {201.0
> > > rows, 205.0 cpu, 0.0 io, 0.0 network, 0.0 memory}, id = 2653
> > > DrillLimitRel(*offset=[0], fetch=[0]*): rowcount = 1.0, cumulative
> cost
> > > = {200.0 rows, 201.0 cpu, 0.0 io, 0.0 network, 0.0 memory}, id = 2651
> > > GelbanaJdbcDrel: rowcount = 100.0, cumulative cost = {200.0 rows,
> > > 201.0 cpu, 0.0 io, 0.0 network, 0.0 memory}, id = 2649
> > > JdbcTableScan(table=[[gelbana, SALES, SALES]]): rowcount = 100.0,
> > > cumulative cost = {100.0 rows, 101.0 cpu, 0.0 io, 0.0 network, 0.0
> > memory},
> > > id = 2572
> >
> >
> > This is the throw error stacktrace
> >
> > [Error Id: 83ea094a-db24-4d6d-bf0d-271db26db933 on 640fb7ebbd1a:31010]
> > at
> > org.apache.drill.common.exceptions.UserException$
> > Builder.build(UserException.java:543)
> > ~[drill-common-1.9.0.jar:1.9.0]
> > at
> > org.apache.drill.exec.work.foreman.Foreman$ForemanResult.
> > close(Foreman.java:825)
> > [drill-java-exec-1.9.0.jar:1.9.0]
> > at org.apache.drill.exec.work.foreman.Foreman.moveToState(
> > Foreman.java:935)
> > [drill-java-exec-1.9.0.jar:1.9.0]
> > at org.apache.drill.exec.work.foreman.Foreman.run(Foreman.java:281)
> > [drill-java-exec-1.9.0.jar:1.9.0]
> > at
> > java.util.concurrent.ThreadPoolExecutor.runWorker(
> > ThreadPoolExecutor.java:1142)
> > [na:1.8.0_131]
> > at
> > java.util.concurrent.ThreadPoolExecutor$Worker.run(
> > ThreadPoolExecutor.java:617)
> > [na:1.8.0_131]
> > at java.lang.Thread.run(Thread.java:748) [na:1.8.0_131]
> > Caused by: org.apache.drill.exec.work.foreman.ForemanException:
> Unexpected
> > exception during fragment initialization: null
> > ... 4 common frames omitted
> > *Caused by: java.lang.NullPointerException: null*
> > at
> > org.apache.drill.exec.planner.sql.handlers.FindHardDistributionScans.
> > visit(FindHardDistributionScans.java:55)
> > ~[drill-java-exec-1.9.0.jar:1.9.0]
> > at org.apache.calcite.rel.core.TableScan.accept(TableScan.java:166)
> > ~[calcite-core-1.4.0-drill-r19.jar:1.4.0-drill-r19]
> > at org.apache.calcite.rel.RelShuttleImpl.visitChild(
> > RelShuttleImpl.java:53)
> > ~[calcite-core-1.4.0-drill-r19.jar:1.4.0-drill-r19]
> > at
> > org.apache.calcite.rel.RelShuttleImpl.visitChildren(
> > RelShuttleImpl.java:68)
> > ~[calcite-core-1.4.0-drill-r19.jar:1.4.0-drill-r19]
> > at org.apache.calcite.rel.RelShuttleImpl.visit(RelShuttleImpl.java:126)
> > ~[calcite-core-1.4.0-drill-r19.jar:1.4.0-drill-r19]
> > at org.apache.calcite.rel.AbstractRelNode.accept(
> AbstractRelNode.java:256)
> > ~[calcite-core-1.4.0-drill-r19.jar:1.4.0-drill-r19]
> > at org.apache.calcite.rel.RelShuttleImpl.visitChild(
> > RelShuttleImpl.java:53)
> > ~[calcite-core-1.4.0-drill-r19.jar:1.4.0-drill-r19]
> > at
> > org.apache.calcite.rel.RelShuttleImpl.visitChildren(
> > RelShuttleImpl.java:68)
> > ~[calcite-core-1.4.0-drill-r19.jar:1.4.0-drill-r19]
> > at org.apache.calcite.rel.RelShuttleImpl.visit(RelShuttleImpl.java:126)
> > ~[calcite-core-1.4.0-drill-r19.jar:1.4.0-drill-r19]
> > at org.apache.calcite.rel.AbstractRelNode.accept(
> AbstractRelNode.java:256)
> > ~[calcite-core-1.4.0-drill-r19.jar:1.4.0-drill-r19]
> > at org.apache.calcite.rel.RelShuttleImpl.visitChild(
> > RelShuttleImpl.java:53)
> > ~[calcite-core-1.4.0-drill-r19.jar:1.4.0-drill-r19]
> > at
> > org.apache.calcite.rel.RelShuttleImpl.visitChildren(
> > RelShuttleImpl.java:68)
> > ~[calcite-core-1.4.0-drill-r19.jar:1.4.0-drill-r19]
> > at org.apache.calcite.rel.RelShuttleImpl.visit(RelShuttleImpl.java:126)
> > ~[calcite-core-1.4.0-drill-r19.jar:1.4.0-drill-r19]
> > at org.apache.calcite.rel.AbstractRelNode.accept(
> AbstractRelNode.java:256)
> > ~[calcite-core-1.4.0-drill-r19.jar:1.4.0-drill-r19]
> > at
> > org.apache.drill.exec.planner.sql.handlers.FindHardDistributionScans.
> > canForceSingleMode(FindHardDistributionScans.java:45)
> > ~[drill-java-exec-1.9.0.jar:1.9.0]
> > at
> > org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.
> > convertToDrel(DefaultSqlHandler.java:262)
> > ~[drill-java-exec-1.9.0.jar:1.9.0]
> > at
> > org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.
> > convertToDrel(DefaultSqlHandler.java:290)
> > ~[drill-java-exec-1.9.0.jar:1.9.0]
> > at
> > org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.getPlan(
> > DefaultSqlHandler.java:168)
> > ~[drill-java-exec-1.9.0.jar:1.9.0]
> > at
> > org.apache.drill.exec.planner.sql.DrillSqlWorker.getPhysicalPlan(
> > DrillSqlWorker.java:123)
> > ~[drill-java-exec-1.9.0.jar:1.9.0]
> > at
> > org.apache.drill.exec.planner.sql.DrillSqlWorker.getPlan(
> > DrillSqlWorker.java:97)
> > ~[drill-java-exec-1.9.0.jar:1.9.0]
> > at org.apache.drill.exec.work.foreman.Foreman.runSQL(Foreman.java:1008)
> > [drill-java-exec-1.9.0.jar:1.9.0]
> > at org.apache.drill.exec.work.foreman.Foreman.run(Foreman.java:264)
> > [drill-java-exec-1.9.0.jar:1.9.0]
> > ... 3 common frames omitted
> >
> > Whic is because this statement
> >
> > unwrap =
> > > scan.getTable().unwrap(DrillTranslatableTable.class).getDrillTable();
> >
> >
> > In *FindHardDistributionScans.java:55* evaluates
> > *scan.getTable().unwrap(DrillTranslatableTable.class)* to null
> >
> > Would someone please explain to me what is Drill trying to do and what
> did
> > I do wrong ?
> >
> > *---------------------*
> > *Muhammad Gelbana*
> > http://www.linkedin.com/in/mgelbana
> >
>
Re: FindHardDistributionScans throws a NPE while visiting a TableScan
Posted by Jinfeng Ni <jn...@apache.org>.
unwrap() essentially is doing a cast. If it returns null for
unwrap(DrillTranslatableTable.class) or unwrap(DrillTable.class), it means
the table associate with this TableScan does not implement either
interface. My suspicion is JDBC storage plugin returns JdbcTable [1],
unlikely other storage plugin which returns an instance implementing
DrillTable.
This seems to indicate FindHardDistributionScans could not be used to
non-DrillTable. I'm not sure if that's the intention of that code, though.
1.
https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/adapter/jdbc/JdbcSchema.java#L233-L234
On Mon, Jun 19, 2017 at 2:19 PM, Muhammad Gelbana <m....@gmail.com>
wrote:
> Everyone,
>
> I made a copy of the Jdbc plugin and made modifications to it by adding a
> few rules. None of the modification I made or the rules I wrote should have
> anything extra to do with handling the following SQL query
>
> SELECT * FROM incorta.SALES.SALES SALES WHERE 1 = 2 LIMIT 1
>
>
> I know the query is useless, but I need to to know how to fix the following
> error thrown while handling this query. This is the final query plan:
>
> DEBUG o.a.d.e.p.s.h.DefaultSqlHandler - HEP_BOTTOM_UP:Convert SUM to $SUM0
> > (0ms):
> > DrillLimitRel(*fetch=[1]*): rowcount = 1.0, cumulative cost = {201.0
> > rows, 205.0 cpu, 0.0 io, 0.0 network, 0.0 memory}, id = 2653
> > DrillLimitRel(*offset=[0], fetch=[0]*): rowcount = 1.0, cumulative cost
> > = {200.0 rows, 201.0 cpu, 0.0 io, 0.0 network, 0.0 memory}, id = 2651
> > GelbanaJdbcDrel: rowcount = 100.0, cumulative cost = {200.0 rows,
> > 201.0 cpu, 0.0 io, 0.0 network, 0.0 memory}, id = 2649
> > JdbcTableScan(table=[[gelbana, SALES, SALES]]): rowcount = 100.0,
> > cumulative cost = {100.0 rows, 101.0 cpu, 0.0 io, 0.0 network, 0.0
> memory},
> > id = 2572
>
>
> This is the throw error stacktrace
>
> [Error Id: 83ea094a-db24-4d6d-bf0d-271db26db933 on 640fb7ebbd1a:31010]
> at
> org.apache.drill.common.exceptions.UserException$
> Builder.build(UserException.java:543)
> ~[drill-common-1.9.0.jar:1.9.0]
> at
> org.apache.drill.exec.work.foreman.Foreman$ForemanResult.
> close(Foreman.java:825)
> [drill-java-exec-1.9.0.jar:1.9.0]
> at org.apache.drill.exec.work.foreman.Foreman.moveToState(
> Foreman.java:935)
> [drill-java-exec-1.9.0.jar:1.9.0]
> at org.apache.drill.exec.work.foreman.Foreman.run(Foreman.java:281)
> [drill-java-exec-1.9.0.jar:1.9.0]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1142)
> [na:1.8.0_131]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
> [na:1.8.0_131]
> at java.lang.Thread.run(Thread.java:748) [na:1.8.0_131]
> Caused by: org.apache.drill.exec.work.foreman.ForemanException: Unexpected
> exception during fragment initialization: null
> ... 4 common frames omitted
> *Caused by: java.lang.NullPointerException: null*
> at
> org.apache.drill.exec.planner.sql.handlers.FindHardDistributionScans.
> visit(FindHardDistributionScans.java:55)
> ~[drill-java-exec-1.9.0.jar:1.9.0]
> at org.apache.calcite.rel.core.TableScan.accept(TableScan.java:166)
> ~[calcite-core-1.4.0-drill-r19.jar:1.4.0-drill-r19]
> at org.apache.calcite.rel.RelShuttleImpl.visitChild(
> RelShuttleImpl.java:53)
> ~[calcite-core-1.4.0-drill-r19.jar:1.4.0-drill-r19]
> at
> org.apache.calcite.rel.RelShuttleImpl.visitChildren(
> RelShuttleImpl.java:68)
> ~[calcite-core-1.4.0-drill-r19.jar:1.4.0-drill-r19]
> at org.apache.calcite.rel.RelShuttleImpl.visit(RelShuttleImpl.java:126)
> ~[calcite-core-1.4.0-drill-r19.jar:1.4.0-drill-r19]
> at org.apache.calcite.rel.AbstractRelNode.accept(AbstractRelNode.java:256)
> ~[calcite-core-1.4.0-drill-r19.jar:1.4.0-drill-r19]
> at org.apache.calcite.rel.RelShuttleImpl.visitChild(
> RelShuttleImpl.java:53)
> ~[calcite-core-1.4.0-drill-r19.jar:1.4.0-drill-r19]
> at
> org.apache.calcite.rel.RelShuttleImpl.visitChildren(
> RelShuttleImpl.java:68)
> ~[calcite-core-1.4.0-drill-r19.jar:1.4.0-drill-r19]
> at org.apache.calcite.rel.RelShuttleImpl.visit(RelShuttleImpl.java:126)
> ~[calcite-core-1.4.0-drill-r19.jar:1.4.0-drill-r19]
> at org.apache.calcite.rel.AbstractRelNode.accept(AbstractRelNode.java:256)
> ~[calcite-core-1.4.0-drill-r19.jar:1.4.0-drill-r19]
> at org.apache.calcite.rel.RelShuttleImpl.visitChild(
> RelShuttleImpl.java:53)
> ~[calcite-core-1.4.0-drill-r19.jar:1.4.0-drill-r19]
> at
> org.apache.calcite.rel.RelShuttleImpl.visitChildren(
> RelShuttleImpl.java:68)
> ~[calcite-core-1.4.0-drill-r19.jar:1.4.0-drill-r19]
> at org.apache.calcite.rel.RelShuttleImpl.visit(RelShuttleImpl.java:126)
> ~[calcite-core-1.4.0-drill-r19.jar:1.4.0-drill-r19]
> at org.apache.calcite.rel.AbstractRelNode.accept(AbstractRelNode.java:256)
> ~[calcite-core-1.4.0-drill-r19.jar:1.4.0-drill-r19]
> at
> org.apache.drill.exec.planner.sql.handlers.FindHardDistributionScans.
> canForceSingleMode(FindHardDistributionScans.java:45)
> ~[drill-java-exec-1.9.0.jar:1.9.0]
> at
> org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.
> convertToDrel(DefaultSqlHandler.java:262)
> ~[drill-java-exec-1.9.0.jar:1.9.0]
> at
> org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.
> convertToDrel(DefaultSqlHandler.java:290)
> ~[drill-java-exec-1.9.0.jar:1.9.0]
> at
> org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.getPlan(
> DefaultSqlHandler.java:168)
> ~[drill-java-exec-1.9.0.jar:1.9.0]
> at
> org.apache.drill.exec.planner.sql.DrillSqlWorker.getPhysicalPlan(
> DrillSqlWorker.java:123)
> ~[drill-java-exec-1.9.0.jar:1.9.0]
> at
> org.apache.drill.exec.planner.sql.DrillSqlWorker.getPlan(
> DrillSqlWorker.java:97)
> ~[drill-java-exec-1.9.0.jar:1.9.0]
> at org.apache.drill.exec.work.foreman.Foreman.runSQL(Foreman.java:1008)
> [drill-java-exec-1.9.0.jar:1.9.0]
> at org.apache.drill.exec.work.foreman.Foreman.run(Foreman.java:264)
> [drill-java-exec-1.9.0.jar:1.9.0]
> ... 3 common frames omitted
>
> Whic is because this statement
>
> unwrap =
> > scan.getTable().unwrap(DrillTranslatableTable.class).getDrillTable();
>
>
> In *FindHardDistributionScans.java:55* evaluates
> *scan.getTable().unwrap(DrillTranslatableTable.class)* to null
>
> Would someone please explain to me what is Drill trying to do and what did
> I do wrong ?
>
> *---------------------*
> *Muhammad Gelbana*
> http://www.linkedin.com/in/mgelbana
>