You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@calcite.apache.org by Dharmendra Chouhan <dh...@yahoo.com.INVALID> on 2016/11/29 14:13:15 UTC
Using Spark engine for processing SQL syntax that cannot be pushed
to the source system
Hi,
In the calcite connection property document , I saw a property SPARK. Its description is - "Specifies whether Spark should be used as the engine for processing that cannot be pushed to the source system"So I thought of testing it over the CSV adapter. i.e anything which is not implemented by csv adapter , use spark for running them. For doing this I followed below steps
1. In pom.xml of csv adapter I added dependency of calcite-spark2. Deleted the target\dependencies folder3. Started sqline.bat4. Used connection string as !connect jdbc:calcite:model=target/test-classes/model.json;spark=true admin admin5. Ran sql query select * from emps;
Is my understanding of this property correct ? Am I setting things up correctly ?Why am I getting the below exception and how to resolve it ?
java.sql.SQLException: Error while executing SQL "select * from emps": null
at org.apache.calcite.avatica.Helper.createException(Helper.java:56) at org.apache.calcite.avatica.Helper.createException(Helper.java:41) at org.apache.calcite.avatica.AvaticaStatement.executeInternal(AvaticaStatement.java:156) at org.apache.calcite.avatica.AvaticaStatement.execute(AvaticaStatement.java:209) at sqlline.Commands.execute(Commands.java:822) at sqlline.Commands.sql(Commands.java:732) at sqlline.SqlLine.dispatch(SqlLine.java:813) at sqlline.SqlLine.begin(SqlLine.java:686) at sqlline.SqlLine.start(SqlLine.java:398) at sqlline.SqlLine.main(SqlLine.java:291)Caused by: java.lang.NullPointerException at org.apache.calcite.interpreter.Interpreter.<init>(Interpreter.java:74) at Baz.bind(Unknown Source) at org.apache.calcite.jdbc.CalcitePrepare$CalciteSignature.enumerable(CalcitePrepare.java:327) at org.apache.calcite.jdbc.CalciteConnectionImpl.enumerable(CalciteConnectionImpl.java:295) at org.apache.calcite.jdbc.CalciteMetaImpl._createIterable(CalciteMetaImpl.java:553) at org.apache.calcite.jdbc.CalciteMetaImpl.createIterable(CalciteMetaImpl.java:544) at org.apache.calcite.avatica.AvaticaResultSet.execute(AvaticaResultSet.java:193) at org.apache.calcite.jdbc.CalciteResultSet.execute(CalciteResultSet.java:67) at org.apache.calcite.jdbc.CalciteResultSet.execute(CalciteResultSet.java:44) at org.apache.calcite.avatica.AvaticaConnection$1.execute(AvaticaConnection.java:607) at org.apache.calcite.jdbc.CalciteMetaImpl.prepareAndExecute(CalciteMetaImpl.java:599) at org.apache.calcite.avatica.AvaticaConnection.prepareAndExecuteInternal(AvaticaConnection.java:615) at org.apache.calcite.avatica.AvaticaStatement.executeInternal(AvaticaStatement.java:148)
Re: Using Spark engine for processing SQL syntax that cannot be
pushed to the source system
Posted by Dharmendra Chouhan <dh...@yahoo.com.INVALID>.
Instead of running from sqlline, I created a sample program to run the program.I added the generated classpath folder into by eclipse project's classpath.But I got a NullPointerExceptionCaused by: java.lang.NullPointerException at org.apache.calcite.interpreter.Interpreter.<init>(Interpreter.java:74) at CalciteProgram151003.bind(Unknown Source) at org.apache.calcite.jdbc.CalcitePrepare$CalciteSignature.enumerable(CalcitePrepare.java:327)
I debugged it and found that in org.apache.calcite.jdbc.createDataContext , for spark it creates SlimDataContext which ignores the parameterValues passed to createDataContext method.
Due to this in the generated code org.apache.calcite.rel.RelNode v0stashed = (org.apache.calcite.rel.RelNode) root.get("v0stashed");
the v0stashed would be null.
So I though let me modify createDataContext to return DataContextImpl(this, parameterValues), even for spark.
Doing so did make the code run, but looking at the generated code and my eclipse's console, nothing is actually getting executed on spark. I can contribute to the spark-calcite project to make it work.
I want to use spark as the engine for processing anything that I am not able to push down to my custom adapter so that in case of large data things don't go out of memory.
Thanks & Regards,Dharmendra
From: Dharmendra Chouhan <dh...@yahoo.com.INVALID>
To: "dev@calcite.apache.org" <de...@calcite.apache.org>
Sent: Tuesday, 29 November 2016 7:43 PM
Subject: Using Spark engine for processing SQL syntax that cannot be pushed to the source system
Hi,
In the calcite connection property document , I saw a property SPARK. Its description is - "Specifies whether Spark should be used as the engine for processing that cannot be pushed to the source system"So I thought of testing it over the CSV adapter. i.e anything which is not implemented by csv adapter , use spark for running them. For doing this I followed below steps
1. In pom.xml of csv adapter I added dependency of calcite-spark2. Deleted the target\dependencies folder3. Started sqline.bat4. Used connection string as !connect jdbc:calcite:model=target/test-classes/model.json;spark=true admin admin5. Ran sql query select * from emps;
Is my understanding of this property correct ? Am I setting things up correctly ?Why am I getting the below exception and how to resolve it ?
java.sql.SQLException: Error while executing SQL "select * from emps": null
at org.apache.calcite.avatica.Helper.createException(Helper.java:56) at org.apache.calcite.avatica.Helper.createException(Helper.java:41) at org.apache.calcite.avatica.AvaticaStatement.executeInternal(AvaticaStatement.java:156) at org.apache.calcite.avatica.AvaticaStatement.execute(AvaticaStatement.java:209) at sqlline.Commands.execute(Commands.java:822) at sqlline.Commands.sql(Commands.java:732) at sqlline.SqlLine.dispatch(SqlLine.java:813) at sqlline.SqlLine.begin(SqlLine.java:686) at sqlline.SqlLine.start(SqlLine.java:398) at sqlline.SqlLine.main(SqlLine.java:291)Caused by: java.lang.NullPointerException at org.apache.calcite.interpreter.Interpreter.<init>(Interpreter.java:74) at Baz.bind(Unknown Source) at org.apache.calcite.jdbc.CalcitePrepare$CalciteSignature.enumerable(CalcitePrepare.java:327) at org.apache.calcite.jdbc.CalciteConnectionImpl.enumerable(CalciteConnectionImpl.java:295) at org.apache.calcite.jdbc.CalciteMetaImpl._createIterable(CalciteMetaImpl.java:553) at org.apache.calcite.jdbc.CalciteMetaImpl.createIterable(CalciteMetaImpl.java:544) at org.apache.calcite.avatica.AvaticaResultSet.execute(AvaticaResultSet.java:193) at org.apache.calcite.jdbc.CalciteResultSet.execute(CalciteResultSet.java:67) at org.apache.calcite.jdbc.CalciteResultSet.execute(CalciteResultSet.java:44) at org.apache.calcite.avatica.AvaticaConnection$1.execute(AvaticaConnection.java:607) at org.apache.calcite.jdbc.CalciteMetaImpl.prepareAndExecute(CalciteMetaImpl.java:599) at org.apache.calcite.avatica.AvaticaConnection.prepareAndExecuteInternal(AvaticaConnection.java:615) at org.apache.calcite.avatica.AvaticaStatement.executeInternal(AvaticaStatement.java:148)