You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by Ted Yu <yu...@gmail.com> on 2016/04/04 01:00:38 UTC
explain codegen
Hi,
Based on master branch refreshed today, I issued 'git clean -fdx' first.
Then this command:
build/mvn clean -Phive -Phive-thriftserver -Pyarn -Phadoop-2.6
-Dhadoop.version=2.7.0 package -DskipTests
I got the following error:
scala> sql("explain codegen select 'a' as a group by 1").head
org.apache.spark.sql.catalyst.parser.ParseException:
extraneous input 'codegen' expecting {'(', 'SELECT', 'FROM', 'ADD', 'DESC',
'WITH', 'VALUES', 'CREATE', 'TABLE', 'INSERT', 'DELETE', 'DESCRIBE',
'EXPLAIN', 'LOGICAL', 'SHOW', 'USE', 'DROP', 'ALTER', 'MAP', 'SET',
'START', 'COMMIT', 'ROLLBACK', 'REDUCE', 'EXTENDED', 'REFRESH', 'CLEAR',
'CACHE', 'UNCACHE', 'FORMATTED', 'DFS', 'TRUNCATE', 'ANALYZE', 'REVOKE',
'GRANT', 'LOCK', 'UNLOCK', 'MSCK', 'EXPORT', 'IMPORT', 'LOAD'}(line 1, pos
8)
== SQL ==
explain codegen select 'a' as a group by 1
--------^^^
Can someone shed light ?
Thanks
Re: explain codegen
Posted by Ted Yu <yu...@gmail.com>.
Thanks to all who have responded.
It turned out that the following command line for maven caused the error (I
forgot to include this in first email):
eclipse:eclipse
Once I omitted the above, 'explain codegen' works.
On Mon, Apr 4, 2016 at 9:37 AM, Reynold Xin <rx...@databricks.com> wrote:
> Why don't you wipe everything out and try again?
>
> On Monday, April 4, 2016, Ted Yu <yu...@gmail.com> wrote:
>
>> The commit you mentioned was made Friday.
>> I refreshed workspace Sunday - so it was included.
>>
>> Maybe this was related:
>>
>> $ bin/spark-shell
>> Failed to find Spark jars directory
>> (/home/hbase/spark/assembly/target/scala-2.10).
>> You need to build Spark before running this program.
>>
>> Then I did:
>>
>> $ ln -s /home/hbase/spark/assembly/target/scala-2.11
>> assembly/target/scala-2.10
>>
>> Cheers
>>
>> On Mon, Apr 4, 2016 at 4:06 AM, Herman van Hövell tot Westerflier <
>> hvanhovell@questtec.nl> wrote:
>>
>>> No, it can''t. You only need implicits when you are using the catalyst
>>> DSL.
>>>
>>> The error you get is due to the fact that the parser does not recognize
>>> the CODEGEN keyword (which was the case before we introduced this in
>>> https://github.com/apache/spark/commit/fa1af0aff7bde9bbf7bfa6a3ac74699734c2fd8a).
>>> That suggests to me that you are not on the latest master.
>>>
>>> Kind regards,
>>>
>>> Herman van Hövell
>>>
>>> 2016-04-04 12:15 GMT+02:00 Ted Yu <yu...@gmail.com>:
>>>
>>>> Could the error I encountered be due to missing import(s) of implicit ?
>>>>
>>>> Thanks
>>>>
>>>> On Sun, Apr 3, 2016 at 9:42 PM, Reynold Xin <rx...@databricks.com>
>>>> wrote:
>>>>
>>>>> Works for me on latest master.
>>>>>
>>>>>
>>>>>
>>>>> scala> sql("explain codegen select 'a' as a group by 1").head
>>>>> res3: org.apache.spark.sql.Row =
>>>>> [Found 2 WholeStageCodegen subtrees.
>>>>> == Subtree 1 / 2 ==
>>>>> WholeStageCodegen
>>>>> : +- TungstenAggregate(key=[], functions=[], output=[a#10])
>>>>> : +- INPUT
>>>>> +- Exchange SinglePartition, None
>>>>> +- WholeStageCodegen
>>>>> : +- TungstenAggregate(key=[], functions=[], output=[])
>>>>> : +- INPUT
>>>>> +- Scan OneRowRelation[]
>>>>>
>>>>> Generated code:
>>>>> /* 001 */ public Object generate(Object[] references) {
>>>>> /* 002 */ return new GeneratedIterator(references);
>>>>> /* 003 */ }
>>>>> /* 004 */
>>>>> /* 005 */ /** Codegened pipeline for:
>>>>> /* 006 */ * TungstenAggregate(key=[], functions=[], output=[a#10])
>>>>> /* 007 */ +- INPUT
>>>>> /* 008 */ */
>>>>> /* 009 */ final class GeneratedIterator extends
>>>>> org.apache.spark.sql.execution.BufferedRowIterator {
>>>>> /* 010 */ private Object[] references;
>>>>> /* 011 */ ...
>>>>>
>>>>>
>>>>> On Sun, Apr 3, 2016 at 9:38 PM, Jacek Laskowski <ja...@japila.pl>
>>>>> wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> Looks related to the recent commit...
>>>>>>
>>>>>> Repository: spark
>>>>>> Updated Branches:
>>>>>> refs/heads/master 2262a9335 -> 1f0c5dceb
>>>>>>
>>>>>> [SPARK-14350][SQL] EXPLAIN output should be in a single cell
>>>>>>
>>>>>> Jacek
>>>>>> 03.04.2016 7:00 PM "Ted Yu" <yu...@gmail.com> napisał(a):
>>>>>>
>>>>>>> Hi,
>>>>>>> Based on master branch refreshed today, I issued 'git clean -fdx'
>>>>>>> first.
>>>>>>>
>>>>>>> Then this command:
>>>>>>> build/mvn clean -Phive -Phive-thriftserver -Pyarn -Phadoop-2.6
>>>>>>> -Dhadoop.version=2.7.0 package -DskipTests
>>>>>>>
>>>>>>> I got the following error:
>>>>>>>
>>>>>>> scala> sql("explain codegen select 'a' as a group by 1").head
>>>>>>> org.apache.spark.sql.catalyst.parser.ParseException:
>>>>>>> extraneous input 'codegen' expecting {'(', 'SELECT', 'FROM', 'ADD',
>>>>>>> 'DESC', 'WITH', 'VALUES', 'CREATE', 'TABLE', 'INSERT', 'DELETE',
>>>>>>> 'DESCRIBE', 'EXPLAIN', 'LOGICAL', 'SHOW', 'USE', 'DROP', 'ALTER', 'MAP',
>>>>>>> 'SET', 'START', 'COMMIT', 'ROLLBACK', 'REDUCE', 'EXTENDED', 'REFRESH',
>>>>>>> 'CLEAR', 'CACHE', 'UNCACHE', 'FORMATTED', 'DFS', 'TRUNCATE', 'ANALYZE',
>>>>>>> 'REVOKE', 'GRANT', 'LOCK', 'UNLOCK', 'MSCK', 'EXPORT', 'IMPORT',
>>>>>>> 'LOAD'}(line 1, pos 8)
>>>>>>>
>>>>>>> == SQL ==
>>>>>>> explain codegen select 'a' as a group by 1
>>>>>>> --------^^^
>>>>>>>
>>>>>>> Can someone shed light ?
>>>>>>>
>>>>>>> Thanks
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
Re: explain codegen
Posted by Reynold Xin <rx...@databricks.com>.
Why don't you wipe everything out and try again?
On Monday, April 4, 2016, Ted Yu <yu...@gmail.com> wrote:
> The commit you mentioned was made Friday.
> I refreshed workspace Sunday - so it was included.
>
> Maybe this was related:
>
> $ bin/spark-shell
> Failed to find Spark jars directory
> (/home/hbase/spark/assembly/target/scala-2.10).
> You need to build Spark before running this program.
>
> Then I did:
>
> $ ln -s /home/hbase/spark/assembly/target/scala-2.11
> assembly/target/scala-2.10
>
> Cheers
>
> On Mon, Apr 4, 2016 at 4:06 AM, Herman van Hövell tot Westerflier <
> hvanhovell@questtec.nl
> <javascript:_e(%7B%7D,'cvml','hvanhovell@questtec.nl');>> wrote:
>
>> No, it can''t. You only need implicits when you are using the catalyst
>> DSL.
>>
>> The error you get is due to the fact that the parser does not recognize
>> the CODEGEN keyword (which was the case before we introduced this in
>> https://github.com/apache/spark/commit/fa1af0aff7bde9bbf7bfa6a3ac74699734c2fd8a).
>> That suggests to me that you are not on the latest master.
>>
>> Kind regards,
>>
>> Herman van Hövell
>>
>> 2016-04-04 12:15 GMT+02:00 Ted Yu <yuzhihong@gmail.com
>> <javascript:_e(%7B%7D,'cvml','yuzhihong@gmail.com');>>:
>>
>>> Could the error I encountered be due to missing import(s) of implicit ?
>>>
>>> Thanks
>>>
>>> On Sun, Apr 3, 2016 at 9:42 PM, Reynold Xin <rxin@databricks.com
>>> <javascript:_e(%7B%7D,'cvml','rxin@databricks.com');>> wrote:
>>>
>>>> Works for me on latest master.
>>>>
>>>>
>>>>
>>>> scala> sql("explain codegen select 'a' as a group by 1").head
>>>> res3: org.apache.spark.sql.Row =
>>>> [Found 2 WholeStageCodegen subtrees.
>>>> == Subtree 1 / 2 ==
>>>> WholeStageCodegen
>>>> : +- TungstenAggregate(key=[], functions=[], output=[a#10])
>>>> : +- INPUT
>>>> +- Exchange SinglePartition, None
>>>> +- WholeStageCodegen
>>>> : +- TungstenAggregate(key=[], functions=[], output=[])
>>>> : +- INPUT
>>>> +- Scan OneRowRelation[]
>>>>
>>>> Generated code:
>>>> /* 001 */ public Object generate(Object[] references) {
>>>> /* 002 */ return new GeneratedIterator(references);
>>>> /* 003 */ }
>>>> /* 004 */
>>>> /* 005 */ /** Codegened pipeline for:
>>>> /* 006 */ * TungstenAggregate(key=[], functions=[], output=[a#10])
>>>> /* 007 */ +- INPUT
>>>> /* 008 */ */
>>>> /* 009 */ final class GeneratedIterator extends
>>>> org.apache.spark.sql.execution.BufferedRowIterator {
>>>> /* 010 */ private Object[] references;
>>>> /* 011 */ ...
>>>>
>>>>
>>>> On Sun, Apr 3, 2016 at 9:38 PM, Jacek Laskowski <jacek@japila.pl
>>>> <javascript:_e(%7B%7D,'cvml','jacek@japila.pl');>> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> Looks related to the recent commit...
>>>>>
>>>>> Repository: spark
>>>>> Updated Branches:
>>>>> refs/heads/master 2262a9335 -> 1f0c5dceb
>>>>>
>>>>> [SPARK-14350][SQL] EXPLAIN output should be in a single cell
>>>>>
>>>>> Jacek
>>>>> 03.04.2016 7:00 PM "Ted Yu" <yuzhihong@gmail.com
>>>>> <javascript:_e(%7B%7D,'cvml','yuzhihong@gmail.com');>> napisał(a):
>>>>>
>>>>>> Hi,
>>>>>> Based on master branch refreshed today, I issued 'git clean -fdx'
>>>>>> first.
>>>>>>
>>>>>> Then this command:
>>>>>> build/mvn clean -Phive -Phive-thriftserver -Pyarn -Phadoop-2.6
>>>>>> -Dhadoop.version=2.7.0 package -DskipTests
>>>>>>
>>>>>> I got the following error:
>>>>>>
>>>>>> scala> sql("explain codegen select 'a' as a group by 1").head
>>>>>> org.apache.spark.sql.catalyst.parser.ParseException:
>>>>>> extraneous input 'codegen' expecting {'(', 'SELECT', 'FROM', 'ADD',
>>>>>> 'DESC', 'WITH', 'VALUES', 'CREATE', 'TABLE', 'INSERT', 'DELETE',
>>>>>> 'DESCRIBE', 'EXPLAIN', 'LOGICAL', 'SHOW', 'USE', 'DROP', 'ALTER', 'MAP',
>>>>>> 'SET', 'START', 'COMMIT', 'ROLLBACK', 'REDUCE', 'EXTENDED', 'REFRESH',
>>>>>> 'CLEAR', 'CACHE', 'UNCACHE', 'FORMATTED', 'DFS', 'TRUNCATE', 'ANALYZE',
>>>>>> 'REVOKE', 'GRANT', 'LOCK', 'UNLOCK', 'MSCK', 'EXPORT', 'IMPORT',
>>>>>> 'LOAD'}(line 1, pos 8)
>>>>>>
>>>>>> == SQL ==
>>>>>> explain codegen select 'a' as a group by 1
>>>>>> --------^^^
>>>>>>
>>>>>> Can someone shed light ?
>>>>>>
>>>>>> Thanks
>>>>>>
>>>>>
>>>>
>>>
>>
>
Re: explain codegen
Posted by Ted Yu <yu...@gmail.com>.
The commit you mentioned was made Friday.
I refreshed workspace Sunday - so it was included.
Maybe this was related:
$ bin/spark-shell
Failed to find Spark jars directory
(/home/hbase/spark/assembly/target/scala-2.10).
You need to build Spark before running this program.
Then I did:
$ ln -s /home/hbase/spark/assembly/target/scala-2.11
assembly/target/scala-2.10
Cheers
On Mon, Apr 4, 2016 at 4:06 AM, Herman van Hövell tot Westerflier <
hvanhovell@questtec.nl> wrote:
> No, it can''t. You only need implicits when you are using the catalyst DSL.
>
> The error you get is due to the fact that the parser does not recognize
> the CODEGEN keyword (which was the case before we introduced this in
> https://github.com/apache/spark/commit/fa1af0aff7bde9bbf7bfa6a3ac74699734c2fd8a).
> That suggests to me that you are not on the latest master.
>
> Kind regards,
>
> Herman van Hövell
>
> 2016-04-04 12:15 GMT+02:00 Ted Yu <yu...@gmail.com>:
>
>> Could the error I encountered be due to missing import(s) of implicit ?
>>
>> Thanks
>>
>> On Sun, Apr 3, 2016 at 9:42 PM, Reynold Xin <rx...@databricks.com> wrote:
>>
>>> Works for me on latest master.
>>>
>>>
>>>
>>> scala> sql("explain codegen select 'a' as a group by 1").head
>>> res3: org.apache.spark.sql.Row =
>>> [Found 2 WholeStageCodegen subtrees.
>>> == Subtree 1 / 2 ==
>>> WholeStageCodegen
>>> : +- TungstenAggregate(key=[], functions=[], output=[a#10])
>>> : +- INPUT
>>> +- Exchange SinglePartition, None
>>> +- WholeStageCodegen
>>> : +- TungstenAggregate(key=[], functions=[], output=[])
>>> : +- INPUT
>>> +- Scan OneRowRelation[]
>>>
>>> Generated code:
>>> /* 001 */ public Object generate(Object[] references) {
>>> /* 002 */ return new GeneratedIterator(references);
>>> /* 003 */ }
>>> /* 004 */
>>> /* 005 */ /** Codegened pipeline for:
>>> /* 006 */ * TungstenAggregate(key=[], functions=[], output=[a#10])
>>> /* 007 */ +- INPUT
>>> /* 008 */ */
>>> /* 009 */ final class GeneratedIterator extends
>>> org.apache.spark.sql.execution.BufferedRowIterator {
>>> /* 010 */ private Object[] references;
>>> /* 011 */ ...
>>>
>>>
>>> On Sun, Apr 3, 2016 at 9:38 PM, Jacek Laskowski <ja...@japila.pl> wrote:
>>>
>>>> Hi,
>>>>
>>>> Looks related to the recent commit...
>>>>
>>>> Repository: spark
>>>> Updated Branches:
>>>> refs/heads/master 2262a9335 -> 1f0c5dceb
>>>>
>>>> [SPARK-14350][SQL] EXPLAIN output should be in a single cell
>>>>
>>>> Jacek
>>>> 03.04.2016 7:00 PM "Ted Yu" <yu...@gmail.com> napisał(a):
>>>>
>>>>> Hi,
>>>>> Based on master branch refreshed today, I issued 'git clean -fdx'
>>>>> first.
>>>>>
>>>>> Then this command:
>>>>> build/mvn clean -Phive -Phive-thriftserver -Pyarn -Phadoop-2.6
>>>>> -Dhadoop.version=2.7.0 package -DskipTests
>>>>>
>>>>> I got the following error:
>>>>>
>>>>> scala> sql("explain codegen select 'a' as a group by 1").head
>>>>> org.apache.spark.sql.catalyst.parser.ParseException:
>>>>> extraneous input 'codegen' expecting {'(', 'SELECT', 'FROM', 'ADD',
>>>>> 'DESC', 'WITH', 'VALUES', 'CREATE', 'TABLE', 'INSERT', 'DELETE',
>>>>> 'DESCRIBE', 'EXPLAIN', 'LOGICAL', 'SHOW', 'USE', 'DROP', 'ALTER', 'MAP',
>>>>> 'SET', 'START', 'COMMIT', 'ROLLBACK', 'REDUCE', 'EXTENDED', 'REFRESH',
>>>>> 'CLEAR', 'CACHE', 'UNCACHE', 'FORMATTED', 'DFS', 'TRUNCATE', 'ANALYZE',
>>>>> 'REVOKE', 'GRANT', 'LOCK', 'UNLOCK', 'MSCK', 'EXPORT', 'IMPORT',
>>>>> 'LOAD'}(line 1, pos 8)
>>>>>
>>>>> == SQL ==
>>>>> explain codegen select 'a' as a group by 1
>>>>> --------^^^
>>>>>
>>>>> Can someone shed light ?
>>>>>
>>>>> Thanks
>>>>>
>>>>
>>>
>>
>
Re: explain codegen
Posted by Herman van Hövell tot Westerflier <hv...@questtec.nl>.
No, it can''t. You only need implicits when you are using the catalyst DSL.
The error you get is due to the fact that the parser does not recognize the
CODEGEN keyword (which was the case before we introduced this in
https://github.com/apache/spark/commit/fa1af0aff7bde9bbf7bfa6a3ac74699734c2fd8a).
That suggests to me that you are not on the latest master.
Kind regards,
Herman van Hövell
2016-04-04 12:15 GMT+02:00 Ted Yu <yu...@gmail.com>:
> Could the error I encountered be due to missing import(s) of implicit ?
>
> Thanks
>
> On Sun, Apr 3, 2016 at 9:42 PM, Reynold Xin <rx...@databricks.com> wrote:
>
>> Works for me on latest master.
>>
>>
>>
>> scala> sql("explain codegen select 'a' as a group by 1").head
>> res3: org.apache.spark.sql.Row =
>> [Found 2 WholeStageCodegen subtrees.
>> == Subtree 1 / 2 ==
>> WholeStageCodegen
>> : +- TungstenAggregate(key=[], functions=[], output=[a#10])
>> : +- INPUT
>> +- Exchange SinglePartition, None
>> +- WholeStageCodegen
>> : +- TungstenAggregate(key=[], functions=[], output=[])
>> : +- INPUT
>> +- Scan OneRowRelation[]
>>
>> Generated code:
>> /* 001 */ public Object generate(Object[] references) {
>> /* 002 */ return new GeneratedIterator(references);
>> /* 003 */ }
>> /* 004 */
>> /* 005 */ /** Codegened pipeline for:
>> /* 006 */ * TungstenAggregate(key=[], functions=[], output=[a#10])
>> /* 007 */ +- INPUT
>> /* 008 */ */
>> /* 009 */ final class GeneratedIterator extends
>> org.apache.spark.sql.execution.BufferedRowIterator {
>> /* 010 */ private Object[] references;
>> /* 011 */ ...
>>
>>
>> On Sun, Apr 3, 2016 at 9:38 PM, Jacek Laskowski <ja...@japila.pl> wrote:
>>
>>> Hi,
>>>
>>> Looks related to the recent commit...
>>>
>>> Repository: spark
>>> Updated Branches:
>>> refs/heads/master 2262a9335 -> 1f0c5dceb
>>>
>>> [SPARK-14350][SQL] EXPLAIN output should be in a single cell
>>>
>>> Jacek
>>> 03.04.2016 7:00 PM "Ted Yu" <yu...@gmail.com> napisał(a):
>>>
>>>> Hi,
>>>> Based on master branch refreshed today, I issued 'git clean -fdx'
>>>> first.
>>>>
>>>> Then this command:
>>>> build/mvn clean -Phive -Phive-thriftserver -Pyarn -Phadoop-2.6
>>>> -Dhadoop.version=2.7.0 package -DskipTests
>>>>
>>>> I got the following error:
>>>>
>>>> scala> sql("explain codegen select 'a' as a group by 1").head
>>>> org.apache.spark.sql.catalyst.parser.ParseException:
>>>> extraneous input 'codegen' expecting {'(', 'SELECT', 'FROM', 'ADD',
>>>> 'DESC', 'WITH', 'VALUES', 'CREATE', 'TABLE', 'INSERT', 'DELETE',
>>>> 'DESCRIBE', 'EXPLAIN', 'LOGICAL', 'SHOW', 'USE', 'DROP', 'ALTER', 'MAP',
>>>> 'SET', 'START', 'COMMIT', 'ROLLBACK', 'REDUCE', 'EXTENDED', 'REFRESH',
>>>> 'CLEAR', 'CACHE', 'UNCACHE', 'FORMATTED', 'DFS', 'TRUNCATE', 'ANALYZE',
>>>> 'REVOKE', 'GRANT', 'LOCK', 'UNLOCK', 'MSCK', 'EXPORT', 'IMPORT',
>>>> 'LOAD'}(line 1, pos 8)
>>>>
>>>> == SQL ==
>>>> explain codegen select 'a' as a group by 1
>>>> --------^^^
>>>>
>>>> Can someone shed light ?
>>>>
>>>> Thanks
>>>>
>>>
>>
>
Re: explain codegen
Posted by Ted Yu <yu...@gmail.com>.
Could the error I encountered be due to missing import(s) of implicit ?
Thanks
On Sun, Apr 3, 2016 at 9:42 PM, Reynold Xin <rx...@databricks.com> wrote:
> Works for me on latest master.
>
>
>
> scala> sql("explain codegen select 'a' as a group by 1").head
> res3: org.apache.spark.sql.Row =
> [Found 2 WholeStageCodegen subtrees.
> == Subtree 1 / 2 ==
> WholeStageCodegen
> : +- TungstenAggregate(key=[], functions=[], output=[a#10])
> : +- INPUT
> +- Exchange SinglePartition, None
> +- WholeStageCodegen
> : +- TungstenAggregate(key=[], functions=[], output=[])
> : +- INPUT
> +- Scan OneRowRelation[]
>
> Generated code:
> /* 001 */ public Object generate(Object[] references) {
> /* 002 */ return new GeneratedIterator(references);
> /* 003 */ }
> /* 004 */
> /* 005 */ /** Codegened pipeline for:
> /* 006 */ * TungstenAggregate(key=[], functions=[], output=[a#10])
> /* 007 */ +- INPUT
> /* 008 */ */
> /* 009 */ final class GeneratedIterator extends
> org.apache.spark.sql.execution.BufferedRowIterator {
> /* 010 */ private Object[] references;
> /* 011 */ ...
>
>
> On Sun, Apr 3, 2016 at 9:38 PM, Jacek Laskowski <ja...@japila.pl> wrote:
>
>> Hi,
>>
>> Looks related to the recent commit...
>>
>> Repository: spark
>> Updated Branches:
>> refs/heads/master 2262a9335 -> 1f0c5dceb
>>
>> [SPARK-14350][SQL] EXPLAIN output should be in a single cell
>>
>> Jacek
>> 03.04.2016 7:00 PM "Ted Yu" <yu...@gmail.com> napisał(a):
>>
>>> Hi,
>>> Based on master branch refreshed today, I issued 'git clean -fdx' first.
>>>
>>> Then this command:
>>> build/mvn clean -Phive -Phive-thriftserver -Pyarn -Phadoop-2.6
>>> -Dhadoop.version=2.7.0 package -DskipTests
>>>
>>> I got the following error:
>>>
>>> scala> sql("explain codegen select 'a' as a group by 1").head
>>> org.apache.spark.sql.catalyst.parser.ParseException:
>>> extraneous input 'codegen' expecting {'(', 'SELECT', 'FROM', 'ADD',
>>> 'DESC', 'WITH', 'VALUES', 'CREATE', 'TABLE', 'INSERT', 'DELETE',
>>> 'DESCRIBE', 'EXPLAIN', 'LOGICAL', 'SHOW', 'USE', 'DROP', 'ALTER', 'MAP',
>>> 'SET', 'START', 'COMMIT', 'ROLLBACK', 'REDUCE', 'EXTENDED', 'REFRESH',
>>> 'CLEAR', 'CACHE', 'UNCACHE', 'FORMATTED', 'DFS', 'TRUNCATE', 'ANALYZE',
>>> 'REVOKE', 'GRANT', 'LOCK', 'UNLOCK', 'MSCK', 'EXPORT', 'IMPORT',
>>> 'LOAD'}(line 1, pos 8)
>>>
>>> == SQL ==
>>> explain codegen select 'a' as a group by 1
>>> --------^^^
>>>
>>> Can someone shed light ?
>>>
>>> Thanks
>>>
>>
>
Re: explain codegen
Posted by Reynold Xin <rx...@databricks.com>.
Works for me on latest master.
scala> sql("explain codegen select 'a' as a group by 1").head
res3: org.apache.spark.sql.Row =
[Found 2 WholeStageCodegen subtrees.
== Subtree 1 / 2 ==
WholeStageCodegen
: +- TungstenAggregate(key=[], functions=[], output=[a#10])
: +- INPUT
+- Exchange SinglePartition, None
+- WholeStageCodegen
: +- TungstenAggregate(key=[], functions=[], output=[])
: +- INPUT
+- Scan OneRowRelation[]
Generated code:
/* 001 */ public Object generate(Object[] references) {
/* 002 */ return new GeneratedIterator(references);
/* 003 */ }
/* 004 */
/* 005 */ /** Codegened pipeline for:
/* 006 */ * TungstenAggregate(key=[], functions=[], output=[a#10])
/* 007 */ +- INPUT
/* 008 */ */
/* 009 */ final class GeneratedIterator extends
org.apache.spark.sql.execution.BufferedRowIterator {
/* 010 */ private Object[] references;
/* 011 */ ...
On Sun, Apr 3, 2016 at 9:38 PM, Jacek Laskowski <ja...@japila.pl> wrote:
> Hi,
>
> Looks related to the recent commit...
>
> Repository: spark
> Updated Branches:
> refs/heads/master 2262a9335 -> 1f0c5dceb
>
> [SPARK-14350][SQL] EXPLAIN output should be in a single cell
>
> Jacek
> 03.04.2016 7:00 PM "Ted Yu" <yu...@gmail.com> napisał(a):
>
>> Hi,
>> Based on master branch refreshed today, I issued 'git clean -fdx' first.
>>
>> Then this command:
>> build/mvn clean -Phive -Phive-thriftserver -Pyarn -Phadoop-2.6
>> -Dhadoop.version=2.7.0 package -DskipTests
>>
>> I got the following error:
>>
>> scala> sql("explain codegen select 'a' as a group by 1").head
>> org.apache.spark.sql.catalyst.parser.ParseException:
>> extraneous input 'codegen' expecting {'(', 'SELECT', 'FROM', 'ADD',
>> 'DESC', 'WITH', 'VALUES', 'CREATE', 'TABLE', 'INSERT', 'DELETE',
>> 'DESCRIBE', 'EXPLAIN', 'LOGICAL', 'SHOW', 'USE', 'DROP', 'ALTER', 'MAP',
>> 'SET', 'START', 'COMMIT', 'ROLLBACK', 'REDUCE', 'EXTENDED', 'REFRESH',
>> 'CLEAR', 'CACHE', 'UNCACHE', 'FORMATTED', 'DFS', 'TRUNCATE', 'ANALYZE',
>> 'REVOKE', 'GRANT', 'LOCK', 'UNLOCK', 'MSCK', 'EXPORT', 'IMPORT',
>> 'LOAD'}(line 1, pos 8)
>>
>> == SQL ==
>> explain codegen select 'a' as a group by 1
>> --------^^^
>>
>> Can someone shed light ?
>>
>> Thanks
>>
>
Re: explain codegen
Posted by Jacek Laskowski <ja...@japila.pl>.
Hi,
Looks related to the recent commit...
Repository: spark
Updated Branches:
refs/heads/master 2262a9335 -> 1f0c5dceb
[SPARK-14350][SQL] EXPLAIN output should be in a single cell
Jacek
03.04.2016 7:00 PM "Ted Yu" <yu...@gmail.com> napisał(a):
> Hi,
> Based on master branch refreshed today, I issued 'git clean -fdx' first.
>
> Then this command:
> build/mvn clean -Phive -Phive-thriftserver -Pyarn -Phadoop-2.6
> -Dhadoop.version=2.7.0 package -DskipTests
>
> I got the following error:
>
> scala> sql("explain codegen select 'a' as a group by 1").head
> org.apache.spark.sql.catalyst.parser.ParseException:
> extraneous input 'codegen' expecting {'(', 'SELECT', 'FROM', 'ADD',
> 'DESC', 'WITH', 'VALUES', 'CREATE', 'TABLE', 'INSERT', 'DELETE',
> 'DESCRIBE', 'EXPLAIN', 'LOGICAL', 'SHOW', 'USE', 'DROP', 'ALTER', 'MAP',
> 'SET', 'START', 'COMMIT', 'ROLLBACK', 'REDUCE', 'EXTENDED', 'REFRESH',
> 'CLEAR', 'CACHE', 'UNCACHE', 'FORMATTED', 'DFS', 'TRUNCATE', 'ANALYZE',
> 'REVOKE', 'GRANT', 'LOCK', 'UNLOCK', 'MSCK', 'EXPORT', 'IMPORT',
> 'LOAD'}(line 1, pos 8)
>
> == SQL ==
> explain codegen select 'a' as a group by 1
> --------^^^
>
> Can someone shed light ?
>
> Thanks
>