You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Nick Chammas <ni...@gmail.com> on 2014/07/13 04:16:55 UTC
Supported SQL syntax in Spark SQL
Is there a place where we can find an up-to-date list of supported SQL
syntax in Spark SQL?
Nick
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Supported-SQL-syntax-in-Spark-SQL-tp9538.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
Re: Supported SQL syntax in Spark SQL
Posted by Nicholas Chammas <ni...@gmail.com>.
FYI: I've created SPARK-2560
<https://issues.apache.org/jira/browse/SPARK-2560> to track creating SQL
reference docs for Spark SQL.
On Mon, Jul 14, 2014 at 2:06 PM, Michael Armbrust <mi...@databricks.com>
wrote:
> You can find the parser here:
> https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/SqlParser.scala
>
> In general the hive parser provided by HQL is much more complete at the
> moment. Long term we will likely stop using parser combinators and either
> write a more complete parser, or adopt one from an existing project.
>
>
> On Mon, Jul 14, 2014 at 12:25 AM, Martin Gammelsæter <
> martingammelsaeter@gmail.com> wrote:
>
>> I am very interested in the original question as well, is there any
>> list (even if it is simply in the code) of all supported syntax for
>> Spark SQL?
>>
>> On Mon, Jul 14, 2014 at 6:41 AM, Nicholas Chammas
>> <ni...@gmail.com> wrote:
>> >> Are you sure the code running on the cluster has been updated?
>> >
>> > I launched the cluster using spark-ec2 from the 1.0.1 release, so I’m
>> > assuming that’s taken care of, at least in theory.
>> >
>> > I just spun down the clusters I had up, but I will revisit this
>> tomorrow and
>> > provide the information you requested.
>> >
>> > Nick
>>
>>
>>
>> --
>> Mvh.
>> Martin Gammelsæter
>> 92209139
>>
>
>
Re: Supported SQL syntax in Spark SQL
Posted by Michael Armbrust <mi...@databricks.com>.
You can find the parser here:
https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/SqlParser.scala
In general the hive parser provided by HQL is much more complete at the
moment. Long term we will likely stop using parser combinators and either
write a more complete parser, or adopt one from an existing project.
On Mon, Jul 14, 2014 at 12:25 AM, Martin Gammelsæter <
martingammelsaeter@gmail.com> wrote:
> I am very interested in the original question as well, is there any
> list (even if it is simply in the code) of all supported syntax for
> Spark SQL?
>
> On Mon, Jul 14, 2014 at 6:41 AM, Nicholas Chammas
> <ni...@gmail.com> wrote:
> >> Are you sure the code running on the cluster has been updated?
> >
> > I launched the cluster using spark-ec2 from the 1.0.1 release, so I’m
> > assuming that’s taken care of, at least in theory.
> >
> > I just spun down the clusters I had up, but I will revisit this tomorrow
> and
> > provide the information you requested.
> >
> > Nick
>
>
>
> --
> Mvh.
> Martin Gammelsæter
> 92209139
>
Re: Supported SQL syntax in Spark SQL
Posted by Martin Gammelsæter <ma...@gmail.com>.
I am very interested in the original question as well, is there any
list (even if it is simply in the code) of all supported syntax for
Spark SQL?
On Mon, Jul 14, 2014 at 6:41 AM, Nicholas Chammas
<ni...@gmail.com> wrote:
>> Are you sure the code running on the cluster has been updated?
>
> I launched the cluster using spark-ec2 from the 1.0.1 release, so I’m
> assuming that’s taken care of, at least in theory.
>
> I just spun down the clusters I had up, but I will revisit this tomorrow and
> provide the information you requested.
>
> Nick
--
Mvh.
Martin Gammelsæter
92209139
Re: Supported SQL syntax in Spark SQL
Posted by Nicholas Chammas <ni...@gmail.com>.
Are you sure the code running on the cluster has been updated?
I launched the cluster using spark-ec2 from the 1.0.1 release, so I’m
assuming that’s taken care of, at least in theory.
I just spun down the clusters I had up, but I will revisit this tomorrow
and provide the information you requested.
Nick
Re: Supported SQL syntax in Spark SQL
Posted by Michael Armbrust <mi...@databricks.com>.
Are you sure the code running on the cluster has been updated? We recently
optimized the execution of LIKE queries that can be evaluated without using
full regular expressions. So it's possible this error is due to missing
functionality on the executors.
> How can I trace this down for a bug report?
>
If the above doesn't fix it, the following would be helpful:
- The full stack trace
- The queryExecution from the SchemaRDD (i.e. println(sql("SELECT
...").queryExecution))
Re: Supported SQL syntax in Spark SQL
Posted by Nicholas Chammas <ni...@gmail.com>.
Actually, this looks like its some kind of regression in 1.0.1, perhaps
related to assembly and packaging with spark-ec2. I don’t see this issue
with the same data on a 1.0.0 EC2 cluster.
How can I trace this down for a bug report?
Nick
On Sun, Jul 13, 2014 at 11:18 PM, Nicholas Chammas <
nicholas.chammas@gmail.com> wrote:
> For example, are LIKE 'string%' queries supported? Trying one on 1.0.1
> yields java.lang.ExceptionInInitializerError.
>
> Nick
>
>
>
> On Sat, Jul 12, 2014 at 10:16 PM, Nick Chammas <nicholas.chammas@gmail.com
> > wrote:
>
>> Is there a place where we can find an up-to-date list of supported SQL
>> syntax in Spark SQL?
>>
>> Nick
>>
>>
>> ------------------------------
>> View this message in context: Supported SQL syntax in Spark SQL
>> <http://apache-spark-user-list.1001560.n3.nabble.com/Supported-SQL-syntax-in-Spark-SQL-tp9538.html>
>> Sent from the Apache Spark User List mailing list archive
>> <http://apache-spark-user-list.1001560.n3.nabble.com/> at Nabble.com.
>>
>
>
Re: Supported SQL syntax in Spark SQL
Posted by Nicholas Chammas <ni...@gmail.com>.
For example, are LIKE 'string%' queries supported? Trying one on 1.0.1
yields java.lang.ExceptionInInitializerError.
Nick
On Sat, Jul 12, 2014 at 10:16 PM, Nick Chammas <ni...@gmail.com>
wrote:
> Is there a place where we can find an up-to-date list of supported SQL
> syntax in Spark SQL?
>
> Nick
>
>
> ------------------------------
> View this message in context: Supported SQL syntax in Spark SQL
> <http://apache-spark-user-list.1001560.n3.nabble.com/Supported-SQL-syntax-in-Spark-SQL-tp9538.html>
> Sent from the Apache Spark User List mailing list archive
> <http://apache-spark-user-list.1001560.n3.nabble.com/> at Nabble.com.
>